Songs Across Borders: Singable and Controllable Neural Lyric Translation
Year: 2023

Abstract:
The development of general-domain neural machine translation (NMT) methods has advanced significantly in recent years, but the lack of naturalness and musical constraints in the outputs makes them unable to produce singable lyric translations. This paper bridges the singability quality gap by formalizing lyric translation into a constrained translation problem, converting theoretical guidance and practical techniques from translatology literature to promptdriven NMT approaches, exploring better adaptation methods, and instantiating them to an English-Chinese lyric translation system. Our model achieves 99.85%, 99.00%, and 95.52% on length accuracy, rhyme accuracy, and word boundary recall. In our subjective evaluation, our model shows a 75% relative enhancement on overall quality, compared against naive finetuning1.

heading: 1 Introduction
text: With the globalization of entertainment, it is becoming increasingly common for people to appreciate songs in foreign languages. Meanwhile, artists are internationalizing their work and building territories worldwide. Nevertheless, an unfriendly barrier exists between the artists and the audience: most commercial songs are not written in multiple languages. Worse still, most existing song translations entirely ignore the music constraints, rendering
1Code available at https://github.com/Sonata165/ControllableLyricTranslation
them unsingable alone with the music. As a result, the language barrier complicates the interaction between artists and their audience.
Obtaining singable lyric translations can facilitate the globalization of the music publishing industry to further promote the growth of its $5.9 billion USD market size (Verified Market Research, 2022). However, song translation is unusually difficult for human translators, due to music constraints and style requirements. If we can construct lyricspecific machine translation (MT) systems that can produce drafts that satisfy these constraints and requirements, the difficulty and cost of lyric translation will be largely reduced, as lyricists and translators can start with such automatic drafts and can focus on post-processing for quality and creativity.
However, obtaining singable lyrics from MT systems is challenging. Figure 1 shows two sentences of lyrics from the song Let It Go, together with an MT output and a singable translation. We observe a notable quality gap between them. While the MT output correctly translates the source, it ignores all the criteria that matter to make the output singable: (1) The second sentence of the MT outputs is unnatural because of incoherent vocabulary selection and lack of aesthetics. (2) Overcrowded syllables in the first sentence of the MT outputs force performers to break music notes in the orange box into multiple pieces to align them with lyrics. The rhythm pattern consequently diverges from the composer’s intention. (3) The two-syllable word in the red box is situated across a musical pause (blue box), let it go in intro Subtitle Composer / arranger
5
15
27





 
  




  


 

 



  
 



  

447
causing an unnatural pronunciation. (4) The endsyllables (purple text) are not of the same rhyme pattern, making the output miss a key chance for being poetic.
In contrast, the singable translation in the third row outperforms the MT output in all four aspects, all while maintaining translation fidelity: it perfectly aligns with each musical note, has the same end-rhyme pattern for the two sentences (green text), a natural stop at the musical pause, and higher naturalness. These properties make it a significantly more performable translation.
To address these quality gaps to obtain singable lyric translations from neural machine translation (NMT) systems, we formalize singable lyric translation as an instance of constrained translation, identify useful constraints, and propose a languagepair independent approach that combines translatology theoretical guidance with prompt-driven NMT. Our contributions are:
• We design an effective and flexible promptbased solution for necessary word boundary position control that enhances the outputs’ singability.
• We find that reverse-order decoding significantly contributes to the accuracy of promptbased rhyme control. With this decoding strategy as the basis, we further design a rhyme ranking scheme to facilitate picking the bestsuitable rhyme for translating input stanzas.
• We conduct comparative studies of different prompt forms’ effectiveness for controlling each aspect—length, rhyme, and necessary word boundary positions—and show the advantage of prompt-based control over control by modifying beam search.
• We show that adding back-translation of target-side monolingual data for fine-tuning is more effective in adapting the model to the lyric domain, compared with the more common practice of in-domain denoising pretraining.

heading: 2 Related Work
text: Lyric/Poetry Translation. Designing domainspecific MT systems for poetic text translation, e.g., poetry and lyrics, is an emerging and underexplored topic in MT. Two previous works conducted pioneering research on lyrics (Guo et al., 2022) and
poetry (Ghazvininejad et al., 2018) translation separately by adopting a similar methodology of adjusting beam scores during beam search (referred to as biased decoding) to encourage the generation of outputs with desired constraints. However, there is plenty of room for improvement. As will be shown in later sections, biased decoding not only fails at effectiveness of control, but also negatively impacts text quality and other simultaneously-controlled aspects. Additionally, the inclusion of controlling aspects is insufficiently comprehensive. For example, GagaST (Guo et al., 2022) omits controls for rhyme, but rhyming is actually a critical desired property for song translations (Strangways, 1921).
Lyric Generation. Research on building lyricspecific language models shows the effectiveness of prompt-based control for outputs’ length, rhyme, stress pattern, and theme (Li et al., 2020; Ma et al., 2021; Xue et al., 2021; Ormazabal et al., 2022; Liu et al., 2022). However, several aspects remain to be enhanced.
First, the prompts’ forms vary: some works add prompts by additive embedding vectors (Li et al., 2020; Ma et al., 2021; Xue et al., 2021; Liu et al., 2022) and others by the prefix of input (Ormazabal et al., 2022; Liu et al., 2022). The lack of comparison makes it difficult to conclude the best prompt form for different control aspects.
In addition, prior works did not control for some aspects in a well-designed manner. For example, (Liu et al., 2022) enhances the music–lyric compatibility by controlling the number of syllables of each word in the output. However, music constraints are usually not that tight so that such finelevel controlling might be unnecessary. Additionally, we found that unfitted rhyme prompts damage the output quality. However, we have not seen research suggesting how to choose the best suitable end-rhyme without naively traversing all possible rhyme prompts.
Translatology: Singable Translation of Songs. We attribute the inability of singable lyric translation from general-domain MT systems to the completely different goal of lyric translation compared with normal interlingual translation (Low, 2005): without considering the rhythm, note values, and stress patterns from music, song translations that seem good on paper may become awkward when singing. When the auditory perception is dominated by music (Golomb, 2005), the goal of trans-
lation is not again predominated by preserving the semantics of source text (Franzon, 2008), but requires skilled handling of non-semantic aspects (Low, 2013) to attain the music–verbal unity, making it even an unusually complex task for human translators (Low, 2003).
Theory and techniques from translatology provide valuable guidelines for our method design. Particularly, the “Pentathlon Principle” (§3.1) from (Low, 2003) is a widely accepted theoretical guidance to obtain singable song translations (Franzon, 2008; Cheng, 2013; Stopar, 2016; Si-yang, 2017; Opperman et al., 2018; Sardiña, 2021; Pidhrushna, 2021). In addition, some practical translation tricks have also been mentioned in (Low, 2003), e.g., determining the last word first and from back to front when translating sentences in rhyme.
Denoising Pretraining. The deficiency of indomain data requires a powerful foundation model to ensure translation quality. We found large-scale denoising sequence-to-sequence pretraining (Lewis et al., 2019) a great candidate in our problem setting because it has been shown to be particularly effective in enhancing model’s performance on text generation tasks such as summarization (Akiyama et al., 2021) and translation (Liu et al., 2020; Tang et al., 2020), and also on domain-specific applications, e.g., (Yang et al., 2020; Soper et al., 2021; Obonyo et al., 2022). However, as indicated in (Liu et al., 2020), the effectiveness of pretraining is related to the amount of monolingual data. In our case where in-domain data are relatively deficient, adopting the same strategy for adaptation might not be optimal.
Back-Translation. Back-translation (BT) and its variants can effectively boost the performance of NMT models (Sennrich et al., 2015; Artetxe et al., 2017; Lample et al., 2018), and also show superior effectiveness in domain adaptation in low-resource settings (Hoang et al., 2018; Wei et al., 2020; Zhang et al., 2022). It is potentially a better adaptation method and may lead to higher output naturalness, which is required by singable translations.
Prompt-based Methods. Adding prompts during fine-tuning shows strong performance on lexical-constrained-MT (Susanto et al., 2020; Chousa and Morishita, 2021; Wang et al., 2022), as well as broad applicability on various controlling aspects such as output length (Lakew et al., 2019) and the beginning word of output (Li et al., 2022).
Compared to some earlier research that adds lexical constraints during beam search (Hokamp and Liu, 2017; Post and Vilar, 2018), the prompt based solution has a faster decoding speed and higher output quality (Susanto et al., 2020), hence might be the better option in our problem setting.

heading: 3 Method
text: To bridge the gaps of previous research, we identify comprehensive controlling aspects from the translatology literature, propose prompt-based solutions for each aspect, and explore more effective foundation models and adaptation methods.

heading: 3.1 Controlling Aspects
text: Are there some universal rules that we can adopt to obtain singable translations? We first rule out some prospective answers. Strictly keeping the positions of stressed syllables (Ghazvininejad et al., 2018) is inappropriate as stressing certain syllables is the property of stress-timed language. In contrast, syllable-timed languages, e.g., French and Mandarin, give syllables approximately equal prominence. Aligning the characters’ tone with the melody (Guo et al., 2022) is also not a good choice. On the one hand, this rule only applies to tonal languages. On the other hand, this rule is increasingly being ignored by the majority of songs composed in recent decades (Gao, 2017), indicating the marginalized importance of the intelligibility of songs, especially in pop2.
To achieve a comprehensive and languageindependent method, we define “singable translation” as following the “Pentathlon Principle” from (Low, 2003): that quality, singable translations are obtained by balancing five aspects—singability, rhythm, rhyme, naturalness, and sense. Table 1 lists these aspects and corresponding requirements, and how we actualize them in our model. Particularly, we identify (1)–(3) as the controlling aspects of our model and realize them with prompt-based control, while (4) and (5) are achieved from the perspectives of adaptation and pretraining.

heading: 3.2 Problem Formulation
text: We define the task that is tackled in this paper, singable and controllable lyric translation, as follows: given one line of lyrics X in a source language Lsrc and a set of desired properties of output
2For example, according to Apple Music, 61 of the 2022 Top 100 Chinese pop songs are songs by Jay Chou, a Chinese artist famous for unintelligible songs.
Aspects Requirements Our Actualization
(1) Singability Outputs are suitable for singing with the given melodies. Enhance music-lyric compatibility by prompt-based necessary word boundary control. (2) Rhythm Outputs follow rhythm patterns in the music. Prompt-based length (number of syllables) control. (3) Rhyme Outputs fulfil certain rhyme patterns. Prompt-based end-rhyme control and paragraph-level rhyme ranking. (4) Naturalness Outputs read like lyrics originally composed in the target language. Adapting with back-translation of in-domain target-side monolingual data. (5) Sense Outputs are fidelity to the meaning of source sentences. Large-scale general-domain pretraining.
Table 1: The “pentathlon principle” and the actualizations in our model.
sentence {ltgt, rtgt, btgt}, generating text translation Y in target language Ltgt for X by modeling P (Y |X, ltgt, rtgt, btgt), where (1) the total number of syllables of sentence Y to be precisely equal to length constraint ltgt; (2) Y ends with a word that is in the same rhyme type of rhyme constraint rtgt; (3) Y has word boundaries—the positions between two consecutive syllables that belong to different words—in all locations indicated in necessary word boundary constraint btgt; (4) Y is of maximal naturalness, and is fidelity to the sense of X .

heading: 3.3 Prompt Methods for Controlling
text: Two types of special tokens are constructed as prompts for sentence-level control. For each sentence, the length and rhyme prompts are single token len_i and rhy_j, indicating the desired number of syllables of the output is i and that the desired end-rhyme type of output is j. The prompt for necessary word boundaries is a sequence of special tokens, bdr = {bdr_0, bdr_1}len_i, indicating the desired word boundary positions.
During the training process, these prompts are derived from the analysis of target-side sentences, guiding the model towards generating sentences with corresponding properties. Consequently, there is no need for accompanying music during training. At the inference stage, prompts can be crafted from either music or source-side sentences. For an overview of the system workflow, please refer to Figures 3b and 3c.
We conducted a group of experiments to test three different prompt methods to determine the best one for each control aspect. They are (1) Encpref: prompts are injected into the encoder’s input as a prefix. (2) Dec-pref: prompts are injected into the decoder’s input as a prefix. (3) Dec-emb: prompts are embedded into a vector and added toward the decoder’s input.

heading: 3.4 Word Boundary Control
text: Intra-word pause is a typical disfluency pattern of beginning language learners (Franco et al., 1998). However, improperly translated lyrics usually con-
tain multi-syllable words that lies across musical pauses, as the blue box in Figure 2, so that the performer has to make awkward intra-word pauses while singing (Guo et al., 2022), causing a drop in pronunciation acceptability. Besides, we observe that positioning highlighted music notes, such as high notes or downbeats, as the orange box in Figure 2, onto a multi-syllable word’s second or later syllables can bring similar adverse effects due to abrupt changes of pitch and tension3.
We address these issues by carefully designing the placement of word boundaries in outputs, i.e., the points between two consecutive syllables that are from different words. Our aim is to ensure that word boundaries align precisely with the boundaries in music, i.e., the melody boundaries, which occur at musical pauses and before highlighted notes (the blue and orange boxes in Figure 2). In this way, we achieve higher compatibility between the output sentences and the accompanying music, enhance the fluency and consistency of pronunciation during singing, and hence lead to the gain of singability.
This solution is achieved by prompt-based word boundary control. We use the prompt bdr to represent melody boundary positions, indicating necessary word boundary positions. bdr is a sequence of special tokens, and each token corresponds to one syllable in the output. There are two types of special interior tokens: bdr_1 and bdr_0, representing after the corresponding syllable “there should be a word boundary” and “we do not care if there
3Stress-timed languages have another solution to this second problem, i.e., put a stressed syllable at the highlighted note. Here we discuss another generic solution.
is a boundary”, respectively. At test time, bdr is obtained from accompanying music and serves as additional inputs. A well-trained word-boundaryaware model can hence place word boundaries at the desired positions to achieve better music–lyric compatibility. For locations where bdr_0 is present (“don’t care”), the translation model operates unconstrained, maximizing translation naturalness.
During training, length and rhyme prompts can be obtained directly from the target sentences in the training samples, but not again for necessary word boundary prompts because they have to be obtained from accompanying music which is absent in training. Nevertheless, we offer a solution: we randomly sample from all actual word boundary positions from the target-side text and use this sampled subset as “pseudo ground truth” to construct bdr for training.

heading: 3.5 Reverse Order Decoding

heading: 3.5.1 Sentence-Level Control
text: We imitate the process of human translators translating texts in rhyme: translating the last word first, and from back to front, which is an old trick to keep rhyming patterns from being forced (Low, 2003). We implement this by reverse-order decoding. During fine-tuning with parallel data, we reverse the word order of target-side text while retaining the source-side text unchanged. This approach minimally changes the structure and workflow of offthe-shelf translation models.

heading: 3.5.2 Paragraph-Level Ranking
text: Controllability alone is not enough. For a given input sentence, the rhyming usually only looks good in certain rhyme types but appears forced in others (see Appendix C.2 for details). No matter how good the controllability is, the output quality will be severely damaged if an ill-fitting rhyme prompt is provided by the user. To avoid such problems, we need to determine the most suitable end-rhyme for translating one sentence, and further one paragraph consisting of multiple sentences. Previous research left this problem unsolved.
Fortunately, our reverse-order decoder simplifies the rhyme ranking process. During training, we use an additional special token rhy_0 to nullify rhyme constraints for output. We achieve this by randomly converting a portion of each type of rhyme prompt to rhy_0 during training. At inference time, for a given source sentence Xi and prompts ltgt, rtgt and btgt, we first use rhy_0 as the rhyme prompt to do
the first step of reverse-order decoding to obtain the end-word probability distribution,
P (y−1|X, ltgt, btgt, rhy_0) = [p(w1), p(w2), . . . , p(wv)], (1)
where the v is the vocabulary size of the target language. Note that the p(wj) not only indicates the end-word probability, but also predicts output text quality and the likelihood of satisfaction of length and word boundary constraints of the rhymeunconstrained model, from a greedy point of view. Intuitively, starting with tokens with low probabilities will pull down the corresponding beams’ scores and degrade the output quality. On the contrary, sentences with higher quality can be obtained by starting decoding with wj with higher p(wj), and we achieve this by giving the model a rhyme prompt that guides it towards starting with such wj . We sum up the probability in Eq. 1 within each rhyme type to obtain the rhyme distribution of given inputs,
pi = ∑
Rhy(wj)∈rhyme i p(wj)
P (Rhy(Y )|X, ltgt, btgt, rhy_0) = P (Rhy(y−1)|X, ltgt, btgt, rhy_0) = [p1, p2, . . . , pu],
where Rhy(·) is a map between a word or the endword of a sentence to its rhyme type, u is the number of rhyme types in the target language. For a certain rhyme type i, a higher pi value indicates a higher probability of successful rhyming and higher output quality.
When translating a paragraph of lyrics, we have multiple sentences together with their corresponding length and boundary prompts as input:
X = [X1, X2, . . . , Xn],with prompts
[(ltgt1 , btgt1), (ltgt2 , btgt2), . . . , (ltgtn , btgtn)].
With the assumption that every sentence is of equal importance, we compute a normalized rhyme distribution for this paragraph by
P (Rhy(Yk)) = f(Xk, ltgtk , btgtk , rhy_0),
P (Rhy(Y)) = softmax( n∑
k=1
P (Rhy(Yk)))
where f refers to the first step of reverse-order decoding. We then use P (Rhy(Y)) as the rhyme ranking score of this paragraph to guide the rhyme selection.

heading: 3.6 Utilizing Monolingual Data
text: In-domain parallel data suffer from two issues. First, its amount is so limited that it is not comparable with general-domain data. Second, there are severe quality issues when target-side lyrics are translated by online communities, including wrong translation (Li, 2020), creative treason (Zhang, 2022), over-domestication (Xie and Lei, 2022), etc.
To mitigate the issue of data quantity and quality, we seek help from target-side monolingual lyric data. Our approach involves incorporating back-translation (Sennrich et al., 2015) of targetside in-domain monolingual data to augment the parallel data for fine-tuning. To demonstrate its effectiveness, we conduct a comparative study with the adaptation method in (Guo et al., 2022), which performs sentence-level denoising pretraining (Lewis et al., 2019) with in-domain data after general-domain pretraining.
Taken together, these innovations form our final control method, which we can apply to any foundation model. In the evaluation that follows, we instantiate our techniques with Multilingual BART (refer to Figure 3 for structure and workflow), producing the Singable Translation (Row 3) in Figure 1. Additional case studies are featured in Appendix C.

heading: 4 Experiment
text: We tested our methods with English–Chinese lyric translation. We obtained a small amount of parallel data (about 102K paired sentences after deduplication) by crawling data of both English–Chinese and Chinese–English pairs from an online lyric transla-
tion sharing platform4. For target-side monolingual data, we adopted lyric data from three publiclyavailable datasets567, resulting in about 5.5M sentences after deduplication. For details of dataset statistics and splits, data preprocessing, and back translation, please refers to Appendix A.

heading: 4.1 Model Configuration
text: We adopted Multilingual BART (Liu et al., 2020) as the foundation model. We set the batch size to the largest possible value to fit into one NVIDIA A5000 GPU (24G), did simple searching for best learning rate, and kept the majority of other hyperparameters as default. For all experiments, models were first trained to converge on back-translated data, and fine-tuned with parallel data afterward. Please refer to Appendix B for implementation details and hyperparameter setting.

heading: 4.2 Evaluation
text: The following metrics are used for objective evaluation: Sacre-BLEU (Post, 2018), TER (Snover et al., 2006), length accuracy (LA), rhyme accuracy (RA), and word boundary recall (BR). BLEU is a standard metric for various translation models. TER is also adopted because it directly reflects how much effort the lyricists need to spend to convert model outputs to perfectly singable lyrics. For length and rhyme control, we compare outputs’ lengths and rhymes with desired constraints and compute the accuracy. For word boundary control, we first obtain outputs’ word boundary locations using the Jieba tokenizer8, and then compute the recall value with the necessary word boundary prompts, indicating the ratio of satisfied desired word boundaries.
For models that are constraint-aware for any controlling aspects, we conducted testing over two groups of experiments, as below:
Target as constraints (tgt-const): For a given sentence pair, the length constraint is equal to the number of syllables of the target-side sentence; the rhyme constraint is equal to the rhyme category of the end-word of the target-side sentence; the boundary constraints are randomly sampled from word boundaries inside the target sentences. In this setting, the BLEU and TER scores represent the text quality directly.
4https://lyricstranslate.com/ 5https://github.com/liuhuanyong/MusicLyricChatbot 6https://github.com/gaussic/Chinese-Lyric-Corpus 7https://github.com/dengxiuqi/ChineseLyrics 8https://github.com/fxsjy/jieba
Source as constraints (src-const): For a given sentence pair, the length constraint is equal to the number of syllables of the source-side sentence; the rhyme constraint is randomly sampled from the real rhyme type distribution of lyrics in the target language, obtained from our monolingual dataset; the boundary constraints are randomly sampled from word boundaries inside the source sentences. This setting simulates real-world lyric translation cases and is more challenging.
In src-const, we do not compare constrained models with unconstrained ones on BLEU or compute TER for outputs, as target-side sentences often possess distinct properties (e.g., # syllables) from prompts generated by source sentences, rendering them not the ground truth. Owing to the divergence between references and prompts, models with more constraints yield lower BLEUs, and TER in srcconst fails to accurately reflect translation quality.
We compare our model with two baselines. The first is the unconstrained and un-adapted Baseline model presented in Table 2. The second is GagaST (Guo et al., 2022), which, to the best of our knowledge, is the only prior work on lyric translation. Due to data acquisition difficulties, we did not perform a model-level comparison with GagaST. Instead, we compared the effectiveness of their adaptation (in-domain denoising pre-training) and control method (biased decoding) with ours (BT and prompt-based control), and compare generation results through subjective evaluation.

heading: 5 Results
text: Table 2 shows the results of our final model. In the tgt-const setting, our model surpasses the baseline model on all objective aspects, not only with much higher BLEU and lower TER scores, but also achieves almost perfect length and rhyme accuracies and a competitive boundary recall score. The success of controlling length, rhyme, and word
boundary while maintaining a high text quality enables our model to generate singable lyric translations. In addition, the controlling mechanism remains effective in the src-const setting, showing the generalizability of our methods.

heading: 5.1 Unconstrained Models
text: As in Table 3, both general-domain pretraining and in-domain fine-tuning are necessary to ensure translation quality. There are performance drops if any of the two components are canceled from the unconstrained model. Meanwhile, fine-tuning with back-translated in-domain monolingual data further contributes to the performance gain, showing higher adaptation effectiveness than in-domain pretraining. We also show BT’s contribution to improving naturalness in §5.5.

heading: 5.2 Best Prompt Methods
text: We select the most effective prompt method for different controlling aspects in our final model. Here are the effectiveness comparisons.
Length Control. As shown in Table 4, the encoder-side prefix is the best prompt method for length control, with the highest length accuracy and higher translation quality than dec-pref.
Rhyme Control. As shown in Table 5, the decoder-side prefix is the best method for rhyme control, with a significantly higher rhyme accuracy than the second-best method encoder-side prefix.
Word Boundary Control.9 As shown in Table 6, enc-pref is the best for word boundary control with much higher effectiveness than dec-pref. It has comparable performance with dec-emb in tgt-const, but shows stronger controllability in the src-const setting, indicating better generalizability.
9BT data are not added to length and rhyme control experiments to maximize the performance differences of different methods, but are added in word boundary control experiments because boundary awareness is much slower to learn.

heading: 5.3 Prompt-Based Word Boundary Control
text: As in Table 7, prompt-based control is much more successful than biased decoding in word boundary control, not only achieving high boundary recall (95.22% and 89.77%) but also slightly raising the length accuracy and text quality. On the contrary, biased decoding contributes limited power to word boundary control with the expense of significant drops in text quality and length control accuracy.

heading: 5.4 Prompt-Based Reverse-Order Decoding
text: Prompt vs. Biased Decoding. As in Table 8, the prompt-based method again shows higher effectiveness in rhyme control, while the biased decoding again negatively impacts text quality. As in Appendix C.3, the prompt-based control enables the model to adjust the expression of the entire sentence according to the given desired rhyme, achieving higher consistency, but the biased decoding sometimes abruptly changes the end-word to fulfill the constraint without considering whether it is compatible with input sentence and target-side context.
Normal vs. Reverse. Reverse-order decoding further raise the performance of prompt-based rhyme control, but conversely, only brings marginal improvement to biased-decoding-based control. A possible explanation is the inability of biased decoding to handle polyphones (see Appendix C.3). We observed multiple cases where one of the pronunciation of the end-word in its output does satisfy the rhyme requirement, but is not the pronunciation in that context. On the contrary, the prompt-based control is aware of the whole target-side sentence, and hence better controllability is achieved.

heading: 5.5 Human Evaluation
text: We employ five students from a local university with music performance or lyric composing back-
grounds. We let participants evaluate outputs on five-point scales and take the average as the final score. Evaluations are from four dimensions: (1) sense, whether the translation output retains the meaning of the input sentence; (2) naturalness, whether the translation output sounds like lyrics composed initially in the target language; (3) music– lyric compatibility, the degree of outputs and music match with each other and the consequent singability gain; (4) Singable Translation Score (STS), the overall quality as singable translations, a singlevalue metric considering the satisfaction of all five perspectives in the Pentathlon Principle (§3.1)10.
Table 9 shows the subjective evaluation results of baseline, GagaST (Guo et al., 2022), our model, and some ablated variants. On the STS metric, which is the ultimate goal of singable lyric translation, our model significantly outperforms the baseline and GagaST by 75.0% and 20.2%, showing its ability to generate singable translations. Besides, our model performs especially well on music–lyric compatibility, by 74.7% and 10.2% higher scores than the baseline and GagaST. In contrast, the baseline model performs worst on the two metrics.
In addition, we show the contributions of different components by the ablated studies. The word boundary control raises music–lyric compatibility (+0.21) and overall quality (+0.11). The contribution from rhyme control is majorly on the overall quality part (+0.22), but with the expense of sense (-0.24) and naturalness (-0.31). Length control is the foundation of music–lyric compatibility (+1.57) and STS (+0.93), but with some expense of sense (- 0.21). Adaptation with BT increases sense (+0.34) and naturalness (+0.16).

heading: 6 Conclusion
text: We discussed how to obtain singable translations with prompt-driven NMT systems with the guid-
10Translation outputs are available at https://www.oulongshen.xyz/lyric_translation
ance of translatology theories. Specifically, we used back-translation to enhance translation quality and naturalness. We compared the effectiveness of different prompt methods in different controlling aspects and showed their advantage over biased decoding. We designed an effective word boundary control approach and presented a training strategy without the help of music data. We demonstrated the effectiveness of reverse-order decoding in NMT models for rhyme control and showed how it helps users to choose the best suitable rhymes for a paragraph of source text.
This work does not explore more detailed prompt manipulation, such as using varied prompts for the same constraint or examining prompt order’s impact on performance. We leave these investigations for future research.
Limitations
The current system may require the user to have some music knowledge to compose the word boundary prompt from music. Hence, more efforts need to be made to fulfill this gap before such a system can operate fully automatically without the human user providing word boundary prompt themselves.
We use the back-translation of mono-lingual data to augment the parallel training data, but the quality, especially the text style of back-translations has room to improve. Although we have tried using iterative BT to gradually refine the backward direction MT model to adapt its outputs to lyric style, we found some errors gradually accumulated in the back-translated data, which finally made our model perform unsatisfactorily for negative sentences, together with the decrease of controlling effectiveness. Further exploration is needed in this aspect.
Similar to chat text, lyrics are usually composed in short sentences. Sometimes it would be challenging to guarantee the consistency of style and meaning for different sentences, if the current sentencelevel translation system are adopted. Hence, for building future lyric translation systems, it would be a better option to translate the lyrics directly at the paragraph level or document level.
Ethics Statement
Our system will help facilitate the creation/recreation of lyrics for song composers. In addition, although our system is implemented in the direction
of English-to-Chinese, the controlling aspects and approaches are universal because we did not take any language-specific aspects into account; hence can be easily implemented in other language pairs. Besides, the method and system discussed in this paper are suitable for creating/re-creating singable song lyrics in languages beyond the original version. They also have the potential to benefit language learning by translating domestic languages into other languages the learner is studying and facilitating learning by singing.
This methodology has limitations by putting the singability into priority. Translations from this system may sometimes not convey the exact meaning of the lyrics in the source language, causing misunderstanding in this case. For cases where conveying the original meaning is crucial, e.g., advertising and serious art songs, the translation outputs need to be checked and revised when necessary by the user before further usage.
For the training and evaluation of our system, all data is publicly available online. Specifically, Chinese Lyric Corpus11 is a public GitHub repository with an MIT license. Lyricstranslate.com is a lyric translation sharing platform, where all parallel lyrics we obtained are publicly available in this website. We adhere to the rules specified in the website’s robots.txt file when crawling. For all existing scientific artifacts used in this research, including datasets, models, and code, we ensure they are used in their original intended usage. For human evaluation, we collect evaluation scores without personal identifiers for subjective evaluation to ensure a fair comparison. We ensure that the questionnaire does not contain any offensive content. Please refer to Appendix E for more details of subjective evaluation.

heading: Acknowledgements
text: This project was funded by research grant A0008150-00-00 from the Ministry of Education, Singapore.

heading: A Data Preprocessing
text: A.1 Dataset Details The monolingual lyric corpus from three sources includes lyrics data in Chinese, and vast majority of
them are in pop genre. Lyrics of one song contains multiple lines. Each line usually corresponds to one utterance in singing. The length of each line is usually short. There are 8.6 Chinese characters each line on average. Only a few cases contains lines longer than 20 Chinese characters.
The crawled parallel lyrics contains two parts. For the first part, the lyrics are created in English originally, and translated to Chinese by online communities. The second part is composed in Chinese originally and translated to English. Similarly, most of them are in pop genre.
A.2 Dataset Splitting
Train/validation/test splitting is performed separately for BT and parallel data. Table 10 shows the detailed statistics.
A.3 Data Preprocessing
We perform text normalization for all Chinese lyric text: all special symbols are removed; traditional characters are substituted with simplified characters12; sentences that are longer than 20 characters are removed; any duplicated sentences are removed. Finally, we split the datasets into train, validation, and test splits while ensuring no same songs exist in different splits.
For in-domain denoising pretraining experiments, text corrupting is performed by sentencelevel mask prediction. There is one mask for each sentence. For the span of masks, for sentences with length in (1, 3] and larger than 3, the mask span is sampled from a Poisson distribution with lambda equals 1 and 3, respectively.
A.4 Back Translation
For back translation, we adopt a Transformer trained with generic-domain Chinese-to-English data13 to obtain sentence-level back translation.
12Follow the implementation of https://github.com/ liuhuanyong/MusicLyricChatbot/blob/master/process_data/ langconv.py
13https://huggingface.co/Helsinki-NLP/opus-mt-zh-en
B Implementation details
Model Configuration At the early stage of our experiment, we found that fine-tuning with genericdomain data does not help with the translation quality of lyrics. Hence we adopt mBART without general-domain fine-tuning as the starting point of training. For the unadapted general-domain model, we use mbart-large-50-one-to-many14.
Our final model is obtained by fine-tuning mbartlarge-5015 (#param: 610,879,488) with both backtranslated monolingual data and parallel data. The tokenizer is modified to be character-level on the Chinese side for better controlling effectiveness. The model is trained on one Nvidia A5000 GPU (24GB) for 10 epochs and 3 epochs on backtranslation and parallel data, respectively, taking about 16 hours and 3 hours. The learning rate is set to 3e-5 and 1e-5, respectively, on BT and parallel data. They are the best value in {1e-5, 3e-5, 1e-4} for the baseline model on the two stages of training. Warm-up steps are set to 2500 and 300 for training with the BT and the parallel data. Dropout and label smoothing are set to 0. For decoding, beam-search with beam size 5 is adopted. The maximum output length is set to 30. All other hyperparameters remain as default values.
For the dec-emb experiments, instead of using sinusoidal encoding for prompts, we use learnable embedding to keep aligned with the positional em-
14https://huggingface.co/facebook/ mbart-large-50-one-to-many-mmt
15https://huggingface.co/facebook/mbart-large-50
bedding of mBART.
Length Prompt. We construct 20 length tokens for length control, len_1 to len_20 for translation output. According to the authors’ observation, only an extremely tiny amount of lyrics in Mandarin have more than 20 characters in one line.
Rhyme Prompt. For rhyme control, we adopt the Chinese 14-rhyme scheme16 for possible rhyme type, implemented as rhy_1 to rhy_14. There is a special token rhy_0 representing “no rhyme control”. This is achieved by randomly setting 1/15 of each type of rhyme prompt to rhy_0 during training.
Word Boundary Prompt We first sample a number n from a categorical distribution with the ratio of 1:4:3:1 for 1, 2, 3, and 4 boundaries, and use n′ = min(number of words, n) as the number of bdr_1 tokens. Then, we uniformly sample n′ times from all syllable boundary locations, without replacement, as the locations of these bdr_1. After that, we initialize the prompt sequence as a sequence of bdr_0 where the length of the sequence equals the number of syllables in the reference sentence. Finally, we substitute bdr_0 with bdr_1 for the sampled locations.
16https://github.com/korokes/chipsongC mposer / arranger
Let It Go verse part

heading: C More Case Studies
text: C.1 Model Outputs
We show the translation comparison of the proposed model and the baseline model in Figure 4. The outputs are perfect in the number of syllables and rhyme constraints. With the guidance of word boundary constraints, the output has much higher music-lyric compatibility than the baseline’s output. For example, there is a downbeat lying on the note of the second word in the source lyrics, "snow", creating a melody boundary between the first and the second note. To get rid of pronunciation interruption, our system successfully places a word boundary here, avoiding the scenario where the second syllable of the word "今夜" is highlighted. Similarly, in the fourth sentence, our system places a word boundary at the place between the translation of "it looks like" and "I’m the queen", where there exists a musical pause.
C.2 Different Rhyming Difficulties We noticed that an improper rhyme prompt will lead to lower text quality and a lower chance of constraints being satisfied. For example, Figure 5 shows the rhyme ranking scores of one paragraph and different outputs when using different rhyme targets. With the 1st-ranked rhyme as prompt (Figure 5b), the output is perfect in length and rhyme control and has a satisfactory translation quality. However, with a rhyme that has a low score (Figure 5c), the rhyme control performance drops (one missing rhyme), and both sense and naturalness become questionable.
C.3 Disadvantage of altering beam search We show the disadvantages of controlling by altering beam search by examples.
Length Forcing Figures 6a and 6b show typical errors when the length constraint is different from the length of the reference sentence, which is usually the case at inference time. If the desired length is shorter than the reference, the beam search might end too soon, so the sentence will be incomplete (Figure 6a). If the desired length is longer than the reference (Figure 6b), there tends to be repetition in the outputs. Both cases significantly damage the translation quality, although the outputs may even
have higher BLEU scores.
Biased decoding for rhyme A type of error frequently happens that the end-words in the outputs are biased toward words that satisfy the rhyme constraints but are irrelevant to the source sentences and are incompatible with other parts of the output sentences, as in Figures 6c and 6d. Such problems are much rarer in translations obtained by promptbased methods.
Figure 6e illustrates a possible explanation for the minor performance improvement observed when using a reverse-order decoder with biased decoding for rhyme control. The highlighted word in the biased decoding output, “落”, has multiple pronunciations. One of these, “lao”, meets the rhyme requirement. However, the correct pronunciation for this specific context is “luo”, which does not fulfill the rhyme constraint.

heading: D Error Bar
text: In order to reduce the randomness in the results of our comparative study, each experiment in §5.2
is run three times. Here we show more detailed results by the error bar charts in Figure 7.

heading: E Subjective Evaluation
text: We select the same five songs as GagaST (Guo et al., 2022) for our subjective testing. When doing this experiments, we ensure these songs are not in the training set.
As mentioned in §5.5, we evaluate the results from four aspects: sense, naturalness, music-lyric compatibility, and the Singable Translation Score (STS), an overall singable translation quality. The four metrics are evaluated at different levels. Sense and naturalness are evaluated for independent textonly sentences, melody compatibility is evaluated for each sentence given the music context, and the last metric is evaluated at the paragraph level. When evaluating STS, we show participants not only the music sheet containing melody notes and lyrics, but also with a singing audio. This audio file contains singing voice synthesized with original melody and generated lyrics, mixed with original
musical accompaniments. The voice part is synthesized by ACE Studio17. The accompaniments is obtained by using a source separation model Demucs v3 mdx_extra (Défossez, 2021).
To test the reliability of our subjective metrics, we computed the inter-rater agreement using intraclass coefficients (two-way mixed-effect, average measure model). The results are as follows: 0.8313 for sense, 0.7817 for naturalness, 0.8663 for music-
lyric compatibility, and 0.7870 for Singable Translation Score. All of these values fall within the "good reliability" range suggested by (Koo and Li, 2016).
E.1 Instructions For Human Evaluation
Example
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
The "Limitation" section.
3 A2. Did you discuss any potential risks of your work? The "Ethics Statement" section.
3 A3. Do the abstract and introduction summarize the paper’s main claims? The "Abstract" section.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Data: Section 4; Section 4.2. Code and model: Appendix B; Appendix E.
3 B1. Did you cite the creators of artifacts you used? Data: Section 4; Section 4.2. Code and model: Section 4.1; Appendix B; Appendix E.
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? The "Ethics Statement" section.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? The "Ethics Statement" section.
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The "Ethics Statement" section.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Appendix A.1.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix A.2.
C 3 Did you run computational experiments? Section 4.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix B.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix B.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5; Appendix D.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4.2; Appendix A.3; Appendix E.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 5.5.
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Appendix E.1.
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 5.5.
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? "Study Information" in Appendix E.1.
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? "Study Information" in Appendix E.1.
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Section 5.5.

heading: Future Work
text: We leave these investigations for future research. Limitations
The current system may require the user to have some music knowledge to compose the word boundary prompt from music. Hence, more efforts need to be made to fulfill this gap before such a system can operate fully automatically without the human user providing word boundary prompt themselves. We use the back-translation of mono-lingual data to augment the parallel training data, but the quality, especially the text style of back-translations has room to improve. Although we have tried using iterative BT to gradually refine the backward direction MT model to adapt its outputs to lyric style, we found some errors gradually accumulated in the back-translated data, which finally made our model perform unsatisfactorily for negative sentences, together with the decrease of controlling effectiveness. Further exploration is needed in this aspect. Similar to chat text, lyrics are usually composed in short sentences. Sometimes it would be challenging to guarantee the consistency of style and meaning for different sentences, if the current sentencelevel translation system are adopted. Hence, for building future lyric translation systems, it would be a better option to translate the lyrics directly at the paragraph level or document level