Composing Features: Compositional Model Augmentation for Steerability of Music Transformers

1 INTRODUCTION . Recent research has focused on ways of adapting unconditional language models to perform well on conditional tasks they were not initially trained on . Methods such as fine-tuning , side-tuning , bias-tuning , and prompt or prefix tuning have emerged as lead candidates for such tasks such as steering the sentiment of or words mentioned in a sentence ( Ben-Zaken et al. , 2021 ; Li & Liang , 2021 ; Lester et al. , 2021 ; Dathathri et al. , 2020 ) . However , such tasks are normally presented in a non- or minimally-compositional approach ( possibly controlling for two orthogonal variables , such as sentiment and topic ( Keskar et al. , 2019 ) , but rarely more than that ) . In contrast , in the domain of music generation , sequence-continuation is inherently a highly compositional problem : the user likely has many aspects of the output they would like to control , such as speed , dynamics , harmony , or texture , each of which can be decomposed into multiple sub-features . Furthermore , the relationship between these features and the output is more diffuse than in the NLP settings : while in examples such as Keskar et al . ( 2019 ) , individual words indicate the different conditions being satisfied , in music the entire sequence of tokens are used in evaluating a single feature ( for instance , average pitch over a span of time depends on all tokens representing that span ) . This makes compositionality even more challenging than in the text generation setting . On the other hand , music is a sequential domain where there are clear logical features which can be examined , such as average dynamics or number of notes per second . This further motivates using music as a test-bed for highly compositional tasks involving simultaneously steering an autoregressive model towards several particular desired attributes according to different classifiers . As a more immediate motivation , consider the following scenario : a composer wants to sample from the pre-trained Music Transformer ( Huang et al. , 2018 ) to complete a musical phrase . In addition to the overall musical quality of the continuation , they want it to stay in key and switch to using block chords ( i.e . a few notes played together at once ) . Repeatedly sampling continuations from the model and cherry-picking a good sample ( rejection sampling ) would be very labor-intensive , but assuming they can compute binary features for “ stays in key ” and “ uses block chords ” , the composer could sample a large number of continuations and cherry-pick from the smaller set of continuations which exhibit all features ( thereby delegating part of the accept/reject step ) . While significantly less labor-intensive , this solution could potentially require sampling enormous amounts of continuations if , for instance , 6 the requested features significantly differ from those of the priming sequence ( such that it and the desired continuation form an unlikely sequence according to the generative model ) . In other words , the pre-trained model could be a poor proposal distribution for some applications . In this work , we are interested in developing better proposal distributions through an adaptation approach which can steer the generative model towards continuations which i ) are significantly more likely to exhibit the requested feature , and ii ) exhibit a satisfactory musical quality . The approach should be able to accommodate a large number of features without adding significant memory or computation overhead . We achieve this by making features composable , making it possible to steer features independently and also multiple features at once . Figure 1 shows that when using a pretrained transformer model augmented with a relatively small number of additional parameters , we are able to steer towards arbitrary logical music features and achieve realistic music generation simply by sampling directly from the model 1 . In contrast , the same unconditional transformer model fails to produce any examples satisfying those features even when generating 100 samples . With our method , a composer can control both the short-chunk features , and the long-form structure , by chaining together chunks steered in different directions ( using different combinations of features ) , while maintaining long-term coherence ( by leveraging the transformers ’ full self-attention receptive field ) . See the demo in the footnote for longer steered examples , and compositions semi-automatically generated using our feature tuning approach in a musician-directed way . 2 PROBLEM FORMULATION . Music Transformer is an autoregressive language model which decomposes the joint probability of a sequence of tokens x1 , . . . , xN ( where xn ∈ K , and K is a set of categorical tokens ) into p ( x = x1 , . . . , xN ) = p ( x1 ) N∏ n=2 p ( xn | x1 , . . . , xn−1 ) . ( 1 ) It leverages a common modeling approach which represents the conditional probabilities p ( xn | x1 , . . . , xn−1 ) using a neural network ( Bengio et al. , 2003 ) . As its name implies , Music Transformer uses a Transformer network architecture ( Vaswani et al. , 2017a ) . Each token xn is first mapped to a real-valued embedding en ( for instance using a lookup table ) , then the network maps each sequence e1 , . . . , en−1 to a probability distribution for the value of xn over the elements of K. Sequence continuation in an autoregressive language model works by repeatedly sampling from its distribution over the next token given the previous tokens . Starting from some priming se- 1Listen to these examples at https : //storage.googleapis.com/composing-features/ index.html quence xp = ( x0 , . . . , xM ) , we first sample yM+1 ∼ p ( · | x0 , . . . , xM ) , then yM+2 ∼ p ( · | x0 , . . . , xM , yM+1 ) , and so on , until the end of the continued sequence xc = ( yM+1 , . . . , yN ) . 2 Many downstream tasks can be cast as sequence continuation problems , including the steerable music generation problem investigated in this work . We are given a set of features Φ = { φj } Jj=1 , φj ∈ KN → { 0 , 1 } . Each φj takes the value 1 if a prime-continuation pair exhibits that feature ( which we note as ( xp , xc ) |= φj ) , and 0 otherwise . Note that the features must take both prime and continuation sequences as input , since some continuation features may be relative to the priming sequence ( e.g . “ significantly higher pitch ” ) . Our true objective with respect to feature φj is to steer the model towards a distribution which maximizes Exc|xp [ ( xp , xc ) |= φj ] ( 2 ) while maintaining musicality . This objective is non-differentiable because ( xp , xc ) |= φj is a nondifferentiable satisfiability criterion . In addition to the single-feature problem , we also consider the problem of composed features Φ̂ , i.e . ( xp , xc ) |= Φ̂ ≡ ∧ φj∈Φ̂⊆Φ ( xp , xc ) |= φj , ( 3 ) to account for scenarios where a user is interested in steering the model towards multiple features ( such as in the “ stays in key ” and “ exhibits extreme dynamical contrast ” scenario dicussed in the introduction ) . 3 PROPOSED APPROACH . We start by describing our proposed approach in the single-feature case and later on explain how we adapt it to the compositional case . 3.1 LIKELIHOOD-BASED TRAINING . While approaches using reinforcement learning—such as KL-regularized deep Q-learning ( Jaques et al. , 2016 ) —could be used to overcome the non-differentiability problem , in this work we consider a proxy loss in the form of the negative log-likelihood l = − log pθ ( xc | xp ) , ( 4 ) which we use in two ways : 1 . Positively : given a prime–continuation pair ( xp , xc ) |= φj , we find an adaptation θj of the model ’ s parameters θ that minimizes lX ( we use the symbol X to denote the fact that l is computed using the correct parameters θj ) . By using prime–continuation examples that sound musical , we ensure that the steered model stays musically grounded . 2 . Negatively : we can also take advantage of other features φi for which ( xp , xc ) 6|= φi , by maximizing l× ( we use the symbol × to denote the fact that l is computed using the incorrect parameters θi ) . The positive case corresponds to maximum-likelihood training . Additionally , we can exploit the intuition that the adapted parameters θj should “ explain ” the prime–continuation pair ( xp , xc ) |= φj better than θi ( for some feature φi for which ( xp , xc ) 6|= φi ) or θ ( the non-adapted model parameters , with a corresponding loss l∅ ) . In other words , we can maximize the probability of choosing θj over θi and θ by minimizing a contrastive loss of the form 2To simplify the discussion , we assume a fixed sequence length N , but the explanation applies to sequences of varying lengths as well . − log ( pθj ( xc | xp ) pθj ( xc | xp ) + pθi ( xc | xp ) + pθ ( xc | xp ) ) = − log ( e−lX e−lX + e−l× + e−l∅ ) ( 5 ) We propose a loss that interpolates between Equations 4 and 5 using an α coefficient ( which is treated as a hyperparameter ) : Lj = α · lX − ( 1− α ) · log ( e−lX e−lX + e−l× + e−l∅ ) ( 6 ) Intuitively , the maximum-likelihood setting should suffice to achieve our adaptation goals , but in practice we find that the approach benefits from the inclusion of negative cases through a contrastive loss term . We tried different α values and found α = 0.8 to work well in practice . See Figure 2 for an illustration of the training setup . 3.2 FEATURE-CONDITIONAL ADAPTATION . Fine-tuning all model parameters can be prohibitive if the number J of features is large ( let alone combinatorially large in the compositional case ) ; however , recent work provides effective and lightweight alternatives : 1 . Prefix tuning ( Li & Liang , 2021 ) works by preprending learnable task embeddings e−K , . . . , e−1 to the priming sequence embeddings e1 , . . . , eM . The loss gradient is then backpropagated through the language model and into the task embeddings . 2 . Bias-tuning ( Ben-Zaken et al. , 2021 ) works by adapting a small subset θb ⊂ θ of the transformer ’ s parameters , namely the biases of its affine transformations . Since these biases amount to a small fraction of the model ’ s parameters , in the case where the number of tasks is relatively small , tuning separate θb for each task becomes feasible . We present an extension to bias-tuning where the number of tasks is exponential in the number of total classification functions , using an approach which nevertheless only requires a number of tuned parameters linear in the number of total classification functions . In practice , while prefix tuning showed promise in the single-feature setting , we were unable to make it work in the compositional setting . We therefore focus our investigation on bias-tuning and present prefix tuning results in the Appendix . In the compositional setting , a naive approach requires learning 2|Φ|−1 model adaptations . Instead , we propose to express the adaptation for a composed feature Φ̂ as the combination of the θj of its underlying features φj ∈ Φ̂ . More specifically , for bias-tuning we average the adapted biases as θb = 1 |Φ̂| ∑ φj∈Φ̂ θb , j . ( 7 ) Note that we do not simply use the above heuristic to compose feature adaptations post-hoc ; we train the model in a compositional setting ( by sampling prime–continuation pairs ( xp , xc ) |= Φ̂ for various composed features ) so that the single-feature adaptations can learn to work well in conjunction with each other . See Figure 3 for an illustration of the methods .

Summary: This work considers how to control sampling from transformer-based autoregressive generative models of music. In particular, it considers constraint satisfaction problems (CSP) given a collection of binary constraints. The technical contributions of this paper are (1) a method for fine-tuning (bias-tuning) a model using a contrastive loss for adaptation to a given CSP (Section 3.1) and (2) an additive approximation to bias-tuning for satisfying combinatorial constraints (Section 3.2).