Some embodiments of the present disclosure relate to generating novel lyrics lines conditioned on music audio. A bimodal neural network model may learn to generate lyric lines conditioned on a given short audio clip. The bimodal neural network model includes a spectrogram variational autoencoder and a text variational autoencoder. Output from the spectrogram variational autoencoder is used to influence output from text variational autoencoder.

TECHNICAL FIELD

The present disclosure relates, generally, to using artificial intelligence to generate lyrics and, in particular embodiments, to using an autoencoder-based approach to lyric generation.

BACKGROUND

Outputs of artificial intelligence models can serve as an inspiration for artists, writers and musicians when they create original artwork or compositions.

There exist a number of known approaches to poetry generation. Some approaches focus on such characteristics as rhyme and poetic meter (see Xingxing Zhang and Mirella Lapata, “Chinese poetry generation with recurrent neural networks,” Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 670-680, 2014). Other approaches focus on generating poetry in the style of a specific poet (see Aleksey Tikhonov and Ivan P Yamshchikov, “Guess who? Multilingual approach for the automated generation of author-stylized poetry,” arXiv preprint arXiv:1807.07147, 2018.). In Wen-Feng Cheng, Chao-Chung Wu, Ruihua Song, Jianlong Fu, Xing Xie, and Jian-Yun Nie, “Image inspired poetry generation in xiaoice,” arXiv preprint arXiv:1808.03090, 2018, the authors propose image-inspired poetry generation. The approach of using style embeddings in controlled text generation has been explored in generating text conditioned on sentiment and persona-conditioned responses in dialogue systems.

SUMMARY

Aspects of the present application relate to generating novel lyrics lines conditioned on music audio. A bimodal neural network model may learn to generate lyric lines conditioned on a given short audio clip. The bimodal neural network model includes a spectrogram variational autoencoder and a text variational autoencoder. Output from the spectrogram variational autoencoder is used to influence output from text variational autoencoder.

According to an aspect of the present disclosure, there is provided a method of generating lyrics. The method includes obtaining, from an encoding portion of a first autoencoder, a representation of a time-limited audio recording, sampling, from a second distribution, a text vector and generating an output lyric line by decoding a text decoder input vector that is based, at least in part, on the representation and the text vector. The decoding uses a decoding portion of a second autoencoder, wherein the second autoencoder is a latent variable model autoencoder. The decoding portion of the second autoencoder has been trained to generate reconstructed output lyric lines based, at least in part, on input that includes: a first latent vector from a latent space of the first autoencoder, the first autoencoder trained with spectrogram input of known musical works; and a second latent vector sampled from a distribution in a latent space of the second autoencoder, the second autoencoder trained with lyric input of the known musical works corresponding to the spectrogram input of known musical works. The distribution in the latent space of the second autoencoder is encoded lyric input corresponding to a spectrogram input encoded to lead to the first latent vector.

According to an aspect of the present disclosure, there is provided a method of constructing a machine learning model for generating lyric lines. The method includes receiving a plurality of known songs that include lyrics, dividing each song of the plurality of known songs into a plurality of intervals, training a first autoencoder to generate a reconstructed spectrogram from an input spectrogram derived from an interval among the plurality of intervals, wherein the training the first autoencoder causes generation of a first latent space including a plurality of first distributions, where each first distribution in the plurality of first distributions corresponds to an interval among the plurality of intervals and training a second variational autoencoder to generate a reconstructed lyric line from an input lyric line derived from an interval among the plurality of intervals, wherein the training the second variational autoencoder causes generation of a second latent space including a plurality of second distributions, where each second distribution in the plurality of second distributions corresponds to an interval among the plurality of intervals. During the training the second variational autoencoder, a decoder portion of the second variational autoencoder is configured to generate the reconstructed lyric line based on input that includes a spectral vector selected from a first distribution in the first latent space, the first distribution corresponding to a given interval, and a text vector from selected from a second distribution in the second latent space, the second distribution corresponding to the given interval.

According to an aspect of the present disclosure, there is provided a method of generating lyrics. The method including obtaining a spectrogram, where the spectrogram is representative of a time-limited audio recording, encoding the spectrogram to, thereby, produce a first distribution, the encoding using an encoding portion of a first trained variational autoencoder, sampling, from the first distribution, an inference spectrogram latent code, generating, by providing the inference spectrogram latent code as input to a trained Generative Adversarial Network, an inference text latent code and generating an output lyric line by decoding a text decoder input vector that is based, at least in part, on the inference spectrogram latent code and the inference text latent code. The decoding uses a decoding portion of a variational autoencoder. The variational autoencoder has been trained to generate reconstructed output lyric lines. Training the variational autoencoder includes encoding a training spectrogram to, thereby, produce a training distribution, the encoding using the encoding portion of the first variational autoencoder, sampling, from the training distribution, a training spectrogram latent code, encoding a training input to, thereby, obtain a second distribution, the encoding using an encoding portion of the variational autoencoder, sampling, from the second distribution, a training text latent code and providing, as input to the decoder portion of the variational autoencoder, a training text decoder input vector that is based, at least in part, on the training spectrogram latent code and the training text latent code. The training input includes a lyric line corresponding to the training spectrogram and the training spectrogram latent code.

According to an aspect of the present disclosure, there is provided a method of generating lyrics. The method includes obtaining a spectrogram, where the spectrogram is representative of a time-limited audio recording, encoding the spectrogram to, thereby, produce a first distribution, the encoding using an encoding portion of a first trained variational autoencoder, sampling, from the first distribution, an inference spectrogram latent vector, sampling, from a second distribution, an inference text latent vector, wherein a location, in a text latent space of a second trained conditional variational autoencoder, of the second distribution corresponds to a location, in a latent space of the first trained variational autoencoder, of the first distribution and generating an output lyric line by decoding a text decoder input vector that is based, at least in part, on the inference spectrogram latent vector and the inference text latent vector. The decoding uses a decoding portion of the conditional variational autoencoder. The conditional variational autoencoder has been trained to generate reconstructed output lyric lines. Training the conditional variational autoencoder includes encoding a training spectrogram to, thereby, produce a training distribution, the encoding using the encoding portion of the first variational autoencoder, sampling, from the training distribution, a training spectrogram latent vector, encoding a training input to, thereby, obtain a second distribution, the encoding using an encoding portion of the conditional variational autoencoder, sampling, from the second distribution, a training text latent vector and providing, as input to the decoder portion of the conditional variational autoencoder, a training text decoder input vector that is based, at least in part, on the training spectrogram latent vector and the training text latent vector. The training input includes a lyric line corresponding to the training spectrogram and the training spectrogram latent vector.

According to an aspect of the present disclosure, there is provided a method of training a machine learning model for generating lyric lines. The method includes receiving a plurality of known songs that include lyrics, dividing each song of the plurality of known songs into a plurality of intervals, training a first autoencoder to generate a reconstructed spectrogram from an input spectrogram derived from an interval among the plurality of intervals, wherein the training the first autoencoder causes generation of a first latent space including a plurality of first distributions, where each first distribution in the plurality of first distributions corresponds to an interval among the plurality of intervals and training a second autoencoder to generate a reconstructed lyric line from an input lyric line derived from a particular interval among the plurality of intervals in combination with a first latent vector sampled from a particular first distribution in the plurality of first distributions, wherein the particular first distribution corresponds to the particular interval and wherein the training the second autoencoder causes generation of a second latent space including a plurality of second distributions, where each second distribution in the plurality of second distributions corresponds to an interval among the plurality of intervals. During the training the second variational autoencoder, a decoder portion of the second variational autoencoder is configured to generate the reconstructed lyric line based on input that includes the first latent vector and a second latent vector selected from a second distribution in the second latent space, the second distribution corresponding to the particular interval.

According to an aspect of the present disclosure, there is provided a method lyric generation. The method includes obtaining a plurality of style vectors, obtaining a plurality of weights, each weight among the plurality of weights corresponding to a style vector among the plurality of style vectors, generating, by weighting each style vector by the corresponding weight, an interpolated style vector, sampling a text vector from a prior distribution of a trained text variational autoencoder, the trained text variational autoencoder having an encoder portion and a decoder portion and generating an output lyric line by decoding, using the decoder portion of the trained text variational autoencoder, a text decoder input vector that is based, at least in part, on the interpolated style vector and the text vector.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

For illustrative purposes, specific example embodiments will now be explained in greater detail in conjunction with the figures.

The embodiments set forth herein represent information sufficient to practice the claimed subject matter and illustrate ways of practicing such subject matter. Upon reading the following description in light of the accompanying figures, those of skill in the art will understand the concepts of the claimed subject matter and will recognize applications of these concepts not particularly addressed herein. It should be understood that these concepts and applications fall within the scope of the disclosure and the accompanying claims.

Moreover, it will be appreciated that any module, component, or device disclosed herein that executes instructions may include, or otherwise have access to, a non-transitory computer/processor readable storage medium or media for storage of information, such as computer/processor readable instructions, data structures, program modules and/or other data. A non-exhaustive list of examples of non-transitory computer/processor readable storage media includes magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, optical disks such as compact disc read-only memory (CD-ROM), digital video discs or digital versatile discs (i.e., DVDs), Blu-ray Disc™, or other optical storage, volatile and non-volatile, removable and non-removable media implemented in any method or technology, random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology. Any such non-transitory computer/processor storage media may be part of a device or accessible or connectable thereto. Computer/processor readable/executable instructions to implement an application or module described herein may be stored or otherwise held by such non-transitory computer/processor readable storage media. Aspects of the present application may be understood to employ processors to carry out various tasks. It is known that graphics processing units (GPUs) are particular well-suited to many of the tasks disclosed herein. However, in implementation, various computation resources other than GPUs may also be employed, such as central processing units (CPUs).

Aspects of the present application relate to generating lyric lines based on a music piece provided by a user. A system that embodies aspects of the present application may suggest, to the user, novel lyric lines that reflect the style and the emotions present in the provided music piece. Responsive to an artist playing a live music piece, or providing a pre-recorded audio clip of a music piece, the system may generate lyric lines that match a detected style of the music piece and have an emotional impact matching the music piece. The user may be shown the lyric lines as the lyric lines are generated in real time. The lyric lines may be seen to suggest phrases and themes that the artist can use, not only to inspire their own lyric composition, but can also use to guide their musical expressions and instrumentation as the artist plays the music piece. The generated lines are not intended to be the complete song lyrics. Instead, the generated lines are intended to act as snippets of ideas and expressions that may inspire the artist's own creativity.

In overview, aspects of a first approach of the present application relate to using generative models to assist songwriters and musicians in the task of writing song lyrics. In contrast to systems that generate lyrics for an entire song, aspects of the present application relate to generating suggestions for lyrics lines in the style of a specified artist. It is expected that unusual and creative arrangements of words in the suggested lyric lines will inspire the songwriter to create original lyrics. Conditioning the generation on the style of a specific artist is done in order to maintain stylistic consistency of the suggestions. Such use of generative models is intended to augment the natural creative process when an artist may be inspired to write a song based on something they have read or heard.

FIG.1illustrates, in a block diagram, a first approach to lyric generation. The first approach includes a converter102configured to receive a 10-second clip of an input song and convert the clip into a mel spectrogram. A spectrogram may be obtained, by the converter102, by computing a Fast Fourier Transform (FFT) on overlapping windowed segments of the input clip. The known mel scale is a perceptual scale of pitches judged by listeners to be equal in distance from one another. Accordingly, a “mel spectrogram” is a spectrogram wherein the frequencies are converted to the mel scale.

The first approach further includes a convolutional neural network (CNN)104that is trained to receive the mel spectrogram and output an artist embedding. The term artist embedding may be used to refer to a vector representative of a particular artist. That is, in a training phase, the CNN104has been provided with a large number of songs by a particular artist. Over the course of the training phase, the CNN104improves at the task of outputting an artist embedding associated with the artist of each input song clip.

The first approach also includes a text variational autoencoder108. The text variational autoencoder108includes a text encoder110, a text latent space112and a text decoder114.

In typical operation of the text autoencoder108, the text encoder110receives, from a text converter106, an input text vector. The input text vector may be understood to exist in a so-called prior distribution of input text vectors. The text converter106receives text and converts the text to vector form. The text converter106works as follows: 1) tokenize the text (i.e., split the text into words); 2) map each word into a vocabulary index; and 3) map each vocabulary index to a word embedding.

During training, providing, to the decoder114, a text decoder input that is a sample obtained from the posterior distribution in the text latent space112is typical of a training phase for a known text variational autoencoder. The text variational autoencoder108differs from the known text variational autoencoder in that the decoder114receives additional input. In the training phase for the text variational autoencoder108ofFIG.1, a text decoder input is formed from a sample obtained from the posterior distribution in the text latent space112along with an artist embedding at the output of the CNN104. When the text input to the text converter106corresponds to a song by a particular artist, the training phase causes the decoder114to learn to produce output lyrics in a manner that is consistent with the particular artist.

An inference phase dispenses with the music converter102, the encoder110and the CNN104and begins with an artist embedding that may be based on a plurality of weights provided by the user. The user may, for example, indicate a weight of 1 for a particular artist embedding and 0 for the rest. Alternatively, the user may indicate a plurality of weights summing to 1, in which case an interpolated artist embedding may be formed. A text sample is obtained from the prior text distribution (e.g., a standard normal distribution) and concatenated with the interpolated artist embedding to form a decoder input vector. The decoder input vector is then decoded by the decoder114, thereby leading to a lyric line.

FIG.2illustrates, in a block diagram, a second approach to lyric generation, in accordance with aspects of the present application.

The second approach includes a spectrogram variational autoencoder208S and a text variational autoencoder208T. Notably,FIG.2does not show equivalent structures to the music converter102and the text converter106ofFIG.1. However, similar structures should be understood to be present as part of the second approach for use in converting music clips into their corresponding spectrograms and text.

As is typical of a generic variational autoencoder, the spectrogram variational autoencoder208S includes a spectrogram encoder210S, a spectrogram latent space212S and a spectrogram decoder214S. In aspects of the present application, the spectrogram encoder210S may be implemented as a convolutional neural network. In aspects of the present application, the spectrogram decoder214S may be implemented as a convolutional neural network.

In a typical training phase of the spectrogram variational autoencoder208S, the spectrogram encoder210S receives an input spectrogram. The structure similar to the music converter102ofFIG.1may be understood to have received a song that includes lyrics. It is expected that the structure has divided the song into a plurality of intervals. The input spectrogram corresponds to an audio signal in a given interval. Lyrics for the given interval are sent toward the text variational autoencoder208T. The spectrogram encoder210S encodes the input spectrogram to produce a posterior distribution in the spectrogram latent space212S. The posterior distribution may be represented as a vector of means and standard deviations. A sample may then be obtained from the posterior distribution corresponding to given interval in the spectrogram latent space212S. The sample may be used as a spectrogram decoder input and provided to the spectrogram decoder214S. An estimated spectrogram at the output of the spectrogram decoder214S is expected, with sufficient training, to approach the input spectrogram for the given interval.

The training of the spectrogram variational autoencoder208S may cause the population of the spectrogram latent space212S with a plurality of spectrogram posterior distributions, where each spectrogram posterior distribution in the plurality of spectrogram posterior distributions corresponds to an interval among the plurality of intervals.

The text variational autoencoder208T ofFIG.2includes a text encoder210T, a text latent space212T and a text decoder214T. In aspects of the present application, the text encoder210T may be implemented as a long short term memory network. In aspects of the present application, the text decoder214T may be implemented as a long short term memory network.

In a training phase for the text variational autoencoder208T ofFIG.2, the text encoder110receives an input text vector. The input text vector may be understood to correspond to an input lyric line derived from an interval and may exist in the prior distribution of input text vectors. The text encoder210T encodes the input text vector to produce a posterior distribution in the text latent space212T. The posterior distribution may be represented as a vector of means and standard deviations. A text vector may then be sampled from the posterior distribution for a particular interval in the text latent space212T. The text vector may be concatenated with a spectrogram vector that has been sampled from the posterior distribution for the particular interval in the spectrogram latent space212S. The text vector concatenated with the spectrogram vector may be used as a text decoder input and provided to the text decoder214T. Reconstructed lyric lines output from a converter (not shown), based on probability distributions at the output of the text decoder214T, are expected, with sufficient training, to approach the input lyric lines. The output probability distributions may be used, by the converter (not shown) to select vocabulary indices and look-up actual words using the vocabulary indices, to generate the reconstructed lyric lines.

The training of the text variational autoencoder208T may cause the population of the text latent space212T with a plurality of text posterior distributions, where each text posterior distribution in the plurality of text posterior distributions corresponds to an interval among the plurality of intervals.

FIG.3illustrates example steps of operation of the second lyric generation approach, ofFIG.2, in an inference phase. In overview, the second lyric generation approach involves receiving a time-limited audio recording and, responsively, generating lyrics. Initially, the spectrogram encoder210S may obtain (step302) a representation of the time-limited audio recording.

The term “representation” may be understood, in some aspects of the present application, to refer to a posterior distribution. In some other aspects of the present application, the spectrogram autoencoder208S (according toFIG.2) need not, necessarily, be variational. In a case wherein the spectrogram autoencoder is not variational, the output of the spectrogram encoder210S may be a vector, rather than a distribution. Furthermore, the spectrogram autoencoder may not even process spectrograms. Instead, other representations of audio may be processed, such as raw waveforms.

The text variational autoencoder208T (according toFIG.2) may sample (step304), from the prior text distribution, a text vector.

The text decoder214T may then generate (step306), with the help of a converter, an output lyric line by decoding a text decoder input vector. The text decoder input vector may be based, at least in part, on the representation obtained in step302and the text vector sampled in step304. Recall that the output of the text decoder214T is, generally, probability distributions. The probability distributions may be converted, by an output converter (not shown) into lyrics lines that may be understood by a user. For simplicity, it may be assumed that an output converter is associated with the text decoder214T for lyric line output purposes.

In some aspects of the present application, the text autoencoder208T need not, necessarily, be variational. However, the text autoencoder208T is expected to fall into a category of autoencoders known as latent variable model autoencoders. Autoencoders in this category include, but are not limited to, a Wasserstein autoencoder and an Adversarially Regularized autoencoder.

FIG.4illustrates, in a block diagram, a third approach to lyric generation, in accordance with aspects of the present application.

The third approach includes a spectrogram variational autoencoder408S and a text variational autoencoder408T. Notably,FIG.4does not show equivalent structures to the music converter102and the text converter106ofFIG.1. However, similar structures should be understood to be present as part of the third approach for use in converting music clips into their corresponding spectrograms and text.

In common with the spectrogram variational autoencoder208S ofFIG.2, the spectrogram variational autoencoder408S ofFIG.4includes a spectrogram encoder410S, a spectrogram latent space412S and a spectrogram decoder414S. In aspects of the present application, the spectrogram encoder410S may be implemented as a convolutional neural network. In aspects of the present application, the spectrogram decoder414S may be implemented as a convolutional neural network.

A training phase for the spectrogram variational autoencoder408S ofFIG.4may proceed in a manner consistent with the training phase, described hereinbefore, for the spectrogram variational autoencoder208S ofFIG.2.

In common with the text variational autoencoder208T ofFIG.2, the text variational autoencoder408T ofFIG.4includes a text encoder410T, a text latent space412T and a text decoder414T. In aspects of the present application, the text encoder410T may be implemented as a long short term memory network. In aspects of the present application, the text decoder414T may be implemented as a long short term memory network.

The third approach ofFIG.4differs from the second approach ofFIG.2in that the third approach aims to align the text latent space412T with the spectrogram latent space412S. To achieve this aim, the third approach ofFIG.4includes Generative Adversarial Network (GAN)420. As is conventional, the GAN420includes a generator network422and a discriminator network424.

A training phase for the text variational autoencoder408T may involve encoding, at the spectrogram encoder410S, a training spectrogram to, thereby, produce a training spectrogram distribution. The spectrogram variational autoencoder408S may then sample, from the training spectrogram distribution, a training spectrogram latent code.

The training phase for the text variational autoencoder408T may further involve encoding, at the text encoder410T, a training input to, thereby, obtain a training text distribution. In aspects of the present application, the training input includes a lyric line corresponding to the training spectrogram and the training spectrogram latent code. Subsequently, the text variational autoencoder408T may sample, from the training text distribution, a training text latent code. A training text decoder input vector may then be decoded by the text decoder414T, thereby leading to a lyric line. The training text decoder input vector may be based, at least in part, on the training spectrogram latent code and the training text latent code.

Subsequent to the training of the spectrogram variational autoencoder408S and the text variational autoencoder408T, training of the GAN420may commence.

Preparing to train the GAN420includes providing an input spectrogram, x(s), to the spectrogram variational autoencoder408S to obtain a spectrogram posterior distribution. The training spectrogram latent code, z(s)=μ(s)+τ(ε·σ(s)), may then be obtained by sampling from the spectrogram posterior distribution. Here, μ(s)denotes the mean predicted by the trained spectrogram variational autoencoder408S and σ(s)denotes the standard deviation predicted by the trained spectrogram variational autoencoder408S. ε˜(0,1) is a random normal noise and τ is a sampling temperature. Preparing to train the GAN420also includes obtaining a training text latent code, z(t)=μ(t)+τ(ε·σ(t)), by providing an input lyric line, x(t), corresponding to the input spectrogram, x(s), to the text variational autoencoder408T.

Training the GAN420involves passing training spectrogram latent code, z(s)through the generator network422. The output, {circumflex over (z)}(t), of the generator network422may be called a predicted text latent code.

A so-called negative sample, {circumflex over (z)}, may be formed by concatenating the predicted text latent code, {circumflex over (z)}(t), with the training spectrogram latent code, z(s).

A so-called positive sample, z, may be formed by concatenating the training text latent code, z(t), with the training spectrogram latent code, z(s).

Upon receipt of the negative sample and the positive sample, the discriminator network424attempts to distinguish between the two inputs. This adversarial training regime may be shown to incentivize the GAN420to match {circumflex over (z)}(t)as closely as possible to z(t).

It is notable that, on the basis that the text encoder410T receives input that includes the training spectrogram latent code, the text variational autoencoder408T may be called a “conditional” variational autoencoder. It is notable that, according to aspects of the present application, the text variational autoencoder408T need not be implemented in a conditional manner.

At inference time, the text encoder410T of the text variational autoencoder408T is no longer needed. The third lyric generation approach is illustrated, inFIG.5, in an inference phase.FIG.6illustrates example steps of operation of the third lyric generation approach as illustrated, inFIG.5, in the inference phase.

A spectrogram that is received (step602) at the spectrogram encoder410, is used to obtain (step604) an inference spectrogram latent code, z(s). The inference spectrogram latent code, z(s), is then received at the generator network422of the GAN420. The generator network422generates (step606), on the basis of the inference spectrogram latent code, z(s), an inference text latent code, z(t). The text decoder414T receives an input vector and, on the basis of the input vector, the text decoder414T generates (step608) an output lyric line. The input vector may be formed by concatenating the inference text latent code, z(t), with the inference spectrogram latent code, z(s).

Notably, the inference method, example steps of which are illustrated inFIG.6, is stochastic, due to the inference spectrogram latent code, z(s), being sampled from the spectrogram posterior distribution in the trained spectrogram latent space412S. Such sampling allows for generation of diverse lyric lines for the same input spectrogram.

FIG.7illustrates, in a block diagram, a fourth approach to lyric generation, in accordance with aspects of the present application.

The fourth approach includes a spectrogram variational autoencoder708S and a text conditional variational autoencoder708T. Notably,FIG.7does not show equivalent structures to the music converter102and the text converter106ofFIG.1. However, similar structures should be understood to be present as part of the third approach for use in converting music clips into their corresponding spectrograms and text.

In common with the spectrogram variational autoencoder208S ofFIG.2, the spectrogram variational autoencoder708S ofFIG.7includes a spectrogram encoder710S, a spectrogram latent space712S and a spectrogram decoder714S. In aspects of the present application, the spectrogram encoder710S may be implemented as a convolutional neural network. In aspects of the present application, the spectrogram decoder714S may be implemented as a convolutional neural network.

A training phase for the spectrogram variational autoencoder708S ofFIG.7may proceed in a manner consistent with the training phase, described hereinbefore, for the spectrogram variational autoencoder208S ofFIG.2.

In common with the text variational autoencoder208T ofFIG.2, the text conditional variational autoencoder708T ofFIG.7includes a text encoder710T, a text latent space712T and a text decoder714T. In aspects of the present application, the text encoder710T may be implemented as a long short term memory network. In aspects of the present application, the text decoder714T may be implemented as a long short term memory network.

The fourth approach aims to induce the text conditional variational autoencoder708T to learn the same latent space topology as the spectrogram variational autoencoder708S. This would mean that data points that are close in the spectrogram latent space712S are expected to be close in the text latent space712T. More concretely, if two audio clips are encoded, by the spectrogram encoder710S, to result in distributions in neighboring regions of the spectrogram latent space712S, their corresponding lyric lines should be encoded, by the text encoder710T, to result in distributions in neighboring regions in the text latent space712T.

In a training phase for the fourth approach, instead of using one prior (standard normal) text distribution to regularize every text posterior distribution, the posterior distribution of the spectrogram variational autoencoder708S may be used as the prior distribution for any given input spectrogram.

More formally, let the input spectrogram be x(s)and let the corresponding input lyric line be x(t). The posterior distribution for the spectrogram in the spectrogram variational autoencoder708S is qϕ(s)(z(s)|x(s)), and the posterior distribution for the lyric line in the text conditional variational autoencoder708T is qϕ(t)(z(t)|x(t), z(s)).

A Kullback-Leibler (KL) term of a loss for the text conditional variational autoencoder708T may be determined between the posterior distribution for the lyric line and a prior distribution. The prior distribution may be set to be the posterior distribution of its corresponding spectrogram in the spectrogram variational autoencoder708S.

The training phase for the text conditional variational autoencoder708T, may involve encoding, at the text encoder710T, a training input to, thereby, obtain a training text distribution, qϕ(t)(z(t)|x(t), z(s)). In aspects of the present application, the training input includes a lyric line corresponding to a training spectrogram and a training spectrogram vector. Subsequently, the text conditional variational autoencoder708T may sample, from the training text distribution, a training text vector. A training text decoder input vector may then be decoded by the text decoder714T, thereby leading to a lyric line. The training text decoder input vector may be based, at least in part, on the training spectrogram vector and the training text vector.

FIG.8illustrates example steps in a lyric generation method using the fourth approach illustrated inFIG.7, in accordance with aspects of the present application.

In an inference phase, initially, the spectrogram encoder710S may encode an input spectrogram to obtain an inference spectrogram distribution. The spectrogram variational autoencoder708S may obtain (step802) an inference spectrogram vector. The obtaining (step802) of the inference spectrogram vector may involve sampling from the inference spectrogram distribution.

The text conditional variational autoencoder708T may sample (step804), from a text distribution in the text latent space712T, a text vector. More particularly, the text conditional variational autoencoder708T may first select the text distribution on the basis that a location, in the text latent space712T, of the text distribution corresponds to a location, in the spectrogram latent space712S, of the inference spectrogram distribution obtained by encoding the input spectrogram.

The text decoder714T may then generate (step806) an output lyric line by decoding a text decoder input vector. The text decoder input vector may be based, at least in part, on the inference spectrogram vector obtained in step802and the text vector sampled in step804.

In contrast to the second, third and fourth approaches outlined hereinbefore, a fifth approach to the task of lyric generation involves receipt of music from a user during model training time, but not during inference.

FIG.9illustrates a globally relevant spectrogram autoencoder908S and a locally relevant text variational autoencoder908T. The globally relevant spectrogram autoencoder908S may be variational or deterministic.FIG.10illustrates a lyric generation system1000configured to implement the globally relevant spectrogram autoencoder908S and the locally relevant text variational autoencoder908T ofFIG.9.

In common with the spectrogram variational autoencoder208S ofFIG.2, the globally relevant spectrogram autoencoder908S ofFIG.9includes a spectrogram encoder910S, a spectrogram latent space912S and a spectrogram decoder914S. In aspects of the present application, the spectrogram encoder910S may be implemented as a convolutional neural network. In aspects of the present application, the spectrogram decoder914S may be implemented as a convolutional neural network.

In common with the text variational autoencoder208T ofFIG.2, the locally relevant text variational autoencoder908T ofFIG.9includes a text encoder910T, a text latent space912T and a text decoder914T. In aspects of the present application, the text encoder910T may be implemented as a long short term memory network. In aspects of the present application, the text decoder914T may be implemented as a long short term memory network.

In a global training phase, a global training data set of spectrograms is used. Notably, the global training data set of spectrograms also includes corresponding lyrics. In the case wherein the globally relevant spectrogram autoencoder908S is deterministic, the output of the spectrogram encoder910S is a vector in the spectrogram latent space912S. In the case wherein the globally relevant spectrogram autoencoder908S is variational, the output of the spectrogram encoder910S is a distribution in the spectrogram latent space912S. In the following, the term vector will be used in places that may also use the term distribution. It should be understood that the term distribution should be substituted for the term vector in the case wherein the globally relevant spectrogram autoencoder908S is variational.

In a local training phase, a user plays a piece of music in a first “style” or a first “mood.” The terms “style” and “mood” may be used interchangeably. The spectrogram encoder910S encodes an interval of the piece of music played by the user to generate a “new” spectrogram vector. The lyric generation system1000compares the new spectrogram vector to each of the spectrogram vectors in the spectrogram latent space912S.

The comparing may, for example, involve determining a cosine distance between the new spectrogram vector and each of the spectrogram vectors in the spectrogram latent space912S.

The lyric generation system1000selects, from among the spectrogram vectors, a set of n top-ranked spectrogram vectors that are “closest” to the new spectrogram vector (e.g., the selected vectors have the lowest cosine distance).

The encoding and comparing may be repeated for all of the intervals that make up the piece of music played by the user. For a three-and-a-half minute piece of music divided into 10 second intervals, there will be 21 intervals to be encoded and compared. For each of the 21 intervals, the lyric generation system1000may be understood to obtain n “closest” vectors from the set vectors in the spectrogram latent space912S. The lyric generation system1000may then create a “pool” of spectrogram vectors (say,6000). The creation of the pool may, for example, involve creating a union of all “closest” vectors for all 21 intervals and creating a single, ranked list. The creation of the single, ranked list may, for example, use a Rank Biased Overlap (RBO) measure. The lyric generation system1000may then then select a representative collection of top-ranked spectrogram vectors. These top-ranked vectors may be the vectors that have the highest cosine similarity to most of the 21 intervals.

The lyric generation system1000may then determine an average of all vectors in the collection of top-ranked vectors. The lyric generation system1000may then determine an average vector. The average vector may, going forward, be considered a “style” vector for association with the given three and a half minute piece of music.

The user may follow up the provision of the first piece of music with provision of further pieces of music in distinct styles. By repeating the encoding, comparing and selecting outlined hereinbefore, the lyric generation system1000may obtain a style vector to associate with each of the pieces and, consequently, the style vector is associated with the style of the music played by the user. If the first musical piece provided by the user was rock, the subsequent pieces can be blues, jazz and reggae. One of the benefits of the training phase is the obtaining of a plurality of style vectors. In the example presented, there are four style vectors including one style vector for each of rock, blues, jazz and reggae.

As discussed hereinbefore, the training phase for the globally relevant spectrogram autoencoder908S involved selecting a representative collection of top-ranked spectrogram vectors. It is understood that each of the top-ranked spectrogram vectors is a result of encoding a spectrogram in the training set and that each of the spectrograms in the training set has a corresponding lyric line.

A training phase may be applied to the locally relevant text variational autoencoder908T ofFIG.9. By using only the lyric lines in the training set that correspond to the top-ranked spectrogram vectors, the training phase for the locally relevant text variational autoencoder908T conserves computing resources while making the result personal to the user. The locally relevant text variational autoencoder908T may be trained as a conditional autoencoder, as discussed hereinbefore, by providing a training spectrogram vector to the text encoder910T as part of an input vector. However, it should be clear that conditionality at the locally relevant text variational autoencoder908T is not essential.

At inference time, illustrated, inFIG.11, as example steps of a lyric generation method using the fifth approach illustrated inFIGS.9and10, in accordance with aspects of the present application, input from the user does not include music. Instead, as illustrated inFIG.10, the lyric generation system1000receives (step1102) input from the user, where the input includes a plurality of weights. Notably, the weights may be normalized to sum to one. Each weight among the plurality of weights corresponds to a style vector among the plurality of style vectors that have been established in the local training phase. The lyric generation system1000may generate (step1104), by weighting each style vector by the corresponding weight, an interpolated style vector. The lyric generation system1000may then obtain (step1106) a text vector by sampling from a prior distribution in the text latent space912T. The lyric generation system1000may then cause the text decoder914T generate (step1108) an output lyric line by decoding a text decoder input vector. The text decoder input vector may be based, at least in part, on the interpolated style vector generated in step1104and the text vector sampled in step1106.

The text autoencoder108,208T,408T,708T,908T in the approaches discussed hereinbefore may be conditioned to generate lyric lines conforming to requirements other than those requirements already discussed. For example, rhyme conditioning may be implemented, as outlined in the following.

One task related to rhyme conditioning relates to processing a dataset. Each lyric line in a training dataset of lyric lines may be converted into a phonetic transcription. Such a conversion may be accomplished using, for example, the known Carnegie Mellon University Pronouncing Dictionary. All syllables beginning from and including a last stressed syllable in a given lyric line may then be extracted. The extracted syllables may then be identified as a rhyming pattern for the given lyric line. Each lyric line in the training dataset may then be processed so the each lyric line may be labelled with a numerical index corresponding to the identified rhyming pattern.

Another task related to rhyme conditioning relates to training the model. For example purposes, consider that a dataset of 30,000 lyric lines has 2,000 rhyming patterns and each rhyming pattern is associated with a rhyming pattern embedding. The 2,000 rhyming pattern embeddings may be randomly initialized, as per usual practice. For example, randomly initializing the rhyming pattern embeddings may be accomplished by performing a random sampling of real numbers from a uniform distribution. The rhyming pattern embeddings may then be set to be trainable, as per a usual stochastic gradient descent training procedure, while training the text autoencoder. The text decoder input vector may be based, at least in part, on the representation obtained in step302and the text vector sampled in step304and the rhyming embedding corresponding to the rhyming pattern of the given lyric line.

Upon completion of rhyme conditioning, generating rhyming lyric lines may involve receiving, from a user, a rhyming pattern. For example, the user may provide a word or a text to rhyme with. A rhyming pattern may be extracted from the user-provided text. From the extracted rhyming pattern, an associated rhyming pattern embedding may be determined. Subsequently, the rhyming pattern embedding may be provided as input to the text decoder114,214T,414T,714T,914T in addition to other decoder inputs described in in the approaches discussed hereinbefore. As a consequence of the training and the input, the text decoder114,214T,414T,714T,914T may generate a lyric line that matches the specified rhyming pattern.

It should be appreciated that one or more steps of the embodiment methods provided herein may be performed by corresponding units or modules. For example, data may be transmitted by a transmitting unit or a transmitting module. Data may be received by a receiving unit or a receiving module. Data may be processed by a processing unit or a processing module. The respective units/modules may be hardware, software, or a combination thereof. For instance, one or more of the units/modules may be an integrated circuit, such as field programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). It will be appreciated that where the modules are software, they may be retrieved by a processor, in whole or part as needed, individually or together for processing, in single or multiple instances as required, and that the modules themselves may include instructions for further deployment and instantiation.