Frost Songs: Using AI to Generate Music from Poems
By: Robert A. Gonsalves
Tags: Music Creation, Editors Pick, Artificial Intelligence, Machine Learning, Poetry

Each line of a selected poem is fed into the system, one line at a time. It uses a module named Pyphen to break each word in the line into syllables using the Hunspell hyphenation dictionary [9]. The results are fed into the Lyrics-to-Melody model. The model is a hybrid between a GAN and a Long Short-Term Memory (LSTM) model which generates notes in MIDI format.

The resulting musical phrase is analyzed using MIT’s Music21 library [10] to determine which key it is in. The phrase is then transposed to C major (or A Minor) and quantized to sixteenth notes using Music21. After all of the lines of music are generated, the resulting MIDI file is fed into the Music Transformer model, which adds an accompanying musical part and creates a human feel with expressive keyboard velocity and timing.

Finally, Google’s Magenta library [11] is used to do some post-processing on the final MIDI file, like assigning instrument voices.

In the next section, I will walk through the steps in more detail and will show the Python code I wrote for the custom processing.

Robert Frost, Photo from the Boston Public Library, Public Domain

System Walkthrough

For the walkthrough, we’ll use a short, complete poem by Frost, called “Plowmen” [12]. I’ll show the key snippets of Python code used to turn this poem into a song.

Preparing the Poems

The first step in processing involves breaking each word into syllables and creating word embeddings to be fed into the LSTM-GAN.

Here’s the sample poem.

Plowmen

A plow, they say, to plow the snow.

They cannot mean to plant it, no–

Unless in bitterness to mock

At having cultivated rock.

- Robert Frost

Here is the code snippet that breaks up each word into syllables, and feeds them into the LSTM-GAN. You can see that it creates and sends in embeddings for both the words and syllables using Word2Vec [13]. According to Google, “Embeddings learned through Word2Vec have proven to be successful on a variety of downstream natural language processing tasks.”

You can also see how I deal with words that are not in the dictionary for the embeddings. If a word is not in the dictionary, I simply substitute the correct number of syllables with “la”. This is a tradition for songwriters when they haven’t written all the lyrics, yet.

Here are the sets of syllables for the lines in the poem.

A plow, they say, to plow the snow.

['A', 'plow', 'they', 'say', 'to', 'plow', 'the', 'snow'] They cannot mean to plant it, no–

['They', 'can', 'not', 'mean', 'to', 'plant', 'it', 'no'] Unless in bitterness to mock

['Un', 'less', 'in', 'bit', 'ter', 'ness', 'to', 'la'] At having cultivated rock.

['At', 'hav', 'ing', 'la', 'la', 'la', 'la', 'rock']

You can see that the words “mock” and “cultivated” got turned into “la”s.

Generating Melodies

Once the embeddings are set up for the words and syllables, it’s fairly easy to generate a melody. Here’s the code.

The length_song variable is set to the number of syllables being passed in for the line of text. The model is hard-coded to take in 20 syllables, so the code will limit the input, and, if necessary, pad the input by repeating the last syllable. Note that this padding will be ignored by the model, and we will get out a vector of notes that equals the number of syllables in the line. Here is the melody.

Plowmen, Original Melody, Poem by Robert Frost, Video and Audio by Author

Processing the Melody

The output from the LSTM-GAN is pretty good, but there are some issues: the music is not quantized and the key changes on every line. The original code for the LSTM-GAN system has functions to “discretize” the melodies and transpose them to a uniform key. But I chose to use the Music21 library to perform these functions.

The code below shows how I quantize each note to sixteenth notes (lines 12 and 13) and extend the last note to the end of a measure (line 22).

You can also see how I transpose each line to be in the key of C using Music21 (line 27 calling the function at line 3). Here is the generated melody.

Plowmen, Preprocessed Melody, Poem by Robert Frost, Video and Audio by Author

The next step is to pass the melody into the Music Transformer to create an accompanying track, and to give the melody a more human feel. Here’s the code.

The first 11 lines of code will set up the transformer. The rest of the code takes in a note sequence called melody_ns and generates an accompanying track with the original melody merged into accompaniment_ns.

The final steps of processing are to assign the musical instruments and create an ending by holding the last notes for an extra measure. Here’s the code for the final steps.

Because the timing of the original melody gets changed in the transformer, I had to write a little function to map the closest notes between the two sequences. I then use the function to find the changed notes to set the instrument to guitar. The last block of code holds the last five notes for an extra measure. This is a little trick to signal the end of the song.

Here’s the final song.

Plowmen, Finished Song, Poem by Robert Frost, Video and Audio by Author

The accompaniment from the Music Transformer really adds depth and color to the piece. The model also gives the piece a more human feel by varying the start times and velocity of the MIDI notes.

Results

In general, this approach to music generation produces decent results.

After transposing and fixing up the timing of the lines of melody, the LSTM-GAN seems to produce good melodies for the poems. Poetry has qualities that make good music, like meter, structure, and lyrical flow. The LSTM-GAN gets both the syllables for the meter and the words for the meaning as inputs. Qualities from both of the inputs lead to good melodies. Note that this model is very consistent, in that it will generate nearly the same melody given the same input text.

The Music Transformer, however, is a bit hit-and-miss. The output for the same input melody varies a lot. Some of the resulting accompaniment is not good at all; most of it is OK; but occasionally it creates a musical part that is very creative and quite excellent!

There are more finished songs in the appendix, so be sure to check them out.

Summary

In this post, I explained how I created an AI system to generate songs based on the poetry of Robert Frost. The system is based largely on other people’s work, primarily the LSTM-GAN by Yi Yu and associates and the Music Transformer by researchers at Google.

Source Code

All source code for this project is available on GitHub, here. You can experiment with the code using this Google Colab. The sources are released under the CC BY-SA license.

Attribution-ShareAlike

Acknowledgments

I would like to thank Yi Yu, Jennifer Lim, and Oliver Strimpel for their help with this project.

Future Work

Future work would include using the initial key determined by the LSTM-GAN, and not forcing everything into C maj / A min. Also, the tempo could be determined using something like the median line length of the poem, e.g. more syllables = faster tempo.

And there is an entire world of poetry that could be fed into the system to see what types of songs the system would create.

References

[1] S. Hendrix, “Robert Frost wrote this masterpiece in about 20 minutes. It belongs to all of us now.”, The Washington Post, 2019, www.washingtonpost.com/history/2019/01/01/robert-frost-wrote-this-masterpiece-about-minutes-it-belongs-all-us-now

[2] R. Frost, “New Hampshire”, 1923, en.wikisource.org/wiki/New_Hampshire_(Frost)

[3] Google, Colab Notebooks, 2019, magenta.tensorflow.org/demos/colab

[4] C. M. Payne, “MuseNet”, OpenAI, 2019, openai.com/blog/musenet

[5] AIVA, “The Artificial Intelligence composing emotional soundtrack music”, 2016, www.aiva.ai

[6] Y. Yu, A. Srivastava, and S. Canales, “Lyrics-Conditioned Neural Melody Generation”, 2019, arxiv.org/pdf/1908.05551.pdf

[7] C.A. Huang, A. Vaswani, J. Uszkoreit, N. Shazeer, I. Simon, C. Hawthorne, A.M. Dai, M.D. Hoffman, M.Dinculescu, and D. Eck, “Music Transformer: Generating Music with Long-Term Structure”, 2018, arxiv.org/pdf/1809.04281.pdf

[8] Yamaha, e-Piano Competition, 2002–2011, www.piano-e-competition.com,

[9] Pyphen, Python Hyphenator, 2014, pyphen.org

[10] MIT, Music21, 1995, web.mit.edu/music21

[11] Google, Magenta, 2016, magenta.tensorflow.org

[12] R. Frost, “Plowmen”, A Miscellany of American Poetry, 1920, en.wikisource.org/wiki/Plowmen

[13] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient Estimation of Word Representations in Vector Space”, 2013, www.tensorflow.org/tutorials/text/word2vec

Appendix

Here are some AI-generated songs based on Robert Frost's poems.