Deep Learning with Tensorflow: Part 3 — Music and text generation | by Matteo Kofler | Towards Data Science

Hi everybody, welcome back to my Tenserflow series, this is part 3. I already described the logic and functionality of neural networks and Tenserflow in the first part as well as I showed you how to perform a image classification in the second part. In this part, I’ll give you a overview on Tensorflow applications as well as I’ll show you how to generate music and text.
Tensorflow allows us to build complex applications. While processing multiple input sources, almost everything can be implemented with Tensorflow, even Tesla’s autopilot could be made with it. To obtain a better overview on the possibilities of Tensorflow, i have collected a short summary of examples that people have already made with it:
Processing text: spam filters, automated answers on emails, chatbots, sports predictions
Processing images: automated cancer detection, street detection
Processing audio and speech: sound generation, speech recognition
Next up, I’ll explain music generation and text generation in more detail. We’ll start looking at music generation, the possibilities of generating music today and we’ll write our own pop music generator. Later we’ll look at text generation, at a concret example and on building our own poem generator. Let’s do it.
Dear songwriters, I am sorry to tell you, but you will be replaced by machine learning algorithms within the next years. Why? Because music can be generated by computers now. Imagine the power such a tool has. Compared to the time, to the hundreds of hours necessary to compose a song, clicking start in a program is like nothing.
I guess that might sound a little bit strange for some people, considering the fact that music is often referred to as “emotional” and something “to feel”, not something “factual” or “mathematical”, but machine learning makes it possible. But how would it work?
So, let’s say you wanted to make a specific song, a “happy” and “funny” song. The first step would be to get a data set of songs, labeled with emotions. The program would convert the speech of the set to text. For each word, the program would create vectors and train the model. After completing the training, the program would find the associated vectors of your input (“happy” and “funny”) and compare them to the vectors that has been created before. The output would be a set of chords representing the emotions you wanted.
Magenta is currently state of the art when it comes to music generation with machine learning, but listen for youself. It’s a project from the Google Brain team that asks: Can we use machine learning to create compelling art and music? Built on top of TensorFlow, Magenta uses a CNN system.
Although Magenta is very powerful, we’ll build our own simple music generator using a Restricted Boltzmann Machine (RBM) to generate short sequences of pop music.
We’ll receive multiple files, if you link them together it’ll sound like this:
Our training data will be around a hundred MIDI files of pop songs (MIDI is a format that directly encodes musical notes). To keep it simple, we won’t label them with emotions as described in the introduction, our output will be another pop melody, just like our training files.
RBM is a neural network with two layers, the visible layer and the hidden layer. Each visible node is connected to each hidden node (and vice versa), but there are no visible-visible or hidden-hidden connections (nodes are simply where calculations take place). This is the restriction. Each visible node takes one chord. Each chord is multiplied by a weight and then the node’s output at the hidden layer. Unlike most of the neural networks, RBMs are generative models that directly model the probability distribution of data. To sample from an RBM, we perform an algorithm known as Gibbs sampling. If you want to learn more about RBM’s, go this way.
Enough theoretical talk, let’s start building by cloning my repo from GitHub. Open up a terminal and type:
git clone https://github.com/koflerm/tensorflow-music-generator.git
Next, we’ll have to install a few libraries:
pip3 install pandaspip3 install msgpack-pythonpip3 install glob2pip3 install tqdmpip3 install py-midi
We’ll use them as follows:
Panda as our data analytics library
Midi as our helper library for the music files
TQDM for printing a progress bar during the process.
The songs are stored in the Pop_Music_Midi folder. You could also use your own training data, but the songs have to be in the MIDI format(!). Find some here.
Basically, that’s the preparation, you can now execute the program. To run the script, simply write:
python rbm_chords.py
Training will take 5–10 minutes. The output will be a collection of MIDI files in the out folder, each about ten seconds long. You could also write a script to link them together, if you wanted to hear just “one song”.
That’s it, you have successfully created music. If you didn’t understand everything I wrote (i’m sorry...), I would highly recommend you to read respectively listen to this two postings:
https://www.youtube.com/watch?v=ZE7qWXX05T0&feature=youtu.be
http://danshiebler.com/2016-08-10-musical-tensorflow-part-one-the-rbm/
As explained earlier, songwriters could be replaced by algorithms in near future. Well, this also applies to poets. We’ll write a program that generates a poem based on the plays of William Shakespeare. The training data (the poems) are stored as text-files, the filename is also the label. But to start off, I’ll show you another example of text generation using Tensorflow.
Replying to emails on mobiles is a real pain, even for short replies. That’s why Google implemented Smart Replies in its Inbox app. They wanted to build a system that could automatically determine if an email was answerable with a short reply (text recognition), and compose a few suitable responses for it (text generation).
They needed to build a machine learning system, because imagine using a system that depends on hand-crafted rules for common reply scenarios for a second. Any engineer’s ability to invent “rules” would be quickly outstripped by the huge diversity of writing styles. A machine learned system, by contrast, captures diverse situations by itself. It generalizes and handles new inputs way better than rule-based systems ever could.
Google choose to build the Smart Reply System on a pair of recurrent neural networks (RNN), one used to encode the incoming email and one to predict possible responses. The encoding network takes the words of the incoming email one at a time, and produces a vector. This so called “thought vector” captures the core of what is being said without getting hung up on how it’s said (e.g. the vector for “Are you free tomorrow?” should be the same as the vector for “Does tomorrow work for you?”). The second network starts from this thought vector and makes a grammatically correct reply one word at a time, like it’s typing it out. The detailed operation of each network is entirely learned, just by training the model to predict likely responses.
Interested in the Smart Reply System? Read a more detailed explanation here.
It’s time to build our text generator on the works of William Shakespeare. We’ll use a Recurrent Neural Network for training our model. After training it, our script will generate a poem itself, which can be viewed in the terminal as well as in a textfile.
First off, I know poems aren’t that interesting at all. But don’t worry, you can easily change the data set ;). Let’s dive into it.
Start by cloning my repository from GitHub:
git clone https://github.com/koflerm/tensorflow-shakespeare-poem-generator.git
As you can see, there is a Shakespeare folder containing multiple poems. Simply put your own TXT files in there, if you want to change the topic.
Now it’s time to train our language model.
> python3 rnn_train.py
This takes hours, so be careful! If you don’t have that much time, you can download the checkpoint files (the final model). You don’t have to train the model first! Simply extract the files into the checkpoints folder.
Everything set up, time to generate some text. Type the following line:
> python3 rnn_play.py
The script rnn_play.py uses our trained checkpoint to generate a new “Shakespeare” play. You can see the output in the terminal as well as in the file output_generated.txt
There it is, your own personal poem, written the Shakespeare way.
Congratulations, you are now able to generate your own music as well as your own version of a Shakespeare play. In the next part, I’ll probably dig in a little bit deeper into image classification and object detection. So stay tuned till the next post!
___
Link to Part 4: https://medium.com/towards-data-science/deep-learning-with-tensorflow-part-4-face-classification-and-video-inputs-fa078f22c1e5