AI as a Movie Maker. How I created an entire short movie... | by Merzmensch | Towards Data Science

It ’s been a blast. I remember watching “Sunspring” — again and again. Fascinated and mesmerized by the absurd dialogues, I was trying to comprehend what’s going on in this short movie. But the meaning used to slip away. Because it was written by AI.
Benjamin was the name of the author. Behind this name was a recurrent neural network LSTM hidden, developed by Ross Goodwin, an AI researcher and poet. The movie was directed by BAFTA-nominated British filmmaker Oscar Sharp and featured a Thomas Middleditch among others.
Even if the plot consisted of absurdist and seemingly random phrases, our human brain was trying to get it. Cogito, ergo sum — I think therefore I am.
Ross’ must-read essays about AI and creativity were highly inspiring for me. If you haven’t a chance, do read this.
medium.com
medium.com
With his experiments (and later with his AI-generated book “1 the Road”), Ross proved what is possible to achieve with Artificial Intelligence. And also this realization was a crucial one: AI is neither just a tool nor an entire replacement for a writer. You need a symbiosis to create art together with AI.
We’ve seen how many creative uses of Machine Learning are possible.
With the 3D Ken Burns Effect, we can make animated dream visions, converting a photo to a spatial camera flight.
With GPT-2 by OpenAI, we can write entire stories.
But what if we combine all the approaches to one single artwork? And so I did it and generated a short movie completely with AI.
...But before you watch it: do you have an idea, what do you need to create a movie?
You surely can waive some elements for arthouse quality, but usually, you need the following ingredients to make a movie (even if a short one):
the Plot
the Camera / Visuals
the Actors
the Music
You also need an original idea and (in most cases) still unused media assets.
So how can we apply the Machine Learning approach to fulfill all of these points?
As previously mentioned, GPT-2 is a powerful NLP model (pre-trained on 40 GB text with 1.5 billion parameters) to generate writings. Its main flaw — and the major advantage — is the surreal character of created contents. They usually don’t make sense (at least for knowledge transfer needs — in a literary context they can be a decent object of interpretation). And: every time a new unique text is generated.
Let’s see which ways GPT-3 will take in the future (OpenAI published this new giant model with 175 billion parameters, which at the moment cannot be fine-tuned due to its excessive dimensions).
For my needs I used a GPT-2 Colab Notebook with Java Interface by Manuel Romero:
colab.research.google.com
There are several different notebooks, allowing you to fine-tune GPT-2 or train on your texts, but I’ve chosen this particular Colab Notebook for its user-friendly and robust structure, and support of the entire 1.5 billion parameter model.
For my movie I entered the following prompt:
Empty room.A man enters the room.He looks out of the window and says: "It still hasn't begun".The woman enters. She has a white dress on and a book in her hand.The woman: "I am waiting as well."The man: "For how long?"The woman: "For a while."
GPT-2 provided me with various texts, but the one I’ve selected was an amazing piece of an absurdist theater play, somewhere between Beckett, Mrozek and David Lynch:
Empty room.A man enters the room.He looks out of the window and says: "It still hasn't begun".The woman enters. She has a white dress on and a book in her hand.The woman: "I am waiting as well."The man: "For how long?"The woman: "For a while."The man: "Ah, you know, I'm a very patient man."The woman: "I know, of course."The man: "I'm going to have to be less patient, then." The woman starts to laugh; the laughter has an ominous edge that only she can feel."In my defence," she says, "I never actually told you what it was I was going to tell you."The man, suddenly, remembers: "That's right."The woman: "So you are lying behind me." The man is surprised by the abruptness of the accusation. He does not, however, feel any anger; as he has noticed in previous encounters, it is the woman who is most convincing when she's saying the lies.The man: "But it was not what you expected."The woman: "Oh...no." She is trying to sound convincing.He wants to ask her about her mother. The woman: "Do you think she'll be willing to lie?"The man turns down the opportunity.The woman: "Maybe...""Why?""Maybe she's still grieving." She is smiling when she says this, and the man doesn't think she'll be the first to smile when the truth comes out.He gives up. "No."The woman: "It's not over." The man has not got to see her family again. He had hoped to return to his work. He has nothing else to do. He wants to be a man.
It convinced me also because of ambiguous communication levels. It’s as if there is a huge story hidden behind all those sparse phrases (human brain does wonders during interpretation, indeed).
So I had a screenplay.
I am a fan of StyleGAN2, but especially in its implementation in ArtBreeder (at least in the section “Portraits”).
You can generate new faces, you can make transitions or even animations (read more about using Artbreeder here). So I created a bunch of images for faces (see here the assets) of the Man, the Woman, and some Room pictures.
It was a tricky thing to align the face modification to the story part (e.g. in the “laughter moments” etc.), but after various trials and errors, it worked for me.
Here are just some assets used for the visuals:
Sure, you can use the First Order Motion Model for dynamical face animations (using yourself with your Andy Serkis skills):
But for my absurdist and minimalist style, the face morphing was most appropriate.
The faces were already made, but something was still missing: voices.
Voice generation is one of probably the oldest Machine Learning approaches. My favorite was MelNet — a Model with unbelievable quality. Just listen to the samples (trained on professional speakers or also celebrities datasets). Alas, MelNet wasn’t available as a working repository or Colab Notebook.
My second choice was Amazon Polly. Being a part of Amazon AWS AI/ML-Services Polly provides a broad quantity of voices in various languages. The downside is: most of them have rather a charisma of an anchorman and are not always suitable for fictional content.
I used one voice for my AI-generated video “Predictions”:
Still, the non-emotional voice is not really convincing, if used in some dialogues.
Then I discovered Replica Studios.
They provide at the moment a small collection, but the voices have astonishing quality. Some of them can be rather used for anchorman need, but another bear already theatrical power within.
I’ve chosen these three voices — Deckard for Narrator, Carlos for Man, and Audrey for Woman.
You also can experiment with the emotional features of various voices, you even can train them your voice, but these three were perfect for my needs.
The use of Replica Studios is pretty easy. You sort the phrases in chronological order and apply the appropriate voices to the characters:
It’s possible to export the voices to mp3-files for your project.
AI-generated music achieved new levels of quality this year after OpenAI released JukeBox — a library of 7k songs and music pieces, being generated with pre-trained models on various musicians and songwriters (read also here).
Many of them are pretty glitchy (like this nightmarish “Mozart” piece). But many of them are beautiful. And especially combined with AI-generated visuals they evoke strange feelings within.
I am using them for my series “Breath ZeroX”:
For our video something cinematic should pass, so I looked up the music trained on Hans Zimmer (and changed the speed a little bit):
The rest was rather hard work of a cutter — for my video I used Premiere Pro and amounts of coffee. Probably the most tricky thing was stretching the videos in time and synchronize them with the plot (original face transitions by ArtBreeder were between 8–30 sec long).
Advice: I used “Time Interpolation: Optical Flow”. It creates new frames between the existent and makes the footage fluent. Sometimes it generates glitches — but they are always welcome if it’s about digitally generated films!
And now: here it is.
I wonder which new audio-visual treasures will come with new ML-approaches. Let‘s stay tuned!