12 Colab Notebooks that matter. StyleGAN, GPT-2, StyleTransfer... | by Merzmensch | Towards Data Science

It took at least two AI Winters to survive. The story is obvious: theoretically, Artificial Intelligence as a concept was already here. Neuronal Networks (as a concept: 1943 — McCulloch&Pitts /functional: 1965 — Ivakhnenko/Lapa), Machine learning (Samuel, 1959), Backpropagation (Werbos — 1975), just to name some of the key researches. Theoretically. But in practice, computational power was still like this:
AI research had still to wait for practical realization. Finally, in the 2000s, every owner of a personal computer became able to do experiments with Machine and Deep Learning.
I remember 2016, as Google Deep Dream emerged (thank Alexander Mordvintsev). It took a good amount of time for me to set it up and to get it running. But the results were astonishing:
I was in love with AI. And I wanted to try out more. Then I discovered Colab Notebooks. And they changed my life.
Google Colab Notebooks enable the democratization of Data Science. They allow it everybody — AI researcher, artist, data scientist et al. — to enjoy the power of Machine and Deep Learning on every device (even on smartphones). Just run the cells, change the parameters, values, and sources, and enjoy the diversity of AI.
I want to share with you some of my favorite Notebooks. Try them out! You can find many great essays here at Towards Data Science about backgrounds and working with Colab Notebooks.
What’s this about?
DeepDream visualizes pattern recognition, interpretation and iterative generation by Neural Networks. By increasing this creative interpretation you can produce dream-alike imagery.
Neural Networks act like our brain in the case of Pareidolia: it looks for familiar patterns, which derive from datasets they were trained on.
The example above (a screen from my presentation on the AI Meetup Frankfurt, November 2019) demonstrates how our brain recognizes a face in the rock formations of Cydonia region on Mars. A user Nixtown transformed Da Vinci’s Mona Lisa by continuous DeepDream iterations — and AI recognizes weird patterns.
Often our brain recognizes patterns or objects which aren’t there. But if our human perception does it, why AI shouldn’t?
Links:
Original Blog post by Alexander Mordvintsev
GitHub
Colab Notebook
Things to try out:
Try to generate different patterns and to resize images (octaves). Use more iterations. And don’t be afraid of insanity — the results could be unsettling.
Fun fact:
Initially, DeepDream used to recognize in every pattern mostly dog faces. According to FastCompany, the Network was trained on...
a smaller subset of the ImageNet database released in 2012 for use in a contest... a subset which contained “fine-grained classification of 120 dog sub-classes (FastCompany).
Read more about my experiments with DeepDream.
What’s this about?
BigGAN was one of the first prominent Generative Adversarial Networks. Trained on ImageNet at a now-humble 128x128 resolution this Network became a standard by its manifold generative abilities.
In this notebook, you can generate samples from a long list of categories.
Links:
BigGAN paper on arXiv (Andrew Brock, Jeff Donahue, and Karen Simonyan. Large Scale GAN Training for High Fidelity Natural Image Synthesis. arxiv:1809.11096, 2018)
Colab Notebook
Things to try out:
The same notebook allows you to create interpolation between images. This approach was — and is — mindblowing and innovative since only Surrealists and Abstract Artists were previously so courageous to combine incompatible things. Now you can do it with photorealistic results, for example, generate an interpolation between a Yorkshire terrier and a Space shuttle.
Read also:
BigGAN as a creative engine
BigGAN and Metamorphosis of Everything
What’s this about:
In this experiment, the Deep Learned systems examine two source images — and transfer their style: not only colors but also shapes and patterns.
Links:
Lucid (TensorFlow)
Colab Notebook
Read also:
Style Transfer (and you can do it as well)
AI&Creativity: Alien Elements with Style
There are many approaches to train AI on Artworks.
One of them was provided via Reddit: StyleGAN trained on Artwork Dataset with 24k images from Kaggle.
You get interesting results, it could even be possible to trace the original artworks the model was trained on. (New art game, everybody? “Visual Etymology”):
Another one is WikiART StyleGAN2 Conditional Model, provided by Peter Baylies (et al) and packed into a notebook by Doron Adler:
This model was trained on WikiART images. It allows even to choose between artists, genres and styles.
And the images are impressive:
Things to try out:
Every new combination produces interesting artworks. Try to select artists from different epochs with an untypical style. Like Picasso & Renaissance or Shishkin & Pop Art. The results are sometimes unexpected and not always comprehensible, but that’s art after all.
Links:
_C0D32_ Colab Notebook (trained on 24k Artworks)
WikiART StyleGAN2 Colab Notebook
Read also:
How to Train Your Artist.
The Non-Treachery of Dataset.
What’s this about?
This network, developed by NVidia, is at the moment (6th March 2020, 4:27 pm) the most advanced generative network for images. It was trained on High-Definition-Datasets (for example, Faces from Flickr-Faces-HQ). StyleGAN2 provides automatically learned, unsupervised separation of high-level attributes, stochastic variation and control of layers with visual features.
There are various StyleGAN2-Notebooks (benefits of crowdsourced research), but my favorite is finetuned by Mikael Christensen.
Links:
Paper: https://arxiv.org/abs/1812.04948
Video: https://youtu.be/kSLJriaOumA
Code: https://github.com/NVlabs/stylegan / https://github.com/NVlabs/stylegan2
Colab Notebook
Things to try out:
There are various default datasets by NVidia available within the notebook (mind the resolution):
Try out new Datasets. Train your own models or use those, provided by various artists and researchers, like Michael Friesen (follow his twitter for new updates).
Bacteria StyleGAN
Art-StyleGAN:
MC Escher-StyleGAN:
Produce videos of interpolations:
Try out StyleGAN2 projection. With the StyleGAN2 notebook you discover (or better: re-cover) images being hidden in the Latent Space of the Network. StyleGAN Projection is a method to trace back StyleGAN2-generated images — so you can detect a photo as an AI-generated product (#DeepFake debunk). It still has some downsides: you have to know the concrete dataset of the particular image; every change in the image will make the process of projection unfeasible.
Here is a successful projection:
And this one will probably deprive you of sleep next night:
Read more:
StyleGAN2 Projection. A Reliable Method for Image Forensics?
If you want to try a bigger diversity of motives, try out ArtBreeder, a powerful image generator by Joel Simon:
artbreeder.com
What’s that about:
DeOldify model by Jason Antic allows organic colorization of photos and videos. You can bring back to vivid life old historical footage. Meanwhile, it’s implemented into MyHeritage.org.
The method is powerful. It recognizes patterns and objects and applies the colors of trained visual databases on it.
For example, these flowers from the 1950ies:
It also works for videos. Watch here #DeOldified and upscaled version of the famous “Arrival of a Train at La Ciotat (The Lumière Brothers, 1896)”
Advice:
Upload your b&w footage to your Google Drive. So you can preserve privacy (if you trust Google Cloud, of course).
Links:
GitHub
DeOldify for images
DeOldify for videos (supports various video platforms)
Read more:
DeOldify: GAN based Image Colorization
Re-animated History
What’s this about:
3D Ken Burns Effect (developed by Simon Niklaus et al) allows generating animated 3D video footage of a single photo.
Things to try out:
In the Colab Notebook you will find the component autozoom.py. Try to finetune it with following parameters:
objectTo = process_autozoom({'dblShift': 10.0,'dblZoom': 10000000000000000000000000000000000000000000000000000000,'objectFrom': objectFrom})numpyResult = process_kenburns({'dblSteps': numpy.linspace(0.0, 8.0, 400).tolist(),'objectFrom': objectFrom,'objectTo': objectTo,'boolInpaint': True})
This will exaggerate the camera movement — and you will fly through the whole scenery. Like in this example:
Do some experiments with parameters, and you will experience your photos from a fully new perspective. The Notebook provides many useful functions, like serial image=>video transfer.
Links:
3D Ken Burns Effect from a Single Image on ArXiv
Colab Notebook (provided by Andi Bayo and improved by Manuel Romero)
Read more:
Very spatial! AI-based parallax 3D videos from photos.
Re-animated History
Beyond the Boundaries
Watch my series dreAIms
towardsdatascience.com
What’s this about:
First Order Motion Model by Aliaksandr Siarohin et al transfers facial movements from video footage to an image.
Things to try out:
Choose your video footage and source image! So many possibilities. Be fair, don’t #DeepFake.
Links:
Project page
GitHub / Paper
Colab Notebook
This language Model, released by OpenAI during the year 2019 is trained on 40 GB text from various sources. There are several GPT-2 Colab notebooks, which work in a similar way: you enter the beginning of the sentence, and GPT-2 continues (or you ask questions to provided texts). The transformer-driven model works with “self-attention”, paying attention to text parts in specified proximity, which allows generating coherent stories, instead of gibberish chaos.
I prefer two GPT-2 notebooks:
“Train a GPT-2 Text-generating Model” by Max Woolf
GPT-2 with Javascript Interface by gpt2ent
Max Woolf’s Notebook allows:
to generate various texts by GPT-2
to train your own texts (up to 355m Model)
I did it in three languages:
English (on “Alice in Wonderland”)German (on “Faust I” by Goethe)Russian (on early poetry by Pushkin)
English (on “Alice in Wonderland”)
German (on “Faust I” by Goethe)
Russian (on early poetry by Pushkin)
As you see, it works to some degree for all languages. Of course, GPT-2 is trained on English sources. For foreign languages, we should apply finetuning and other assets, but this proof of concept was convincing for me. With some interesting observations:
The more I trained German on Faust, the closer to original the texts became. The reason is probably in a small dataset (just one single text). If you want to train on your texts, provide wider data amounts.
Russian Texts are not really comprehensible, but you can nevertheless recognize the style and even form by Pushkin's poetry. And the coinages and neologisms are perfect, every literary Avant-gardist would be proud of such inventions.
“GPT-2 with Javascript Interface”-Notebook allows:
Text generation, not more, not less. But you can control the text length (which is a very relevant factor):
With Temperature and top_k you can modify the randomness, repeatedness, and “weirdness” of the text.
With Generate how much you can generate longer texts (I am using the value of 1000).
Links:
First OpenAI post about GPT2
GPT-2: 1.5B Release
Max Woolf’s Blog
Colab Notebook by Max Woolf
GPT-2 with Javascript Interface
You also can use the web-implementation of GPT-2 by Adam King:
TalkToTransformer.com
I asked this application about the meaning of life. The answer was very wise and mature.
Read also:
Lakrobuchi! (my Trilogy about GPT-2)
AI can write music as well. In case of TensorFlow based Magenta, it uses Transformers, like in GPT-2, with self-attention, to allow the harmonic coherence and consistent composition.
Here is a sample, being generated with Magenta:
Things to try out:
The notebook provides a lot of possibilities, for example for continuation, modulation, it allows .wav and .midi export. Magenta is a project by GoogleAI. A huge thing, it even includes hardware.
And if you just want to listen, here is Magenta-Radio:
magenta.github.io
Links:
Magenta by Google.ai
Colab Notebook
Magenta-Radio
This is the mission of Colab Notebooks — to provide the possibility of work with ML/DL for the broad audience. To make AI accessible. To raise digital awareness and competence.
And this is just the beginning of AI Spring. After years of snowy landscapes. And after frozen systems.
In these Libraries you can find more Notebooks:
github.com
github.com
Let’s experiment, explore, create!
What are your favorite Colab Notebooks? Share them in comments!