Next Level Art and the Future of Work and Leisure
By: Max Frenzel
Tags: Artificial Intelligence, Work, Art, Creativity, Deep Learning

The last artist I want to introduce here is computational design lecturer Tom White.

His project Perception Engines focuses, as the name suggests, on the role of perception in creativity. In his words,

“Human perception is an often under-appreciated component of the creative process, so it is an interesting exercise to devise a computational creative process that puts perception front and center.”

Again, it is essentially an exercise in tricking neural networks to do things they were initially not intended to do by cleverly changing the domain on which they are applied.

It uses the idea of adversarial examples and puts an interesting artistic twist on it. Particularly, White sets up a feedback loop where the perception of a network guides the creative process, which then in turn again influences the perception.

In a nutshell (and a bit simplified), White took a neural network trained to recognize objects in images, and then used a second system that can generate abstract shapes and search for a result that can “trick” the network into a high certainty prediction of a particular object class. The results are seemingly abstract shapes (which White later turned into real screen-prints) that still convince the networks to be photorealistic representations of certain objects.

What is interesting is that once we know what the network thinks it sees, we can in most cases suddenly also see the objects in most of the images (although I doubt anyone is tricked into confusing it with the real thing).

The true process White used is actually even more clever and deep than the abbreviated outline of the project I gave here. If you are interested in the details I highly recommend checking out his article on the piece.

Qosmo: Computational Creativity and Beyond

Now that you have a little bit of an idea of what AI art can be and gotten to know a few people working in this emerging field, let me briefly give you my own story of how I got involved in this.

I actually started my academic career as a physicist, doing my PhD in Quantum Information Theory. But while doing this I realized that I wanted to do something a bit more applied. And having had some entrepreneurial experience as well through a startup I co-founded, I decided that AI seemed both interesting from a purely academic perspective, but also very promising tool to solve some really cool real-world problems (and make some money).

So after my PhD I worked for a few years at a startup that was applying AI to business problems in a wide range of fields, for example finance and health care. While there are certainly interesting problems to be solved in these domains, I personally got more and more interested in the creative side of AI.

Eventually, in February 2019 I finally decided to quit my previous job and join my friend Nao Tokui at his company Qosmo. If you are interested in the full story of how all this unfolded (and how I also became an author and musician at the same time), I wrote about all this in detail recently:

Qosmo is a small team of creatives based in Tokyo. The core concept of the company is “Computational Creativity”, with a strong focus on AI and music (but certainly not limited to these areas).

Here I want to briefly introduce you to three of our past projects.

AI DJ

Probably Qosmo’s most famous project so far is our AI DJ Project.

Originally started in 2016, AI DJ is a musical dialogue between human and AI.

In DJing, playing “back to back” means that two DJs take turns at selecting and mixing tracks. In our case we have a human playing back to back with an AI.

Specifically, a human (usually Nao) selects a track and mixes it, then the AI takes over and selects a track and mixes it, and so on, creating a natural and continuous cooperative performance.

This idea of augmenting human creativity and playing with the relationship of human and machine creativity is at the heart of what we do at Qosmo. We are not particularly interested in autonomously creative machines (nor do we really believe they are possible in the near future), but rather in how humans can interact with AI and machines for creative purposes.

AI DJ consists of several independent neural networks. At the core are a system that can select a track based on the history of previously played tracks, as well as a system that can do the beat matching and mixing.

Crucially, we are not using digital audio but actual vinyl records. The AI has to learn how to physically manipulate the disc (through a tiny robot arm trained using reinforcement learning) in order to align the beats and get the tempo to match.

While the project is already several years old, we are still constantly developing the system. For example using cameras to analyze crowd behavior and trying to encourage people do dance more by adjusting the track selection to this information.

We have taken this performance to many venues in the past, both local as well as global. Our biggest performance so far was at Google I/O 2019, where we did a one hour show on the main stage warming up the crowd right before CEO Sundar Pichai’s keynote.

You can read more about the details of AI DJ on our website.

Imaginary Soundscapes

As humans we have quite deeply linked visual and auditory experiences. Look at an image of a beach and you can easily imagine the sounds of waves and sea gulls. Looking at a busy intersection might bring car horns and construction noises to mind.

Imaginary Soundscapes is an experiment in giving AI a similar sense of imagined sounds associated with images. It is a web-based sound installation that lets users explore Google Street View while immersing themselves into the imaginary soundscapes dreamed up by the AI.

Technically it is based on the idea of cross-modal information retrieval techniques, such as image-to-audio or text-to-image.

The system was trained with two models on video (i.e. visual and audio) inputs: one well-established, pre-trained image recognition model processes the frames, while another convolutional neural network reads the associated audio as spectrogram images, with a loss that forces the distribution of its output to get as close as possible to that of the first model.

Once trained, the two networks allow us to retrieve the best-matching sound file for a particular scene out of our massive environmental sound dataset.

The generated soundscapes are at times interesting, at times amusing, and at times thought provoking. Many of them match human expectations, while others surprise us. We encourage you to get lost in the imaginary soundscapes yourself.

Neural Beatbox

Our most recent art project is Neural Beatbox, an audio-visual installation which is currently featured at the Barbican in London as part of the exhibition “AI: More Than Human” (which also features works by Mario Klingemann and Memo Akten).

Just like AI DJ this piece is centred around a musical conversation. However, other than in AI DJ, here the AI is not a participant but only the facilitator, and the dialogue takes place between different viewers of the installation.

Rhythm and beats are some of the most fundamental and ancient means of communication among humans. Neural Beatbox enables anyone, no matter their musical background and ability, to create complex beats and rhythms with their own sounds.

As viewers approach the installation they are encouraged to record short clips of themselves, making sounds and pulling funny faces. Using that video, one neural network segments, analyses and classifies a viewer’s sounds into various categories of drum sounds, some of which are then integrated in the currently playing beat.

Simultaneously, another network continuously generates new rhythms.

By combining the contributions of subsequent viewers in this way, an intuitive musical dialogue between people unfolds, resulting in a constantly evolving piece.

The slight imperfections of the AI, such as occasional misclassifications or unusual rhythms actually enhance the creative experience and result in interesting and unique musical experiences. As a viewer, trying to push the system beyond its intended domain by making “non-drum sounds” can lead to really interesting results, some of which are actually surprisingly musical and inspiring.

Currently Neural Beatbox is limited to being displayed in public places like the Barbican exhibition, but we are considering opening it up as an interactive web-based piece as well. We are just a bit concerned what kind of sounds and videos people on the internet might contribute to this installation… While the results could be hilarious and entertaining, they would probably also fairly quickly contain some NSFW content. ;)

Generative Models and VAEs

Besides my (still very recent) work at Qosmo, I have also done a few of my own artworks and more general creativity related projects with AI. Before showing you some of these, I’d briefly like to go into a quick and simple technical excursion.

Many of the models used in the creative scenes fall under the broad category of “Generative Models”. The GANs introduced above are one variant of this.

Generative models are essentially models that, as their name suggest, learn how to generate more or less realistic data. The general idea behind this is very nicely captured in a quote by physicist Richard Feynman.

“What I cannot create, I do not understand.” — Richard Feynman

As researchers and engineers working with AI, we hope that if we can teach our models to create data that is at least vaguely realistic, these models must have gained some kind of “understanding” of what the real world looks like, or how it behaves.

In other words, we use the ability to create and generate meaningful output as a sign of intelligence.

Unfortunately, this “understanding” or “intelligence” still often looks like the following image.

While our models are definitely learning something about the real world, their domain of knowledge is often severely limited, as we have already seen in the examples of bias above.

In my previous job working on practical business applications this was bad news. You do not want your financial or medical predictions to look like the image above!

Now as an artist however, I find this exciting and inspiring. In fact, as already pointed out, many artists deliberately seek these breaking points or edge cases of generative models.