Computational Creativity: The Role of the Transformer
By: Prashanth Rao
Tags: Machine Learning, Deep Learning, Generative Art, Neural Networks, Artificial Intelligence

Over the years, computers have become increasingly sophisticated in their ability to identify more and more complex patterns. The field of computational creativity, a multidisciplinary endeavour to build software that can assist humans in a variety of tasks in the arts, science and the humanities, has seen much progress since the early days of computers where instructions had to be explicitly programmed.

In this article, we will attempt to unravel some of the recent developments in generative modelling that have shown significant improvements in computers’ ability to generate useful patterns that appeal to human observers. In particular, one type of neural network architecture, the Transformer, will be discussed in detail with regard to its ability to capture longer-term dependencies in text, music and images. Some future directions that this technology could lead to are also discussed.

What is creativity, anyway?

In her illuminating essay, Margaret Boden describes that creativity is “the ability to come up with ideas or artifacts that are new, surprising, and valuable”. Historically, this has been viewed as a unique aspect of human intelligence. As described by Boden, for something to be deemed creative, it must deliver both novelty and value.

A common criticism about modelling creativity is that machines (including the most recent deep learning methods) just use “brute-force pattern matching” to achieve an end goal — if the machine is simply identifying existing patterns in the data that it was trained on, then what “novelty” is it truly adding? In addition, “values” are known to be highly variable, and “any arguments about creativity are rooted in disagreements about value”.

While it may be true that computers might “just be pattern-matching” on a large-scale, it is important to realize that even great scientists and mathematicians are in essence, using their finely tuned pattern-matching abilities (albeit in a highly sophisticated form) to discover patterns in reality that nobody else thought of before. In the below sections, we will look at how transformers could help address both the issues of novelty and value, and how they might combine with other tools to aid humans in combinatorial creativity.

The Transformer

The 2017 Google AI paper “Attention is All You Need” first introduced the concept of the transformer neural network architecture. In essence, a transformer is a stack of encoders that can process a sequence of arbitrary length, which connects to a stack of decoders that outputs another sequence. Below is an example of such a structure used for machine translation.

Image source: Illustrated Transformer by Jay Alammar

In recent times, there have been a number of extensions and modifications made to the vanilla transformer that allow it to generalize to other tasks by using encoder-only stacks (BERT), or decoder-only stacks (OpenAI GPT), resulting in large-scale improvements to downstream language modelling tasks.

A deep technical description of the transformer is out of the scope of this article — however there are excellent visual explanations (Illustrated Transformer) and line-by-line descriptions of the implementation (Annotated Transformer) that supplement the original paper.

Self-Attention

At the core of the transformer is its self-attention mechanism — i.e. the ability to model relationships between parts of a sequence (for example, words in a sentence), regardless of their respective positions in the sequence.

Image source: Google AI blog post on transformers

The above image shows two examples where the meaning of the word “it” could be ambiguous to a machine— in the example on the left, “it” refers to the word “animal”, whereas on the right, “it” refers to “street”. The term self-attention used in this context means that the encoder, while inspecting its input sequence, is able to focus on different parts of the input in relation to itself, allowing it to “attend” to the right representation based on observed patterns in the training data.

Self-attention is incorporated into an encoder/decoder structure as follows.

Image source: Illustrated Transformer by Jay Alammar

Via this design, the encoder’s input tokens first flow through the self-attention layer, which allows it to focus on different parts of the sequence in relation to itself. The output of the self-attention layer is then passed to a feed-forward neural network, through which the model “learns” the representation of each word.

Multiple Attention Heads

The transformer expands on the idea of self-attention by incorporating multiple “attention heads”, i.e. instances of self-attention applied to different parts of the sequence simultaneously. In Google AI’s transformer, there are eight attention heads focusing on different positions at the same time step, applied to each encoder.

Image source: Illustrated Transformer by Jay Alammar

In essence, having multiple attention heads (eight, in the above image) makes parallelizing the computation relatively trivial, since each head can identify the relationships between words independently of the other. This greatly reduces the amount of training effort, allowing the transformer to scale efficiently to very large training sets, unlike more sequential architectures like Recurrent Neural Networks (RNNs).

Over each training epoch, the individual attention heads update their vector representations, with the final representation being calculated as the weighted average of attentions.

Image source: Self-Attention for Generative Models, By Ashish Vaswani and Anna Huang

The decoder side works in a much similar way, and the final output vector representation is fed through a linear layer (which projects the vector output to a much larger “vocabulary space”), followed by a softmax layer (which converts the float representation of the word into a probability distribution for each word in the complete vocabulary).