representation is often termed the “latent space,” implying that here hidden traits of the data-generating process are coded, which are essential to the reconstruction process. This is very akin to the latent variables estimated by EM. In the autoencoder, the encoder will learn to code its input in terms of these latent variables, while the decoder will learn to represent them again in the source domain. In the following, we will be discussing the application to images though, in principle, both autoencoders and their variational variant are general mechanisms working for any domain.

We will later be interested in a behind-the-scene understanding of their modeling approach, which will be related to the employed loss function. We will then look at VAEs more extensively from the same vantage point: to understand their loss function—which is closest to the loss formulation of early GANs, the Kullback-Leibler divergence or KL divergence, $D_{KL}$.

With this tool in hand, we will examine how to optimize (train) a network with regard to KL divergence as the loss and understand key problems with this particular loss function. This will lead us to the motivation for a more powerful alternative.

### 3.2.1 From AE to VAE

VAEs are an interesting subject to study to emphasize the limits a loss function like KL divergence may place on a model. We will begin with a recourse to plain autoencoders to introduce the concept of learning a latent representation. We will then proceed to modify the autoencoder into a variational formulation which brings about the switch to a divergence measure as a loss function. From these grounds, we will then show how GANs again modified the loss function to succeed in high-quality image generation.

Figure 8 shows the schematic of a plain autoencoder (AE). As indicated in the sketch, input and output are of potentially very high dimensionality, like images. In between the encoder and decoder networks lies a “bottleneck” representation, which is, for example, a convolutional layer of orders of magnitude lower dimensionality (represented, for example, by a convolutional layer with only a few channels or a dense layer with a given low number of weights), which forces the network to find an encoding that preserves all information required for reconstruction.

A typical loss function to use when training the autoencoder is, for example, cross entropy, which is applicable for sigmoid activation functions, or simply the mean squared error (MSE). Any loss shall essentially force the AE to learn the identity function between input and output.

Let us introduce the notation for this. Let $X$ be the input image tensor and $X'$ the output image tensor. With $f_w$ being the encoder function given as a neural network parameterized by weights and biases $w$ and $g_v$ the decoder function parameterized by $v$, the loss hence works to make $X = X' = g_v(f_w(X))$.
In a variational autoencoder,<fnref n="2" /> things work differently. Auto-encoders like before use a fixed (deterministic) latent code to map the input to, while variational autoencoders will replace this with a distribution. We can call this distribution $p_w$, indicating the parameterization by $w$. It is crucial to understand that a choice was made here that imposes conditions on the latent code. It is meant to represent the input data in a variational way: in a way following Bayes' laws. Our mapping of the input image tensor $X$ to the latent variable $z$ is by this choice defined by

* The prior probability $p_w(z)$

* The likelihood (conditional probability) $p_w(X|z)$

* The posterior probability $p_w(z|X)$

Therefore, once we have obtained the correct parameters $\hat{w}$ by training the VAE, we can produce a new output $X'$ by sampling a $z^{(i)}$ from the prior probability $p_{\hat{w}}(z)$ and then generate the example from the conditional probability through $X^{(i)} = p_{\hat{w}}(X|z=z^{(i)})$.

Obtaining the optimal parameters, however, isn't possible directly. The searched optimal parameters are those that maximize the probability that the generated example $X'$ looks real. This probability can be rewritten as the aggregated conditional probabilities:

$$p_w(X^{(i)}) = \int p_w(X^{(i)}|z)p_w(z)dz.$$

This, however, does not make the search any easier since we need to enumerate and sum up all $z$. Therefore, an approximation is made through a surrogate distribution, parameterized by another set of parameters, $q_v$. Weng [15] shows in her explanation of the VAE the graphical model highlighting how $q_v$ is a stand-in for the unknown searched $p_w$ (see Fig. 9).

The reason to introduce this surrogate distribution actually comes from our wish to train neural networks for the decoding/encoding functions, and this requires us to back-propagate through the random variable, $z$, which of course cannot be done. Instead, if we have control over the distribution, we can select it such that the reparameterization trick can be employed. We define $q_v$ to be a multivariate Gaussian distribution with means and a covariance matrix that can be learned and a stochastic element multiplied to the covariance matrix for sampling [15, 16]. With this, we can back-propagate through the sampling process.

<footnote n="2">²Though variational autoencoders are in general not necessarily neural networks, in our context, we restrict ourselves to this implementation and stick to the notation with parameters $w$ and $v$, where in many publications they are denoted $\theta$ and $\phi$.</footnote>
<figure><img src="image_84.png" /><figcaption>**Fig. 9** The graphical model of the variational autoencoder. In a VAE, the variational decoder is $p_w(X|z)$, while the variational encoder is $q_v(z|X)$ (Figure after [15])</figcaption></figure>

At this point, the two distributions need to be made to match: $q_v$ should be as similar to $p_w$ as possible. Measuring their similarity can be done in a variety of ways, of which Kulback-Leibler divergence (KL divergence or KLD) is one.

### 3.2.2 KL Divergence

A divergence can be thought of as an asymmetric distance function between two probability distributions, $P$ and $Q$, measuring the similarity between them. It is a statistical distance which is not symmetric, which means it will not yield the same value if measured from $P$ to $Q$ or the other way around:

$$D_{KL}(P||Q) \neq D_{KL}(Q||P)$$

This can be seen when looking at the definition of KL divergence:

$$D_{KL}(P||Q) = \sum_{x} P(x) \log \frac{P(x)}{Q(x)} \quad (2)$$

Sometimes, the measure $D_{KL}$ is also called the relative entropy or information gain of $P$ over $Q$, which also indicates the asymmetry.

To give the two distributions more meaning, let us associate them with a use case. $P$ is usually the probability distribution of the example data, which can be our real images we wish to model, and is assumed to be unknown and high-dimensional. $Q$ on the other hand, is the modeled distribution, for example, parameterized by $\theta$, similar to Eq. 1. Hence, $Q$ is the distribution we can play with (in our case, optimize its parameters) to make them more similar to $P$. This means $Q$ will get more informative with respect to the true $P$ when we approach the optimal parameters.
### Box 3: Example: Calculating $D_{KL}$

When comparing the two distributions given in Fig. 10, the calculation of the Kullback-Leibler divergence, $D_{KL}$, can explicitly be given by reading off the y values of the nine elements (columns) from Fig. 11 and inserting them into Eq. 2.

The result of this calculation is for

$$ 
\begin{aligned}
D_{KL}(P||Q) &= \sum_{x} P(x) \log \frac{P(x)}{Q(x)} \\
&= 0.02 * \log \frac{.02}{.01} + 0.04 * \log \frac{.04}{.12} + \dots + 0.02 * \log \frac{.02}{.022} \\
&= 0.004 - 0.01 + \dots - 0.0002 \\
&= 0.0801
\end{aligned}
$$

which we call “forward KL” as it calculates in the direction from the actual distribution $P$ to the model distribution $Q$ and for

$$ 
\begin{aligned}
D_{KL}(Q||P) &= \sum_{x} Q(x) \log \frac{Q(x)}{P(x)} \\
&= 0.01 * \log \frac{0.01}{0.02} + 0.12 * \log \frac{0.12}{0.04} + \dots + 0.022 * \log \frac{0.022}{0.02} \\
&= -0.002 - 0.05 + \dots + 0.0002 \\
&= 0.0899
\end{aligned}
$$

which we call “reverse KL.”

Note that in the example in Box 3, there is both a $P(X=x_i)$ and $Q(X=x_i)$ for each $i \in \{0, 1, \dots, 8\}$. This is crucial for KL divergence to work as a loss function.

### 3.2.3 Optimizing the KL Divergence

Examine what happens in forward and reverse KL if this condition is not satisfied for some $i$. If in forward KL $P$ has values everywhere but $Q$ has not (or extremely small values), the quotient in the log

<figure><img src="image_85.png" /><figcaption>**Fig. 10** Two distributions $P$ and $Q$, here scaled to identical height</figcaption></figure>
<figure><img src="image_86.png" /><figcaption>**Fig. 11** The distributions *P* and *Q*, scaled to unit density, with added labels</figcaption></figure>

<figure><img src="image_87.png" /><figcaption>**Fig. 12** The distributions *P* (solid) and *Q*<sub>θ</sub> (dashed), in the initial configuration and after minimizing reverse KL D<sub>KL</sub>(*Q*<sub>θ</sub>|*P*). This time, in the initial configuration, *Q*<sub>θ</sub> has values greater than 0 where *P* has not (marked with green shading)</figcaption></figure>

function will tend to infinity by means of the division by almost
zero, and the term will be very large.

In Fig. 12, we assume $Q_{\theta}$ to be a unimodal normal distribution, i.e., a Gaussian, while $P$ is any empirical distribution. In the left plots of the figure, we show a situation before minimizing the forward/reverse KL divergence between $P$ and $Q_{\theta}$, in the right plots, the resulting shape of the Gaussian after minimization.

When in the minimization of forward KL D<sub>KL</sub>(P|Q<sub>θ</sub>) Q<sub>θ</sub> is zero where P has values greater zero, KL goes to infinity in these regions (marked area in the start configuration of the top row in Fig. 12), since the denominator in the log function goes to zero. This, in turn, drives the parameters of Q<sub>θ</sub> to broaden the Gaussian to cover these areas, thereby removing the large loss contributions. This is known as the *mean-seeking* behavior of forward KL.

Conversely, in reverse KL (bottom row in Fig. 12), in the marked areas of the initial configuration, *P* is zero in regions where *Q*<sub>θ</sub> has values greater than zero. This yields high-loss
contributions from the log denominator, in this case driving the Gaussian to remove these areas from $Q_\theta$. Since we assumed a unimodal Gaussian $Q$, the minimization will focus on the largest mode of the unknown $P$. This is known as the *mode-seeking* behavior of reverse KL.

Forward KL tends to overestimate the target distribution, which is exaggerated in the right plot in Fig. 12. In contrast, reverse KL tends to underestimate the target distribution, for example, by dropping some of its modes. Since underestimation is the more desirable property in practical settings, reverse KL is the loss function of choice, for example, in variational autoencoders. The downside is that as soon as target distribution $P$ and model distribution $Q_\theta$ have no overlap, KL divergence evaluates to infinity and is therefore uninformative. One countermeasure to take is to add noise to $Q_\theta$, so that there is guaranteed overlap. This noise, however, is not desirable in the model distribution $Q_\theta$ since it disturbs the generated output.

Another way to remedy the problem of KL going to infinity is to adjust the calculation of the divergence, which is done in Jensen-Shannon divergence (JS divergence, $D_{JS}$) defined as

$$D_{JS} = \frac{1}{2} (D_{KL}(P \| M) + D_{KL}(Q_{\theta} \| M)), \quad (3)$$

where $M = \frac{P+Q_\theta}{2}$. In the case of nonoverlapping $P$ and $Q_\theta$, this evaluates to constant $\log 2$, which is still not providing information about the closeness but is computationally much friendlier and does not require the addition of a noise term to achieve numerical stability.

### 3.2.4 The Limits of VAE

In the VAE, reverse KL is used. Our optimization goal is maximizing the likelihood to produce realistic looking examples—ones with a high $p_w(x)$. Simultaneously, we want to minimize the difference between the real and estimated posterior distributions $q_v$ and $p_w$. This can only be achieved through a reformulation of reverse KL [15]. After some rearranging of reverse KL, the loss of the variational autoencoder becomes

$$ \begin{aligned} L_{\text{VAE}}(w, v) &= -\log p_w(X) + D_{\text{KL}}(q_v(\mathbf{z}|X)\|p_w(\mathbf{z}|X)) \\ &= -\mathbb{E}_{\mathbf{z} \sim q_v(\mathbf{z}|X)} \log p_w(X|\mathbf{z}) + D_{\text{KL}}(q_v(\mathbf{z}|X)\|p_w(\mathbf{z})) \end{aligned} \quad (4) $$

$\hat{w}$ and $\hat{v}$ are the parameters maximizing the loss.

We have seen how mode-seeking reverse KL divergence limits the generative capacity of variational autoencoders through the potential underrepresentation of all modes of the original distribution.
KL divergence and minimizing the ELBO also have a second fundamental downside: there is no way to find out how close our solution is to the obtainable optimum. We measure the similarity to the target distribution up to the KL divergence, but since the true $p_{\hat{w}}(.)$ is unknown, the stopping criterion in the optimization has to be set by another metric, e.g., to a maximum number of iterations or corresponding to an improvement of the loss below some $\epsilon$.

The original presentation of the variational autoencoder was given as one example of the general framework called the autoencodding variational Bayes. This publication presented the above ideas in a thorough mathematical formulation, starting from a directed graphical model that poses the abstract problem. The authors also develop the seminal “reparameterization trick” to make the loss formulation differentiable and with this to make the search for the autoencoder parameters amenable to gradient descent optimizers [16]. The details are beyond this introductory treatment.

## 3.3 The Fundamental GAN Approach

At the core of the adversarial training paradigm is the idea to create two players competing in a minimax game. In such games, both players have access to the same variables but have opposing goals, so that they will manipulate the variables in different directions.

Referring to Fig. 13, we can see the generative part in orange color, where random numbers are drawn from the latent space and, one by one, converted into a set of “fake images” by the generator

<figure><img src="image_88.png" /><figcaption>**Fig. 13** Schematic of a GAN network. Generator (orange) creates fake images based on random numbers drawn from a latent space. These together with a random sample of real images are fed into the discriminator (blue, right). The discriminator looks at the batch of real/fake images and tries to assign the correct label (“0” for fake, “1” for real)</figcaption></figure>
network, in the figure implemented by a CNN. Simultaneously, from a database of real images, a matching number of examples are randomly drawn. The real and fake images are composed into one batch of images which are fed into the discriminator. On the right side, the discriminator CNN is indicated in blue. It takes the batch of real and fake images and decides for each if it appears real (yielding a value close to “1”) or fake (“0”).

The error signal is computed from the number of correct assignments the discriminator can do on the batch of generated and real images. Both the generator and the discriminator can then update their parameters based on this same error signal. Crucially, the generator has the aim to *maximize* the error, since this signifies that it has successfully fooled the discriminator into taking the fake images for real, while the discriminator weights are updated to *minimize* the same error, indicating its success in telling true and fake examples apart. This is the core of the competitive game between generator and discriminator.

Let us introduce some abbreviations to designate GAN components. We will denote the generator and discriminator networks with G and D, respectively. The objective of GAN training is a game between generator and discriminator, where both affect a common loss function J, but in opposed directions. Formally, this can be written as

$$ \min_G \max_D J(G, D), $$

with the GAN objective function

$$ J(G, D) = \mathbb{E}_{x \sim p_{\text{data}}} [\log D(x)] + \mathbb{E}_{z \sim p_G} [1 - \log D(G(z))] \quad (5) $$

D will attempt to maximize J by maximizing the probability to assign the correct labels to real and generated examples: this is the case if $D(x)=1$, maximizing the first loss component, and if $D(G(z))=0$, maximizing the second loss component. The generator G, instead, will attempt to generate realistic examples that the discriminator labels with “1,” which corresponds to a minimization of $\log(1 - D(G(z)))$.

**3.4 Why Early GANs Were Hard to Train**

GANs with this training objective implicitly use JS divergence for the loss, which can be seen by examining the GAN training objective. Consider the ideal discriminator D for a fixed generator. Its loss is minimal for the optimal discriminator given by [1]

$$ \hat{D}(x) = \frac{p_{\text{data}}(x)}{p_{\text{data}}(x) + p_G(x)}. \quad (6) $$

Substituting $\hat{D}$ in Eq. 5 yields (without proof) the implicit use of the Jensen-Shannon divergence if the above training objective is employed:
$$J(G, \hat{D}) = 2D_{JS}(p_{\text{data}}||p_G) - \log 4. \quad (7)$$

This theoretical result shows that a minimum in the GAN training can be found when the Jensen-Shannon divergence is zero. This is achieved for identical probability distributions $p_{\text{data}}$ and $p_G$ or, equivalently, when the generator perfectly matches the data distribution [17].

Unfortunately, it also shows that this loss is, like KL divergence, only helpful when target distribution (i.e., data distribution) and model distribution have overlapping support. Therefore, added noise can be required to approximate the target distribution. In addition, the training criterion saturates if the discriminator in the early phase of training perfectly distinguishes between fake and real examples. The generator will therefore no longer obtain a helpful gradient to update its weights. An approach thought to prevent this was proposed by Goodfellow et al. [1]. The generator loss was turned from the minimization problem into a maximization problem that has the same fixed point in the overall minimax game but prevents saturation: instead of minimizing $\log(1 - D(G(z)))$, one maximizes $\log(D(G(z)))$ [1].

## 3.5 Improving GANs

GAN training has quickly become notorious for the difficulties it posed upon the researchers attempting to apply the mechanism to real-world problems. We have qualitatively attributed a part of these problems to the inherently difficult task of density estimation and motivated the intuition that while fewer samples might suffice to learn a decision boundary in a discriminative task, many more examples are required to build a powerful generative model.

In the following, some more light shall be shed on the reasons why GAN training might fail. Typical GAN problems comprise the following:

**Mode dropping** is the phenomenon in forward KL caused by regions of the data distribution not being covered by the generator distribution, which implies large probabilities of samples coming from $P_{\text{data}}$ and very small probabilities of originating from $P_G$. This drives forward KL toward infinity and punishes the generator for not covering the entire data distribution [18]. If all modes but one are dropped, one can call this mode collapse: the generator only generates examples from one mode of the distribution.

**Poor convergence** can be caused by a discriminator learning to distinguish real and fake examples very early—which is also very likely to happen throughout the GAN training. This is rooted in the
observation that by the generative process
that projects from a low-dimensional latent
space into the high-dimensional $p_G$, the sam-
ples in $p_G$ are not close to each other but
rather inhabit “islands” [18]. The discrimina-
tor can learn to find them and thereby differ-
entiate between true and false samples easily,
which causes the gradients driving generator
optimization to vanish [17].

**Poor sample quality**

despite a high log likelihood of the model is a
consequence of the practical independence of
sample quality and model log likelihood.
Theis et al. [19] show that neither does a
high log likelihood imply generated sample
fidelity nor do visually pleasing samples
imply a high log likelihood. Therefore, train-
ing a GAN with a loss function that effectively
implement maximizing a log likelihood term
is not an ideal choice—but exactly corre-
sponds to KL minimization.

is a consequence of reformulating the genera-
tor loss into maximizing $\log D(G(z))$. It can
be shown [18] that this choice effectively
makes the generator struggle between a
reverse KL divergence favoring mode-seeking
behavior and a negative JS divergence actually
driving the generator into examples different
from the real data distribution.

**Unstable training**

There have been many subsequent authors touching these
topics, but already Arjovsky and Bottou [18] have shown best
practices of how to overcome these problems.

Among the solutions proposed for GAN improvements are
some that prevent the generator from producing only too similar
samples in one batch, some that keep the discriminator insecure
about the true labels of real and fake examples, and more, which
Creswell et al. [17] have summarized in their GAN overview. A
collection of best practices compiled from these sources is pre-
sented in Box 4. It is almost impossible to write a cookbook for
successful, converging, stable GAN training. For almost every tip,
there is a caveat or situation where it cannot be applied. The
suggestions below therefore are to be taken with a grain of salt
but have been used by many authors successfully.
**Box 4: Best Practices for Stable GAN Training**

**General measures.** GAN training is sensitive to hyperparameters, most importantly the learning rate. Mode collapse might already be mitigated by a lower learning rate. Also, different learning rates for generator and discriminator might help. Other typical measures are batch normalization (or instance normalization in case of small batch sizes; mind however that batch normalization can taint the randomness of latent vector sampling and in general should not be used in combination with certain GAN loss functions), use of transposed convolutions instead of parameter-free upsampling, and strided convolutions instead of down-sampling.

**Feature matching.** One typical observation is that neither discriminator nor generator converges. They play their “cat-and-mouse” game too effectively. The generator produces a good image, but the discriminator learns to figure it out, and the generator shifts to another good image, and so on.

A remedy for this is feature matching, where the $l_2$ distance between the average feature vectors of real and fake examples is computed instead of a cross-entropy loss on the logits. Because per batch the feature vectors change slightly, this introduces randomness that helps to prevent discriminator overconfidence.

**Minibatch discrimination.** When the generator only produces very convincing but extremely similar images, this is an indication for mode collapse.

This can be counteracted by calculating a similarity metric between generated samples and penalizing the generator for too little variation. Minibatch discrimination is considered to be superior in performance to feature matching.

**One-sided label smoothing.** Deep classification models often suffer from overconfidence, focusing on only very few features to classify an image. If this happens in a GAN, the generator might figure this out and only produce the feature the discriminator uses to decide for a real example.

A simple measure to counteract this is to provide not a “1” as a label for the real images in the batch but a lower value. This way, the discriminator is penalized for overconfidence (when it returns a value close to “1”).

**Cost function selection.** Several sources list possible GAN cost functions. Randomly trying them one by one might work, but often some of the above measures, in particular learning rate and hyperparameter tuning, might be more successful first steps.
Besides these methods, one area of discussion concerned the question if there is a need of balancing discriminator and generator learning and convergence at all. The argument was that a converged discriminator will as well yield a training signal to the generator as a non-converged discriminator. Practically, however, many authors described carefully designed update schedules, e.g., updating the generator once per a given number of discriminator updates.

Many more ideas exist: weight updating in the generator using an exponential moving average of previous weights to avoid “forgetting,” different regularization and conditioning techniques, and injecting randomness into generator layers anew. Some we will encounter later, as they have proven to be useful in more recent GAN architectures.

Despite the recent advances in stabilizing GAN training, even the basic method described so far, with the improvements made in the seminal DCGAN publication [20], finds application until today, e.g., for the de novo generation of PET color images [21]. The usefulness of an approach as presented in their publication might be doubted, since the native PET data is obviously not colored. The authors use 2D histograms of the three-color channel combinations to compare true and fake examples. As we have discussed earlier, this is likely a poor metric since it does not allow insights into the high-dimension joint probability distribution underlying the data-generating process. Figure 14 shows an example comparison of some generated examples compared to original PET images.

<figure><img src="image_89.png" /><figcaption>**Fig. 14** PET images generated from random noise using a DCGAN architecture. Image taken from [21] (CC-BY4.0)</figcaption></figure>
To address many of the GAN training dilemmas, Arjovsky and Bottou [18] have proposed to employ the Wasserstein distance as a replacement for KL or JS divergence already in their examination of the root causes of poor GAN training results and have later extended this into their widely anticipated approach we will focus on next [22, 23]. We will also see more involved and recent approaches to stabilize and speed up GAN training in later sections of this chapter (Subheading 4).

## 3.6 Wasserstein GANs

Wasserstein GANs were appealing to the deep learning and GAN scene very quickly after Arjovsky et al.'s [22] seminal publication because of a number of traits their inventors claimed they'd have. For one, Wasserstein GANs are based on the theoretical idea that the change of the loss function to the Wasserstein distance should lead to improved results. This combined with the reported benchmark performance would already justify attention. But Wasserstein GANs additionally were reported to train much more stably, because, as opposed to previous GANs, the discriminator would be trained to convergence in every iteration, instead of demanding a carefully and heuristically found update schedule for generator and discriminator. In addition, the loss was directly reported to correlate with visual quality of generated results, instead of being essentially meaningless in a minimax game.

Wasserstein GANs are therefore worth an in-depth treatment in the following sections.

### 3.6.1 The Wasserstein (Earthmover) Distance

The Wasserstein distance figuratively measures how, with an optimal transport plan, mass can be moved from one configuration to another configuration with minimal work. Think, for example, of heaps of earth. Figure 15 shows two heaps of earth, *P* and *Q* (discrete probability distributions), both containing the same amount of earth in total, but in different concrete states *x* and *y* out of all possible states.

Work is defined as the shovelfuls of earth times the distance it is moved. In the three rows of the figure, earth is moved (only within one of *P* or *Q*, not from one to the other), in order to make the configuration identical. First, one shovelful of earth is moved one pile further, which adds one to the Wasserstein distance. Then, two shovelfuls are moved three piles, adding six to the final Wasserstein distance of $D_W = 7$.

Note that in an alternative plan, it would have been possible to move two shovelfuls of earth from $p_4$ to $p_1$ (costing six) and one from $p_4$ to $p_3$, which is the inverse transport plan of the above, executed on *P*, and leading to the same Wasserstein distance. The Wasserstein distance is in fact a distance, not a divergence, because it yields the same result regardless of the direction. Also note that
<figure><img src="image_90.png" /><figcaption>**Fig. 15** One square is one shovel full of earth. Transporting the earth shovel-wise from pile to pile amasses performed work: the Wasserstein (earthmover) distance. The example shows a Wasserstein distance of $D_W = 7$</figcaption></figure>

we implicitly assumed that *P* and *Q* share their support,<fnref n="3" /> but that in case of disjunct support, only a constant term would have to be added, which grows with the distance between the support regions.

Many other transport plans are possible, and others can be equally cheap (or even cheaper—it is left to the reader to try this out). Transport plans need not modify only one of the stocks but can modify both to reach the optimal strategy to make them identical. Algorithmically, the optimal solution to the question of the optimal transport plan can be found by formulating it as a linear programming problem. However, enumerating all transport plans and computing the linear programming algorithm are intractable for larger and more complex “heaps of earth.” Any nontrivial GAN will need to estimate transport of such complex “heaps,” so they

<footnote n="3">³ The support, graphically, is the region where the distribution is not equal to zero.</footnote>
suffer this intractability problem. Consequently, in practice, a different approach must be taken, which we will sketch below.<fnref n="4" />

Formalizing the search for the optimal transport plan, we look at all possible joint distributions of our $P$ and $Q$, forming the set of all possible transport plans, and denote this set $\Pi(P, Q)$, implying that for all $\gamma \in \Pi(P, Q)$, $P$ and $Q$ will be their marginal distributions.<fnref n="5" /> This, in turn, means that by definition $\Sigma_{x\gamma}(x, y) = P(y)$ and $\Sigma_{y\gamma}(x, y) = Q(x)$.

For one concrete transport plan $\gamma$ that works between a state $x$ in $P$ and a state $y$ in $Q$, we are interested in the optimal transport plan $\gamma(x, y)$. Let $\|x-y\|$ be the Euclidian distance to shift earth between $x$ and $y$, and then multiplying this with every value of $\gamma$ (the amount of earth shifted) leads to

$$D_W(P, Q) = \inf_{\gamma \in \Pi} \sum_{x,y} \|x-y\| \gamma(x,y),$$

which can be rewritten to obtain

$$D_W(P, Q) = \inf_{\gamma \sim \Pi(P, Q)} \mathbb{E}_{(x,y) \sim \gamma} \|x-y\|. \quad (8)$$

It measures both the distance of two distributions with disjoint support and the difference between distributions with perfectly overlapping support because it includes both, the shifting of earth and the distance to move it.

Practically, though, this result cannot be used directly, since the Linear Programming problem scales exponentially with the number of dimensions of the domain of $P$ and $Q$, which are high for images. To our disadvantage, we additionally need to differentiate the distance function if we want to use it for deep neural network training using backpropagation. However, we cannot obtain a derivative from our distance function in the given form, since, in the linear programming (LP) formulation, our optimized distribution (as well as the target distribution) end up as constraints, not parameters.

Fortunately, we are not interested in the transport plan $\gamma$ itself, but only in the distance (of the optimal transport plan). We can therefore use the dual form of the LP problem, in which the constraints of the primal form become parameters. With some clever definitions, the problem can be cast into the dual form, finally yielding

<footnote n="4">⁴ An extensive treatment of Wasserstein distance and optimal transport in general is given in the 1.000-page treatment of Villani's book [24], which is freely available for download.</footnote>

<footnote n="5">⁵ This section owes to the excellent blog post of Vincent Herrmann, at https://vincentherrmann.github.io/blog/wasserstein/. Also recommended is the treatment of the “Wasserstein GAN” paper by Alex Irpan at https://www.alexirpan.com/2017/02/22/wasserstein-gan.html. An introductory treatment of Wasserstein distance is also found in [25, 26].</footnote>
$$D_W(P, Q) = \|f\|_{L \le 1} \sup_{E_{x \sim P} f(x) - E_{x \sim Q} f(x)}$$

with a function $f$ that has to adhere to a constraint called the 1-Lipschitz continuity constraint, which requires $f$ to have a slope of at most magnitude 1 everywhere. $f$ is the neural network, and more specifically for a GAN, the discriminator network. 1-Lipschitzness can be achieved trivially by clipping the weights to a very small interval around 0.

### 3.6.2 Implementing WGANs

To implement the distance as a loss function, we rewrite the last result again as

$$D_W(P, Q) = \max_{w \in W} \mathbb{E}_{x \sim P}[D_w(x)] - \mathbb{E}_{z \sim Q}[D_w(G_w(z))]. \quad (9)$$

Note that in opposition to other GAN losses we have seen before, there is no logarithm anymore, because, this time, the “discriminator” is no longer a classification network that should learn to discriminate true and fake samples but rather serves as a “blank” helper function that during training learns to estimate the Wasserstein distance between the sets of true and fake samples.

## Box 5: Spectral Normalization

Spectral normalization is applied to the weight matrices of a neural network to ensure a boundedness of the error function (e.g., Lipschitzness of the discriminator network in the WGAN context). This helps convergence like any other normalization method, as it provides a guaranty that gradient directions are stable around the current point, allowing larger step widths.

The **spectral norm** (or matrix norm) measures how far a matrix $A$ can stretch a vector $x$:

$$||A|| = \max_{x \neq 0} \frac{||Ax||}{||x||}$$

The numerical value of the spectral norm of $A$ can be shown to be just its maximum singular value. To compute the maximum singular value, an algorithmic idea helps: the power iteration method, which yields the maximal eigenvector.

**Power iteration** uses the fact that any matrix will rotate a random vector toward its largest eigenvector. Therefore, by iteratively calculating $\frac{Ax}{||Ax||}$, the largest eigenvector is obtained eventually.

In practice, it is observed that a single iteration is already sufficient to achieve the desired normalizing behavior.
Consequently, the key ingredient is the Lipschitzness con-
straint of the discriminator network,<fnref n="6" /> and how to enforce this in a
stable and regularized way. It soon turned out that weight clipping
is not an ideal choice. Rather, two other methods have been pro-
posed: the gradient penalty approach and normalizing the weights
with the spectral norm of the weight matrices.

Both have been added to the standard catalogue of
performance-boosting measures in GAN training ever since,
where in particular spectral normalization (cf. Box 5) is attractive
as it can be implemented very efficiently, has a sound theoretical and
mathematical foundation, and ensures stable and efficient training.

One of the first applications of Wasserstein GANs in a practical use
case was presented in the medical domain, specifically in the context
of attributing visible changes of a diseased patient with respect to a
normal control to locations in the images [27]. The way this
detection problem was cast into a GAN approach (and then solved
with a Wasserstein GAN) was to delineate the regions that make the
images of a diseased patient look “diseased,” i.e., find the residual
region, that, if subtracted from the diseased-looking image, would
make it look “normal.”

Figure 16 shows the construction of the VA-GAN architecture
with images from a mocked dataset for illustration. For the authors’
results, see their publication and code repository.<fnref n="7" />

For their implementation, the authors note that neither batch
normalization nor layer normalization helped convergence and
hypothesize that the difference between real and generated exam-
ples may be a reason that in particular batch normalization may in
fact have an adverse effect especially during the early training phase.
Instead, they impose an l₁ norm loss component on the U-Net-
generated “visual (feature) attribution” (VA) map to ensure it to be
a minimal change to the subject. This serves to prevent the genera-
tor from changing the subject into some “average normal” image
that it may otherwise learn. They employ an update regime that
trains the critic network for more iterations than the generator, but
doesn’t train it to convergence as proposed in the original WGAN
publications. Apart from these measures, in their code repository,
the authors give several practical hints and heuristics that may
stabilize the training, e.g., using a tanh activation for the generator
or exploring other dropout settings and in general using a large
enough dataset. They also point out that the Wasserstein distance
isn’t suited for model selection since it is too unstable and not
directly correlated to the actual usefulness of the trained model.

<footnote n="6">⁶ The discriminator network in the context of continuous generator loss functions like the Wasserstein-based loss is called a “critique” network, as it no longer discriminates but yields a metric. For ease of reading, this chapter sticks to the term “discriminator.”</footnote>

<footnote n="7">⁷ https://github.com/baumgach/vagan-code.</footnote>
<figure><img src="image_91.png" /><figcaption>**Fig. 16** An image of a diseased patient is run through a U-Net with the goal to yield a map that, if added to the input image, results in a modified image that fools the discriminator (“critique”) network into classifying it as a “normal” control. The map can be interpreted as the regions attributed to appear abnormal, giving rise to the name of the architecture: visual attribution GAN (VA-GAN).</figcaption></figure>

This is one more reason to turn in the next section to an important topic in the context of validation for generative models: How to quantify their results?

## 3.7 GAN Performance Metrics

One imminent question has so far been postponed, though it implicitly plays a crucial role in the quest for “better” GANs: How to actually measure the success of a GAN or the performance in terms of result quality?

GANs can be adapted to solve image analysis tasks like segmentation or detection (cf. Subheading 3.6.3). In such cases, the quality and success can be measured in terms of task-related performance (Jaccard/Dice coefficient for segmentation, overlap metrics for detection etc.).

Performance assessment is less trivial if the GAN is meant to generate unseen images from random vectors. In such scenarios, the intuitive criterion is how convincing the generated results are. But convincing to whom? One could expose human observers to the real and fake images, ask them to tell them apart, and call a GAN better than a competing GAN if it fools the observer more consistently.<fnref n="8" /> Since this is practically infeasible, metrics were sought that provide a more objective assessment.

<footnote n="8">⁸ In fact, there is only very little research on the actual performance of GANs in fooling human observers, though guides exist on how to spot “typical” GAN artifacts in generated images. These are older than the latest GAN models, and it can be hypothesized that the lack of such literature is indirect confirmation of the overwhelming capacity of GANs to fool human observers.</footnote>
The most widely used way to assess GAN image quality is the Fréchet inception distance (FID). This distance is conceptually related to the Wasserstein distance. It has an analytical solution to calculate the distance of Gaussian (normal) distributions. In the multivariate case, the Fréchet distance between two distributions X and $\Upsilon$ is given by the squared distance of their means $\mu_X$ (resp. $\mu_\Upsilon$) and a term depending on the covariance matrix describing their variances $\Sigma_X$ (resp. $\Sigma_\Upsilon$):

$$d(X, \Upsilon) = \|\mu_X - \mu_\Upsilon\|^2 + \mathrm{Tr}(\Sigma_X + \Sigma_\Upsilon - 2\sqrt{\Sigma_X \Sigma_\Upsilon}). \quad (10)$$

The way this distance function is being used is often the score, which is computed as follows:

* Take two batches of images (real/fake, respectively).

* Run them through a feature extraction or embedding model. For FID, the inception model is used, pretrained on ImageNet. Retain the embeddings for all examples.

* Fit each one multivariate normal distribution to the embedded real/fake examples.

* Calculate their Fréchet distance according to the analytical formula in Eq. 10.

This metric has a number of downsides. Typically, if computed for a larger batch of images, it decreases, although the same model is being evaluated. This bias can be remedied, but FID remains the most used metric still. Also, if the inception network cannot capture the features of the data FID should be used on, it might simply be uninformative. This is obviously a grave concern in the medical domain where imaging features look much different from natural images (although, on the other hand, transfer learning for medical classification problems proved to work surprisingly well, so that apparently convolutional filters trained on photographs also extract applicable features from medical images). In any case, the selection of the pretrained embedding model brings a bias into the validation results. Lastly, the assumption of a multivariate normal distribution for the inception features might not be accurate, and only describing it through their means and covariances is a severe reduction of information. Therefore, a qualitative evaluation is still required.

One obvious additional question arises: If the ultimate metric to judge the quality of the generator is given by, for example, the FID, why can't it be used as the optimization goal instead of minimizing a discriminator loss? In particular, as the Fréchet distance is a variant of the Wasserstein distance, an answer to this question is not obvious. In fact, feature matching as described in Box 4 exactly uses this type of idea, and likewise, it has been partially adopted in recent GAN architectures to enhance the stability of training with a more fine-grained loss component than a pure categorical cross-entropy loss on the “real/fake” classification of the discriminator.
Related recent research is concerned with the question how generated results can automatically be detected to counteract fraudulent authors. So-called forensic algorithms detect patterns that point out generated images. This research puts up the question how to detect fake images reliably. Solutions based on different analysis directions encompass image fingerprinting and frequency-domain analysis [28–31].

# 4 Selected GAN Architectures You Should Know

In the following, we will examine some GAN architectures and GAN developments that were taken up by the medical community or that address specific needs that might make them appealing, e.g., for limited data scenarios.

## 4.1 Conditional GAN

GANs cannot be told what to produce—at least that was the case with early implementations. It was obvious, though, that a properly trained GAN would imprint the semantics of the domain onto its latent space, which was evidenced by experiments in which the latent space was traversed and images of certain characteristics could be produced by sampling accordingly. Also, it was found that certain dimensions of the latent space can correspond to certain features of the images, like hair color or glasses, so that modifying them alone can add or take away such visible traits.

With the improved development of conditional GANs [32] following a number of GANs that modeled the conditioning input more explicitly, another approach was introduced that was based on the U-Net architecture as a generator and a favorable discriminator network that values local style over a full-image assessment.

Technically, the formulation of a conditional GAN is straightforward. Recalling the value function (learning objective) of GANs from Eq. 5,

$$J(G, D) = \mathbb{E}_{x \sim p_{\text{data}}} [\log D(x)] + \mathbb{E}_{z \sim p_{G}} [1 - \log D(G(z))],$$

We now want to condition the generation on some additional knowledge or input. Consequently, both the generator G and the discriminator D will receive an additional “conditioning” input, which we call x. This can be a class label but also any other associated information. Very commonly, the additional input will be an image, as, for example, for image translation application (e.g., transforming from one image modality to another such as, for instance, MRI to CT). The result is the cGAN objective function:

$$J_{\text{cGAN}}(G, D) = \mathbb{E}_{x \sim p_{\text{data}}} [\log D(x|y)] + \mathbb{E}_{z \sim p_{G}} [1 - \log D(G(z|y))] \quad (11)$$
<figure><img src="image_92.png" /><figcaption>**Fig. 17** A possible architecture for a cGAN. Left: the generator network takes the base images $x$ as input and generates a translated image $\hat{y}$. The discriminator receives either this pair of images or a true pair $x, y$ (right). The additional generator reconstruction loss (often a $l_1$ loss) is calculated between $y$ and $\hat{y}$.</figcaption></figure>

Isola et al. [32] describe experiments with MNIST handwritten digits, where a simple generator with two layers of fully connected neurons was used, and similarly for the discriminator. $x$ was set to be the class label. In a second experiment, a CNN creates a feature representation of images, and the generator is trained to generate textual labels (choosing from a vocabulary of about 250.000 encoded terms) for the images conditioned on this feature representation.

Figure 17 shows a possible architecture to employ a cGAN architecture for image-to-image translation. In this diagram, the conditioning input is the target image that the trained network shall be able to produce based on some image input. The generator network therefore is a U-Net. The discriminator network can be implemented, for example, by a classification network. This network always receives two inputs: the conditioning image ($x$ in Fig. 17) and either the generated output $\hat{y}$ or the true paired image $y$.
<figure><img src="image_93.png" /><figcaption>Fig. 18 Input and output of a pix2pix experiment. Online demo at https://affinelayer.com/pixsrv/</figcaption></figure>

Note that the work of Isola et al. [32] introduces an additional loss term on the generator that measures the $l_1$ distance between the generated and ground truth image, which is (with variables as in Eq. 11)

$$J_{l_1}(G) = E_{x,y,z} \|y - G(x, z)\|_1,$$

where $\|\cdot\|_1$ is the $l_1$ norm.

The authors do not further justify this loss term apart from stating that $l_1$ is preferred over $l_2$ to encourage less blurry results. It can be expected that this loss component provides a good training signal to the generator when the discriminator loss doesn't, e.g., in the beginning of the training with little or no overlap of target and parameterized distributions. The authors propose to give the $l_1$ loss orders of magnitudes more weight than the discriminator loss component to value accurate translations of images over “just” very plausible images in the target domain.

The cGAN, namely, in the configuration with a U-Net serving as the generative network, was very quickly adopted by artists and scientists, thanks to the free implementation pix2pix.<fnref n="9" /> One example created with pix2pix is given in Fig. 18, where the cGAN was trained to produce cat images from line drawings.

One application in the medical domain was proposed, for example, by Senaras et al. [33]. The authors used a U-Net as a generator to produce a stained histopathology image from a label image that has two distinct labels for two kinds of cell nuclei. Here, the label image is the conditioning input to the network. Consequently, the discriminator network, a classification CNN tailored to

<footnote n="9">⁹ https://github.com/phillipi/pix2pix.</footnote>
the patch-based classification of slides, receives two inputs: the histopathology image and a label image.

Another example employed an augmented version of the conditional GAN to translate CT to MR images of the brain, including a localized uncertainty estimate about the image translation success. In this work, a Bayesian approach to model the uncertainty was taken by including dropout layers in the generator model [34].

Lastly, a 3D version of the pix2pix approach with a 3D U-Net as a generative network was devised to segment gliomas in multi-modal brain MRI using data from the 2020 International Multi-modal Brain Tumor Segmentation (BraTS) challenge [35]). The authors called their derived model vox2vox, alluding to the extension to 3D data [36].

More conditioning methods have been developed over the years, some of which will be sketched further on. It is common to this type of GANs that paired images are required to train the network.

## 4.2 CycleGAN

While cGANs require paired data for the gold standard and conditioning input, this is often hard to come by, in particular in medical use cases. Therefore, the development of the CycleGAN set a milestone as it alleviates this requirement and allows to train image-to-image translation networks without paired input samples.

The basic idea in this architecture is to train two mapping functions between two domains and to execute them in sequence so that the resulting output is considered to be in the origin domain again. The output is compared against the original input, and their $l_1$ or $l_2$ distance establishes a novel addition to the otherwise usual adversarial GAN loss. This might conceptually remind one of the autoencoder objectives: reproduce the input signal after encoding and decoding; only this time, there is no bottleneck but another interpretable image space. This can be exploited to stabilize the training, since the sequential concatenation of image translation functions, which we will call *G* and *F*, can be reversed. Figure 19 shows a schematic of the overall process (left) and one incarnation of the cycle, here from image domain *X* to $\mathcal{Y}$ and back (middle).

CycleGANs employ several loss terms in training: two adversarial losses $J(G, D_\mathcal{Y})$ and $J(F, D_X)$ and two cycle consistency losses, of which one $J_{\text{cyc}}(G, F)$ is indicated rightmost in Fig. 19. Zhu et al. [37] presented the initial publication with a participation of the cGAN author Isola [37]. The cycle consistency losses are $l_1$ losses in their implementation, and the GAN losses are least square losses instead of negative log likelihood, since more stable training was observed with this choice.

Almahairi et al. [38] provided an augmented version [38], noting that the original implementation suffers from the inability to generate stochastic results in the target domain $\mathcal{Y}$ but rather learns a one-to-one mapping between $X$ and $\mathcal{Y}$ and vice versa. To
<figure><img src="image_94.png" /><figcaption>**Fig. 19 Cycle GAN.** Left: image translation functions *G* and *F* convert between two domains. Discriminators $D_X$ and $D_Y$ give adversarial losses in both domains. Middle: for one concrete translation of an image *x*, the translation to *Y* and back to *X* is depicted. Right: after the translation cycle, the original and back-translated result are compared in the *cycle consistency loss*</figcaption></figure>

alleviate this problem, the generators are conditioned on one latent space each for both directions, so that, for the same input $x \in X$, *G* will now produce multiple generated outputs in $\mathcal{Y}$ depending on the sample from the auxiliary latent space (and similarly in reverse). Still, *F* has to recreate a $\hat{x}$ minimizing the cycle consistency loss for each of these samples. This also remedies a second criticism brought forward against vanilla CycleGANs: these networks can learn to hide information in the (intermediate) target image domain that fool the discriminator but help the backward generator to minimize the cycle consistency loss more efficiently [39]. Chu et al. [39] use adaptive histogram equalization to show that in visually empty regions of the intermediate images information is present. This is a finding reminiscent of adversarial attacks, which the authors elaborate on in their publication.

Zhang et al. [40] show a medical application. In their work, a CycleGAN has been used to train image translation and segmentation models on unpaired images of the heart, acquired with MRI and CT and with gold standard expert segmentations available for both imaging datasets. The authors proposed to learn more powerful segmentation models by enriching both datasets with artificially generated data. To this end, MRIs are converted into CT contrast images and vice versa using GANs. Segmentation models for MRI and CT are then trained on dataset consisting of original images and their expert segmentations and augmented by the converted images, for which expert segmentations can be carried over from their original domain. To achieve this, it is of importance that the converted (translated) images accurately depict the shape of the organs as expected in the target domain, which is enforced using the shape consistency loss.

In the extended setup of the CycleGAN with shape and cycle consistency, three different loss types instead of the original two are combined during training:

**Adversarial GAN losses** $J_{GAN}$. This loss term is the same as defined, e.g., in Eq. 5.
<figure><img src="image_95.png" /><figcaption>**Fig. 20 Cycle GAN with shape consistency loss (rightmost part of figure).** Note that the figure shows only one direction to ease readability</figcaption></figure>

<table><tr><td>Cycle consistency losses $J_{cyc}$.</td><td>This is the $l_1$ loss presented by the original CycleGAN authors discussed above.</td></tr><tr><td>Shape consistency losses $J_{shape}$.</td><td>The shape consistency loss is a new addition proposed by the authors. A cross-correlation loss takes into account two segmentations, the first being the gold standard segmentation $m_x$ for an $x \in X$ and one segmentation produced by a segmenter network $S$ that was trained on domain $\mathcal{Y}$ and receives the translated image $\hat{y} = G(x)$.</td></tr></table>

Figure 20 depicts the three loss components, of which the first two are known already from Fig. 19.

Note that the description as well as Fig. 20 only show one direction for cycle and shape consistency loss. Both are duplicated into the other direction and combined into the overall training objective, which then consists of six components.

In several other works, the CycleGAN approach was extended and combined with domain adaption methods for various segmentation tasks and also extended to volumetric data [41–43].

## 4.3 StyleGAN and Successor

One of the most powerful image synthesis GANs to date is the successor of StyleGAN, StyleGAN2 [44, 45]. The authors, at the time of writing researching at Nvidia, deviate from the usual GAN approach in which an image is generated from a randomly sampled vector from a latent space. Instead, they use a latent space that is created by a mapping function $f$ which is in their architecture implemented as a multilayer perceptron which maps from a 512-dimensional space $Z$ into a 512-dimensional space $W$. The second major change consisted of the so-called adaptive instance normalization layer, AdaIN, which implements a normalization to zero-mean and unit variance of each feature map, followed by a multiplicative factor and an additive bias term. This serves to
<figure><img src="image_96.png" /><figcaption>**Fig. 21** StyleGAN architecture, after [44]. Learnable layers and transformations are shown in green, the AdalN function in blue</figcaption></figure>

reweight the importance of feature maps in one layer. To ensure the locality of the reweighting, the operation is followed by the non-linearity. The scaling and bias are two components of $y = (y_s, y_b)$, which is the result of a learnable affine transformation $A$ applied to a sample from $W$.

In their experiments, Karras et al. [44] recognized that after these changes, the GAN actually no longer depended on the input vector drawn from $W$ itself, so the random latent vector was replaced by a static vector fed into the GAN. The $y$, which they call *styles*, remained to be results from a vector randomly sampled from the new embedding space $W$.

Lastly, noise is added in each layer, which serves to allow the GAN to produce more variation without learning to produce it from actual image content. The noise, like the latent vector, is fed through learnable transformations $B$, before it is added to the unnormalized feature maps. The overall architecture is sketched in Fig. 21.

In the basic setup, one sample is drawn from $W$ and fed through per-layer learned $A$ to gain per-layer different interpretations of the style, $y = (y_s, y_b)$. This can be changed, however, and the authors show how using one random sample $w_1$ in some of the layer blocks and another sample $w_2$ in the remaining; the result will be a mixture of styles of both individual samples. This way, the coarse attributes of the generated image can stem from one sample and the fine detail from another. Applied to a face generator, for example, pose and shape of the face are determined in the coarse early layers of the network, while hair structure and skin texture are the fine
<figure><img src="image_97.png" /><figcaption>**Fig. 22** Images created with StyleGAN; https://thisperson-artwork-cat-horse-chemicaldoesnotexist.com. Last accessed: 2022-01-14</figcaption></figure>

details of the last layers. The architecture and results gained widespread attention through a website,<fnref n="10" /> which recently was followed up by further similar pages. Results are depicted in Fig. 22.

The crucial finding in StyleGAN was that the mapping function *F* transforming the latent space vector from *Z* to *W* serves to ensure a disentangled (flattened) latent space. Practically, this means that if interpolating points *z*ᵢ between two points *z*₁ and *z*₂ drawn from *Z* and reconstructing images from these interpolated points *z*ᵢ, semantic objects might appear (in a StyleGAN-generating faces, for example, a hat or glasses) that are neither part of the generated images from the first point *z*₁ nor the second point *z*₂ between which it has been interpolated. Conversely, if interpolating in *W*, this “semantic discontinuity” is no longer the case, as the authors show with experiments in which they measure the visual change of resulting images when traversing both latent spaces.

In their follow-up publications, the same authors improve the performance even further. They stick to the basic architecture but redesign the generative network pertaining to the AdaIN function. In addition, they add their metric from [44] that was meant to quantify the entanglement of the latent space as a regularizer. The discriminator network was also enhanced, and the mechanisms of StyleGAN that implement the progressive growing have been successively replaced by more performance-efficient setups. In their experiments, they show a growth of visual and measured quality and removal of several artifacts reported for StyleGAN [45].

## 4.4 Stabilized GAN for Few-Shot Learning

GAN training was very demanding both regarding GPU power, in particular for high-performance architectures like StyleGAN and StyleGAN2, and, as importantly, availability of data. StyleGAN2, for example, has typical training times of about 10 days on a Nvidia 8-GPU Tesla V100. The datasets comprised at least tens of thousands of images and easily orders of magnitude more. Particularly in the medical domain, such richness of data is typically hard to find.

<footnote n="10">¹⁰ https://thispersondoesnotexist.com/.</footnote>
<figure><img src="image_98.png" /><figcaption>**Fig. 23** The FastGAN generator network. Shortcut connections through feature map weighting layers (called skip-layer excitation, SLE) transport information from low-resolution feature maps into high-resolution feature maps. For details regarding the blocks, see text</figcaption></figure>

The authors of [46] propose simple measures to stabilize the training of a specific GAN architecture, which they design from scratch using a replacement for residual blocks, arranged in an architecture with very few convolutional layers, and a loss that drives the discriminator to be less certain when it gets closer to convergence. In sum, this achieves very fast training and yields results competitive with prior GANs [46] and outperforming them in low-data situations.

The key ingredients to the architecture are shortcut connections in the generator model that rescale feature maps of higher resolution with learnable weights derived from low resolutions. The effect is to make fine details simultaneously more independent of direct predecessor feature maps and yet ensure consistency across scales.

A random seed vector of length 256 enters the first block ("Up Conv"), where it is upscaled to a 256 × 4 × 4 tensor. In Fig. 23, the further key blocks of the architecture are "upsample" and "SLE" blocks.

**Upsample** blocks consist of a nearest-neighbor upsampling followed by a 3×3 convolution, batch normalization, and nonlinearity.

**SLE** blocks (seen in the top right inset in the architecture diagram) don't touch the incoming high-resolution input (entering from top into the block) but comprise a pooling layer that in each SLE block is set up to yield a
<figure><img src="image_99.png" /><figcaption>**Fig. 24** The FastGAN self-supervision mechanism of the discriminator network. Self-supervision manifests through the loss term indicated by the curly bracket between reconstructions from feature maps and resampled/cropped versions of the original real image, $J_{recon}$</figcaption></figure>

4 × 4 stack of feature maps, followed by a convolution to reduce to a 1 × 1 tensor, which is then in a 1 × 1 convolution brought to the same number of channels as the high-resolution input. This vector is then multiplied to the channels of the high-resolution input.

Secondly, the architecture introduces a self-supervision feature in the discriminator network. The discriminator network (see Fig. 24) is a simple CNN with strided convolutions in each layer, halving resolution in each feature map. In the latest (coarsest) feature maps, simple up-scaling convolutional networks are attached that generate small images, which are then compared in loss functions ($J_{recon}$ in Fig. 24) to down-sampled versions of the real input image. This self-supervision of the discriminator is only performed for real images, not for generated ones.

The blocks in the figure spell out as follows:

**Down Conv Block** consists of two convolutional layers with strided 4 × 4 convolutions, effectively reducing the resolution from 1024² to 256².

**Residual Blocks** have two sub-items, “Conv Block A” being a strided 4 × 4 convolution to half resolution, followed by a padded 3 × 3 convolution. “Conv Block B” consists of a strided 2 × 2 average pooling that quarters resolution, followed by a 1 × 1 convolution, so that both blocks result in identically shaped tensors, which are then added.
<figure><img src="image_100.png" /><figcaption>**Fig. 25** FastGAN as implemented by the authors has been used to train a CT slice generative model. Images are not cherry-picked, but arranged by similar anatomical regions</figcaption></figure>

<table><tr><td>Conv Block C</td><td>consists of a 1 × 1 convolution followed by a 4 × 4 convolution without strides or padding, so that the incoming 8² feature map is reduced to 5².</td></tr><tr><td>Decoder</td><td>The decoder networks are four blocks of upsampling layers each followed by 3 × 3 convolutions.</td></tr></table>

The losses employed in the model are the discriminator loss consisting of the hinge version of the usual GAN loss, with the added regularizing reconstruction loss between original real samples and their reconstruction, and the generator loss plainly being $J_G = E_{z \sim Z}[D(G(z))]$.

The model is easy to train on modest hardware and little data, as evidenced by own experiments on a set of about 30 chest CTs (about 2500 image slices, converted to RGB). Figure 25 shows randomly picked generated example slices, roughly arranged by anatomical content. It is to be noted that organs appear mirrored in some images. On the other hand, no color artifacts are visible, so that the model has learned to produce only gray scale images. Training time for 50,000 iterations on a Nvidia TitanX GPU was approximately 10 hours.
<figure><img src="image_101.png" /><figcaption>**Fig. 26** The VQGAN+CLIP combination creates images from text inputs, here: "A child drawing of a dark garden full of animals"</figcaption></figure>

**4.5 VQGAN**

In a recent development, a team of researchers combined techniques for text interpretation with a dictionary of elementary image elements feeding into a generative network. The basic architecture component that is employed goes back to vector quantization variational autoencoders (VQ-VAE), where the latent space is no longer allowed to be continuous, but is quantized. This allows to use the latent space vectors in a look-up table: the visual elements.

Figure 26 was created using code available online, which demonstrates how images of different visual styles can be created using the combination of text-based conditioning and a powerful generative network.

The basis for image generation is the VQGAN ("vector quantization generative adversarial network") [47], which learns representations of input images that can later steer the generative process, in an adversarial framework. The conditioning is achieved with the CLIP ("Contrastive Image-Language Pretraining") model that learns a discriminator that can judge plausible images for a text label or vice versa [48].

The architecture has been developed with an observation in mind that puts the benefits and drawbacks of convolutional and transformer architectures in relation to each other. While the locality bias of convolutional architectures is inappropriate if overall structural image relations should be considered, it is of great help in capturing textural details that can exist anywhere, like fur, hair, pavement, or grass, but where the exact representation of hair
positions or pavement stones is irrelevant. On the other hand, image transformers are known to learn convolutional operators implicitly, posing a severe computational burden without a visible impact on the results. Therefore, Esser et al. [47] suggest to combine convolutional operators for local detail representation and transformer-based components for image structure.

Since the VQGAN as a whole is no longer a pure CNN but for a crucial component uses a transformer architecture, this model will be brought up again briefly in Subheading 5.2.

The VQGAN architecture is derived from the VQ-VAE (vector quantization variational autoencoder) [49], adding a reconstruction loss through a discriminator, which turns it into a GAN. At the core of the architecture is the quantization of estimated codebook entries. Among the quantized entries in the codebook, the closest entry to the query vector coding, an image patch is determined. The found codebook entry is then referred to by its index in the codebook. This quantization operation is non-differentiable, so for end-to-end training, gradients are simply copied through it during backpropagation.

The transformer can then efficiently learn to predict codebook indices from those comprising the current version of the image, and the generative part of the architecture, the decoder, produces a new version of the image. Learning expressive codebook entries is enforced by a perceptual loss that punishes inaccurate local texture, etc. Through this, the authors can show that high compression levels can be achieved—a prerequisite to enable efficient, yet comprehensive, transformer training.

# 5 Other Generative Models

We have already seen how GANs were not the first approach to image generation but have prevailed for a time when they became computationally feasible and in consequence have been better understood and improved to accomplish tasks in image analysis and image generation with great success. In parallel with GANs, other fundamentally different generative modeling approaches have also been under continued development, most of which have precursors from the “before-GAN” era as well. To give a comprehensive outlook, we will sketch in this last section the state of the art of a selection of these approaches.<fnref n="11" />

<footnote n="11">¹¹ The research on the so-called flow-based models, e.g., normalizing flows, has been omitted in this chapter, though acknowledging their emerging relevance also in the context of image generation. Flow-based models are built from sequences of invertible transformations, so that they learn data distributions explicitly at the expense of sometimes higher computational costs due to their sequential architecture. When combined, e.g., with a powerful GAN, they allow innovative applications, for example, to steer the exploration of a GAN’s latent space to achieve fine-grained control over semantic attributes for conditional image generation. Interested readers are referred to the literature [11, 13, 50–52].</footnote>
## 5.1 Diffusion and Score-Based Models

Diffusion models take a completely different approach to distribution estimation. GANs implicitly represent the target distribution by learning a surrogate distribution. Likelihood-based models like VAE approximate the target distribution explicitly, not requiring the surrogate. In diffusion models, however, the gradient of the log probability density function is estimated, instead of looking at the distribution itself (which would be the unfathomable integral of the gradient). This value is known as the Stein score function, leading to the notion that diffusion models are one variant of score-based models [53].

The simple idea behind this class of models is to revert a sequential noising process. Consider some image. Then, perform a large number of steps. In each step, add a small amount of noise from a known distribution, e.g., the normal distribution. Do this until the result is indistinguishable from random noise.

The denoising process is then formulated as a latent variable model, where $T-1$ latents successively progress from a noise image $\mathbf{x}_T \sim \mathcal{N}(\mathbf{x}_T; \mathbf{0}, \mathbf{I})$ to the reconstruction that we call $\mathbf{x}_0 \sim q(\mathbf{x}_0)$. The reconstructed image, $\mathbf{x}_0$, is therefore obtained by a *reverse process* $q_0(\mathbf{x}_{0:T})$. Note that each step in this chain can be evaluated in closed form [54]. Several model implementations of this approach exist, one being the deep diffusion probabilistic model (DDPM). Here, a deep neural network learns to perform one denoising step given the so-far achieved image and a $t \in \{1, \dots, T\}$. Iterative application of the model to the result of the last iteration will eventually yield a generated image from noise input.

Autoregressive diffusion models (ARDMs) [55] follow yet another thought model, roughly reminiscent of PixelRNNs we have briefly mentioned above (see Subheading 3.2). Both share the approach to condition the prediction of the next pixel or pixels on the already predicted ones. Other than in the PixelRNN, however, the specific ARDM proposed by the authors does not rely on a predetermined schedule of pixel updates, so that these models can be categorized as latent variable models.

As of late, the general topic of score-based methods, among which diffusion models are one variant, received more attention in the research community, fueled by a growing body of publications that report image synthesis results that outperform GANs [53, 56, 57]. Score function-based and diffusion models superficially share the similar concept of sequentially adding/removing noise but achieve their objective with very different means: where score function-based approaches are trained by score-matching and their sampling process uses Langevin dynamics [58], diffusion models are trained using the evidence lower bound (ELBO) and sample with a decoder, which is commonly a neural network. Figure 27 visualizes an example for a score function.

Score function-based (sometimes also score-matching) generative models have been developed to astounding quality levels, and
<figure><img src="image_102.png" /><figcaption>**Fig. 27** The Stein score function can be conceived of as the gradient of the log probability density function, here indicated by two Gaussians. The arrows represent the score function</figcaption></figure>

the recent works of Yang Song and others provide accessible blog posts,<fnref n="12" /> and a comprehensive treatment of the subject in several publications [53, 58, 59].

In the work of Ho et al. [54], the stepwise reverse (denoising) process is the basis of the denoising diffusion probabilistic models (DDPM). The authors emphasize that a proper selection of the noise schedule is crucial to fast, yet high-quality, results. They point out that their work is a combination of diffusion probabilistic models with score-matching models, in this combination also generalizing and including the ideas of autoregressive denoising models. In an extension of Ho et al.'s [54] work by Nichol and Dhariwal [57], an importance sampling scheme was introduced that lets the denoising process steer the most easy to predict next image elements. Equipped with this new addition, the authors can show that, in comparison to GANs, a wider region of the target distribution is covered by the generative model.

## 5.2 Transformer-Based Generative Models

The basics of how attention mechanisms and transformer architectures work will be covered in the subsequent chapter on this promising technology (Chapter 6). Attention-based models, predominantly transformers, have been used successfully for some time in sequential data processing and are now considered the superior alternative to recurrent networks like long-short-term memory (LSTM) networks. Transformers have, however, only recently made their way into the image analysis and now also the image generation world. In this section, we will only highlight some developments in the area of generative tasks.

<footnote n="12">¹² https://yang-song.github.io/blog/.</footnote>
Google Brain/Google AI's 2018 publication on so-called image transformers [60], among other tasks, shows successful conditional image generation for low-resolution input images to achieve super-resolution output images, and for image inpainting, where missing or removed parts of input images are replaced by content produced by the image transformer.

OpenAI have later shown that even unmodified language transformers can succeed to model image data, by dealing in sheer compute power for hand modeling of domain knowledge, which was the basis for the great success of previous unsupervised image generation models. They have trained Image GPT (or iGPT for short), a multibillion parameter language transformer model, and it excels in several image generation tasks, though only for fairly small image sizes [61].

In the recent past, StyleSwin has been proposed by Microsoft Research Asia [62], enabling high-resolution image generation. However, the approach uses a block-wise attention window, thereby potentially introducing spatial incoherencies at block edges, which they have to correct for.

"Taming transformers" [47], another recent publication already mentioned above, uses what the authors call a learned template code book of image components, which is combined with a vector quantization GAN (VQGAN). The VQGAN is structurally modeled after the VQ-VAE but adds a discriminator network. A transformer model in this architecture composes these code book elements and is interrogated by the GAN variational latent space, conditioned on a textual input, a label image, or other possible inputs. The GAN reconstructs the image from the so-quantized latent space using a combination of a perceptual loss assessing the overall image structure and a patch-based high-resolution reconstruction loss. By using a sliding attention window approach, the authors prevent patch border artifacts known from StyleSwin. Conditioning on textual input makes use of parts of the CLIP [48] idea ("Contrastive Language-Image Pretraining"), where a language model was trained in conjunction with an image encoder to learn embeddings of text-image pairs, sufficient to solve many image understanding tasks with competitive precision, without specific domain adaption.

It is evidenced by the lineup of institutions that training image transformer models successfully is nothing that can be achieved with modest hardware or on even a medium-scale image database. In particular for the medical area, where data is comparatively scarce even under best assumptions, the power of such models will only be available in the near future if domain transfer learning can be successfully achieved. This, however, is a known strength of transformer architectures.
Acknowledgements

I thank my colleague at the Fraunhofer Institute for Digital Medi-
cine MEVIS, Till Nicke, for his thorough review of the chapter and
many valuable suggestions for improvements. I owe many thanks
more to other colleagues for their insights both in targeted discus-
sions and most importantly in everyday work life.

References

[1] Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Proceedings of the 27th international conference on neural information processing systems - volume, NIPS’14 . MIT Press, Cambridge, pp 2672–2680

[2] Casella G, Berger RL (2021) Statistical inference. Cengage Learning, Boston

[3] Grinstead C, Snell LJ (2006) Introduction to probability. Swarthmore College, Swarthmore

[4] Severini TA (2005) Elements of distribution theory, vol 17. Cambridge University Press, Cambridge

[5] Murphy KP (2012) Machine learning: a probabilistic perspective. MIT Press, Cambridge

[6] Murphy KP (2022) Probabilistic machine learning: an introduction. MIT Press, Cambridge. http://doi.org/probml.ai

[7] Do CB, Batzoglou S (2008) What is the expectation maximization algorithm? Nat Biotechnol 26:8, 26:897–899. https://doi.org/10.1038/nbt1406. https://www.nature.com/articles/nbt1406

[8] Dempster AP, Laird NM, Rubin DB (1977) Maximum likelihood from incomplete data via the em algorithm. J Roy Statist Soc Ser B (Methodolog) 39:1–22. https://doi.org/10.1111/j.2517-6161.1977.TB01600.X. https://onlinelibrary.wiley.com/doi/full/10.1111/j.2517-6161.1977.tb01600.x. https://onlinelibrary.wiley.com/doi/abs/10.1111/j.2517-6161.1977.tb01600.x. https://rss.onlinelibrary.wiley.com/doi/10.1111/j.2517-6161.1977.tb01600.x

[9] van den Oord A, Kalchbrenner N, Kavukcuoglu K (2016) Pixel recurrent neural networks. ArXiv abs/1601.06759

[10] Magnusson K (2020) Understanding maximum likelihood: an interactive visualization. https://rpsychologist.com/likelihood/

[11] Rezende DJ, Mohamed S (2015) Variational inference with normalizing flows. In: ICML

[12] van den Oord A, Kalchbrenner N, Espeholt L, Kavukcuoglu K, Vinyals O, Graves A (2016) Conditional image generation with PixelCNN decoders. In: NIPS

[13] Dinh L, Sohl-Dickstein J, Bengio S (2017) Density estimation using Real NVP. ArXiv abs/1605.08803

[14] Salakhutdinov R, Hinton G (2009) Deep Boltzmann machines. In: van Dyk D, Welling M (eds) Proceedings of the twelfth international conference on artificial intelligence and statistics, PMLR, hilton clearwater beach resort, clearwater beach, Florida USA, Proceedings of Machine Learning Research, vol 5, pp 448–455. https://proceedings.mlr.press/v5/salakhutdinov09a.html

[15] Weng L (2018) From autoencoder to Beta-VAE. lilianwenggithubio/lil-log. http://lilianweng.github.io/lil-log/2018/08/12/from-autoencoder-to-beta-vae.html

[16] Kingma DP, Welling M (2014) Auto-encoding variational bayes. ArXiv 1312.6114

[17] Creswell A, White T, Dumoulin V, Arulkumaran K, Sengupta B, Bharath AA (2018) Generative adversarial networks: an overview. IEEE Signal Process Mag 35(1): 53–65. https://doi.org/10.1109/MSP.2017.2765202

[18] Arjovsky M, Bottou L (2017) Towards principled methods for training generative adversarial networks. ArXiv abs/1701.04862

[19] Theis L, van den Oord A, Bethge M (2016) A note on the evaluation of generative models. CoRR abs/1511.01844

[20] Radford A, Metz L, Chintala S (2015) Unsupervised representation learning with
deep convolutional generative adversarial networks. ArXiv http://arxiv.org/abs/1511.06434

[21] Islam J, Zhang Y (2020) GAN-based synthetic brain PET image generation. Brain Inform 7:1–12. https://doi.org/10.1186/s40708-020-00104-2/FIGURES/9. https://braininformatics.springeropen.com/articles/10.1186/s40708-020-00104-2

[22] Arjovsky M, Chintala S, Bottou L (2017) Wasserstein GAN. ArXiv http://arxiv.org/abs/1701.07875v3. 1701.07875

[23] Gulrajani I, Ahmed F, Arjovsky M, Dumoulin V, Courville A (2017) Improved training of Wasserstein GANs. ArXiv http://arxiv.org/abs/1704.00028v3. nIPS camera-ready, 1704.00028

[24] Villani C (2009) Optimal transport, old and new. Springer, Berlin. https://doi.org/10.1007/978-3-540-71050-9. https://www.cedricvillani.org/wp-content/uploads/2012/08/preprint-1.pdf

[25] Basso G (2015) A Hitchhiker's guide to Wasserstein distances. https://homeweb.unifr.ch/BassoG/pub/A%20Hitchhikers%20guide%20to%20Wasserstein.pdf

[26] Weng L (2019) From GAN to WGAN. ArXiv 1904.08994

[27] Baumgartner CF, Koch LM, Tezcan KC, Ang JX, Konukoglu E (2018) Visual feature attribution using Wasserstein GANs. In: The IEEE conference on computer vision and pattern recognition (CVPR)

[28] Dzanic T, Shah K, Witherden FD (2020) Fourier spectrum discrepancies in deep network generated images. In: 34th conference on neural information processing systems (NeurIPS)

[29] Joslin M, Hao S (2020) Attributing and detecting fake images generated by known GANs. In: Proceedings - 2020 IEEE symposium on security and privacy workshops, SPW 2020. Institute of Electrical and Electronics Engineers, Piscataway, pp 8–14. https://doi.org/10.1109/SPW50608.2020.00019

[30] Le BM, Woo SS (2021) Exploring the asynchronous of the frequency spectra of GAN-generated facial images. ArXiv https://arxiv.org/abs/2112.08050v1. 2112.08050

[31] Goebel M, Nataraj L, Nanjundaswamy T, Mohammed TM, Chandrasekaran S, Manjunath BS, Maya (2021) Detection,

attribution and localization of GAN generated images. Electron Imag. https://doi.org/10.2352/ISSN.2470-1173.2021.4. MWSF-276

[32] Isola P, Zhu JY, Zhou T, Efros AA (2016) Image-to-image translation with conditional adversarial networks. ArXiv http://arxiv.org/abs/1611.07004

[33] Senaras C, Sahiner B, Tozbikian G, Lozanski G, Gurcan MN (2018) Creating synthetic digital slides using conditional generative adversarial networks: application to Ki67 staining. In: Medical imaging 2018: digital pathology, society of photo-optical instrumentation engineers (SPIE) conference series, vol 10581, p 1058103. https://doi.org/10.1117/12.2294999

[34] Zhao G, Meyerand ME, Birn RM (2021) Bayesian conditional GAN for MRI brain image synthesis. ArXiv 2005.11875

[35] Bakas S, Reyes M, ..., Menze B (2019) Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge. ArXiv 1811.02629

[36] Cirillo MD, Abramian D, Eklund A (2020) Vox2Vox: 3D-GAN for brain tumour segmentation. ArXiv 2003.13653

[37] Zhu JY, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: 2017 IEEE international conference on computer vision (ICCV), IEEE, pp 2242–2251. http://ieeexplore.ieee.org/document/8237506/papers3://publication/doi/10.1109/ICCV.2017.244

[38] Almahairi A, Rajeswar S, Sordoni A, Bachman P, Courville A (2018) Augmented CycleGAN: Learning many-to-many mappings from unpaired data. ArXiv https://arxiv.org/pdf/1802.10151.pdf. 1802.10151

[39] Chu C, Zhmoginov A, Sandler M (2017) CycleGAN, a master of steganography. ArXiv http://arxiv.org/abs/1712.02950

[40] Zhang Z, Yang L, Zheng Y (2018) Translating and segmenting multimodal medical volumes with cycle- and shape-consistency generative adversarial network. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, IEEE, pp 9242–9251. https://doi.org/10.1109/CVPR.2018.00963. https://ieeexplore.ieee.org/document/8579061/
[41] Hoffman J, Tzeng E, Park T, Zhu JY, Isola P, Saenko K, Efros AA, Darrell T (2017) CyCADA: Cycle-consistent adversarial domain adaptation. ArXiv 1711.03213

[42] Huo Y, Xu Z, Bao S, Assad A, Abramson RG, Landman BA (2018) Adversarial synthesis learning enables segmentation without target modality ground truth. In: 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), pp 1217-1220. https://doi.org/10.1109/ISBI.2018.8363790

[43] Yang D, Xiong T, Xu D, Zhou SK (2020) Segmentation using adversarial image-to-image networks. In: Handbook of medical image computing and computer assisted intervention, pp 165-182. https://doi.org/10.1016/B978-0-12-816176-0.00012-0

[44] Karras T, Laine S, Aila T (2018) A style-based generator architecture for generative adversarial networks. IEEE Trans Pattern Analy Mach Intell 43:4217-4228. https://doi.org/10.1109/TPAMI.2020.2970919. https://arxiv.org/abs/1812.04948v3

[45] Karras T, Laine S, Aittala M, Hellsten J, Lehtinen J, Aila T (2020) Analyzing and improving the image quality of StyleGAN. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp 8107-8116. https://doi.org/10.1109/CVPR42600.2020.00813. https://arxiv.org/abs/1912.04958v2

[46] Liu B, Zhu Y, Song K, Elgammal A (2021) Towards faster and stabilized GAN training for high-fidelity few-shot image synthesis. In: International conference on learning representations. https://openreview.net/forum?id=1Fqg133qRaI

[47] Esser P, Rombach R, Ommer B (2021) Taming transformers for high-resolution image synthesis. In: 2021 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 12868-12878. https://doi.org/10.1109/CVPR46437.2021.01268

[48] Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, Sastry G, Askell A, Mishkin P, Clark J, Krueger G, Sutskever I (2021) Learning transferable visual models from natural language supervision. ArXiv 2103.00020

[49] van den Oord A, Vinyals O, Kavukcuoglu K (2017) Neural discrete representation learning. CoRR abs/1711.00937. http://arxiv.org/abs/1711.00937

[50] Weng L (2018) Flow-based deep generative models. lilianwenggithubio/lil-log. http://lilianweng.github.io/lil-log/2018/10/13/flow-based-deep-generative-models.html

[51] Kingma DP, Dhariwal P (2018) Glow: generative flow with invertible 1x1 convolutions. ArXiv https://doi.org/10.48550/ARXIV.1807.03039. https://arxiv.org/abs/1807.03039

[52] Abdal R, Zhu P, Mitra NJ, Wonka P (2021) StyleFlow: attribute-conditioned exploration of StyleGAN-generated images using conditional continuous normalizing flows. ACM Trans Graph 40(3):1-21. https://doi.org/10.1145/3447648. https://doi.org/10.1145/2F3447648

[53] Song Y, Sohl-Dickstein J, Kingma DP, Kumar A, Ermon S, Poole B (2021) Score-based generative modeling through stochastic differential equations. In: International conference on learning representations. https://openreview.net/forum?id=PxTIG12RRHS

[54] Ho J, Jain A, Abbeel P (2020) Denoising diffusion probabilistic models. ArXiv 2006.11239

[55] Hoogeboom E, Gritsenko AA, Bastings J, Poole B, van den Berg R, Salimans T (2021) Autoregressive diffusion models. ArXiv 2110.02037

[56] Dhariwal P, Nichol A (2021) Diffusion models beat GANs on image synthesis. ArXiv http://arxiv.org/abs/2105.05233

[57] Nichol A, Dhariwal P (2021) Improved denoising diffusion probabilistic models. ArXiv http://arxiv.org/abs/2102.09672

[58] Song Y, Ermon S (2019) Generative modeling by estimating gradients of the data distribution. In: Advances in neural information processing systems, pp 11895-11907

[59] Song Y, Garg S, Shi J, Ermon S (2019) Sliced score matching: a scalable approach to density and score estimation. In: Proceedings of the thirty-fifth conference on uncertainty in artificial intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019, p 204. http://auai.org/uai2019/proceedings/papers/204.pdf
[60] Parmar N, Vaswani A, Uszkoreit J, Łukasz Kaiser, Shazeer N, Ku A, Tran D (2018) Image transformer. ArXiV 1802.05751

[61] Chen M, Radford A, Child R, Wu J, Jun H, Luan D, Sutskever I (2020) Generative pre-training from pixels. In: Daumé III H, Singh A (eds) Proceedings of the 37th international conference on machine learning,

PMLR, proceedings of machine learning research, vol 119, pp 1691–1703. https://proceedings.mlr.press/v119/chen20s.html

[62] Zhang B, Gu S, Zhang B, Bao J, Chen D, Wen F, Wang Y, Guo B (2021) StyleSwin: transformer-based GAN for high-resolution image generation. ArXiV 2112.10762

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
**Transformers and Visual Transformers**

Robin Courant, Maika Edberg, Nicolas Dufour, and Vicky Kalogeiton

**Abstract**

Transformers were initially introduced for natural language processing (NLP) tasks, but fast they were adopted by most deep learning fields, including computer vision. They measure the relationships between pairs of input tokens (words in the case of text strings, parts of images for visual transformers), termed attention. The cost is exponential with the number of tokens. For image classification, the most common transformer architecture uses only the transformer encoder in order to transform the various input tokens. However, there are also numerous other applications in which the decoder part of the traditional transformer architecture is also used. Here, we first introduce the attention mechanism (Subheading 1) and then the basic transformer block including the vision transformer (Subheading 2). Next, we discuss some improvements of visual transformers to account for small datasets or less computation (Subheading 3). Finally, we introduce visual transformers applied to tasks other than image classification, such as detection, segmentation, generation, and training without labels (Subheading 4) and other domains, such as video or multimodality using text or audio data (Subheading 5).

**Key words** Attention, Transformers, Visual transformers, Multimodal attention

# 1 Attention

Attention is a technique in Computer Science that imitates the way in which the brain can focus on the relevant parts of the input. In this section, we introduce attention: its history (Subheading 1.1), its definition (Subheading 1.2), its types and variations (Subheadings 1.3 and 1.4), and its properties (Subheading 1.5).

To understand what attention is and why it is so useful, consider the following film review:

> While others claim the story is boring, I found it fascinating.

Is this film review positive or negative? The first part of the sentence is unrelated to the critic's opinion, while the second part suggests a positive sentiment with the word 'fascinating'. To a human, the answer is obvious; however, this type of analysis is not necessarily obvious to a computer.
Typically, sequential data require *context* to be understood. In natural language, a word has a meaning because of its position in the sentence, with respect to the other words: its *context*. In our example, while “boring” alone suggests that the review is negative, its contextual relationship with other words allows the reader to reach the appropriate conclusion. In computer vision, in a task like object detection, the nature of a pixel alone cannot be identified: we need to account for its neighborhood, its *context*. So, how can we formalize the concept of *context* in sequential data?

## 1.1 The History of Attention

This notion of *context* is the motivation behind the introduction of the attention mechanism in 2015 [1]. Before this, language translation was mostly relying on encoder-decoder architectures: recurrent neural networks (RNNs) [2] and in particular long-short-term memory (LSTMs) networks were used to model the relationship among words [3]. Specifically, each word of an input sentence is processed by the encoder sequentially. At each step, the past and present information are summarized and encoded into a fixed-length vector. In the end, the encoder has processed every word and outputs a final fixed-length vector, which summarizes all input information. This final vector is then decoded and finally translates the input information into the target language.

However, the main issue of such structure is that all the information is compressed into one fixed-length vector. Given that the sizes of sentences vary and as the sentences get longer, a fixed-length vector is a real bottleneck: it gets increasingly difficult not to lose any information in the encoding process due to the vanishing gradient problem [1].

As a solution to this issue, Bahdanau et al. [1] proposed the attention module in 2015. The attention module allows the model to consider the parts of the sentence that are relevant to predicting the next word. Moreover, this facilitates the understanding of relationships among words that are further apart.

## 1.2 Definition of Attention

Given two lists of tokens, $X \in \mathbb{R}^{N \times d_x}$ and $\mathcal{Y} \in \mathbb{R}^{N \times d_y}$, attention encodes information from $\mathcal{Y}$ into $X$, where $N$ is the length of inputs $X$ and $\mathcal{Y}$ and $d_x$ and $d_y$ are their respective dimensions. For this, we first define three linear mappings, query mapping $W^Q \in \mathbb{R}^{d_x \times d_q}$, key mapping $W^K \in \mathbb{R}^{d_y \times d_k}$, and value mapping $W^V \in \mathbb{R}^{d_y \times d_v}$, where $d_q, d_k$, and $d_v$ are the embedding dimensions in which the query, key, and value are going to be computed, respectively.

Then, we define the query $Q$, key $K$, and value $V$ [4] as:

$$Q = X W^Q$$

$$K = \mathcal{Y} W^K$$

$$V = \mathcal{Y} W^V$$
Next, the attention matrix is defined as:

$$A(Q, K) = \text{Softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right). \qquad (1)$$

This is illustrated in the left part of Fig. 1. The nominator $QK^T \in \mathbb{R}^{N \times N}$ represents how each part of the input in $X$ attends to each part of the input in $Y$.<fnref n="1" /> This dot product is then put through the softmax function to normalize its values and get positive values that add to 1. However, for large values of $d_k$, this may result in the softmax to have incredibly small gradients, so it is scaled down by $\sqrt{d_k}$.

The resulting $N \times N$ matrix encodes the relationship between $X$ with respect to $Y$: it measures how important a token in $X$ is with respect to another one in $Y$.

Finally, the attention output is defined as:

$$\text{Attention}(Q, K, V) = A(Q, K)V. \qquad (2)$$

Figure 1 displays this. The attention output encodes the information of each token by taking into account the contextual information. Therefore, through the learnable parameters—queries, keys, and values—the attention layers learn a token embedding that takes into account their relationship.

**Contextual Relationships** How does Eq. 2 encode contextual relationships? To answer this question, let us reconsider analyzing the sentiment of film reviews. To encode contextual relationships into the word embedding, we first want a matrix representation of the relationship between all words. To do so, given a sentence of length $N$, we take each word vector and feed it to two different linear layers, calling one output “query” and the other output “key”. We pack the queries into the matrix $Q$ and the keys into the matrix $K$, by taking their product $(QK^T)$. The result is a $N \times N$ matrix that explains how important the $i$-th word (row-wise) is to understand the $j$-th word (column-wise). This matrix is then scaled and normalized by the division and softmax. Next, we feed the word vectors into another linear layer, calling its output “value”. We multiply these two matrices together. The results of their product are attention vectors that encode the meaning of each word, by including their contextual meaning as well. Given that each of these queries, keys, and values is learnable parameter, as the attention layer is trained, the model learns how relationships among words are encoded in the data.

<footnote n="1">¹ Note that in the literature, there are two main attention functions: additive attention [1] and dot-product attention (Eq. 1). In practice, the dot product is more efficient since it is implemented using highly optimized matrix multiplication, compared to the feed-forward network of the additive attention; hence, the dot product is the dominant one.</footnote>
<figure><img src="image_103.png" /><figcaption>**Fig. 1** Attention block. Next to each element, we denote its dimensionality. Figure inspired from [4]</figcaption></figure>

**1.3 Types of Attention**

There exist two dominant types of attention mechanisms: self-attention and cross attention [4]. In self-attention, the queries, keys, and values come from the same input, i.e., $X= \Upsilon$; in cross attention, the queries come from a different input than the key and value vectors, i.e., $X \neq \Upsilon$. These are described below in Subheadings 1.3.1 and 1.3.2, respectively.

**1.3.1 Self-Attention**

In self-attention, the tokens of $X$ attend to themselves ($X= \Upsilon$). Therefore, it is modeled as follows:

$$SA(\mathbf{X}) = \text{Attention}(\mathbf{X} \mathbf{W}^Q, \mathbf{X} \mathbf{W}^K, \mathbf{X} \mathbf{W}^V). \quad (3)$$

Self-attention formalizes the concept of context. It learns the patterns underlying how parts of the input correspond to each other. By gathering information from the same set, given a sequence of tokens, a token can attend to its neighboring tokens to compute its output.

**1.3.2 Cross Attention**

Most real-world data are multimodal—for instance, videos contain frames, audios, and subtitles, images come with captions, etc. Therefore, models that can deal with such types of multimodal information have become essential.
Cross attention is an attention mechanism designed to handle multimodal inputs. Unlike self-attention, it extracts queries from one input source and key-value pairs from another one ($X \neq \Upsilon$). It answers the following question: “Which parts of input $X$ and input $\Upsilon$ correspond to each other?” Cross attention (CA) is defined as:

$$CA(X, \Upsilon) = \text{Attention}(X W^Q, \Upsilon W^K, \Upsilon W^V). \quad (4)$$

## 1.4 Variation of Attention

Attention is typically employed in two ways: (1) multi-head self-attention (MSA, Subheading 1.4) and (2) masked multi-head attention (MMA, Subheading 1.4).

**Attention Head** We call attention head the mechanism presented in Subheading 1.2, i.e., query-key-value projection, followed by scaled dot product attention (Eqs. 1 and 2).

When employing an attention-based model, relying only on a single attention head can inhibit learning. Therefore, the multi-head attention block is introduced [4].

**Multi-head Self-Attention (MSA)** MSA is shown in Fig. 2 and is defined as:

$$ 
\begin{aligned}
\text{MSA}(\mathbf{X}) &= \text{Concat}(\text{head}_1(\mathbf{X}), \dots, \text{head}_b(\mathbf{X})) \mathbf{W}^O, \\
\text{head}_i(\mathbf{X}) &= \text{SA}(\mathbf{X}), \quad \forall i \in \{1, b\},
\end{aligned}
\quad (5) $$

where Concat is the concatenation of $b$ attention heads and $\mathbf{W}^O \in \mathbb{R}^{b d_v \times d}$ is projection matrix. This means that the initial embedding dimension $d_x$ is decomposed into $b \times d_v$ and the computation per head is carried out independently. The independent attention heads are usually concatenated and multiplied by a linear layer to match the desired output dimension. The output dimension is often the same as the input embedding dimension $d$. This allows an easier stacking of multiple blocks.

**Multi-head Cross Attention (MCA)** Similar to MSA, MCA is defined as:

$$ 
\begin{aligned}
\text{MCA}(\mathbf{X}, \mathbf{\Upsilon}) &= \text{Concat}(\text{head}_1(\mathbf{X}, \mathbf{\Upsilon}), \dots, \text{head}_b(\mathbf{X}, \mathbf{\Upsilon})) \mathbf{W}^O, \\
\text{head}_i(\mathbf{X}, \mathbf{\Upsilon}) &= \text{CA}(\mathbf{X}, \mathbf{\Upsilon}), \quad \forall i \in \{1, b\}.
\end{aligned}
\quad (6) $$

**Masked Multi-head Self-Attention (MMSA)** The MMSA layer [4] is another variation of attention. It has the same structure as the multi-head self-attention block (Subheading 1.4), but all the later vectors in the target output are masked. When dealing with sequential data, this can help make training parallel.
<figure><img src="image_104.png" /><figcaption>**Fig. 2** Multi-head self-attention block (MSA). First, the input **X** is projected to queries, keys, and values and then passed through **h** attention blocks. The **h** resulting attention outputs are then concatenated together and finally projected to a **d**-dimensional output vector. Next to each element, we denote its dimensionality. Figure inspired from [4]</figcaption></figure>

**1.5 Properties of Attention**

While attention encodes contextual relationships, it is *permutation equivalent*, as the mechanism does not account for the order of the input data. As shown in Eq. 2, the attention computations are all matrix multiplication and normalizations. Therefore, a permuted input results in a permuted output. In practice, however, this may not be an accurate representation of the information. For instance, consider the sentences “the monkey ate the banana” and “the banana ate the monkey.” They have distinct meanings because of the order of the words. If the order of the input is important, various mechanisms, such as the positional encoding, discussed in Subheading 2.1.2, are used to capture this subtlety.

# 2 Visual Transformers

The transformer architecture was introduced in [4] and is the first architecture that relies purely on attention to draw connections between the inputs and outputs. Since its debut, it revolutionized deep learning, making breakthroughs in numerous fields, including
natural language processing, computer vision, chemistry, and biology, thus making its way to becoming the *default* architecture for learning representations. Recently, the standard transformer [4] has been adapted for vision tasks [5]. And again, visual transformer has become one of the central architectures in computer vision.

In this section, we first introduce the basic architecture of transformers (Subheading 2.1) and then present its advantages (Subheading 2.2). Finally, we describe the vision transformer (Subheading 2.3).

## 2.1 Basic Transformers

As shown in Fig. 3, the transformer architecture [4] is an encoder-decoder model. First, it embeds input tokens $X = (x_1, ..., x_N)$ into a latent space, resulting in latent vectors $Z = (z_1, ..., z_N)$, which are fed to the decoder to output $\Upsilon = (y_1, ..., y_M)$. The encoder is a stack of $L$ layers, with each one consisting of two sub-blocks: multi-head self-attention (MSA) layers and a multilayer perceptron (MLP). The decoder is also a stack of $L$ layers, with each one consisting of three sub-blocks: masked multi-head self-attention (MMSA), multi-head cross attention (MCA), and MLP.

**Overview** Below, we describe the various parts of the transformer architecture, following Fig. 3. First, the input tokens are converted into the embedding tokens (Subheading 2.1.1). Then, the positional encoding adds a positional token to each embedding token to denote the order of tokens (Subheading 2.1.2). Then, the transformer encoder follows (Subheading 2.1.3). This consists of a stack of $L$ multi-head attention, normalization, and MLP layers and encodes the input to a set of semantically meaningful features. After, the decoder follows (Subheading 2.1.4). This consists of a stack of $L$ masked multi-head attention, multi-head attention, and MLP layers followed by normalizations and decodes the input features with respect to the output embedding tokens. Finally, the output is projected to linear and softmax layers.

### 2.1.1 Embedding

The first step of transformers consists in converting input tokens<fnref n="2" /> into embedding tokens, i.e., vectors with meaningful features. To do so, following standard practice [6], each input is projected into an embedding space to obtain embedding tokens $\mathcal{Z}^i$. The embedding space is structured in a way that the distance between a pair of vectors is relative to the semantic similarity of their associated words. For the initial NLP case, this means that we get a vector of each word, such that the vectors that are closer together have similar meanings.

<footnote n="2">² Note the initial transformer architecture was proposed for natural language processing (NLP), and therefore the inputs were words.</footnote>
<figure><img src="image_105.png" /><figcaption>**Fig. 3** The transformer architecture. It consists of an encoder (left) and a decoder (right) block, each one consisting from a series of attention blocks (multi-head and masked multi-head attention) and MLP layers. Next to each element, we denote its dimensionality. Figure inspired from [4]</figcaption></figure>

2.1.2 Positional Encoding

As discussed in Subheading 1.5, the attention mechanism is posi-
tional agnostic, which means that it does not store the information
on the position of each input. However, in most cases, the order of
input tokens is relevant and should be taken into account, such as
the order of words in a sentence matter as they may change its
meaning. Therefore, [4] introduced the *Positional Encoding*
PE ∈ ℝ^{N × d_x}, which adds a positional token to each embedding
token Z^e ∈ ℝ^{N × d_x}.
**Sinusoidal Positional Encoding**

The sinusoidal positional encoding [4] is the main positional encoding method, which encodes the position of each token with sinusoidal waves of multiple frequency. For an embedding token $\mathbf{Z}^e \in \mathbb{R}^{N \times d_x}$, its positional encoding $\mathbf{PE} \in \mathbb{R}^{N \times d_x}$ is defined as:

$$ 
\begin{aligned}
\mathbf{PE}(i, 2j) &= \sin\left(\frac{i}{10000^{2j/d}}\right) \\
\mathbf{PE}(i, 2j + 1) &= \cos\left(\frac{i}{10000^{2j/d}}\right), \forall i, j \in [|1, n|] \times [|1, d|].
\end{aligned}
\qquad (7) $$

**Learnable Positional Encoding**

An orthogonal approach is to let the model learn the positional encoding. In this case, $\mathbf{PE} \in \mathbb{R}^{N \times d_x}$ becomes a learnable parameter. This, however, increases the memory requirements, without necessarily bringing improvements over the sinusoidal encoding.

**Positional Embedding**

After its computation, either the positional encoding $\mathbf{PE}$ is added to the embedding tokens or they are concatenated as follows:

$$ 
\begin{aligned}
\mathbf{Z}^{\text{pe}} &= \mathbf{Z}^e + \mathbf{PE}, \text{ or} \\
\mathbf{Z}^{\text{pe}} &= \text{Concat}(\mathbf{Z}^e, \mathbf{PE}),
\end{aligned}
\qquad (8) $$

where Concat denotes vector concatenation. Note that the concatenation has the advantage of not altering the information contained in $\mathbf{Z}^e$, since the positional information is only added to the unused dimension. Nevertheless, it augments the input dimension, leading to higher memory requirements. Instead, the addition does preserve the same input dimension while altering the content of the embedding tokens. When the input dimension is high, this content altering is trivial, as most of the content is preserved. Therefore, in practice, for high dimension, summing positional encodings is preferred, whereas for low dimensions concatenating them prevails.

### 2.1.3 Encoder Block

The encoder block takes as input the embedding and positional tokens and outputs features of the input, to be decoded by the decoder block. It consists of a stack of L multi-head self-attention (MSA) layers and a multilayer perceptron (MLP). Specifically, the embedding and positional tokens, $\mathbf{Z}_x^{\text{pe}} \in \mathbb{R}^{N \times d_x}$, go through a multi-head self-attention block. Then, a residual connection with layer normalization is deployed. In the transformer, this operation is performed after each sub-layer. Next, we feed its output to an MLP and a normalization layer. This operation is performed L times, and each time the output of each encoder block (of size $N \times d_x$) is the input of the subsequent block. In the L-th time, the output of the normalization is the input of the cross-attention block in the decoder (Subheading 2.1.4).
### 2.1.4 Decoder Block

The decoder has two inputs: first, an input that constitutes the queries $Q \in \mathbb{R}^{N \times d}$ of the encoder, and, second, the output of the encoder that constitutes the key-value $K, V \in \mathbb{R}^{N \times d}$ pair. Similar to Subheadings 2.1.1 and 2.1.2, the first step constitutes encoding the output token to output embedding token and output positional token. These tokens are fed into the main part of the decoder, which consists of a stack of $L$ masked multi-head self-attention (MMSA) layers, multi-head cross-attention (MCA) layers, and multilayer perceptron (MLP) followed by normalizations. Specifically, the embedding and positional tokens, $Z_y^{pe} \in \mathbb{R}^{N \times d}$, go through a MMSA block. Then, a residual connection with layer normalization follows. Next, an MCA layer (followed by normalization) maps the queries to the encoded key values before forwarding the output to an MLP. Finally, we project the output of the $L$ decoder blocks (of dimension $N \times d_y$) through a linear layer and get output probability through a softmax layer.

## 2.2 Advantages of Transformers

Since their introduction, the transformers have had a significant impact on deep learning approaches.

In natural language processing (NLP), before transformers, most architectures used to rely on recurrent modules, such as RNNs [2] and in particular LSTMs [3]. However, recurrent models process the input sequentially, meaning that, to compute the current state, they require the output of the previous state. This makes them tremendously inefficient, as they are impossible to parallelize. On the contrary, in transformers, each input is processed independent of the others, and the multi-head attention can perform multiple attention computations at once. This makes transformers highly efficient, as they are highly parallelizable.

This results in not only exceptional scalability, both in the complexity of the model and the size of datasets, but also relatively fast training. Notably, the recent switch transformers [7] was pretrained on 34 billion tokens from the C4 dataset [8], scaling the model to over 1 trillion parameters.

This scalability [7] is the principal reason for the power of the transformer. While it was originally introduced for translation, it refrains from introducing many inductive biases, i.e., the set of assumptions that the user makes about the structure of the model input. In doing so, the transformer relies on data to learn how they are structured. Compared to its counterparts with more biases, the transformer requires much more data to produce comparable results [5]. However, if a sufficient amount of data is available, the lack of inductive bias becomes a strength. By learning the structure of the data from the data, the transformer is able to learn better without human assumptions hindering [9].

In most tasks involving transformers, the model is first pretrained on a large dataset and then fine-tuned for the task at hand on a smaller dataset. The pretraining phase is essential for
transformers to learn the global structure of the specific input modality. For fine-tuning, typically fewer data suffice as the model is already rich. For instance, in natural language processing, BERT [10], a state-of-the-art language model, is pretrained on a Wikipedia-based dataset [11], with over 6 million articles and Book Corpus [12] with over 10,000 books. Then, this model can be fine-tuned on much more specific tasks. In computer vision, the vision transformer (ViT) is pretrained on the JFT-300M dataset, containing over 1 billion labels for 300 million images [5]. Hence, with a sufficient amount of data, transformers achieve results that were never possible before in various areas of machine learning.

## 2.3 Vision Transformer

Transformers offer an alternative to CNNs that have long held a stranglehold on computer vision. Before 2020, most attempts to use transformers for vision tasks were still highly reliant on CNNs, either by using self-attention jointly with convolutions [13, 14] or by keeping the general structure of CNNs while using self-attention [15, 16].

The reason for this is rooted in the two main weaknesses of the transformers. First, the complexity of the attention operation is high. As attention is a quadratic operation, the number of parameters skyrockets quickly when dealing with visual data, i.e., images—and even more so with videos. For instance, in the case of ImageNet [17], inputting a single image with $256 \times 256 = 65,536$ pixels in an attention layer would be too heavy computationally. Second, transformers suffer from lack of inductive biases. Since CNNs were specifically created for vision tasks, their architecture includes spatial inductive biases, like translation equivariance and locality. Therefore, the transformers have to be pretrained on a significantly large dataset to achieve similar performances.

The vision transformer (ViT) [5] is the first systematic approach that uses directly transformers for vision tasks by addressing both aforementioned issues. It rids the concept of convolutions altogether, using purely a transformer-based architecture. In doing so, it achieves the state of the art on image recognition on various datasets, including ImageNet [17] and CIFAR-100 [18].

Figure 4 illustrates the ViT architecture. The input image is first split into $16 \times 16$ patches, flattened, and mapped to the expected dimension through a learnable linear projection. Since the image size is reduced to $16 \times 16$, the complexity of the attention mechanism is no longer a bottleneck. Then, ViT encodes the positional information and attaches a learnable embedding to the front of the sequence, similarly to BERT's classification token [10]. The output of this token represents the entirety of the input—it encodes the information from each part of the input. Then, this sequence is fed into an encoder block, with the same structure as in the standard transformers [4]. The output of the classification token is then fed into an MLP that outputs class probabilities.
<figure><img src="image_106.png" /><figcaption>**Fig. 4** The vision transformer architecture (ViT). First, the input image is split into patches (bottom), which are linearly projected (embedding), and then concatenated with positional embedding tokens. The resulting tokens are fed into a transformer, and finally the resulting classification token is passed through an MLP to compute output probabilities. Figure inspired from [5]</figcaption></figure>

Due to the lack of inductive biases, when ViT is trained only on mid-sized datasets such as ImageNet, it scores some percentage points lower than the state of the art. Therefore, the proposed model is first pretrained on the JFT-300M dataset [19] and then fine-tuned on smaller datasets, thereby increasing its accuracy by 13%.

For a complete overview of visual transformers and follow-up works, we invite the readers to study [9, 20].

## 3 Improvements over the Vision Transformer

In this section, we present transformer-based methods that improve over the original vision transformer (Subheading 2.3) in two main ways. First, we introduce approaches that are trained on smaller datasets, unlike ViT [5] that requires pretraining on 300 million labeled images (Subheading 3.1). Second, we present extensions over ViT that are more computational-efficient than ViT, given that training a ViT is directly correlated to the image resolution and the number of patches (Subheading 3.2).
## 3.1 Data Efficiency

As discussed in Subheading 2.3, the vision transformer (ViT) [5] is pretrained on a massive proprietary dataset (JFT-300M) which contains 300 million labeled images. This need arises with transformers because we remove the inductive biases from the architecture compared to convolutional-based networks. Indeed, convolutions contain some translation equivariance. ViT does not benefit from this property and thus has to learn such biases, requiring more data. JFT-300M is an enormous dataset, and to make ViT work in practice, better data-efficiency is needed. Indeed, collecting that amount of data is costly and can be infeasible for most tasks.

**Data-Efficient Image Transformers (DeiT) [21]** The first work to achieve an improved data efficiency is DeiT [21]. The main idea of DeiT is to distill the inductive biases from a CNN into a transformer (Fig. 5). DeiT adds another token that works similarly to the class token. When training, ground truth labels are used to train the network according to the class token output with a cross-entropy (CE) loss. However, for the distillation network, the output labels are compared to the labels provided from a teacher network with a

<figure><img src="image_107.png" /><figcaption>**Fig. 5** The DeiT architecture. The architecture features an extra token, the distillation token. This token is used similarly to the class token. Figure inspired from [21]</figcaption></figure>
cross-entropy loss. The final loss for a N-categorical classification task is defined as follows:

$$ 
\begin{aligned}
\mathcal{L}_{\text{hardDistill}}^{\text{global}} &= \frac{1}{2} (\mathcal{L}_{\text{CE}}(\Psi(\mathbf{Z}_{\text{class}}), \mathbf{y}) + \mathcal{L}_{\text{CE}}(\Psi(\mathbf{Z}_{\text{distill}}), \mathbf{y}_{T})), \\
\mathcal{L}_{\text{CE}}(\hat{\mathbf{y}}, \mathbf{y}) &= - \frac{1}{N} \sum_{i=1}^{N} [y_i \log \hat{y}_i + (1 - y_i) \log (1 - \hat{y}_i)]
\end{aligned}
\quad (9) $$

with $\Psi$ the softmax function, $\mathbf{Z}_{\text{class}}$ the class token output, $\mathbf{Z}_{\text{distill}}$ the class token output, $\mathbf{y}$ the ground truth label, and $\mathbf{y}_{T}$ the teacher label prediction.

The teacher network is a Convolutional Neural Network (CNN). The main idea is that the distillation head will provide the inductive biases needed to improve the data efficiency of the architecture. By doing this, DeiT achieves remarkable performance on the ImageNet dataset, by training “only” on ImageNet-1K [17], which contains 1.3 million images.

**Convit [22]** The main disadvantage of DeiT [21] is that it requires a pretrained CNN, which is not ideal, and it would be more convenient to not have this requirement. The CNN has a hard inductive bias constraint that can be a major limitation. Indeed, if enough data is available, learning the biases from the data can result in better representations.

Convit [22] overpasses this issue by including the inductive bias of CNNs into a transformer in a soft way. Specifically, if the inductive bias is limiting the training, the transformer can discard it. The main idea is to include the inductive bias into the ViT initialization. Therefore, before beginning training, the ViT is equivalent to a CNN. Then, the network can progressively learn the needed biases and diverge from the CNN initialization.

**Compact Convolutional Transformer [23]**, DeiT [21], and Convit [22] successfully achieve data efficiency at the ImageNet scale. However, ImageNet is a big dataset with 1.3 million images, whereas most datasets are significantly smaller.

To reach higher data efficiency, the compact convolutional transformer [23] uses a CNN operation to extract the patches and then uses these patches in a transformer network (Fig. 6). The compact convolutional transformer comes with some modifications that lead to major improvements. First, by having a more complex encoding of patches, the system relies on the convolutional inductive biases at the lower scales and then uses a transformer network to remove the locality constraint of the CNN. Second, the authors show that discarding the “class” token results in higher efficiency. Specifically, instead of the class token, the compact convolutional transformer pools together all the patches token and classifies on top of this pooled token. These two modifications enable using
<figure><img src="image_108.png" /><figcaption>**Fig. 6** Compact convolutional transformers. This architecture features a convolutional-based patch extraction to leverage a smaller transformer network, leading to higher data efficiency. Figure inspired from [23]</figcaption></figure>

smaller transformers while improving both the data efficiency and the computational efficiency. Therefore, these improvements allow the compact convolutional transformer to be successfully trained on smaller datasets, such as CIFAR or MNIST.

## 3.2 Computational Efficiency

The vision transformer architecture (Subheading 2.3) suffers from a $O(n^2)$ complexity with respect to the number of tokens. When considering small resolution images or big patch size, this is not a limitation; for instance, for an image of 224 x 224 resolution with 16 x 16 patches, this amounts to 196 tokens. However, when needing to process larger images (for instance, 3D images in medical imaging) or when considering smaller patches, using and training such models becomes prohibitive. For instance, in tasks such as segmentation or image generation, it is needed to have more granular representations than 16 x 16 patches; hence, it is crucial to solve this issue to enable more applications of vision transformer.

**Swin Transformer** [24] One idea to make transformers more computation-efficient is the Swin transformer [24]. Instead of attending every patch in the image, the Swin transformer proposes to add a locality constraint. Specifically, the patches can only attend other patches that are limited to a vicinity window *K*. This restores the local inductive bias of CNNs. To allow communication across
<figure><img src="image_109.png" /><figcaption>**Fig. 7** Shifting operation in the Swin transformer [24]. Between each attention operation, the attention window is shifted so that each patch can communicate with different patches than before. This allows the network to gain more global knowledge with the network's depth. Figure inspired from [24]</figcaption></figure>

patches throughout the network, the Swin transformer shifts the attention windows from one operation to another (Fig. 7). Therefore, the Swin transformer is quadratic with regard to the size of the window *K* but linear with respect to the number of tokens *n* with complexity $O(nK^2)$. In practice, however, *K* is small, and this solves the quadratic complexity problem of attention.

**Perceiver [25, 26]** Another idea for more computation-efficient visual transformers is to make a more drastic change to the architecture. If instead of using self-attention the model uses cross attention, the problem of the quadratic complexity with regard to the number of tokens can be solved. Indeed, computing the cross attention between two sets of length *m* and *n*, respectively, has complexity $O(mn)$. This idea is introduced in the perceiver [25, 26]. The key idea is to have a smaller set of latent variables that will be used as queries and that will retrieve information in the image token set (Fig. 8). Since this solves the quadratic complexity issue, it also removes the need of using patches; hence, in the case of transformers, each pixel is mapped to a single token.

# 4 Vision Transformers for Tasks Other than Classification

Subheadings 1–3 introduce visual transformers for one main application: classification. Nevertheless, transformers can be used for numerous other tasks than classification.

In this section, we present some fundamental vision tasks where transformers have had a major impact: object detection in images (Subheading 4.1), image segmentation (Subheading 4.2), training
<figure><img src="image_110.png" /><figcaption>**Fig. 8** The perceiver architecture [25, 26]. A set of latent tokens retrieve information from the image through cross attention. Self-attention is performed between the tokens to refine the learned representation. These operations are linear with respect to the number of image tokens. Figure inspired from [25, 26]</figcaption></figure>

visual transformers without labels (Subheading 4.3), and image generation using generative adversarial networks (GANs) (Subheading 4.4).

**4.1 Object Detection with Transformers**

Detection is one of the early tasks that have seen improvements thanks to transformers. Detection is a combined recognition and localization problem; this means that a successful detection system should both recognize whether an object is present in an image and localize it spatially in the image. Carion et al. [14] is the first approach that uses transformers for detection.

**DEtection TRansformer (DETR) [14]** DETR first extracts visual representations with a convolutional network (Fig. 9).<fnref n="3" /> Then, the encodings are processed by a transformer network. Finally, the processed tokens are provided to a transformer decoder. The decoder uses cross attention between a set of learned tokens and the image tokens encoded by the encoder and outputs a set of tokens. Each output token is then passed through a feed-forward network that predicts if an object is present in an image or not; if the object is indeed present, the network also predicts the class and spatial location of the object, i.e., coordinates within the image.

**4.2 Image Segmentation with Transformers**

The goal of image segmentation is to assign to each pixel of an image the label of the object it belongs to. The segmenter [27] is a purely ViT approach addressing image segmentation. The idea is to first use ViT to encode the image. Then, the segmenter learns a token per

<footnote n="3">³ Note that, in DETR, the transformer is not directly used to extract the visual representation. Instead, it focuses on refining the visual representation to extract the object information.</footnote>
<figure><img src="image_111.png" /><figcaption>**Fig. 9** The DETR architecture. It refines a CNN visual representation to extract object localization and classes. Figure inspired from [14]</figcaption></figure>

<figure><img src="image_112.png" /><figcaption>**Fig. 10** The segmenter architecture. It is a purely ViT-based approach to perform semantic segmentation. Figure inspired from [27]</figcaption></figure>

semantic label. The encoded patch tokens and the semantic tokens are then fed to a second transformer. Finally, by computing the scalar product between the semantic tokens and the image tokens, the network assigns a label to each patch. Figure 10 displays this.

**4.3 Training Transformers Without Labels**

Visual transformers have initially been trained for classification tasks. However, this tasks requires having access to massive amounts of labeled data, which can be hard to obtain (as discussed in Subheading 3.1). Subheadings 3.1 and 3.2 present ways to train ViT more efficiently. However, it would also be interesting to be able to train this type of networks with “cheaper” data. Therefore, the goal of this part is to introduce unsupervised learning with transformers, i.e., training transformers without any labels.
<figure><img src="image_113.png" /><figcaption>**Fig. 11** The DINO training procedure. It consists in matching the outputs between two networks ($p_1$ and $p_2$) having two different augmentations ($X_1$ and $X_2$) of the same image as input ($\mathbf{X}$). The parameters of the teacher model are updated with an exponential moving average (ema) of the student parameters. Figure inspired from [28]</figcaption></figure>

**Self-DIstillation with NO labels (DINO) [28]** DINO is one of the first works that trains a ViT with self-supervised learning (Fig. 11). The main idea is to have two ViT models following the teacher-student paradigm: the first model is updated through gradient descent, and the second is an exponential moving average of the first one. Then, the whole two-stream DINO network is trained using two augmentations of the same image, which are each passed to one of the two networks. The goal of the training is to match the output between the two networks, i.e., no matter the augmentation in the input data, both networks should produce the same result. The main finding of DINO is that the ViT is capable of learning a semantic understanding of the image, as the attention matrices display some semantic information. Figure 12 visualizes the attention matrix of the various ViT heads trained with DINO.

**Masked Autoencoders (MAE) [29]** Another way to train a ViT without supervision is by using an autoencoder architecture. Masked autoencoders (MAE) [29] perform a random masking of the input token and give the task to reconstruct the original image to a decoder. The encoder learns a representation that performs
<figure><img src="image_114.png" /><figcaption>**Fig. 12** DINO samples. Visualization of the attention matrix of ViT heads trained with DINO. The ViT discovers the semantic structure of an image in an unsupervised way</figcaption></figure>

<figure><img src="image_115.png" /><figcaption>**Fig. 13** The MAE training procedure. After masking some tokens of an image, the remaining tokens are fed to an encoder. Then a decoder tries to reconstruct the original image from this representation. Figure inspired from [29]</figcaption></figure>

well in a given downstream task. This is illustrated in Fig. 13. One of the key observations of the MAE work [29] is that the decoder does not need to be very good for the encoder to achieve good performance: by using only a small decoder, MAE successfully trains a ViT in an autoencoder fashion.

## 4.4 Image Generation with Transformers and Attention

Attention and vision transformers have also helped in developing fresh ideas and creating new architectures for generative models and in particular for generative adversarial networks (GANs).

**GANsformers** [30] GANsformers are the most representative work of GANs with transformers, as they are a hybrid architecture using both attention and CNNs. The GANsformer architecture is illustrated in Fig. 14. The model first splits the latent vector of a GAN into multiple tokens. Then, a cross-attention mechanism is used to improve the generated feature maps, and at the same time, the GANsformer architecture retrieves information from the generated feature map to enrich the tokens. This mechanism allows the GAN to have better and richer semantic knowledge, which is showed to be useful for generating multimodal images.

**StyleSwin** [31] Another approach for generative modeling is to purely use a ViT architecture like StyleSwin [31]. StyleSwin is a GAN that leverages a similar type of attention as the Swin transformer [24]. This allows to generate high-definition images without having to deal with the quadratic cost problem.
<figure><img src="image_116.png" /><figcaption>Fig. 14 GANsformer architecture. A set of latents contribute to bring information to a CNN feature map. Figure inspired from [30]</figcaption></figure>

# 5 Vision Transformers for Other Domains

In this section, we present applications of visual transformers to other domains. First, we describe multimodal transformers operating with vision and language (Subheading 5.1), then we describe video-level attention and video transformers (Subheadings 5.2 and 5.3), and finally we present multimodal video transformers operating with vision, language, and audio (Subheading 5.4).

## 5.1 Multimodal Transformers: Vision and Language

As transformers have found tremendous success in both natural language processing and computer vision, their use in vision-language tasks is also of interest. In this section, we describe some representative multimodal methods for vision and language: ViLBERT (Subheading 5.1.1), DALL-E (Subheading 5.1.3), and CLIP (Subheading 5.1.2).

### 5.1.1 VILBERT

Vision-and-language BERT (ViLBERT) [32] is an example of architecture that fuses two modalities. It consists of two parallel streams, each one working with one modality. The vision stream extracts
bounding boxes from images via an object detection network, by encoding their position. The language stream embeds word vectors and extracts feature vectors using the basic transformer encoder block [4] (Fig. 3 left). These two resulting feature vectors are then fused together by a cross-attention layer (Subheading 1.3.2). This follows the standard architecture of the transformer encoder block, where the keys and values of one modality are passed onto the MCA block of the other modality. The output of the cross-attention layer is passed into another transformer encoder block, and these two layers are stacked multiple times.

The language stream is initialized with BERT trained on Book Corpus [12] and Wikipedia [11], while the visual stream is initialized with Faster R-CNN [33]. On top of the pretraining of each stream, the whole architecture is pretrained on the Conceptual Captions dataset [34] on two pretext tasks.

ViLBERT has been proven powerful for a variety of multimodal tasks. In the original paper, ViLBERT was fined-tuned to a variety of tasks, including visual question answering, visual commonsense reasoning, referring expressions, and caption-based image retrieval.

### 5.1.2 CLIP

Connecting Text and Images (CLIP) [35] is designed to address two major issues of deep learning models: costly datasets and inflexibility. While most deep learning models are trained on labeled datasets, CLIP is trained on 400 million text-image pairs that are scraped from the Internet. This reduces the labor of having to manually label millions of images that are required to train powerful deep learning models. When models are trained on one specific dataset, they also tend to be difficult to extend to other applications. For instance, the accuracy of a model trained on ImageNet is generally limited to its own dataset and cannot be applied to real-world problems. To optimize training, CLIP models learn to perform a wide variety of tasks during pretraining, and this task allows for zero-shot transfer to many existing datasets. While there are still several potential improvements, this approach is competitive to supervised models that are trained on specific datasets.

#### CLIP Architecture and Training

CLIP is used to measure the similarity between the text input and the image generated from a latent vector. At the core of the approach is the idea of learning perception from supervision contained in natural language. Methods which work on natural language can learn passively from the supervision contained in the vast amount of text on the Internet.

Given a batch of *N* (image, text) pairs, CLIP is trained to predict which of the *N*×*N* possible (image, text) pairings across a batch actually occurred. To do this, CLIP learns a multimodal embedding space by jointly training an image encoder and a text encoder to maximize the cosine similarity of the image and text
embeddings of the *N* real pairs in the batch while minimizing the
cosine similarity of the embeddings of the *N*²−*N* incorrect pair-
ings. A symmetric cross-entropy loss over these similarity scores is
optimized.

Two different architectures were considered for the image
encoder. For the first, ResNet-50 [36] is used as the base architec-
ture for the image encoder due to its widespread adoption and
proven performance. Several modifications were made to the origi-
nal version of ResNet. For the second architecture, ViT is used with
some minor modifications: first, adding an extra layer normaliza-
tion to the combined patch and position embeddings before the
transformer and, second, using a slightly different initialization
scheme.

The text encoder is a standard transformer [4] (Subheading 2.1) with the architecture modifications described in [35]. As a base size, CLIP uses a 63M-parameter 12-layer 512-wide model with eight attention heads. The transformer operates on a lowercased byte pair encoding (BPE) representation of the text with a 49,152 vocab size [37]. The max sequence length is capped at 76. The text sequence is bracketed with [SOS] and [EOS] tokens,<fnref n="4" /> and the activations of the highest layer of the transformer at the [EOS] token are treated as the feature representation of the text which is layer normalized and then linearly projected into the multimodal embedding space.

5.1.3 DALL-E and
DALL-E 2

DALL-E [38] is another example of the application of transformers
in vision. It generates images from a natural language prompt—
some examples include “an armchair in the shape of an avocado”
and “a penguin made of watermelon.” It uses a decoder-only
model, which is similar to GPT-3 [39]. DALL-E uses 12 billion
parameters and is pretrained on Conceptual Captions [34] with
over 3.3 million text-image pairs. DALL-E 2 [40] is the upgraded
version of DALL-E, based on diffusion models and CLIP (Sub-
heading 5.1.2), and allows better performances with more realistic
and accurate generated images. In addition to producing more
realistic results with a better resolution than DALL-E, DALL-E
2 is also able to edit the outputs. Indeed, with DALL-E 2, one can
add or remove realistically an element in the output and can also
generate different variations of the same output. These two models
clearly demonstrate the powerful nature and scalability of transfor-
mers that are capable of efficiently processing a web-scale amount
of data.

<footnote n="4">⁴ [SOS], start of sequence; [EOS], end of sequence</footnote>
5.1.4 Flamingo
Flamingo [41] is a visual language model (VLM) tackling a wide range of multimodal tasks based on few-shot learning. This is an adaptation of large language models (LLMs) handling an extra visual modality with 80B parameters.

Flamingo consists of three main components: a vision encoder, a perceiver resampler, and a language model. First, to encode images or videos, a vision convolutional encoder [42] is pretrained in a contrastive way, using image and text pairs.<fnref n="5" /> Then, inspired by the perceiver architecture [25] (detailed in Subheading 1.3.2), the perceiver resampler takes a variable number of encoded visual features and outputs a fixed-length latent code. Finally, this visual latent code conditions the language model by querying language tokens through cross-attention blocks. Those cross-attention blocks are interleaved with pretrained and frozen language model blocks.

The whole model is trained using three different kinds of datasets without annotations (text with image content from web-pages [41], text and image pairs [41, 43], and text and video pairs [41]). Once the model is trained, it is fine-tuned using few-shot learning techniques to tackle specific tasks.

**5.2 Video Attention**

Video understanding is a long-standing problem, and despite incredible computer vision advances, obtaining the best video representation is still an active research area. Videos require employing effective spatiotemporal processing of RGB and time streams to capture long-range interactions [44, 45] while focusing on important video parts [46] with minimum computational resources [47].

Typically, video understanding benefits from 2D computer vision, by adapting 2D image processing methods to 3D spatiotemporal methods [48]. And through the Video Vision Transformer (ViViT) [49], history repeats itself. Indeed, with the rise of transformers [4] and the recent advances in image classification [5], video transformers appear as logical successors of CNNs.

However, in addition to the computationally expensive video processing, transformers also require a lot of computational resources. Thus, developing efficient spatiotemporal attention mechanisms is essential [25, 49, 50].

In this section, we first describe the general principle of video transformers (Subheading 5.2.1), and then, we detail three different attention mechanisms used for video representation (Subheadings 5.2.2, 5.2.3, and 5.2.4).

<footnote n="5">⁵ The text is encoded using a pretrained BERT model [10].</footnote>
### 5.2.1 General Principle

Generally, inputs of video transformers are RGB video clips $X \in \mathbb{R}^{F \times H \times W \times 3}$, with $F$ frames of size $H \times W$.

To begin with, video transformers split the input video clip $X$ into $ST$ tokens $x_i \in \mathbb{R}^K$, where $S$ and $T$ are, respectively, the number of tokens along the spatial and temporal dimension and $K$ is the size of a token.

To do so, the simplest method extracts nonoverlapping 2D patches of size $P \times P$ from each frame [5], as used in TimeSformer [50]. This results in $S=HW/P^2$, $T=F$, and $K=P^2$.

However, there exist more elegant and efficient token extraction methods for videos. For instance, in ViViT [49], the authors propose to extract 3D volumes from videos (involving $T \neq F$) to capture spatiotemporal information within tokens. In TokenLearner [47], they propose a learnable token extractor to select the most important parts of the video.

Once raw tokens $x_i$ are extracted, transformer architectures aim to map them into $d$-dimensional embedding vectors $Z \in \mathbb{R}^{ST \times d}$ using a linear embedding $E \in \mathbb{R}^{d \times K}$:

$$Z = [z_{cls}, Ex_1, Ex_2, \dots, Ex_{ST}] + PE, \quad (10)$$

where $z_{cls} \in \mathbb{R}^d$ is a classification token that encodes information from all tokens of a single sample [10] and $PE \in \mathbb{R}^{ST \times d}$ is a positional embedding that encodes the spatiotemporal position of tokens, since the subsequent attention blocks are permutation invariant [4].

In the end, embedding vectors $Z$ pass through a sequence of $L$ transformer layers. A transformer layer $l$ is composed of a series of multi-head self-attention (MSA) [4], layer normalization (LN) [51], and MLP blocks:

$$\begin{aligned} \boldsymbol{\gamma}^l &= \text{MSA}(\text{LN}(\boldsymbol{Z}^l)) + \boldsymbol{Z}^l, \\ \boldsymbol{Z}^{l+1} &= \text{MLP}(\text{LN}(\boldsymbol{\gamma}^l)) + \boldsymbol{\gamma}^l. \end{aligned} \quad (11)$$

In this way, as shown in Fig. 2, we denote four different components in a video transformer layer: the query-key-value (QKV) projection, the MSA block, the MSA projection, and the MLP. For a layer with $b$ heads, the complexity of each component is [4]:

* QKV projection: $O(h.(2Std_k + Std_v))$

* MSA: $O(hS^2T^2.(d_k + d_v))$

* MSA projection: $O(SThd_vd)$

* MLP: $O(STD^2)$

We note that the MSA complexity is the most impacting component, with a quadratic complexity with respect to the number of tokens. Hence, for comprehension and clarity purposes, in the rest of the section, we consider the global complexity of a video transformer with $L$ layers to equal to $O(LS^2T^2)$.
<figure><img src="image_117.png" /><figcaption>**Fig. 15** Full space-time attention mechanism. Embedding tokens at layer $l-1$, $Z^{(l-1)}$ are all fed simultaneously through a unique spatiotemporal attention block. Finally, the spatiotemporal embedding is passed through an MLP and normalized to output embedding tokens of the next layer, $Z^l$. Figure inspired from [50]</figcaption></figure>

### 5.2.2 Full Space-Time Attention

As described in [49, 50], *full space-time attention* mechanism is the most basic and direct spatiotemporal attention mechanism. As shown in Fig. 15, it consists in computing self-attention across all pairs of extracted tokens.

This method results in a heavy complexity of $O(LS^2T^2)$ [49, 50]. This quadratic complexity can fast be memory-consuming, in which it is especially true when considering videos. Therefore, using full space-time attention mechanism is impractical [50].

### 5.2.3 Divided Space-Time Attention

A smarter and more efficient way to compute spatiotemporal attention is the *divided space-time attention* mechanism, first described in [50].

As shown in Fig. 16, it relies on computing spatial and temporal attention separately in each transformer layer. Indeed, we first compute the spatial attention, i.e., self-attention within each temporal index, and then the temporal attention, i.e., self-attention across all temporal indices.
<figure><img src="image_118.png" /><figcaption>**Fig. 16** Divided space-time attention mechanism. Embedding tokens at layer $l-1$, $Z^{(l-1)}$ are first processed along the temporal dimension through a first MSA block, and the resulting tokens are processed along the spatial dimension. Finally, the spatiotemporal embedding is passed through an MLP and normalized to output embedding tokens of the next layer, $Z^l$. Figure inspired from [50]</figcaption></figure>

The complexity of this attention mechanism is $O(LST.(S+T))$ [50]. By separating the calculation of the self-attention over the different dimensions, one tames the quadratic complexity of the MSA module. This mechanism highly reduces the complexity of a model with respect to the full space-time complexity. Therefore, it is reasonable to use it to process videos [50].

An even more refined way to reduce the computational cost of attention calculation consists of using cross attention as a bottleneck. For instance, as shown in Fig. 17 and mentioned in Subheading 3.2, the perceiver [25] projects the extracted tokens $x_i$ into a
<figure><img src="image_119.png" /><figcaption>**Fig. 17** Attention bottleneck mechanism. Raw input patches and embedding tokens at layer $l-1$, $Z^{(l-1)}$ are fed to a cross-attention block (CA) and then normalized and projected. Finally, the resulting embedding is passed through a transformer to output embedding tokens of the next layer, $Z^l$. Figure inspired from [25]</figcaption></figure>

very low-dimensional embedding through a cross-attention block placed before the transformer layers.

Here, the cross-attention block placed before the $L$ transformer layers reduce the input dimension from $ST$ to $N$, where $N \ll ST$,<fnref n="6" /> thus resulting in a complexity of $O(STN)$. Hence, the total complexity of this attention block is $O(STN + LN^2)$. It reduces again the complexity of a model with respect to the *divided space-time attention* mechanism. We note that it enables to design deep architectures, as in the perceiver [25], and then it enables the extraction of higher-level features.

### 5.2.5 Factorized Encoder
Lastly, the *factorized encoder* [49] architecture is the most efficient with respect to the complexity/performance trade-off.

As in *divided space-time attention*, the *factorized encoder* aims to compute spatial and temporal attention separately. Nevertheless, as shown in Fig. 18, instead of mixing spatiotemporal tokens in each transformer layer, here, there exist two separate encoders:

<footnote n="6">⁶ In practice, $N \le 512$ for perceiver [25], against $ST = 16 \times 16 \times (32/2) = 4096$ for ViViT-L [49]</footnote>
<figure><img src="image_120.png" /><figcaption>**Fig. 18** Factorized encoder mechanism. First, a spatial transformer processes input tokens along the spatial dimension. Then, a temporal transformer processes the resulting spatial embedding along the temporal dimension. Figure inspired from [25]</figcaption></figure>

First, a representation of each temporal index is obtained, thanks to
a spatial encoder with $L_s$ layers. Second, these tokens are passed
through a temporal encoder with $L_t$ layers (i.e., $L = L_s + L_t$).

Hence, the complexity of such architecture has two main
components: the spatial encoder complexity of $O(L_s S^2)$ and the
temporal encoder complexity of $O(L_t T^2)$. It results in a global
complexity of $O(L_s S^2 + L_t T^2)$. Thus, it leads to very lightweight
models. However, as it first extracts per-frame features and then
aggregates them to a final representation, it corresponds to a late-
fusion mechanism, which can sometimes be a drawback as it does
not mix spatial and temporal information simultaneously [52].

**5.3 Video Transformers**

In this section, we present two modern transformer-based archi-
tectures for video classification. We start by introducing the Time-
Sformer architecture in Subheading 5.3.1 and then the ViViT
architecture in Subheading 5.3.2.

5.3.1 TimeSformer

TimeSformer [50] is one of the first architectures with space-time
attention that impacted the video classification field. It follows the
same structure and principle described in Subheading 5.2.1.

First, it takes as input an RGB video clip sampled at a rate of
1/32 and decomposed into 2D 16 × 16 patches.

As shown in Fig. 19, the TimeSformer architecture is based on
the ViT architecture (Subheading 2.3), with 12 12-headed MSA
layers. However, the added value compared to the ViT is that
TimeSformer uses the *divided space-time attention* mechanism (Sub-
heading 5.2.3). Such attention mechanism enables to capture high-
level spatiotemporal features while taming the complexity of the
model. Moreover, the authors introduce three variants of the archi-
tecture: (i) TimeSformer, the standard version of the model, that
operates on 8 frames of 224 × 224; (ii) TimeSformer-L, a configu-
ration with high spatial resolution, that operates on 16 frames of
448 × 448; and (iii) TimeSformer-HR, a long temporal range setup,
that operates on 96 frames of 224 × 224.
<figure><img src="image_121.png" /><figcaption>**Fig. 19** TimeSformer architecture. The TimeSformer first projects input to embedding tokens, which are summed to positional embedding tokens. The resulting tokens are then passed through *L* divided space-time attention blocks and then linearly projected to obtain output probabilities</figcaption></figure>

Finally, the terminal classification token embedding is passed through an MLP to output a probability for all video classes. During inference, the final prediction is obtained by averaging the output probabilities from three different spatial crops of the input video clip (top left, center, and bottom right).

TimeSformer achieves similar state-of-the-art performances as the 3D CNNs [53, 54] on various video classification datasets, such as Kinetics-400 and Kinetics-600 [55]. Note the TimeSformer is much faster to train (416 training hours against 3840 hours [50] for a SlowFast architecture [54]) and, also, more efficient (0.59 TFLOPs against 1.97 TFLOPs [50] for a SlowFast architecture [53]).

5.3.2 ViViT

ViViT [49] is the main extension of the ViT [5] architecture (Subheading 2.3) for video classification.

First, the authors use a 16 tubelet embedding instead of a 2D patch embedding, as mentioned in Subheading 5.2.1. This alternate embedding method aims to capture the spatiotemporal
<figure><img src="image_122.png" /><figcaption>**Fig. 20** ViViT architecture. The ViViT first projects input to embedding tokens, which are summed to positional embedding tokens. The resulting tokens are first passed through $L_s$ spatial attention blocks and then through $L_t$ temporal attention blocks. The resulting output is linearly projected to obtain output probabilities</figcaption></figure>

information from the tokenization step, unlike standard architectures that fuse spatiotemporal information from the first attention block.

As shown in Fig. 20, the ViViT architecture is based on *factorized encoder* architecture (Subheading 5.2.5) and consists of one spatial and one temporal encoder operating on input clips with 32 frames of 224 × 224. The spatial encoder uses one of the three ViT variants as backbone.<fnref n="7" /> For the temporal encoder, the number

<footnote n="7">⁷ ViT-B: 12 12-headed MSA layers; ViT-L: 24 16-headed MSA layers; and ViT-H: 32 16-headed MSA layers.</footnote>
of layers does not impact much the performance, so that, according
to the performance/complexity trade-off, the number MSA layers
is fixed at 4. The authors show that such architecture reaches high
performances while reducing drastically the complexity.

Finally, as in TimeSformer (Subheading 5.3.1), ViViT outputs
probabilities for all video classes through the last classification token
embedding and averages the obtained probabilities across three
crops of each input clip (top left, center, and bottom right).

ViViT outperforms both 3D CNNs [53, 54] and TimeSformer
[50] on the Kinetics-400 and Kinetics-600 datasets [55]. Note the
complexity of this architecture is highly reduced in comparison to
other state-of-the-art models. For instance, the number of FLOPs
for a ViViT-L/16×16×2 is 3.89×10¹² against 7.14×10¹² for a
TimeSformer-L [50] and 7.14×10¹² for a SlowFast [53]
architecture.

5.4 **Multimodal Video**
**Transformers**

Nowadays, one of the main gaps between artificial and human
intelligence is the ability for us to process multimodal signals and
to enrich the analysis by mixing the different modalities. Moreover,
until recently, deep learning models have been focusing mostly on
very specific visual tasks, typically based on a single modality, such as
image classification [5, 17, 18, 56, 57], audio classification [25, 52,
58, 59], and machine translation [10, 60–63]. These two factors
combined have pushed researchers to take up multimodal
challenges.

The *default* solution for multimodal tasks consists in first cre-
ating an individual model (or network) per modality and then in
fusing the resulting single-modal features together [64, 65]. Yet,
this approach fails to model interactions or correlations among
different modalities. However, the recent rise of attention [4, 5,
49] is promising for multimodal applications, since attention per-
forms very well at combining multiple inputs [25, 52, 66, 67].

Here, we present two main ways of dealing with several
modalities:

1. Concatenating tokens from different modalities into one vector [25, 66]. The multimodal video transformer (MM-ViT) [66] combines raw RGB frames, motion features, and audio spectrogram for video action recognition. To do so, the authors fuse tokens from all different modalities into a single-input embedding and pass it through transformer layers. However, a drawback of this method is that it fails to distinguish well one modality to another. To overcome this issue, the authors of the perceiver [25] propose to learn a modality embedding in addition to the positional embedding (see Subheadings 3.2 and 5.2.1). This allows associating each token
with its modality. Nevertheless, given that (i) the complexity of
a transformer layer is quadratic with respect to the number of
tokens (Subheading 5.2.1) and (ii), with this method, the
number of tokens is multiplied by the number of modalities,
it may lead to skyrocketing computational cost [66].

2. **Exploiting cross attention** [52, 67, 68]. Several modern approaches exploit cross attention to mix multiple modalities, such as [52] for audio and video, [67] for text and video, and [68] for audio, text, and video. The commonality among all these methods is that they exploit the intrinsic properties of cross attention by querying one modality with a key-value pair from the other one [52, 67]. This idea can be easily generalized to more than two modalities by computing cross attention across each combination of modalities [68].

6 Conclusion

Attention is an intuitive and efficient technique that enables
handling local and global cues.

On this basis, the first pure attention architecture, the trans-
former [4], has been designed for NLP purposes. Quickly, the
computer vision field has adapted the transformer architecture for
image classification, by designing the first visual transformer model:
the vision transformer (ViT) [5].

However, even if transformers naturally lead to high perfor-
mances, the raw attention mechanism is a computationally greedy
and heavy technique. For this reason, several enhanced and refined
derivatives of attention mechanisms have been proposed [21–26].

Then, rapidly, a wide variety of other tasks have been con-
quered by transformer-based architectures, such as object detection
[14], image segmentation [27], self-supervised learning [28, 29],
and image generation [30, 31]. In addition, transformer-based
architectures are particularly well suited to handle multidimen-
sional tasks. This is because multimodal signals are easily combined
through attention blocks, in particular vision and language cues
[32, 35, 38] and spatiotemporal signals are also easily tamed, as in
[25, 49, 50].

For these reasons, transformer-based architectures enabled
many fields to make tremendous progresses in the last few years.
In the future, transformers will need to become more and more
computationally efficient, e.g., to be usable on cellphones, and will
play a huge role to tackle multimodal challenges and bridge
together most AI fields.
References

1. Bahdanau D, Cho K, Bengio Y (2015) Neural machine translation by jointly learning to align and translate. In: International conference on learning representations

2. Cho K, van Merriënboer B, Gulcehre C, Bahdanau D, Bougares F, Schwenk H, Bengio Y (2014) Learning phrase representations using RNN encoder–decoder for statistical machine translation. In: Empirical methods in natural language processing, association for computational linguistics, pp 1724–1734

3. Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. In: Advances in neural information processing systems, vol 27

4. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: Advances in neural information processing systems, vol 30

5. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, Uszkoreit J, Houlsby N (2021) An image is worth 16×16 words: transformers for image recognition at scale. In: International conference on learning representations

6. Press O, Wolf L (2017) Using the output embedding to improve language models. In: Proceedings of the 15th conference of the European chapter of the association for computational linguistics: volume 2, short papers, association for computational linguistics, pp 157–163

7. Fedus W, Zoph B, Shazeer N (2021) Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:210103961

8. Raffel C, Shazeer N, Roberts A, Lee K, Narang S, Matena M, Zhou Y, Li W, Liu PJ (2020) Exploring the limits of transfer learning with a unified text-to-text transformer. J Mach Learn Res 21(140):1–67

9. Khan S, Naseer M, Hayat M, Zamir SW, Khan FS, Shah M (2021) Transformers in vision: a survey. ACM Comput Surv 24:200

10. Devlin J, Chang MW, Lee K, Toutanova K (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the north American chapter of the association for computational linguistics: human language technologies, volume

1 (long and short papers), association for computational linguistics, pp 4171–4186

11. Wikimedia Foundation (2019) Wikimedia downloads. https://dumps.wikimedia.org

12. Zhu Y, Kiros R, Zemel R, Salakhutdinov R, Urtasun R, Torralba A, Fidler S (2015) Aligning books and movies: towards story-like visual explanations by watching movies and reading books. In: Proceedings of the international conference on computer vision, pp 19–27

13. Wang X, Girshick R, Gupta A, He K (2018) Non-local neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7794–7803

14. Carion N, Massa F, Synnaeve G, Usunier N, Kirillov A, Zagoruyko S (2020) End-to-end object detection with transformers. In: Proceedings of the European conference on computer vision. Springer, Berlin, pp 213–229

15. Ramachandran P, Parmar N, Vaswani A, Bello I, Levskaya A, Shlens J (2019) Standalone self-attention in vision models. In: Advances in neural information processing systems, vol 32

16. Wang H, Zhu Y, Green B, Adam H, Yuille A, Chen LC (2020) Axial-deeplab: stand-alone axial-attention for panoptic segmentation. In: Proceedings of the European conference on computer vision. Springer, Berlin, pp 108–126

17. Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 248–255

18. Krizhevsky A, Hinton G (2009) Learning multiple layers of features from tiny images. Tech rep University of Toronto, Toronto, ON

19. Sun C, Shrivastava A, Singh S, Gupta A (2017) Revisiting unreasonable effectiveness of data in deep learning era. In: Proceedings of the international conference on computer vision, pp 843–852

20. Selva J, Johansen AS, Escalera S, Nasrollahi K, Moeslund TB, Clapés A (2022) Video transformers: a survey. arXiv preprint arXiv:220105991

21. Touvron H, Cord M, Douze M, Massa F, Sablayrolles A, Jégou H (2021) Training data-efficient image transformers & distillation through attention. In: International conference on machine learning. PMLR, pp 10347–10357
22. d'Ascoli S, Touvron H, Leavitt ML, Morcos AS, Biroli G, Sagun L (2021) Convit: improving vision transformers with soft convolutional inductive biases. In: International conference on machine learning. PMLR, pp 2286-2296

23. Hassani A, Walton S, Shah N, Abuduweili A, Li J, Shi H (2021) Escaping the big data paradigm with compact transformers. arXiv preprint arXiv:210405704

24. Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, Lin S, Guo B (2021) Swin transformer: hierarchical vision transformer using shifted windows. In: Proceedings of the international conference on computer vision

25. Jaegle A, Gimeno F, Brock A, Vinyals O, Zisserman A, Carreira J (2021) Perceiver: general perception with iterative attention. In: International conference on machine learning. PMLR, pp 4651-4664

26. Jaegle A, Borgeaud S, Alayrac JB, Doersch C, Ionescu C, Ding D, Koppula S, Zoran D, Brock A, Shelhamer E et al (2021) Perceiver IO: a general architecture for structured inputs & outputs. arXiv preprint arXiv:210714795

27. Strudel R, Garcia R, Laptev I, Schmid C (2021) Segmenter: transformer for semantic segmentation. In: Proceedings of the international conference on computer vision, pp 7262-7272

28. Caron M, Touvron H, Misra I, Jégou H, Mairal J, Bojanowski P, Joulin A (2021) Emerging properties in self-supervised vision transformers. In: Proceedings of the international conference on computer vision

29. He K, Chen X, Xie S, Li Y, Dollár P, Girshick R (2021) Masked autoencoders are scalable vision learners. arXiv preprint arXiv:211106377

30. Hudson DA, Zitnick L (2021) Generative adversarial transformers. In: International conference on machine learning. PMLR, pp 4487-4499

31. Zhang B, Gu S, Zhang B, Bao J, Chen D, Wen F, Wang Y, Guo B (2022) Styleswin: transformer-based GAN for high-resolution image generation. In: Proceedings of the IEEE conference on computer vision and pattern recognition

32. Lu J, Batra D, Parikh D, Lee S (2019) Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In: Advances in neural information processing systems, vol 32

33. Ren S, He K, Girshick R, Sun J (2015) Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in neural information processing systems, vol 28

34. Sharma P, Ding N, Goodman S, Soricut R (2018) Conceptual captions: a cleaned, hypernymed, image alt-text dataset for automatic image captioning. In: Proceedings of ACL

35. Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, Sastry G, Askell A, Mishkin P, Clark J et al (2021) Learning transferable visual models from natural language supervision. In: International conference on machine learning. PMLR, pp 8748-8763

36. He X, Peng Y (2017) Fine-grained image classification via combining vision and language. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5994-6002

37. Sennrich R, Haddow B, Birch A (2016) Neural machine translation of rare words with sub-word units. In: Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), association for computational linguistics, pp 1715-1725

38. Ramesh A, Pavlov M, Goh G, Gray S, Voss C, Radford A, Chen M, Sutskever I (2021) Zero-shot text-to-image generation. In: International conference on machine learning. PMLR, pp 8821-8831

39. Brown T, Mann B, Ryder N, Subbiah M, Kaplan JD, Dhariwal P, Neelakantan A, Shyam P, Sastry G, Askell A, et al (2020) Language models are few-shot learners. In: Advances in neural information processing systems, vol 33, pp 1877-1901

40. Ramesh A, Dhariwal P, Nichol A, Chu C, Chen M (2022) Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:220406125

41. Alayrac JB, Donahue J, Luc P, Miech A, Barr I, Hasson Y, Lenc K, Mensch A, Millican K, Reynolds M et al (2022) Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:220414198

42. Brock A, De S, Smith SL, Simonyan K (2021) High-performance large-scale image recognition without normalization. In: International conference on machine learning. PMLR, pp 1059-1071

43. Jia C, Yang Y, Xia Y, Chen YT, Parekh Z, Pham H, Le Q, Sung YH, Li Z, Duerig T (2021) Scaling up visual and vision-language
representation learning with noisy text supervision. In: International conference on machine learning. PMLR, pp 4904–4916

44. Epstein D, Vondrick C (2021) Learning goals from failure. In: Proceedings of the IEEE conference on computer vision and pattern recognition

45. Marin-Jimenez MJ, Kalogeiton V, Medina-Suarez P, Zisserman A (2019) Laeo-net: revisiting people looking at each other in videos. In: Proceedings of the IEEE conference on computer vision and pattern recognition

46. Nagrani A, Yang S, Arnab A, Jansen A, Schmid C, Sun C (2021) Attention bottlenecks for multimodal fusion. In: Advances in neural information processing systems

47. Ryoo M, Piergiovanni A, Arnab A, Dehghani M, Angelova A (2021) Tokenlearner: Adaptive space-time tokenization for videos. In: Advances in neural information processing systems, vol 34

48. Hara K, Kataoka H, Satoh Y (2018) Can spatiotemporal 3d CNNs retrace the history of 2d CNNs and imagenet? In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6546–6555

49. Arnab A, Dehghani M, Heigold G, Sun C, Lučić M, Schmid C (2021) Vivit: a video vision transformer. In: Proceedings of the international conference on computer vision, pp 6836–6846

50. Bertasius G, Wang H, Torresani L (2021) Is space-time attention all you need for video understanding? In: International conference on machine learning

51. Ba JL, Kiros JR, Hinton GE (2016) Layer normalization. arXiv preprint arXiv:160706450

52. Nagrani A, Yang S, Arnab A, Jansen A, Schmid C, Sun C (2021) Attention bottlenecks for multimodal fusion. In: Advances in neural information processing systems, vol 34

53. Feichtenhofer C, Fan H, Malik J, He K (2019) Slowfast networks for video recognition. In: Proceedings of the international conference on computer vision, pp 6202–6211

54. Carreira J, Zisserman A (2017) Quo vadis, action recognition? A new model and the kinetics dataset. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6299–6308

55. Kay W, Carreira J, Simonyan K, Zhang B, Hillier C, Vijayanarasimhan S, Viola F, Green T, Back T, Natsev P et al (2017) The

kinetics human action video dataset. arXiv preprint arXiv:170506950

56. Wu H, Xiao B, Codella N, Liu M, Dai X, Yuan L, Zhang L (2021) CvT: introducing convolutions to vision transformers. In: Proceedings of the international conference on computer vision, pp 22–31

57. Touvron H, Cord M, Sablayrolles A, Synnaeve G, Jégou H (2021) Going deeper with image transformers. In: Proceedings of the international conference on computer vision, pp 32–42

58. Gemmeke JF, Ellis DP, Freedman D, Jansen A, Lawrence W, Moore RC, Plakal M, Ritter M (2017) Audio set: an ontology and human-labeled dataset for audio events. In: IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, pp 776–780

59. Gong Y, Chung YA, Glass J (2021) AST: audio spectrogram transformer. In: Proceedings of interspeech 2021, pp 571–575

60. Bojar O, Buck C, Federmann C, Haddow B, Koehn P, Leveling J, Monz C, Pecina P, Post M, Saint-Amand H, et al (2014) Findings of the 2014 workshop on statistical machine translation. In: Proceedings of the 9th workshop on statistical machine translation, pp 12–58

61. Liu X, Duh K, Liu L, Gao J (2020) Very deep transformers for neural machine translation. arXiv preprint arXiv:200807772

62. Edunov S, Ott M, Auli M, Grangier D (2018) Understanding back-translation at scale. In: Empirical methods in natural language processing, association for computational linguistics, pp 489–500

63. Lin Z, Pan X, Wang M, Qiu X, Feng J, Zhou H, Li L (2020) Pre-training multilingual neural machine translation by leveraging alignment information. In: Empirical methods in natural language processing, association for computational linguistics, pp 2649–2663

64. Owens A, Efros AA (2018) Audio-visual scene analysis with self-supervised multisensory features. In: Proceedings of the European conference on computer vision, pp 631–648

65. Ramanishka V, Das A, Park DH, Venugopalan S, Hendricks LA, Rohrbach M, Saenko K (2016) Multimodal video description. In: Proceedings of the ACM international conference on multimedia, pp 1092–1096
66. Chen J, Ho CM (2022) Mm-vit: Multi-modal video transformer for compressed video action recognition. In: Proceedings of the IEEE/ CVF winter conference on applications of computer vision, pp 1910–1921

67. Narasimhan M, Rohrbach A, Darrell T (2021) Clip-it! language-guided video summarization. In: Advances in neural information processing systems, vol 34

68. Liu ZS, Courant R, Kalogiton V (2022) Fun- nyNet: Audiovisual Learning of Funny Moments in Videos. In: Proceedings of the Asian conference on computer vision. Springer, Macau, China, pp 3308–3325

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution, and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third-party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
# Part II

## Data
**Clinical Assessment of Brain Disorders**

Stéphane Epelbaum and Federica Cacciamani

**Abstract**

The clinical evaluation of brain diseases strictly depends on patient's complaint and observation of their behavior. The specialist, often the neurologist, chooses whether and how to assess cognition, motor system, sensory perception, and autonomic nervous system. They may also decide to request a more in-depth examination, such as neuropsychological and language assessments and imaging or laboratory tests. From the synthesis of all these results, they will be able to make a diagnosis. The neuropsychological assessment in particular is based on the collection of medical history, on the clinical observation, and on the administration of standardized cognitive tests validated in the scientific literature. It is therefore particularly useful when a neurological disease with cognitive and/or behavioral manifestation is suspected. The introduction of machine learning methods in neurology represents an important added value to the evaluation performed by the clinician to increase the diagnostic accuracy, track disease progression, and assess treatment efficacy.

**Key words** Clinical assessment, Neurological examination, Neuropsychology, Cognitive scores

# 1 Introduction

## 1.1 What Is a Disease? Why Are Clinical Assessments Important?

A *disease* is a specific set of processes, often biological or histological, that induce *symptoms* (subjectively felt), which negatively affect the individual's normal functioning (e.g., discomfort, pain, suffering), are often associated with a complaint, and will manifest by *signs* (objectively measured), for instance, decreased motor strength or slowed speech. Symptoms and signs taken together define a *syndrome* (e.g., headache, vomiting, stiff neck point to a meningeal syndrome), and the syndromes are contextually interpreted by physicians to hypothesize on a given disease. If, for instance, the meningeal syndrome appears brutally and is very intense, the suspected disease will be meningeal hemorrhage. If it appears subacutely over a few hours and is accompanied by a fever, the physician will rather surmise a meningitis. Box 1 introduces the main medical definitions.
A clinical evaluation is therefore requested by the patient himself/herself or by a clinician (general practitioner, specialist, psychologist, etc.). The aim is to better characterize the symptoms and the underlying disease.

Box 1 Main Medical Definitions

<table><tr><td>Disease</td><td>Physiological (biological and/or pathological) process (es) causing pejorative clinical manifestations</td></tr><tr><td>Symptom</td><td>Subjective manifestation of a disease (pain, memory complaint, nausea, etc.)</td></tr><tr><td>Sign</td><td>Objective manifestation of a disease upon medical examination (decreased reflex, elevated blood pressure, etc.)</td></tr><tr><td>Syndrome</td><td>Association of symptoms and signs that can be related to a set of diseases (e.g., headache, nausea, and neck stiffness are a meningeal syndrome that can correspond to either meningitis or meningeal hemorrhage)</td></tr><tr><td>Clinical assessment</td><td>Stereotyped interrogation, observation, and examination of an individual by a trained healthcare provider in order to collect his/her symptoms and signs to determine a syndrome and hypothesize a main disease diagnosis and differential diagnoses</td></tr></table>

During their studies, physicians learn over a few years a large quantity of diagnostic and prognostic “decision trees” based on the co-occurrence of every set of symptoms and signs. The learning is structured so that frequent and severe diseases are more studied, while rare or orphan diseases and those considered less severe are covered more briefly. For instance, the few symptoms described above will most likely be recognized and diagnosed well by any physician as well as the degree of urgency they imply. This learning is based on aggregated knowledge at one point in time which is always susceptible to change. A clear example of such changes is Alzheimer’s disease (AD) which was considered a rare form of dementia of the young from its seminal description in 1906 [1] until the 1980s when it was finally identified by numerous pathological studies to be the predominant cause of dementia in the elderly [2]. Importantly, clinical assessment requires tools to be performed, such as the famous reflex hammer used by neurologists or cognitive tests used by the neuropsychologist. Machine learning, and the decision support system that it entails, may be considered as such a tool, although it has the peculiarity of being harder to comprehend for most clinicians which may be a specific challenge for its implementation.
Every clinical assessment, whether conducted in the routine practice of medicine or in biomedical research, has to adhere to strict ethical rules that warrant the trust the patient puts in their healthcare providers. The main rules are that of beneficence; non-maleficence; respect for any individual notwithstanding their race, gender, religion, or personal beliefs; and medical confidentiality.

Finally, the current development of digital and information
technologies is rapidly changing the scope of clinical assessments.
Prior to consultation, auto-assessment and patient empowerment
are promoted through the development of specific applications to
explicitly diagnose or monitor a disease [3, 4] and patient education
and access to relevant information [5]. The main issue concerning
this last point is the exponential growth of these digital solutions
and the risk of misinformation that can sometime lead the patient
toward unethical care [6].

## 1.2 Peculiarities of Clinical Assessment of Brain Disorders

The brain has functionally distinct regions, so there is a topographical correspondence between the location of the lesion in the brain and the symptom. The characterization of symptoms therefore allows to trace which brain region is affected. This helps in identifying the underlying disease. The motor and sensory cortices are perfect examples of this functional topography often depicted as homunculi [7].

Clinical evaluations for brain disorders thus follow a standar-
dized procedure. In addition to the symptoms and signs appraisal,
the physician often makes an assumption as to where the nervous
system is affected. This often overlaps with the syndromic defini-
tion: “frontotemporal dementia” implies that the lesions are in the
frontotemporal cortices. However, this is not always the case as
some diseases and syndromes still bear the name of the physician
who was the first to describe it. While most neurologists know that
a parkinsonism (or Parkinson syndrome) is due to basal ganglia
lesions, it is not implied in its name.

# 2 The Neurological Examination

## 2.1 General
Information on the
Neurological
Examination

The neurological examination begins with the collection of anam-
nestic data, that is, the complete history recalled and recounted by a
patient or their entourage, including complaint, medical history,
lifestyle, concurrent treatments, etc. During the collection of anam-
nestic data, the clinician also carefully observes patient's behavior.
The neurologist then proceeds with the examination of brain func-
tion, which is oriented by the complaint, and often includes cogni-
tive screening tests and examination of motor system, sensitivity,
and autonomic nervous system. Usually, this examination has more
formal and structured parts (this can be, for example, a systematic
evaluation of reflexes always in the same order or the use of a
specific scale to assess sensory or cognitive function) and other
more informal ones. In fact, the clinician chooses case by case on the basis of what is required and what is available to the physician at the time of the said assessment. For example, they may use a lay journal in their office to ask a patient to describe a complex photograph in order to get a general idea of their visuospatial perception skills. This is quite time-consuming, and, depending on the patient's case, presence of entourage, and thoroughness of the clinician, an initial visit can take from 0.5 h to 2 h to capture the essential features necessary to formulate a diagnosis, prognosis, and care plan. If the neurologists deem it necessary, they may request additional tests or examinations, such as a neuropsychological evaluation, language assessment, laboratory tests, imaging tests, etc.

For applying machine learning techniques, the results of formal exams are usually more adequate because they offer quantitative measures. However, this may change over the coming years as solutions are being developed to analyze informal material. This may include clinical reports or videos of patient examinations. Another example is natural language processing tools that may help in identifying semantic deficits in patients suffering from incipient dementia [8]. The context of data acquisition is very important and can greatly impact its quality. Among the different contexts, we can cite “routine clinical practice,” “retrospective or prospective observational studies,” and “clinical trials” that have increasing levels of quality due to the level of standardization of data acquisition and monitoring.

**2.2 Clinical Interview** A clinical interview precedes any objective assessment. It is adapted to the patient's complaint and as standardized as possible so as not to forget any question. It consists of:

- Personal and family history with, if necessary, a family tree.

- Lifestyle (including alcohol intake and smoking).

- Past or current treatments.

- As accurate as possible description of the illness made by the patient and/or their informant. It is important to know the intensity of the symptoms, their frequency, the chronological order of their appearance, the explorations already carried out, and the treatments undertaken as well as their effectiveness.

In a formal evaluation, especially in cohort studies and clinical trials, symptoms can be assessed thanks to different scales, some of which will be presented in this chapter, depending on the clinical variable of interest. These scales' results will also be used to monitor the disease evolution, notably in order to test new treatments.

The interview process is probably the most important part of the whole clinical assessment. It will allow delineating the patient's medical issue, which in turn will determine the next steps of the examination and management plan. It also creates a relation of trust that is essential for the future adhesion of the patient to the physician's propositions.
## 2.3 Evaluation of Cognition and Behavior

The assessment of cognition and behavior can be carried out by the neurologist using more or less in-depth tests depending on the situation, or a complete neuropsychological assessment can be requested and carried out separately by a neuropsychologist (see Subheading 3 of this chapter). The assessment of cognition is guided by the cognitive complaint of the patient and/or the infomant [9]. However, on the one hand, it is possible that the patient is not fully aware of their deficits. This is a symptom called *anosognosia* (which literally means *lack of knowledge of the disease*) and is typical of various forms of dementia, including AD and frontotemporal dementia, but also brain damage due, for example, to stroke in certain regions of the brain. On the other hand, a cognitive complaint can be due to anxiety, depression, and personality traits and may have no neurological basis. The medical doctor can use simple tests in their daily practice such as the Mini-Mental State Examination (MMSE) [10]. For a more detailed description of the MMSE, please refer to Subheading 3 of this chapter.

## 2.4 Evaluation of Motor System

The examination of motor function starts as soon as the physician greets their patient in the waiting room. They will immediately observe the patient's walk and their bodily movements. Then, in their office, the observation will continue to search, for example, for a muscular atrophy or fascicules (i.e., muscular shudder detected by looking at the skin of the patient). This purely observational phase is followed by a formal examination, provoking objective signs.

One goal of motor assessment is to assess muscle strength. This is done segmentally, that is, carried out by evaluating the function of muscle groups that perform the same action, for example, the muscles that allow the elbow to flex. The neurologist gives a score ranging from 0 to 5, where 0 indicates that they did not detect any movement and 5 indicates normal movement strength.

A second aspect which is assessed is muscle tone. It is explored by passively mobilizing the patient joints. Hypertonia, or rigidity, is an increase in the tone. When the neurologist moves the joint, it may remain rigidly in that position (*plastic* or *parkinsonian* hypertonia), or the limb may immediately return to the resting position as soon as the neurologist stops manipulating it (*spastic* or *elastic* hypertonia). Hypotonia is a reduction of muscle tone, i.e., lack of tension or resistance to passive movement. This is observed in cerebellar lesions and chorea.

Another goal of motor assessment is evaluating deep tendon reflexes. Using a reflex hammer, the neurologist taps the tendons (e.g., Achilles' tendon for the Achillean reflex). The deep tendon reflexes will be categorized as (1) normal, (2) increased and polykinetic (i.e., a single tap provokes more than one movement), (3) diminished or abolished (as in peripheral nervous system diseases), and (4) pendular (as in cerebellar syndrome). Often evidenced in case of increased reflexes, Babinski's sign is the lazy and
majestic extension of the big toe followed by the other toes in response to the scraping of the outer part of the foot plant. It is pathognomonic (i.e., totally specific) of a pyramidal syndrome, which is named after the axonal fiber tract that is altered: the pyramidal fasciculus. Motor assessment also includes evaluation of tremors and posture.

Once again, specific scales exist to robustly and homogeneously assess some of these signs such as the Unified Parkinson's disease rating scale (UPDRS) in Parkinson's disease [11]. For more information, the Movement Disorder Society UPDRS Revision Task Force has made the questionnaire available [12]. We report MDS-UPDRS items in Box 2. There are 65 items, 60 of which with a score from 0 to 4 (0, normal; 1, slight; 2, mild; 3, moderate; and 4, severe) and 5 with yes/no responses.

Box 2 MDS-UPDRS Structures

Part I: Non-motor experiences of daily living

13 items. Less than 10 min

1. Cognitive impairment

2. Hallucinations and psychosis

3. Depressed mood

4. Anxious mood

5. Apathy

6. Features of dopamine dysregulation syndrome

7. Nighttime sleep problems

8. Daytime sleepiness

9. Pain and other sensations

10. Urinary problems

11. Constipation problems

12. Lightheadedness on standing

13. Fatigue

Part II: Motor experiences of daily living

13 items. It does not involve examiner time; items are answered by the patient or caregiver independently.

1. Speech

2. Salivation and drooling

3. Chewing and swallowing

4. Eating tasks

5. Dressing

6. Hygiene

7. Handwriting

8. Doing hobbies and other activities

9. Turning in bed

10. Tremor

11. Getting out of bed, car, or deep chair

12. Walking and balance

13. Freezing

Part III: Motor examination

33 items (18 items with different duplicates corresponding to the right or left side or to different body parts).

15 min

1. Speech

2. Facial expression

3. Rigidity of neck and four extremities

4. Finger taps

5. Hand movements

Part IV: Motor complications

Six items. 5 min

1. Time spent with dyskinesia

2. Functional impact of dyskinesias

3. Time spent in the OFF state

4. Functional impact of fluctuations

5. Complexity of motor fluctuations

6. Painful OFF-state dystonia

(continued)
**Box 2 (continued)**

6. Pronation/supination

7. Toe tapping

8. Leg agility

9. Arising from chair

10. Gait

11. Freezing of gait

12. Postural stability

13. Posture

14. Global spontaneity of movement

15. Postural tremor of hands

16. Kinetic tremor of hands

17. Rest tremor amplitude

18. Constancy of rest tremor

**2.5 Evaluation of Sensitivity**

Sensitivity is the ability to feel different tactile sensations: normal (or crude) tact, pain, hot, or cold. Once again, it depends on the anatomical regions and tracts affected by a pathological process. The anterior spinothalamic tract carries information about crude touch. The lateral spinothalamic tract conveys pain and temperature. Assessment includes measuring:

- Epicritic sensitivity: test the patient's ability to discriminate two very close stimuli.

- Deep sensitivity: test the direction of position of the joints by the blind prehension. The doctor can also ask the patient if the vibrations of a diapason on joint bones (knee, elbow) are felt.

- Discrimination of hot and cold; sensitivity to pain.

**2.6 Other Evaluations**

The physician evaluation will also assess the autonomic nervous system which, when impaired, can induce tensile disorders: hypo-/hypertension, orthostatic hypotension (without compensatory acceleration of pulse), diarrhea, sweating disorders, accommodation disorders, and sexual disorders. They will also evaluate cerebellar functions: balance, coordination (which when impaired causes ataxia), and tremor.

Finally, clinicians will assess cranial nerves' functions. Cranial nerves are those coming out of the brainstem and have various functions including olfaction, vision, eye movements, face sensorimotricity, and swallowing. They are tested once again in a standardized way from the first one to the twelfth.

**2.7 Summary of the Neurological Evaluation**

At the end of this examination, the signs and symptoms are described in the report, and the physician specifies:

- A syndromic group of signs and symptoms

- The presumed location of brain damage
- A main diagnostic hypothesis

- Possibly, secondary hypotheses (differential diagnosis)

- Additional examination strategy through neuroimaging or additional examinations to refine disease diagnosis

- A therapeutic program

# 3 Neuropsychological Assessment

## 3.1 Generalities on Neuropsychological Assessment

Neuropsychology is concerned with how cognitive functions (see Box 3) and behavior are correlated with anatomo-physiological brain mechanisms. Thanks to the scientific-technological advances made in recent decades and the advent of increasingly sensitive structural and functional imaging techniques, we have discovered that human cognition has a modular architecture in which each module—whose operationalization depends on the reference framework—corresponds to a specific function [13]. This allowed us to understand which brain regions or structures we expect to be damaged when we observe a certain cognitive deficit [14–17]. The role of the neuropsychologist can be summarized in two core activities: assessment and intervention. In this chapter, we will focus on neuropsychological assessment, which produces data that is typically used by machine learning algorithms.

Neuropsychological assessment includes a clinical interview, followed by the measurement of cognitive functions using standardized tests and finally the interpretation of the results. This is applicable in diagnostic settings, to monitor disease progression if the diagnosis has previously been made or to measure the effectiveness of a treatment.

### Box 3 Main Cognitive Functions

#### Memory

Short-term memory or working memory temporarily retains few pieces of information for the time needed to perform a certain task, using mechanisms such as mental repetition

Episodic memory allows long-term conscious memory of a potentially infinite number of events (episodes) and contexts (time and place) in which they occurred

Semantic memory allows the long-term conscious memory of a potentially infinite number of facts, concepts, and vocabulary, which constitute the knowledge that the individual has of the world

Procedural memory is the memory of how things are done (e.g., tying shoelaces) and how objects are used

(continued)
<table><thead><tr><th colspan="2">Box 3 (continued)</th></tr></thead><tbody><tr><td>Attention</td><td>Selective attention is the ability to select relevant information from the environment<br>Sustained attention is the ability to persist for a relatively long time on a certain task</td></tr><tr><td>Visuospatial abilities</td><td>Estimation of spatial relationships between the individual and the objects and between the objects themselves and identification of visual characteristics of a stimulus such as its orientation</td></tr><tr><td>Language</td><td>Oral and written production and comprehension, at a phonological, morphological, syntactic, semantic, and pragmatic level</td></tr><tr><td>Executive functions</td><td>Superior cognitive functions such as planning, organization, performance monitoring, decision-making, mental flexibility, etc.</td></tr><tr><td>Social cognition</td><td>Using information previously learned more or less explicitly to explain and predict one's own behavior and that of others in social situations</td></tr></tbody></table>

Neuropsychology is therefore an interdisciplinary discipline. It is first and foremost a branch of psychology. The clinical interview that precedes the administration of tests is typical of psychological disciplines. The clinician collects anamnestic information (i.e., regarding medical history, lifestyle, and familiarity), observes patient behavior, and builds a relationship of trust and collaboration with him/her. All of these are crucial aspects in any type of psychological interview. In addition, the neuropsychologist must also be able to understand whether the cognitive complaint or the deficits detected are linked to brain damage or whether they are psychogenic. To do this, they assess, qualitatively or quantitatively depending on the situation, the mood of the patient and the presence of any anxiety syndromes, psychotic symptoms, etc.

Neuropsychology also has obvious points in common with neurology, since it is interested in the evaluation and intervention on the cognitive-behavioral manifestation of pathologies of the central nervous system. Over the past decades, much knowledge has been gained on the relationship between cognition and brain, and many tests have been developed. As a result, neuropsychological assessment has split off from neurological examination, assuming a separate role [18].
## 3.2 Psychometric Properties of Neuropsychological Tests

The use of cognitive tests is the specificity of the neuropsychological assessment.

Each new test is developed according to a rigid and rigorous methodology, trying to minimize all possible sources of error or bias, and based on scientific evidence. For example, a test that aims to assess learning skills might include a list of words for the participant to memorize and then recall. These words will not be randomly selected but carefully chosen based on characteristics such as frequency of use, length, phonology, etc. The procedures for administering neuropsychological tests are also standardized. The situation (i.e., materials, instructions, test conditions, etc.) is the same for all individuals and dictated by the administration manuals provided with each test.

All tests, before being published, are validated for their psychometric properties and normed. A normative sample is selected according to certain criteria which may change depending on the situation [19]. In most cases, these are large samples of healthy individuals from the general population, stratified by age, sex, and/or level of education. In other cases, more specific samples are preferred. The goal is to identify how the score is distributed in the normative sample. In this way, we can determine if the score obtained by a hypothetical patient is normal (i.e., around the average of the normative distribution) or pathological (i.e., far from the average). Establishing how far from the average an observation must be in order to be considered abnormal is a real matter of debate [20]. Many neuropsychological scores, as well as many biological or physical attributes, follow a normal distribution in the general population. The most used metrics to determine pathology thresholds are z scores and percentiles. For a given patient, the neuropsychologist usually computes the z score by subtracting the mean of the normative sample from the raw score obtained by the patient and then dividing the result by the standard deviation (SD) of the normative sample. The distribution of z scores will have a mean of 0 and a SD of 1. We can also easily find the percentile corresponding to the z score. Most often, a score below the fifth percentile (or z score = -1.65) or the second percentile (or z-score = -2) is considered pathological. As an example, intelligence, or intelligence quotient (IQ), is an attribute that follows a normal distribution. It is conventionally measured with the Wechsler Adult Intelligence Scale, also known as WAIS [21], or the Wechsler Intelligence Scale for Children, also known as WISC [22]. The distribution of IQs has a mean of 100 and a SD of 15 points. Around 68% of individuals in the general population achieve an IQ of 100 ± 15 points. Scores between 85 and 115 are therefore considered to be average IQs (therefore normal). Ninety-five percent of individuals are in a range within 30 points of 100, thus