Magician's Corner: 5. Generative Adversarial Networks.

G adversarial networks (GANs) are a fascinating new technology first described by Goodfellow et al in 2014 (1). Compared with most deep learning applications which are focused on classifying an image (2) or segmenting structures within an image (3), GANs are focused on the task of creating images. It is the technology behind DeepFakes (4), where images are generated that look completely real. For instance, the website http:// ThisPersonDoesNotExist.com repeatedly generates images of humanlike faces, but there is no human used to create the photograph, or at least not directly. GANs also can be used for “style transfer,” where a given image is transformed into a different style (5). For example, one may take any photograph and apply a style like van Gogh’s paintings to that photograph. Figure 1 shows an example of a photograph I took and then applied the van Gogh style to it, using the styleGan transfer routine. Finally, GANs also can create “superresolution” images (6) from lower-resolution images—essentially creating detail out of nothing. Although the task of a radiologist is mostly to interpret images of real life, it is becoming apparent that GANs will have a substantial impact on the way we practice radiology in the not-too-distant future. Each of the tasks described above could impact how we practice radiology. The first task, to create images from nothing, would seem dangerous, and indeed it would be if totally unconstrained. However, if we are able to constrain it to some reasonable degree, it may be possible to create much larger training sets than could be feasibly obtained from existing databases. It has been shown that GAN-based generation of datasets results in better training of both traditional machine learning and deep learning systems than traditional image augmentation of real data for both medical and nonmedical tasks (7). Many articles have been published on the use of GANs to “translate” images from one modality to another: for instance, to convert MR images to CT images used to create attenuation-correction maps for PET/MR scanners (8). GANs have been developed to convert CT images to PET images (9), create MR images from other contrast types of MRI, and reduce radiation dose or contrast material dose (10,11). This article introduces how GANs work, provides some simple one-dimensional (1D) examples that the user can experiment with and then moves on to produce two-dimensional (2D) MR images of the head. There are three critical components of a GAN (see Fig 2): (a) a collection of real-world examples that the GAN is tasked with simulating; (b) a generator that seeks to create artificial examples that look “realistic;” and (c) a discriminator that determines if an example presented to it is real (that is, from the collection of real images) or fake (from the generator). The approach is called an adversarial network because the generator and the discriminator are in a contest—each tries to beat the other. At the start, the discriminator has never seen an image, and so essentially it will be guessing randomly. Similarly, our generator has not seen an image before, so initially it will create something that looks nothing like an image. Both components receive feedback about whether they are right, so the discriminator will start to learn what real images look like, and the generator will also start to learn which examples it produces are more real.