In this work, we are enhancing photo realism enhancement. To understand what that means, let us have a look at some footage from the game Grand Theft Auto 5. The game looks great, but it is not quite photo realistic. Let us now turn on our method. Our method modifies the images from the game to look more realistic. It is a convolutional network which produces images frame by frame and can be run at interactive rates. This model was trained to translate GTA 5 to the cityscapes dataset. Cityscapes contains mostly images of German cities recorded with an automotive grade camera. Here are a few examples from the dataset. With a model of reality induced by cityscapes, our method paves the roads with smoother asphalt, increases the glossiness of car paint and greens the sun-dried hills of GTA's California. The results are geometrically and semantically consistent with the input images and temporally stable. Let us have a look at the state of the art in comparison to our work. Here is contrastive, unpaired image to image translation, or cut for short. Cut is less temporally stable and hallucinates objects, such as trees in the sky or Mercedes stars on the hood of the car. Here is another state of the art approach. Similar to cut, T-set is less temporally stable and adds many artifacts to the images. Let us now look at our method in detail to understand why it is more stable and produces less artifacts. Our method takes a rendered image from the game and passes it through an image enhancement network to produce an enhanced image. We further extract a set of rendering buffers called G-Buffers produced by the game engine. G-Buffers contain geometric information, such as surface normals or distance to the camera, material information such as albedo or glossiness. They may also contain information about the lighting in a scene. We pass the buffers through a G-Buffer and Coder network to produce feature tensors at multiple scales. The tensors encode the information about the rendered scene. To train our networks for enhancing realism, we employ a perceptual discriminator. For each image it takes in, it produces a realism score. To make it understand what the real world looks like, we also feed it real photos. To ensure that our networks retain the structure of rendered images, we further add them to the discriminator inputs and employ a perceptual loss. Let us now zoom in on the G-Buffer encoder. From the G-Buffers, we can derive a semantic class label map, which assigns an object ID to every pixel. We pass the G-Buffers through multiple convolutional network streams and fuse the streams again based on the object IDs. This allows the encoder to learn different ways of processing for each type of object. For example, trees will be treated differently from cars. The resulting feature tensors are further processed by residual blocks, which output tensors at multiple scales. Let us now look at the Image Enhancement Network to see what we can do with these multi-scale tensors. The Enhancement Network is based on HRNet, which processes images at multiple scales in parallel. Each scale of the feature tensors matches a scale in the network. The network consists of many residual blocks at each scale. Within the residual blocks, we replace the batch normalization layers with our own rendering-aware denormalization blocks, or RAD for short. The RAD layers modulate features based on the rendering information in the G-Buffer feature tensors. For more details, we refer to our paper. Let us now look at how the perceptual discriminator works. The discriminator contains a robust semantic segmentation network and a network for extracting perceptual features, in this case a VGG-16. We extract features from the VGG at multiple levels and pass them through a convolutional network. This differs from prior work that trains discriminators directly on images. Training on VGG features allows our network to quickly learn a model of realism at multiple perceptual levels. We further pass the unmodified rendered image, as well as real images, through the segmentation network to obtain a label map for each. The label map is used to specialize the discriminator on individual object classes. Since we use a pre-trained, robust segmentation network, the object classes are consistent for rendered and real images. For more details, we refer to our paper. There is one more thing. For training discriminator networks, the common choice is to take in as much of an image as possible. The reasoning is that the more context of a scene the network sees, the better. We started with this strategy as well and found it to be suboptimal because scene layouts between real and rendered images can be quite different. Here we see the probabilities for each pixel to contain a certain type of object. The top row shows cityscapes, the bottom GTA 5. Images from GTA contain much more sky at the top, where cityscapes images contain vegetation. Images from cityscapes have a Mercedes star at the bottom. If we train our network with full images from both datasets, it will start adding Mercedes stars and place trees in the sky. We observe this effect in many other approaches. One part of our solution is to decrease the field of view for the network and show it corresponding scenes from both datasets. When training our network, we sample rendered and real image patches that roughly contain the same things. This is done automatically and requires no access to semantic label maps. More details can be found in our paper. Let us now look at more results and comparisons. Here we see GTA again. And on the right, the results produced by our method. Here we see the results from classic color transfer. Color transfer matches the color distribution of GTA to cityscapes. But it does not modify images beyond the colors. Roads and trees keep their synthetic appearance. Unlike many other colors, we see the results from classic color transfer. Color transfer matches the color distribution of GTA to cityscapes. Unlike many other approaches, color transfer is temporarily stable. Here is WCT2, a state-of-the-art photo-style transfer approach. WCT2 requires a reference photo. It can become temporarily unstable if rendered images and the reference diverge. Here we see again results from cut. Cut commonly hallucinates objects like trees in the sky and the star on the hood. The sampling strategy of our approach alleviates such artifacts. Here we see T-SIT, an image-to-image translation approach. T-SIT also requires a favorable reference photo, which may cause artifacts and temporal instability if rendered images and the reference diverge. Finally, let us look at GTA again. And here is our method trained on the Mapillary Vista's data set. Vista's features diverse images from around the world recorded with different types of cameras. Our method captures the style of this data set with its vibrant colors while keeping the semantic and geometric content of the underlying game. Thanks for watching.