Cyclic generative adversarial network for unsupervised cross-domain image generation

A system is provided for unsupervised cross-domain image generation relative to a first and second image domain that each include real images. A first generator generates synthetic images similar to real images in the second domain while including a semantic content of real images in the first domain. A second generator generates synthetic images similar to real images in the first domain while including a semantic content of real images in the second domain. A first discriminator discriminates real images in the first domain against synthetic images generated by the second generator. A second discriminator discriminates real images in the second domain against synthetic images generated by the first generator. The discriminators and generators are deep neural networks and respectively form a generative network and a discriminative network in a cyclic GAN framework configured to increase an error rate of the discriminative network to improve synthetic image quality.

BACKGROUND

Technical Field

The present invention relates to image recognition, and more particularly to a cyclic generative adversarial network for unsupervised cross-domain image generation.

Description of the Related Art

Generating images in target domains while having labels only in the source domain allows learning image recognition classifiers in the target domain without need for target domain labels. In fields that can involve image generation, the test (target) and source (training) domains for the image generator can often vary in a multitude of ways. As such, the quality of the images generated by the image generator may be lacking and paired training data of corresponding images from the two domains may not be available. Accordingly, there is a need for domain adaptation in order to reduce such variations and provide enhanced classification accuracy.

SUMMARY

According to an aspect of the present invention, a system is provided for unsupervised cross-domain image generation relative to a first image domain and a second image domain that each include real images. The system includes a first image generator for generating synthetic images having a similar appearance to one or more of the real images in the second image domain while including a semantic content of one or more of the real images in the first image domain. The system further includes a second image generator for generating synthetic images having a similar appearance to at least one of the real images in the first image domain while including a semantic content of at least one of the real images in the second image domain. The system also includes a first discriminator for discriminating the real images in the first image domain against the synthetic images generated by the second image generator. The system additionally includes a second discriminator for discriminating the real images in the second image domain against the synthetic images generated by the first image generator. The discriminators and the generators are deep neural networks and respectively form a generative network and a discriminative network in a cyclic Generative Adversarial Network (GAN) framework that is configured to increase an error rate of the discriminative network to improve a quality of the synthetic images.

According to another aspect of the present invention, a computer-implemented method is provided for unsupervised cross-domain image generation relative to a first image domain and a second image domain that each include real images. The method includes generating, by a first image generator, synthetic images having a similar appearance to one or more of the real images in the second image domain while including a semantic content of one or more of the real images in the first image domain. The method further includes generating, by a second image generator, synthetic images having a similar appearance to at least one of the real images in the first image domain while including a semantic content of at least one of the real images in the second image domain. The method also includes discriminating, by a first discriminator, the real images in the first image domain against the synthetic images generated by the second image generator. The method additionally includes discriminating, by a second discriminator, the real images in the second image domain against the synthetic images generated by the first image generator. The generators are each neural network based and respectively form a generative network and a discriminative network in a cyclic Generative Adversarial Network (GAN) framework. The method further includes increasing an error rate of the discriminative network to improve a quality of the synthetic images.

According to yet another aspect of the present invention, a computer program product is provided for unsupervised cross-domain image generation relative to a first image domain and a second image domain that each include real images. The computer program product includes a non-transitory computer readable storage medium having program instructions embodied therewith. The program instructions are executable by a computer to cause the computer to perform a method. The method includes generating, by a first image generator of the computer, synthetic images having a similar appearance to one or more of the real images in the second image domain while including a semantic content of one or more of the real images in the first image domain. The method further includes generating, by a second image generator of the computer, synthetic images having a similar appearance to at least one of the real images in the first image domain while including a semantic content of at least one of the real images in the second image domain. The method also includes discriminating, by a first discriminator of the computer, the real images in the first image domain against the synthetic images generated by the second image generator. The method additionally includes discriminating, by a second discriminator of the computer, the real images in the second image domain against the synthetic images generated by the first image generator. The discriminators and the generators are each neural network based and respectively form a generative network and a discriminative network in a cyclic Generative Adversarial Network (GAN) framework. The method further includes increasing an error rate of the discriminative network to improve a quality of the synthetic images.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

The present invention is directed to a cyclic generative adversarial network for unsupervised cross-domain image generation.

In an embodiment, a cyclic generative adversarial network is proposed that takes images from a source domain to generate images in a different target domain and then back to the source domain, without having any corresponding pairs of images in the source and target domains. This is used for image recognition applications such as object detection and semantic segmentation, with labels available in the source domain, but no labels in the target domain, whereby the generated images become available as training data with labels preserved across the source and target domains while the image properties change.

In an embodiment, the present invention provides an image generation algorithm that can transfer an image in one domain to another domain. For example, the domain transfer can involve, but is not limited to, for example, generating realistic images from synthetic images, generating night images from day images, and so forth. In an embodiment, the generation process keeps the high level semantic concepts in the input image while transforming the image characteristics to make it indistinguishable from the images in the target domain.

In an embodiment, an unsupervised domain-to-domain translation model is provided that could be learned without any supervision in the training dataset. This enables us to learn a high quality image generation model for many valuable applications, such as real image generation from synthetic images, rainy scene generation from bright day images, and so forth, where it is not possible to have corresponding images in both domains (that is, supervised as in image-to-image translation).

In an embodiment, the present invention employs a cyclic Generative Adversarial Network (GAN) framework that enforces recovering the entire image content when applied to two domain transfers cyclically such as, for example, but not limited to, rainy image to bright image and back to a rainy image. Enforcing such cyclic consistency aids in domain transfer model learning which keeps the semantic contents consistent across the generation process while adapting image properties to the target domain.

Moreover, it is to be appreciated that framework200described below with respect toFIG. 2is a framework for implementing respective embodiments of the present invention. Part or all of processing system100may be implemented in one or more of the elements of framework200.

Further, it is to be appreciated that processing system100may perform at least part of the method described herein including, for example, at least part of method400ofFIGS. 4-5. Similarly, part or all of framework200may be used to perform at least part of method400ofFIGS. 4-5.

FIG. 2shows an exemplary cyclic Generative Adversarial Network (GAN) framework200of the present invention, in accordance with an embodiment of the present invention.

The cyclic GAN framework (hereinafter “framework” in short)200includes a first domain input (hereinafter “input A” in short)201and a second domain input (hereinafter “input B” in short)251corresponding to a first image domain (hereinafter “domain A” in short)291and a second image domain (hereinafter “domain B” in short)292, respectively. Domain A291and domain B292are respective image domains that include real images. Hence, input A201and input B251are implemented as respective real images. Thus, the two domains are not required to be supervised (include the same images).

The framework200further includes a neural network based discriminator (hereinafter “discriminator A” in short)210, a neural network based discriminator (hereinafter “discriminator B” in short)220, a neural network based image generator (hereinafter “generator A2B” in short)230, a neural network based image generator (hereinafter “generator B2A” in short)240, and a cyclic consistency loss (also referred to herein as “L2 loss”)250. The generator A2B230can be implemented as a generative model that is trained with domain A291. In an embodiment, the generators can be implemented by convolutional neural networks, while the discriminators can be implemented by de-convolutional neural networks. Of course, other types of neural networks can also be used in accordance with the teachings of the present invention, while maintaining the spirit of the present invention.

The generator A2B230generates images ABA277that are looks similar to the images in domain B292but including the semantic contents of the input image from domain A291. The generator B2A240generates images AB278based on the output of the generator A2B230. The discriminator A210and discriminator B220are trained to discriminate real image from domain A291(or domain B292) against the generated images for domain A291(or domain B292). That is, the discriminator A210discriminates a real image from the domain A291against a generated image for the domain A291, while the discriminator B220discriminates a real image from domain B292against a generated image for domain B292.

The Generative Adversarial Network (GAN) framework200, together with cyclic consistency loss L2250, learn the neural network based elements (210,220,230, and240). The GAN loss encourages the generated outputs to look similar to the images in the corresponding target domain, which is achieved by the gradient coming from the discriminators. On the other hand, the cyclic consistency loss (L2 in our case) helps keep the semantic contents of the image. Note that we also learn the image cyclic GAN for the BAB directions (that is, from B to A and back to B) simultaneously. The GAN loss encourages images synthesized by the generator to have similar statistics as images from the target domain, which is achieved by gradients from the discriminator. The L2 loss results from the cyclic approach used for the GAN framework and represents the cyclic consistency loss in the cross-domain image generation. It compares the original image with the image synthesized using the output of the first generator as input. While an L2 loss is used in this embodiment, equivalent formulation may be derived using alternative losses such as L1, SSIM, perceptual loss, or other objectives that impose consistency for certain image statistics such as edge distributions.

In an embodiment the GAN framework200is configured to learn both the generative model and the discriminator at the same using the combination of GAN and L2 loss functions. The resulting training dynamics are usually described as a game between a generator(s) (i.e., the generative model(s)) and a discriminator(s) (i.e., the loss function(s)).

The discriminators (210and220) and the generators (230and240) respectively form a generative network and a discriminative network in the Generative Adversarial Network (GAN) framework200, where the GAN framework is configured to increase an error rate of the discriminative network (i.e., “fool” the discriminator network by producing novel synthetic images that appear to have come from the true data distribution). That is, the goal of the generators (230and240) is to produce realistic samples that fool the discriminators (210and220), while the discriminators (210and220) are trained to distinguish between the true training data and samples from the generators (230and240).

In an embodiment, the framework200can be based on deep learning and is trainable, rather than relying on a handcrafted generation algorithm. Thus, it can be applied to many different domain transfer tasks as long as there exists suitable datasets. Also, since the present invention does not require any supervision, it is widely applicable to many different image generation tasks.

FIG. 3shows a portion of the cyclic GAN framework200ofFIG. 2during a testing phase300, in accordance with an embodiment of the present invention.

Once the generator A2B230is trained with domain A291, the generator A2B230can be deployed to produce images in domain B292with any images from domain A291.

FIGS. 4-5show an exemplary method400for unsupervised cross-domain image generation relative to a first image domain and a second image domain that each include real images, in accordance with an embodiment of the present principles.

The method400is performed by a cyclic Generative Adversarial Network (GAN) having a first image generator (e.g., generator A2B230), a second image generator (e.g., generator B2A240), a first discriminator (e.g., discriminator A210), and a second discriminator (e.g., discriminator B220). The discriminators and the generators are each neural network based and respectively form a generative network and a discriminative network in the cyclic Generative Adversarial Network (GAN) framework. The cyclic GAN framework is configured to increase an error rate of the discriminative network to improve a quality of the synthetic images. In an embodiment, blocks410-440can correspond to a training phase of the cyclic GAN framework, and blocks450and460can correspond to a test phase of the cyclic GAN framework.

At block410, generate, by the first image generator, synthetic images having a similar appearance to one or more of the real images in the second image domain while including a semantic content of one or more of the real images in the first image domain.

At block420, generate, by the second image generator, synthetic images having a similar appearance to at least one of the real images in the first image domain while including a semantic content of at least one of the real images in the second image domain.

At block430, discriminate, by the first discriminator, the real images in the first image domain against the synthetic images generated by the second image generator.

In an embodiment, block430can include block430.

At block430A, obtain gradients from a discrimination process applied to the real images in the first image domain versus the synthetic images generated by the second image generator.

At block440, discriminate, by the second discriminator, the real images in the second image domain against the synthetic images generated by the first image generator.

In an embodiment, block440can include block440.

At block440A, obtain gradients from a discrimination process applied to the real images in the second image domain versus the synthetic images generated by the first image generator.

At block450, generate, by the generative network (now trained per blocks410-440), one or more additional synthetic images using an input image from the first image domain. The one or more additional synthetic images are generated to appear similar to at least a subset of the images in the second image domain while including a semantic content of the input image from the first image domain. The additional synthetic images will be of a higher quality than the previously generated synthetic images due the learning process implemented by the training of the cyclic GAN framework. For example, the additional synthetic images can use the obtained gradients from blocks430A and440A in order to exploit the GAN loss to obtain similar appear, while a cyclic consistency loss (L2) is exploited to preserve semantic content from a source domain.

The additional synthetic images can be used for a myriad of applications, as readily appreciated by one ordinary skill in the art. For example, other applications to which the present invention can be applied include, but are not limited to, training other supervised learning elements in an object category detection or other type of detection/classification network (see, e.g., block450A), the generation of datasets for different weather conditions (see, e.g., block450B), a cyclic domain transfer (see, e.g., block450C), annotation extraction and corresponding response action performance (e.g., block450D), and so forth.

In an embodiment, block450can include blocks450A-C.

At block450A, train another supervised learning element in an object category detection or other type of detection/classification network using one or more of the additional synthetic images.

At block450B, generate the additional synthesized images for different weather and/or other environmental conditions.

At block450C, perform a cyclic domain transfer with respect to the first image domain and the second image domain using the additional synthesized images.

In an embodiment, block450C can include block450C1.

At block450C1, enforce, by the cyclic GAN framework, cyclic consistency across the cyclic domain transfer while adapting the image properties from one of the domains to another one of the domains.

At block450D, perform an annotation operation using the additional synthetic images, perform a matching operation between the resultant annotations and a predetermined set of action words, and initiate a response action when one or more matches occur.

A further description will now be given regarding various aspects of the present invention, in accordance with an embodiment of the present invention.

The present invention incorporates a principled deep generative model to learn high quality image generation models.

The present invention introduces a novel cyclic GAN framework that does not require supervised datasets, e.g., same image in the two domains.

At test time, new images can be efficiently generated using the already trained generative network.

In an embodiment, the generated images could be used for training other supervised learning modules such as semantic segmentation or object category detection networks. Particularly, using the image generation network trained for domain transfer from synthetic to real, a dataset with detailed annotations can be obtained almost for free. The annotations can be obtained and/or otherwise derived from the semantic content, as preserved by the L2 loss. In an embodiment, a processor (e.g., CPU104) can be used to receive the annotations and perform matching against the annotations such that matches between the annotations and a predetermined set of actions can be used to initiate a response action. For example, in the case of classifying an action as dangerous (e.g., due to the presence of a weapon (e.g., a firearm or knife), an action can be initiated by the processor such as locking a door to keep the person holding the weapon out of an area or contained within an area.

In an embodiment, the present invention can be applied to train an image generation network for different weather conditions. It is to be appreciated that having a large dataset for all possible weather condition can be prohibitively costly. However, using the present invention, datasets can be generated for different weather conditions without additional labors.

These and other applications to which the present invention can be applied are readily determined by one of ordinary skill in the art, given the teachings of the present invention provided herein, while maintaining the spirit of the present invention.

Some of the many advantages and/or contributions made by the present invention include, but are not limited to, the following.

The present invention does not require a supervised dataset to train the model, noting that a supervised dataset is often not available in many important application domains.

Moreover, the present invention can generate higher quality images that conventional approaches.

Additionally, the present invention can be used to generate image data for other supervised learning methods, such as object detection, semantic segmentation, and so forth. This can significantly reduce the cost for data acquisition.