DINO: A Conditional Energy-Based GAN for Domain Translation

1 INTRODUCTION . Domain translation methods exploit the information redundancy often found in data from different domains in order to find a mapping between them . Successful applications of domain translation include image style transfer ( Zhu et al. , 2017a ) and speech-enhancement ( Pascual et al. , 2017 ) . Furthermore , these systems are increasingly being used to translate across modalities in applications such as speech-driven animation ( Chung et al. , 2017 ) and caption-based image generation ( Reed et al. , 2016 ) . Some of the most popular methods for domain translation are based on conditional Generative Adversarial Networks ( cGANs ) ( Mirza & Osindero , 2014 ) . The conditional information in cGANs is used to drive the generation and to enforce the correspondence between condition and sample . Various alternatives have been proposed for how the condition should be included in the discriminator ( Miyato & Koyama , 2018 ; Reed et al. , 2016 ) but the majority of frameworks provide it as an input , hoping that the sample ’ s correlation with the condition will play a role in distinguishing between synthesized and genuine samples . The main drawback of this approach is that it does not encourage the use of the conditional information and therefore its contribution can be diminished or even ignored . This may lead to samples that are not semantically consistent with the condition . In this paper , we propose the Dual Inverse Network Optimisation ( DINO ) framework1 which is based on energy-based GANs ( Zhao et al. , 2017 ) and consists of two networks that perform translation in opposite directions as shown in Figure 1 . In this framework , one network ( Forward network ) translates data from the source domain to the target domain while the other ( Reverse Network ) performs the inverse translation . The Reverse network ’ s goal is to minimize the reconstruction error for genuine data and to maximize it for generated data . The Forward network aims to produce samples that can be accurately reconstructed back to the source domain by the Reverse Network . Therefore , during training the Forward network is trained as a generator and the Reverse as a discriminator . Since discrimination is based on the ability to recover source domain samples , the Forward network is driven to produce samples that are not only realistic but also preserve the shared semantics . We show that this approach is effective across a broad range of supervised translation problems , capturing the correspondence even when domains are from different modalities ( i.e. , video-audio ) . In detail , the contributions of this paper are : 1Source code : https : //github.com/DinoMan/DINO • A domain translation framework , based on a novel conditioning mechanism for energybased GANs , where the adversarial loss is based on the prediction of the condition . • An adaptive method for balancing the Forward and Reverse networks , which makes training more robust and improves performance . • A method for simultaneously training two networks to perform translation in inverse direc- tions , which requires fewer parameters than other domain translation methods . • The first end-to-end trainable model for video-driven speech reconstruction capable of pro- ducing intelligible speech without requiring task-specific losses to enforce correct content . 2 RELATED WORK . Domain translation covers a wide range of problems including image-to-image translation ( Isola et al. , 2017 ) , caption-based image synthesis ( Qiao et al. , 2019 ) , and text-to-speech synthesis ( Arik et al. , 2017 ) . Unsupervised translation methods attempt to find a relationship between domains using unpaired training data . However , finding correspondence without supervision is an ill-posed problem which is why these methods often impose additional constraints on their networks or objectives . The majority of unsupervised methods are applied to image-to-image translation problems . The CoGAN model ( Liu & Tuzel , 2016 ) imposes a weight-sharing constraint on specific layers of two GANs , which are trained to produce samples from different domains . The motivation is that sharing weights in layers associated with high-level features should help preserve the overall structure of the images . This approach is extended in the UNIT framework ( Liu et al. , 2017 ) , where the generative networks are Variational Autoencoders ( VAEs ) with a shared latent space . The weight-sharing used in the CoGAN and UNIT frameworks restricts them to problems where both domains are of the same modality . A more generic method of achieving domain-correspondence is presented in the CycleGAN model proposed by Zhu et al . ( 2017a ) . The CycleGAN objective includes a cycle-consistency loss to ensure that image translation between two domains is invertible . Recently , Chen et al . ( 2020 ) showed that reusing part of the discriminators in CycleGAN as encoders for the generators achieves parameter reduction as well as better results . Although it is possible to apply the cycle consistency loss for cross-modal translation it has not been widely used in such scenarios . Unlike unsupervised methods , supervised approaches rely on having a one-to-one correspondence between the data from different domains . The Pix2Pix model ( Isola et al. , 2017 ) uses cGANs to perform image-to-image translation and has inspired many subsequent works ( Zhu et al. , 2017a ; Wang et al. , 2018 ; Park et al. , 2019 ) . Compared to unsupervised methods , supervised approaches have had more success in translating across different modalities . Notable applications include speechdriven facial animation ( Vougioukas et al. , 2020 ) and text-to-image synthesis ( Reed et al. , 2016 ; Qiao et al. , 2019 ) . It is important to note that the adversarial loss in cGANs alone is often not capable of establishing domain correspondence , which is why these approaches also rely on additional reconstruction or perceptual losses ( Johnson et al. , 2016 ) in order to accurately capture semantics . In many scenarios , the relationship between domains is not bijective ( e.g . one-to-many mapping ) hence it is desirable for translation systems to produce a diverse set of outputs for a given input . Achieving this diversity is a common issue with GAN-based translation systems ( Isola et al. , 2017 ; Liu et al. , 2017 ) since they often suffer from mode collapse . The Pix2Pix model ( Isola et al. , 2017 ) proposes using dropout in both training and inference stages as a solution to this problem . Another successful approach is to apply the diversity regularisation presented in Yang et al . ( 2019 ) . Furthermore , many works ( Zhu et al. , 2017b ; Huang et al. , 2018 ; Chang et al. , 2018 ) attempt to solve this issue by enforcing a bijective mapping between the latent space and the target image domain . Finally , adding a reconstruction loss to the objective also discourages mode collapse ( Rosca et al. , 2017 ) , by requiring that the entire support of the distribution of training images is covered . 2.1 CONDITIONAL GANS . The most common method for conditioning GANs is proposed by Mirza & Osindero ( 2014 ) and feeds the conditional information as input to both the generator and the discriminator . Using the condition in the discriminator assumes that the correlation of samples with the condition will be considered when distinguishing between real and fake samples . However , feeding the condition to the discriminator does not guarantee that the correspondence will be captured and could even lead to the condition being ignored by the network . This issue is shared across all methods which use the condition as input to the discriminator ( Miyato & Koyama , 2018 ; Reed et al. , 2016 ) . Furthermore , it explains why these models perform well when there is structural similarity between domains ( e.g . image-to-image translation ) but struggle to maintain semantics in cases where domains are significantly different such as cross-modal applications ( e.g . video-to-speech ) . Another method presented in Park et al . ( 2019 ) proposes generator conditioning through spatiallyadaptive normalisation layers ( SPADE ) . This approach has been used to produce state of the art results in image generation . It should be noted that this approach requires that source domain data be one-hot encoded semantic segmentation maps and is therefore limited to specific image-translation problems ( i.e . segmentation maps to texture image translations ) . More importantly , conditioning of the discriminator is still done by feeding the condition as an input and hence will have similar drawbacks as other cGAN based methods with regards to semantic preservation . In some cases it is possible to guide the discriminator to learn specific semantics by performing a self-supervised task . An example of this is the discriminator proposed in Vougioukas et al . ( 2020 ) which enforces audio-visual synchrony in facial animation by detecting in and out of sync pairs of video and audio . However , this adversarial loss alone can not fully enforce audio-visual synchronization which is why additional reconstruction losses are required . Finally , it is important to note that finding a self-supervised task capable of enforcing the desired semantics is not always possible . 2.2 ENERGY-BASED GANS . Energy-based GANs ( Mathieu et al. , 2015 ; Berthelot et al. , 2017 ) use a discriminator D which is an autoencoder . The generator G synthesizes a sample G ( z ) from a noise sample z ∈ Z . The discriminator output is fed to a loss function L in order to form an energy function LD ( · ) = L ( D ( · ) ) . The objective of the discriminator is to minimize the energy assigned to real data x ∈ X and maximize the energy of generated data . The generator has the opposite objective , leading to the following minimax game : min D max G V ( D , G ) = LD ( x ) − LD ( G ( z ) ) ( 1 ) The EBGAN model proposed by Mathieu et al . ( 2015 ) uses the mean square error ( MSE ) to measure the reconstruction and a margin loss to limit the penalization for generated samples . The resulting objective thus becomes : min D max G V ( D , G ) = ‖D ( x ) − x‖+max ( 0 , m− ‖D ( G ( z ) ) −G ( z ) ‖ ) , ( 2 ) The marginm corresponds to the maximum energy that should be assigned to a synthesized sample . Performance depends on the magnitude of the margin , with large values causing instability and small values resulting in mode collapse . For this reason , some approaches ( Wang et al. , 2017 ; Mathieu et al. , 2015 ) recommend decaying the margin during training . An alternative approach is proposed by Berthelot et al . ( 2017 ) which introduces an equilibrium concept to balance the generator and discriminator and measure training convergence . Energy-based GANs have been successful in generating high quality images although their use for conditional generation is limited . 3 METHOD . The encoder-decoder structure used in the discriminator of an energy-based GAN gives it the flexibility to perform various regression tasks . The choice of task determines how energy is distributed and can help the network focus on specific characteristics . We propose a conditional version of EBGAN where the generator ( Forward network ) and discriminator ( Reverse network ) perform translations in opposite directions . The Reverse network is trained to minimize the reconstruction error for real samples ( low energy ) and maximize the error for generated samples ( high energy ) . The Forward network aims to produce samples that will be assigned a low energy by the Reverse network . Generated samples that do not preserve the semantics can not be accurately reconstructed back to the source domain and are thus penalized . Given a condition x ∈ X and its corresponding target y ∈ Y and networks F : X → Y and R : Y → X the objective of the DINO framework becomes : min R max F V ( R , F ) = L ( R ( y ) , x ) −L ( R ( F ( x ) ) , x ) , ( 3 ) where L ( · , · ) is a loss measuring the reconstruction error between two samples . Multiple choices exist for the loss function and their effects are explained in Lecun et al . ( 2006 ) . We propose using the MSE to measure reconstruction error and a margin loss similar to that used in EBGAN . However , as shown in Mathieu et al . ( 2015 ) this method is sensitive to the value of margin parameter m , which must be gradually decayed to avoid instability . We propose using an adaptive method inspired by BEGAN ( Berthelot et al. , 2017 ) which is based on maintaining a fixed ratio γ ∈ [ 0 , 1 ) between the reconstruction of positive and negative samples . γ = L ( R ( y ) , x ) L ( R ( F ( x ) ) , x ) ( 4 ) Balancing is achieved using a proportional controller with gain λ . A typical value for the gain is λ = 0.001 . The output of the controller kt ∈ [ 0 , 1 ] determines the amount of emphasis that the Reverse network places on the reconstruction error of generated samples . The balance determines an upper bound for the energy of fake samples , which is a fixed multiple of the energy assigned to real samples . When the generator is producing samples with a low energy they are pushed to this limit faster than when the generator is already producing high-energy samples . Since the ratio of reconstruction errors is kept fixed this limit will decay as the reconstruction error for real samples improves over time . This achieves a similar result to a decaying margin loss without the necessity for a decay schedule . The output of the controller as well as the reconstruction error for real and fake samples during training is shown in Figure 2 . We notice that the controller output increases at the start of training in order to push generated samples to a higher energy value and reduces once the limit determined by γ is reached . Although this approach is inspired by BEGAN there are some key differences which prevent the BEGAN from working with the predictive conditioning proposed in this paper . These are discussed in detail in Section A.4 of the appendix . In practice we find it advantageous to use the margin loss in combination with adaptive balancing . In this case the margin parameter serves as a hard cutoff for the energy of generated samples and helps stabilize the system at the beginning of training . As training progresses and reconstruction of real samples improves training relies more on the soft limit enforced by the energy balancing mechanism . In this case we can set γ = 0 to fall back to a fixed margin approach . The training objective is shown in Equation 5 . When dealing with one-to-many scenarios we find that adding a reconstruction loss to the generator ’ s objective can help improve sample diversity . LR = ‖R ( y ) − x‖+ kt ·max ( 0 , m− ‖R ( F ( x ) ) − x‖ ) LF = ‖R ( F ( x ) ) − x‖ kt+1 = kt + λ · [ ‖R ( y ) − x‖ − γ · ‖R ( F ( x ) ) − x‖ ] ( 5 )

Summary: The paper proposes an adversarial framework DINO to train translation models from source to target and target to source. The basic idea is to replace generator and discriminator in the energy based GAN with two source-to-target generation models. The discriminator(reverse generator) and the generator competes in a minimax game to reconstruct the data. The framework is further extended with duplicate output heads for both discriminator and generator to enhance the training robustness.