DINO: A Conditional Energy-Based GAN for Domain Translation
Authors: Konstantinos Vougioukas
Stavros Petridis
Maja Pantic
Venue: ICLR 2021 Poster (2021)
Keywords: Generative Modelling
Domain Translation
Conditional GANs
Energy-Based GANs

Abstract: Domain translation is the process of transforming data from one domain to another while preserving the common semantics. Some of the most popular domain translation systems are based on conditional generative adversarial networks, which use source domain data to drive the generator and as an input to the discriminator. However, this approach does not enforce the preservation of shared semantics since the conditional input can often be ignored by the discriminator. We propose an alternative method for conditioning and present a new framework, where two networks are simultaneously trained, in a supervised manner, to perform domain translation in opposite directions. Our method is not only better at capturing the shared information between two domains but is more generic and can be applied to a broader range of problems. The proposed framework performs well even in challenging cross-modal translations, such as video-driven speech reconstruction, for which other systems struggle to maintain correspondence.

Published as a conference paper at ICLR 2021
DINO: A CONDITIONAL ENERGY-BASED GAN FOR DOMAIN TRANSLATION
Konstantinos Vougioukas, Stavros Petridis & Maja Pantic
Department of Computing, Imperial College London, UK (k. vougioukas, stavros, Scavros.petriie petridii00 m. (..panticceimppeee ial al.ac.
ABSTRACT
Domain translation is the process of transforming data from one domain another while preserving the common semantics. Some of the most popular domain trans- lation systems are based on conditional generative adversarial networks, which use source domain data to drive the generator and as an input to the discrimina- tor. However, this approach does not enforce the preservation of shared semanties since the conditional input can often be ignored by the discriminator. We propose an alternative method for conditioning and present new framework, where two networks are simultaneously trained, in supervised manner, to perform domain translation in opposite directions. Our method not only better at capturing the shared information between two domains but is more generic and can be applied to a broader range of problems. The proposed framework performs well even challenging cross-modal translations, such as video-driven speech reconstruction, for which other systems struggle to maintain correspondence.
INTRODUCTION
Domain translation methods exploit the information redundancy often found in data from different domains in order to find a mapping between them. Successful applications of domain translation include image style transfer (Zhu al.) 2017a) and specch-enhancemmet (Pascual al. 2017).
Furthermore, these systems are increasingly being used to translate across modalities in applications such as speech-driven animation (Chung et al. 2017) and caption-based image generation (Reed) let al. 2016). Some of the most popular methods for domain translation are based on conditional Generative Adversarial Networks (cGANs) (Mirza Osindero 2014). The conditional information in cGANs is used to drive the generation and to enforce the correspondence between condition and sample. Various alternatives have been proposed for how the condition should be included in the discriminator (Miyato& & Koyama| 2018 Reed et al. (2016) but the majority of frameworks provide it as an input, hoping that the sample's 's correlation with the condition will play a role in distinguishing between synthesized and genuine samples. The main drawback of this approach is that it does not encourage the use of the conditional information and therefore its contribution can be diminished or even ignored. This may lead to samples that are not semantically consistent with the condition.
In this paper. we propose the Dual Inverse Network Optimisation (DINO) framework!] which is based on energy-based GANs (Zhao et al. (017) and consists of two networks that perform transla- tion in opposite directions as shown in Figure In this framework, one network (Forward network) translates data from the source domain the target domain while the other (Reverse Network) per- forms the inverse translation. The Reverse network's ' goal is to minimize the reconstruction error for genuine data and maximize it for generated data. The Forward network aims to produce samples that can be accurately reconstrueted back to the source domain by the Reverse Network. Therefore, during training the Forward network trained as a generator and the Reverse as a discrininator: Since discrimination is based on the ability to recover source domain samples, the Forward network driven to produce samples that are not only realistic but also preserve the shared semanties. We show that this approach is effective across broad range of supervised translation problems, cap- turing the correspondence even when domains are from different modalities (i.c., video-audio). In detail, the contributions of this paper are:
Source code: httpp:///gtth// com/Di httm//DinMaa//DDa

---

Published as a conference paper at ICLR 2021
A domain translation framework, based on novel conditioning mechanism for energy- based GANs, where the adversarial loss is based on the prediction of the condition.
An adaptive method for balancing the Forward and Reverse networks, which makes training more robust and improves performance.
A method for simultaneously training two networks to perform translation in inverse direc- tions, which requires fewer parameters than other domain translation methods.
The first end-to-end trainable model for video-driven speech reconstruction capable of pro- ducing intelligible speech without requiring task-specifie losses to enforce correct content.
Source
Source
Target Domain
Reconstruction
Revrar Netwod
Notaork
High
Kaho
Recemirruzion
Lao Rarrgy
Figure I: Architecture for the DINO framework. The Forward network performs a translation from the source to the target domain. The Reverse network performs the opposite translation and assigns an energy based on the its ability to recover source domain samples from real and generated samples.
RELATED WORK
Domain translation covers a wide range of problems including image-to-image translation (Isola et al. 2017). caption-based image synthesis (Qiao et al.) 2019). and text-to-speech synthesis (Arik
ct al. 2017). Unsupervised translation methods attempt to find a relationship between domains using unpaired training data. However, finding correspondence without supervision is an ill- ill-posed prob- lem which why these methods often impose additional constraints on their networks or objectives. The majority of unsupervised methods are applied to
translation problems. The CoGAN model (Liu & Tuzel 2016) imposes a weight-sharing constraint on specific layers of two GANs, which are trained to produce samples from different domains. The motivation is that sharing weights layers associated with high-level features should help preserve the overall structure images. This approach is extended in the UNIT framework (Liu al. 2017). where the generative networks are Variational Autoencoders (VAEs) with a shared latent space. The weight-sharing used in the CoGAN and UNIT frameworks restricts them problems where both domains of the same modality. A more generic method of achieving comain-correspoddence is presented in the Cycle- GAN model proposed by Zhu al. (2017a). The CycleGAN objective includes a cycle-consistency loss to ensure that image translation between two domains is invertible. Recently, Chen et al. (2020) showed that reusing part of the discriminators in CycleGAN as encoders for the generators achieves parameter reduction well as better results. Although it is possible to apply the cycle consistency loss for gross-modal translation has been widely used in such scenarios.
Unlike unsupervised methods, supervised approaches rely on having one-to-one correspondence between the data from different domains. The Pix2Pix model (Isola al., 2017) uses cGANs to per- form image-to-imane iomeiageieiiaaee translation and has inspired many subsequent works (Zhu et al. 2017a Wang cet al.) 2018 8 Park et al.) 2019). Compared to unsupervised methods, supervised approaches have had more success in translating across different modalities. Notable applications include speech- driven facial animation /Vougioukas et al.] 2020) and text-to-image synthesis (Reed et al., 2016 Qiao et al. 2019). It is important to note that the adversarial loss in cGANs alone is often not capa- ble of establishing domain correspondence, which is why these approaches also rely on additional reconstruction perceptual losses (Johnson al. 2016) in order accurately capture semanties.
2

---

Published as a conference paper at ICLR 2021
In many scenarios, the relationship between domains is not bijective (e.g. mee-oo-many mapping)
hence it is desirable for translation systems to produce a diverse set of outputs for a given input.
Achieving this diversity is common issue with GAN-based translation systems (Isola et al.
2017
Liu et al., 2017) since they often suffer from mode collapse. The Pix2Pix model (Isola et al. 2017)
proposes using dropout in both training and inference stages solution this problem. Another successful approach is to apply the diversity regularisation presented in Yang et al. (2019). Fur-
thermore, many works (Zhu et al. 2017b] Huang al. 2018 Chang et al. (018) attempt solve this issue by enforcing a bijective mapping between the latent space and the target image domain. Finally, adding a reconstruction loss to the objective also discourages mode collapse (Rosca et al.)
2017). by requiring that the entire support of the distribution of training images is covered.
2.1 CONDITIONAL GANS
The most common method for conditioning GANs proposed by Mirza & Osindero (014 and feeds the conditional information input to both the generator and the discriminator. Using the condition in the discriminator assumes that the correlation of samples with the condition will be considered when distinguishing between real and fake samples. However, feeding the condition to the discriminator does not guarantee that the correspondence will be captured and could even lead to the condition being ignored by the network. This issue shared across all methods which use the condition as input to the discriminator (Miyato & Koyama| 2018 Reed et al. 2016). Furthermore, it explains why these models perform well when there is structural similarity between domains (e.g. image-to-image translation) but struggle to maintain semantics in cases where domains are significantly different such as cross-modal applications (e.g. video-to- video-to-speech).
Another method presented in Park et al. (2019) proposes generator conditioning through spatially- adaptive normalisation layers (SPADE). This approach has been used to produce state of the art results in image generation. It should be noted that this approach requires that source domain data be one-hot encoded semantic segmentation maps and is therefore limited to specific amage-translation problems (i.e. segmentation maps to texture image translations). More importantly, conditioning of the discriminator is still done by feeding the condition as an input and hence will have similar drawbacks other cGAN based methods with regards to semantic preservation.
In some cases it is possible to guide the discriminator to learn specific semantics by performing a self-supervised task. An example of this is the diseriminator proposed in Vougioukas al. (2020) which enforces audio-visual synchrony in facial animation by detecting in and out of sync pairs of video and audio. However, this adversarial loss alone can not fully enforce audio-visual synchro- nization which is why additional reconstruction losses are required. Finally, it is important to note that finding self-supervised task capable of enforcing the desired semantics is not always possible.
2.2 ENERGY-BASER GANS
Energy-based GANs Maahhieu et al. 2015 Berthelot et al.) 2017) use discriminator D which is an autoencoder. The generator G synthesizes a sample G(z) from a noise sample z E Z. The discriminator output is fed to loss function L in order to form L(D(.)). The objective of the discriminator is to minimize the energy assigned to real data x X and maximize the energy of generated data. The generator has the opposite objective, leading to the following minimax game:
energy function (D(-)
minmaxV (D. = = Ln(x) CDD
(1)
D
The EBGAN model proposed by Mathicu et al. (2055) uses the mean square error (MSE) to measure the reconstruction and a margin loss to limit the penalization for generated samples. The resulting objective thus becomes:
minmaxV((D,G) = (1D D
max(0, m - - G(z)I),
The margin corresponds to the maximum energy that should be assigned to synthesized sample. Performance depends on the magnitude of the margin, with large values causing instability and small values resulting in mode collapse. For this reason, some approaches (Wang al.) 2017 Mathieu et al., 2013) recommend decaying the margin during training. An alternative approach is proposed by Berthelot et al. (017)) which introduces equilibrium concept to balance the generator
3

---

Published as a conference paper at ICLR 2021
and discriminator and measure training convergence. Energy-based GANs have been successful in generating high quality images although their use for conditional generation is limited.
3 METHOD
The encoder-decoder structure used in the discriminator of an energy-based GAN gives it the flexi- bility to perform various regression tasks. The choice of task determines how energy is distributed and can help the network focus on specific characteristics. We propose a conditional version of EBGAN where the generator (Forward network) and discriminator (Reverse network) perform trans- lations in opposite directions. The Reverse network is trained to minimize the reconstruction error for real samples (low energy) and maximize the error for generated samples (high energy). The For- ward network aims to produce samples that will be assigned a low energy by the Reverse network. Generated samples that do not preserve the semantics can not be accurately reconstructed back to the source domain and are thus penalized. Given a condition E X' and its corresponding target y E y and networks F : X + y and y - X the objective of the DINO framework becomes:
minmaxV (R, F²) = (R()) - 2 (R(((z)))
where (,,,)) loss measuring the reconstruction error between two samples. Multiple choices exist for the loss function and their effects are explained in Lecun et al. (006). We propose using the MSE to measure reconstruction error and margin loss similar to that used in EBGAN. However, as shown in Mathieu et al. (2015) this method is sensitive to the value of margin parameter m. which must be gradually decayed avoid instability. We propose using an adaptive method inspired by BEGAN (Berthelot et al. 2017) which is based on maintaining fixed ratio y E [0, 1) between the reconstruction of positive and negative samples.
7 L(R(y), x)
(4)
L(R(F())).z)
Balancing is achieved using a proportional controller with gain X. A typical value for the gain is = 0.001. The output of the controller k E [0, 1] determines the amount of emphasis that the Reverse network places on the reconstruction error of generated samples. The balance determines an upper bound for the energy of fake samples, which real samples. When the generator is producing samples with low energy they are pushed to this limit faster than when the generator already producing high-energy samples. Since the ratio of reconstruction errors is kept fixed this limit will decay as the reconstruction error for real samples improves over time. This achieves a similar result to a decaying margin loss without the necessity for a decay schedule. The output of the controller as well as the reconstruction error for real and fake samples during training is shown in Figure 2 We notice that the controller output increases at the start of training in order to push generated samples higher energy value and reduces once the limit determined by 7 is reached. Although this approach is inspired by BEGAN there are some key differences which prevent the BEGAN from working with the predictive conditioning proposed in this paper. These are discussed in detail in Section A.4 of the appendix.
fixed multiple of the energy assigned to
0.3
Reel Reconstuction Britr
0.25
[LAnntral Catyol)
0.2
0.05
10000
20000
30003
50000
60000
70003
Iterations
Figure 2: The emphasis placed by the Reverse network (discriminator) on generated samples during training.
In practice we find it advantageous to use the margin loss in combination with adaptive balancing. In this case the margin parameter serves hard cutoff for the energy of generated samples and
4

---

Published as a conference paper at ICLR 2021
helps stabilize the system the beginning of training. As training progresses and reconstruction of real samples improves training relies more on the soft limit enforced by the energy balancing mechanism. In this case objective is shown Equation | When dealing with one-to-many scenarios we find that adding a reconstruction loss to the generator's objective help improve sample diversity.
set to fall back to fixed margin approach. The training
LR [ =11 + Kr max [[[(F(x)) ==l)
LF [ =11
(5)
ke+1 kiz + [|R(v) xl1 - [[(((x)) x=]]
3.1 BIDIRECTIONAL TRANSLATION
evident from Equation that the two networks have different goals and that only the Forward network will produce realistic translations since the Reverse network is trained only using an MSE loss. This prohibits Forward network. For the DINO framework, since the Forward and Reverse network have the same structure we can swap the roles of the networks and retrain to obtain realistic translation in the opposite direction. However, it is also possible train both networks simultancously by combining the objectives for both roles (i.e. discriminator and generator). This results in the following zero-sum two player game:
use for domain translation and limits to facilitating the training of
minmaxV(R,F) ( - + (((()))))))(((((L,.)))) (6)
R
In this game both players have the same goal which is minimize the reconstruction error for real samples and to maximize it for fake samples while also ensuring that their samples are assigned low energy by the other player. Each player therefore behaves both generator and discriminator. However, in practice we find that is difficult for a network to achieve the objectives for both roles, causing instability during training. The work proposed by Chen et al. (2020), where discriminators and generators share encoders, comes to a similar conclusion and proposes decoupling the training for different parts of the networks. This is not possible our framework since the discriminator for one task is the generator for the other. To solve this problem we propose branching the decoders of the networks to create two heads which are exclusively used for either discrimination or generation. We find empirically that the best performance in our image-to-image experiments is achieved when branching right after the third layer of the decoder. Additionally, the network encoders are frozen during the generation stage. The bidirectional training paradigm is illustrated in Figure 3
Frazon
Tratsabl
Energy Resonstrction
High Energy Reconstructoon
Lou-Knsry Reconstrusion
Bie Reed
High Energy Ressnstruction
Figure 3: Bidirectional training using the DINO framework. The two players are trained for both the generator and discriminator objectives. The top half of the diagram shows training with the green player as the generator and the bottom of the diagram shows its training as the discriminator. both cases the blue player assumes the opposite role.
When training network R as a discriminator we use the stream that passes through the discrimina- tive head Rutse and when training generator the stream that uses the generative head

---

Published as a conference paper at ICLR 2021
Ryen. The same applies for player F and which uses streams Faise and Fgen for discrimination and generation, respectively. To maintain balance during training we use a different controller for each player which results the objective shown in Equation The first two terms in each players objective represent the player's S goal as a discriminator and the last term reflects its goal as generator.
LR L (Paissc))) x) ke L (Rdisc (F (1)) x) + L (FFdisel (Rgen (1)),
discriminator objective
generator objective
LF (F(iiis (z), Me L (isio diso R (1)). + L (Retio gen (2)).
discriminator objective
generatio objective
he+1 = kt + +AR- (Rassssse (y), 2) ID- LL (ReiioclFeenn gen (x)) 1441 = 14 + Xp [5e (Faisc (T), "G (F (Rgen())))))))) 9)
3.2 COMPARISON WITH OTHER METHODS
As mentioned in section 2.1 the cGAN conditioning mechanism, used in most supervised translation systems, struggles to preserve the shared semantics cases where there is no structural similarity between domains. The DINO framework attempts to overcome this limitation by using different paradigm, where the condition is predicted by the discriminator instead of being fed as an additional input, forcing the generator maintain the common semantics. Our approach is inspired by several semi-supervised training techniques for GANs (Salimans et al. 2016 Odena et al. 2017 7, Springen- berg 2015), which have showed that specializing the discrimmator by performing classification task adds structure to its latent space and improves the quality of generated samples. However, these approaches are not designed for conditional generation and use classification only as proxy task. This differs from our approach where discrimination is driven directly by the prediction of condition.
Another advantage of our system stems from its use of encoder-decoder structure for the Reverse network. This provides flexibility since the Reverse network can be easily adapted to perform a variety of different translation tasks. In contrast, the multi-stream discriminators used in cross- modal cGANs require fusing representations from different streams. The fusion method as well as the stage at which embeddings are fused is an important design decision that must be carefully chosen depending on the task since it can greatly affect the performance of these models.
The objective of the generator in Equation 5 resembles the cycle-consistency loss used in many unsupervised methods such as CycleGAN (Zhu et al.) 2017a) and NICE-GAN (Chen et al.) 2020). This also bears resemblance to the back- translation used in bidirectional neural machine translation methods (Artetxe et al., 2018 Lample et al. 2018). However, it is important note that the cycle- consistency loss used in these approaches is not an adversarial loss since it is optimized with respect to both networks' parameters. The most similar work to is MirrorGAN (Qiao et al., 2019), which improves the generation of images through re-description. This model however uses pre- trained network for re-description in addition to an adversarial loss. Compared all aforementioned approaches the DINO framework is the only one in which the adversarial loss alone can both achieve sample realism while enforcing correspondence. Finally, since our bidirectional framework uses the generators for discrimination it requires far fewer parameters than these approaches.
EXPERIMENTS
We evaluate the DINO framework image-to-image translation since this the most typical appli- cation for domain- traiinn-trnllation systems. Additionally, we tackle the problem of video-dríven speech reconstruction, which involves synthesising intelligible speech from silent video. In all of the exper- iments focus is placed not only on evaluating the quality of the generated samples but also verifying that the semantics are preserved after translation.
4.1 IMAGE-TO-IMAGE TRANSLATION
The majority of modern domain translation methods have been applied to image-to-imanl translation problems, since it is common for both domains to share high-level structure and therefore easier to capture their correspondence. We evaluate the DINO framework on the CelebAMask- MaQ (Lee et al.] 2020) and the Cityscapes (Cordts et al. 2016) datasets, using their recommended training-test splits.

---

Published as a conference paper at ICLR 2021
When judging the performance of image-to-imade translation systems must consider multiple factors including the perceptual quality, the semantic consistency and the diversity of the generated images. We therefore rely combination of full-reference reconstruction metrics and perceptual metrics for image assessment.
Reconstruction metrics such the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM) measure the deviation of generated images from the ground truth. Although these metrics are good at measuring image distortion they usually poor indicators of realism and they penalize diversity. For this reason, we also measure the perceptual quality of the images by using the Fréchet Inception Distance (FID), which compares the statisties of the embeddings of real and fake images in order to measure the quality and diversity. Furthermore, we use the cumulative probability blur detection (CPBD) metric (Narvekar & Karam (0009) to assess image sharpness. Finally, we use pre-trained semantic segmentation models to verity that image semantics are accurately captured in the images. For the GelebAMask--Ree dataset we use the segmentation model from Lee et al.) (2020) and for the Cityscapes dataset use a DeepLabv3+ model (Chen et al. 2018). We report the pixel accuracy as well as the average intersection over union (mIOU).
We compare our method to other supervised image-to-image translation models, such Pix2Pix and BiCycleGAND Since DINO is generic translation method comparing it to translation methods that are tailored to specific type of translation (Yi et al. 2019) is unfair comparison since these methods make use of additional information or use task-specific losses. Nevertheless, we present the results for SPADE (Par et al. 2019) the Cityscapes dataset in order see how well approach performs compared to state-of-the-art task-specific translation methods. Since the pre- trained SPADE model generates images resolution of 512 256 we resize images to 256 256 for a fair comparison.
When training the DINO model we resize images 256 256 and use a networks with a similar U-Net architecture to the Pix2Pix model to ensure a fair comparison. The architecture of the net- works used in these experiments can be found in section A.I.I of the appendix. Additionally, like Pix2Pix we use an additional L1 loss to train the Forward network (generator), which helps improve image diversity. The balance parameter y set 0.8 for image-to-imas We train using the Adam optimizer (Kingma & Ba 2015), with learning rate of 0.0002, and mo- mentum parameters 31 = 0.5, 82 = 0.999. The quantitative evaluation on the CelebAMask-HHC and Cityscapes datasets is shown in Tables and 2 Qualitative results are presented in Section A.5. of the appendix.
translation experiments.
Table 1: Evaluation CelebAMask--H dataset when translating in the direction labels photos.
Method
PSNR
SSTM t
CPBD
FID 4
Pix. Acc.
mloU
Ground Truth
DC
1.00
0.54
93.4 %
62.1 %
Pix2Pix BiCycleGAN
11.61
0.35
0.52
59.1
93.9 %
63.1
10.47
0.31
0.50
90.5
52.6
DINO
12.08
0.38
0.49
51.5
96.8 %
69.7 %
DINO (Bidirectional)
12.18
0.37
0.42
55.0
96.2 %
67.3 %
Table 2: Evaluation on Cityscupes dataset when translating in the direction labels - photos.
Method
PSNR
SSIM
CPBD1 1
Pix. Acc. +
mloU t
Ground Truth
1.00
0.72
93.9 %
44.1 %
Pix2Pix BiCycleGAN
15.96
0.37
0.68
89.4 %
30.7 %
14.77
0.32
0.66
79.9 %
21.3 %
SPADE DINO DINO (Bidirectional)
15.97
0.38
0.72
92.8 %
38.3 %
16.44
0.41
0.71
91.4%
35.9%
16.78
0.42
0.72
91.3
35.2 %
https: ://g thub. com http::/wwww.//mana///////llleeea
PPeetrained model from: wttps::/ggtttbb..octtoo

---

Published as a conference paper at ICLR 2021
The results on Tables D and show that our method outperforms the Pix2Pix and BicycleGAN models both in terms of perceptual quality as well as reconstruction error. More importantly, our approach is better at preserving the image semantics as indicated by the higher pixel accuracy and mloU. We notice that for the CelehAMask-HO dataset the segmentation accuracy is better for gen- erated images than for real images. This phenomenon is due to some inconsistent labelling and is explained in Section A.2 of the appendix. We also note that the bidirectional DINO framework can simultaneously train networks perform translation in both directions without a sacrificing quality and with fewer parameters. Finally, an ablation study for our model is performed in A.3 of the appendix.
When comparing our results to those achieved by the SPADE network on the Cityscapes dataset we notice that our model performs similarly, achieving slightly better performance on reconstruction metrics (PSNR, SSIM) and slightly worse performance for preserving the image semantics. This is expected since the SPADE model has been specifically designed for translation from segmentation maps images. Furthermore, the networks used in these experiments, for the DINO framework, are far simpler (37 million parameters in the generator compared 97 million). More importantly, unlike SPADE our network can be applied any task and perform the translation in both directions.
4.2 VIDEO-DRIVER SPEECH RECONSTRUCTION
Many problems require finding a mapping between signals from different modalities (e.g. speech- driven facial animation, caption-based image generation). This far more challenging than image- to-image translation since signals from different modalities do not have structural similarities, mak- ing it difficult capture their correspondence. We evaluate the performance of our method on video-driven speech reconstruction, which involves synthesising speech from silent video. This is a notoriously difficult problem due ambiguity which is attributed to the existence of homophe- nous words. Another reason for choosing this problem is that common reconstruction losses (e.g. LI, MSE), which are typically used image-to-imes image translation to enforce low-freat www-frquency cor- rectness (Isola et al., 2017) are not helpful for the generation of raw waveforms. This means that methods must rely only on the conditional adversarial loss to enforce semantic consistency.
We show that the DINO framework the adversarial loss described in Equation Adjusting the framework for this task requires using encoders and decoders that can handle audio and video shown in Figure 4 The Forward network transforms a sequence of video frames centered around the mouth to its corresponding waveform. The Reverse network is fed waveform and the initial video frame to produce video sequence of the speaker. The initial frame is provided to enforce the speaker identity and ensures that the reconstruction error will be based on the facial animation and on any differences in appearance. This forces the network to focus capturing the content of the speech and not the speaker's 's identity.
synthesize intelligible speech from silent video using only
Audia
Gill!
Audio
Excoder
Deeader
Decoder
Discrininttrr
Figure 4: DINO framework architecture used for video-dríven speech reconstruction. The Forward network (generator) takes a video as input and outputs waveform. The Reverse network (dis- criminator) takes input waveform and outputs video. Components that make up the Forward network are shown in blue and components belonging to the Reverse network are shown in green. More details shown in Section A.1.2 f. of the appendix.
8

---

Published as a conference paper at ICLR 2021
Experiments performed on the GRID dataset (Cooke et al. 2006), which contains short phrases spoken by 33 speakers. There are 1000 phrases per speaker, each containing 6 words from a vocab- ulary of 51 words. The data is split according to Vougioukas et al. (2019) that the test set contains unseen speakers and phrases. As baselines for comparison we use a conditional version of Wave- GAN (Donahue al., 2019) and CycleGAN framework adapted for video-to-audio translation. Additionally, we compare with the model proposed by Vougioukas et al. (2019), which is designed for video-driven speech reconstruction and uses perceptual loss accurately capture the spoken content. An Adam optimiser is used with a learning rate of 0.0001 for the video-to-aude to-dudio network and learning rate of 0.001 for the audio-to- -vdeo network. The balancing parameter ry is set 0.5.
We evaluate the quality of the synthesized audio based on intelligibility and spoken word accuracy. We measure speech quality using the mean Mel Cepstral Distance (MCD) (Kubichek) 1993), which measures the distance between two signals in the mel-frequency cepstrum and is often used to assess synthesized speech. Furthermore, we use the Short-Time Objective Intelligibility (STOI) (Taal et al.) 2011) and Perceptual Evaluation of Speech Quality (PESQ) (Rix et al. 2001 metries, which mea- sure the intelligibility of the spoken message the Word Error Rate (WER). The results for the speech recoostrtccnnt task are shown in Table 3
the synthesized audio. Finally, order to verity the semantic consistency
pretrained automatic speech recognition (ASR) model and measure
Table 3: Evaluation of video-driven speech reconstruction the GRID dataset (unseen subjects).
Method
PESQ f
STOI
MCD 4
WER
Ground Truth
4.5
1.00
0.0
5.4%
Conditional WaveGAN CycleGAN
0.96
0.37
43.7
81.7%
1.01
0.29
28.3
88.6%
Perceptual GAN (Vougioukas et al. 2019 DINO
1.24
0,45
24.3
40.5 %
1.21
0.51
23.0
32.6 %
The results of Table 3) that our method is capable of producing intelligible speech and achiev- ing similar performance to the model proposed by VVougiouua et al. (2019). Furthermore, the large WER for both baselines highlights the limitations of cGANs and CycleGANs for cross-modal trans- lation. Although our approach better at capturing the content and audio-visual correspondence, notice that samples all share the same robotic voice compared the other methods. This is expected since discrimination using our approach focuses mostly on audio-visual correspondence and not capturing the speaker identity. Examples of synthesized waveforms and their spectrograms are shown in Section of the appendix and samples are provided in the supplementary material.
Ethical considerations: We have tested the DINO model on this task as an academic investigation to test its ability to capture common semantics even across modalities. Video-driven speech recon- struction has many practical applications especially digital communications. enables videocon- ferencing in noisy or silent environments and can improve assiiiig-ssisttvve devices. However, this used in surveillance systems which raises privacy concerns. There-
technology can potentially fore, although we believe that this topic is worth exploring, future researchers should be careful when developing features that will enable this technology to be used for surveillance purposes.
5 CONCLUSIONS
In this paper we have presented a domain translation framework, based on predictive conditioning. Unlike other conditional approaches, predicting the condition forces the discriminator to learn the the relationship between domains and ensures that the generated samples preserve cross-domain se- mantics. The results on image-to-image translation verify that our approach capable of producing sharp and realistic images while strongly enforcing semantic correspondence between domains. Fur- thermore, results on video-driven speech reconstruction show that our method is applicable to a wide range of problems and that correspondence can be maintained even when translating across different modalities. Finally, we present a method for bidirectional translation and show that it achieves the same performance while reducing the number of training parameters compared to other models.

---

Published as a conference paper at ICLR 2021
REFERENCES
Sercan O. Arik, Gregory Diamos, Andrew Gibiansky, John Miller, Kainan Peng, Wei Ping. Jonathan Raiman, and Yanqi Zhou. Deep voice 2: Multi-speaker neural texe-to-speech. In NeurIPS, pp. 2963-2971, 2017.
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. translation. In ICLR, 2018.
Unsupervised neural machine
David Berthelot, Thomas Schumm, and Luke Metz. Began: Boundary equilibrium generative ad- versarial networks. In arXiv preprint: 1703, 10717, pp. 1-10, 2017.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https:/// www. Software available from wandb.com.
wandb. com/
Chia Che Chang, Chich Hubert Lin, Che Rung Lee, Da Cheng Juan, Wei Wei, and Hwann Tzong Chen. Escaping from collapsing modes in constrained space. Lecture Notes in Computer Science, 11211 LNCS:212-227, 2018.
Liang-Chies Chich Chen, Yukun Zhu, George Papandreou. Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In ECCV, pp. 801- 818, 2018
Runfa Chen, Wenbing Huang, Binghui Huang. Fuchun Sun, and Bin Fang. Reusing discriminators for encoding: Towards unsupervised image-to-image translation. In CVPR, 2020.
Joon Son Chung, Amir Jamaludin, and Andrew Zisserman. You said that? In BMVC, pp. 1-12, 2017.
Martin Cooke, Jon Barker, Stuart Cunningham, and Xu Shao. An audio-visual corpus for speech perception and automatic speech recognition. The Journal of the Acoustical Society of America, 12065):2421-2422) 2006.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, pp. 3213-3223, 2016.
Chris Donahue, Julian McAuley, and Miller Puckette. Adversarial audio synthesis. ICLR 2019, pp- 1-16, 2019.
Xun Huang, Ming Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to- image translation. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11207 1NCS::79-116,, 2018.
Phillip Isola, Jun Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional adversarial networks. In CVPR, volume 2017-Janua, pp. 5967-5976, 2017.
Justin Johnson, Alex Alahi, re, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, pp. 694-711, 2016.
Diederik P Kingma and Jimmy Lei Ba. Adam: A method for stochastic gradient descent. In ICLR, 2015.
R. Kubichek. Mel-cepstral distance measure for objective speech quality assessment. Proceedings OEIE Pucific Rim Conference Communications Computers and Signal Processing, volume 1, pp. 125-128 1, 1993.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc' Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. In ICLR, 2018.
Yann Lecun, Sumit Chopra, Raia Hadsell, Mare Aurelio Ranzato, and Fu Jie Huang. Energy-based models. In Predicting Structured Data pp. 191-249 2006..
Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In CVPR, pp. 5548-5557 2020.
10

---

Published as a conference paper at ICLR 2021
Ming Yu Liu and Oncel Tuzel. Coupled generative adversarial networks. In NeurIPS, pp. 469-477, 2016.
Ming Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. In NeurfPS, pp. 701-709, 2017.
Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. (2015):1-14, 2015.
Mehdi Mirza and Simon Osindero.
Conditional
generative adversarial nets.
arXiv
preprint: (eprintt:4411.. 1.111.114474 2014.
Takeru Miyato and Masanori Koyama. Cgans with projection discriminator. In ICLR, 2018.
Niranjan D. Narvekar and Lina Karam. no-reference perceptual image sharpness metric based cumulative probability of blur detection. International Workshop on Quality of Multimedia Experience, QoMEx, 20(9):87-91, 2009.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxil- iary classifier gans. In ICML, pp. 2642-26666662 2,51, 2017.
Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun- Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In CVPR, pp. 2337-2346, 2019.
Santiago Pascual, Antonio Bonafonte, and Joan Serrà. Segan: Speech enhancement generative adversarial network. In INTERSPEECH, number D, 2017.
Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. Mirrorgan: Learning text-to-image generation by redescription. In CVPR, pp. 1505-1514, 2019.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schicle, and Honglak Lec. Generative adversarial text to image synthesis. In ICML, pp. 1060-1069, 2016.
A. W. Rix, J. G. Beerends, M. P. Hollier, and A. Hekstra. Perceptual evaluation of speech qual- ity (pesq) a method for speech quality assessment of telephone networks and codecs. In ICASSP, volume 2, pp. 749-752, 2001.
Mihaela Rosca, Balaji Lakshminarayanan, David Warde-Farley, and Shakir Mohamed. Variational approaches for auto-encoding generative adversarial networks. 2017.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.
Improved techniques for training gans. pp. 1-10, 2016.
Jost Tobias Springenberg. Unsupervised and supe-suped learning with categorical generative adversarial networks. arXiv preprint arXiv: (0w::5511.0333 0663990 2015.
Coes H. Taal, Richard C. Hendriks, Richard Heusdens, and Jesper Jensen. An algorithm for intelligi- bility prediction of time- time-frequency weighted noisy speech. IEEE Transactions Audio, Speech and Language Processing, 19(7):2125221) 2011.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprini arXiv: 2016.
Konstantinos Vougioukas, Pingchuan Ma. Stavros Petridis, and Maja Pantic. Video-driven speech reconstruction using generative adversarial networks. In INTERSPEECH, pp- 4125 4129, 2019.
Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. Realistic speech-driven facial anima- tion with gans. I/CV, 128(5): 13855):338-11433 2020.
Ruohan Wang, Antoine Cully, Hyung Jin Chang, Yiannis Demiris, and United Kingdom. Magan Margin adaptation for generative adversarial networks. arXiv preprint:1704.033 2017.
Ting Chun Wang, Ming Yu Liu, Jun Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In CVPR, pp. 8798- 8807, 2018.
11

---

Published as a conference paper at ICLR 2021
Dingdong Yang, Hong Seunghoon, Ynseok Jang, Tianchen Zhao, and Honglak Lee. Diversity- sensitive conditional generative adversarial networks. In ICLR, pp. 1-21, 2019.
Ran Yi, Yong Jin Liu, Yu Kun Lai, and Paul L. Rosin. Apdrawinggan: Generating artistic portrait drawings from face photos with hierarchical gans. CVPR, pp. 10735-10744, 2019.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. ICML, pp. 12744-12753, 2019.
Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial network. In ICLR, pp. 1-17, 2017.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexci A. Efros. Unpaired mage-tt--image translation using sycle-consistent adversarial networks. In ICCV, pp. 2242-2251, 2017a.
Jun Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A. Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. NeurlPS, number 1, pp. 666-477, 2017b.
A APPENDIX
A.1 NETWORK ARCHITECTURE
A.1.1 IMAGE-TO-IMAGE TRANSLATION
This section describes the network architecture used for the image-to-imagee translation experiments in Section The two networks used in the DINO framework are identical and both use U-Net encoder-decoder architecture similar to that used in Pix2Pix (Isola al. 2017). The encoder is 7-layer Convolutional Neural Network (CNN) made of strided 2D convolutions. The decoder is a 12-layer CNN made of 2D convolutions and up-sampling layers. We use Instance Normalization Ulyanov et al., 2016), which has been shown to work well style transfer applications. The
network is shown in detail in Figure
Conv 20
Instance
B
Leakey BeLU
Conv 20 K S: 1
B2
fanh
B1
8,
B
256
B
8
8,
B5
Conv' 2D
Instance Norm
ReLU
4 512
512
512
x x 1024
x 8 8 512
32
16 16 x 1024
16 512
32 x 32 1024
128x 128 x 64
64 64 x 128
32 x 32 x 256
noise
64 x 64 x
128x 128 256
Conv ZD
Kc 3 5:
Instance Norm
ReLu
Up 2x
K: 3 S:
Instance Norm
ReLU
256 x 256
256 x 256
conv 20
B₃
Up
lanh
Figure Architecture used the DINO framework for image-to-image translation experiments.
12

---

Published as a conference paper at ICLR 2021
A.1.2
SPEECH RECONSTRUCTION
This section describes the architecture of the networks used for video-driven speech reconstruction in the experiments of Section 4.2 In this scenario the Forward network synthesizes speech and the Reverse network performs speech-driven facial animation. The Forward network is made up of Video Encoder, a single-layer GRU and an Audio Decoder. The video sequence is fed to the Video Encoder, which uses spatio-temporal convolutions to produce an embedding per video frame. The embeddings are fed to single-layer GRU to create coherent sequence of representations which then passed an Audio Decoder network which will produce 640 audio samples per embedding. Concatenating these chunks of samples without overlap forms waveform. Both the Video Encoder and Audio Decoder are fully convolutional networks, with the Audio Decoder using an additional self- seff-attention layer (Zhang et al. 2019) before the last layer shown in Figure
Conv3o
7.7.4.4
$:1.2.2
Conv3o
w. 1.1.4.4
$:1.2.2
Corv30
k: 1.1.444 4
$:1.2.2
CorvaD
k:: 1.11. 4.4.4
51 1.1.2.2
ConvaD 4,1,4
8:1.1.1
ConuTIn 5:
ConVTID 54.554 p: 10
ConOTID 24.5 s: 4::10
COnVTID 4
Self-Attesiio
12.5:22::1
(a)
(b)
The architecture of the Video Encoder and Audio Encoder used in the Forward network for video-to-audio translation.
Figure 6:
The Reverse network is made up of two encoders responsible for capturing the speaker identity and content. The content stream uses a sliding window approach to create a sequence of embeddings for the audio using an Audio Encoder and a 2-layer GRU. The identity stream consists of an Identity Encoder which captures the identity of the person and enforces on the generated video. The two embeddings are concatenated and fed to a Frame Decoder which produces a video sequence. Skip connections between the Identity Encoder and Frame Decoder ensure that the face is accurately reconstructed. A detailed illustration of the Reverse network is shown in Figure
sllp connaction
Identity Encoder
Frame Decader
channall-viie concatenation
96
Sx 98 64
120 2 18
128 24
12
x
512x6x4
02
1200
16x 1100
32 400
125
Figure 7: The model for the Reverse network of the DINO framework when performing video-driven speech reconstruction.
A.2 CELEBA SEGMENTATION
In Table I we notice that the segmentation evaluation on generated images surpasses that of real images. The reason for this are some inconsistencies in the labelled images. Examples in Figure
13

---

Published as a conference paper at ICLR 2021
show that in these cases some objects labeled despite being occluded in the real image. However, these objects will appear in the generated images since the labelled images are used to drive their generation. These small inconsistencies in the data annotations explain why segmentation is slightly better for synthesized samples.
Figure 8: Examples where the annotations in the segmentation map are consistent with the real image. However, the generated images will remain true to the segmentation map since they are constructed from
A.3 ABLATION STUDY
In order to measure the effect of the reconstruction loss and adaptive balancing used in the DINO framework we perform an ablation study on the CelebAMask-HHC dataset. The results the study shown in Table 4 As expected the addition of the L1 loss results in higher PSNR and SSIM since these metrics depend on the reconstruction error, which is directly optimised by this loss. More importantly, note that the addition of the L1 loss improves the FID score since it prevents mode collapse. This evident when observing the examples shown in Figure which shows that mode- dropping that occurs in both DINO and Pix2Pix when this loss is omitted. Finally, notice that the adaptive balancing used in DINO allows for more stable training and improves performance, which reflected across all metrics.
Table 4: Ablation study for DINO performed on the HelcbAMaskk-H dataset
Model
PSNR
SSIM
CPBD
FID J.
Pix. Ace.
mIoU
DINO w LI (- = 0.8)
12.08
0.38
0.49
51.5
96.8 %
69.7 %
DINO w/o L1 (7 = 0.8)
11.34
0.36
0.41
69.9
96.8 %
69.4 %
DINO w/o 1.1 no halance
10.25
0.31
0.31
89.5
93.3 %
58.8 %
fixed margin
0.2)
A.4 ADAPTIVE BALANCING
As mentioned in Section 3 DINO uses controller to ensure that the energy of generated samples always a fixed multiple of the energy of real samples. Although this approach is similar that used by BEGAN (Berthelot et al.) 2017) there is key difference. BEGANs perform autoencoding and therefore assume that the discriminator's reconstruction error will be larger for real samples since they have more details which are harder to reconstruct. For this reason, the controller used by BEGAN tries to maintain a balance throughout training where L(Ireat) > L(I In the DINO framework the discriminator performs domain translation therefore it is natural to assume that real samples should produce better reconstructions since they contain useful information regarding the semantics. For this reason choose to maintain balance where L(x take) > [(Ireat). This is reflected in the controller update as well as the balance parameter of DINO which is the inverse of that used in BEGANs.
14

---

Published as a conference paper at ICLR 2021
(a) DINO
(b) DINO w/o LI
(c) Pix2Pix
(d) Pix2Pix w/o LI
Figure 9: Examples of images generated with and without an L. reconstruction loss. The loss improves diversity both in the case of Pix2Pix and our proposed approach. Models trained without this loss will generate faces with similar attributes (e.g. hair color, skin color, age).
As mentioned the core difference with the adaptive balancing in BEGAN is that DINO maintains a balance where fake) > L L(Zreat) whereas BEGAN maintains a balance for which L(Ireat) > ((xje))). This makes BEGAN unsuitable for use with the predictive conditioning proposed in this paper since it allows the generator "hide" information about the condition in synthesized samples. The generator thus tricks the discriminator into producing a much better reconstruction of the condition for fake samples without the need for them to be realistic. Since the controller prevents the discriminator from pushing fake samples to a higher energy than real samples (i.c. the controller output is zero when fake samples have higher energy) this behaviour is not prohibited BEGANs throughout training.
The method used in DINO however does not have this problem since it encourages the discriminator to assign higher energies to unrealistic samples thus penalizing them and preventing the generator from "cheating" in the same way BEGAN. To show this effect train conditional BEGAN
15

---

Published as a conference paper at ICLR 2021
and the DINO framework to perform translation from photo to sketch using the APDrawings dataset from (Yi et al. 2019). Figure IO shows how the balancing used in DINO allows the network to penalize unrealistic images by encouraging the discriminator to assign them energies larger than the real samples. We note that this problem occurs only in cases where the source domain is more informative than the target domain (i.e. photo sketch). This does not occur in cases where the source domain in more generic than the target domain (i.e. segmentation map + photo)
Fabe Recommaacio Grar [Coerol DutuO)
0.2
0.33
0.55
Test Set Images Target Source
Target Source
Set
Thining Progress Oheest
Thiriing Pregrou Boe stiona)
(a) DINO
(b) Conditional BEGAN
Figure 10: Example showing how the balancing used in BEGAN fails to produce realistic images. This happens because BEGAN does not incentivize the discriminator to assign larger energies to the fake samples. The figures below the plots shows how the performance on the test set progresses during training for each method.
16

---

Published as a conference paper at ICLR 2021
A.5
QUALITATIVE RESULTS
A.5.1
IMAGE-TO-IMAGE TRANSLATION
CelebAMask-HI
Examples of image-to-image translation from segmentation maps to photos for the CelebMask-HQ dataset are shown in Figure [I] We note that our approach able to maintain semanties and produce realistic results even in cases with extreme head poses and facial expressions.
DINO (2-way) DINO((2-way)) Pabol
Labels
Photo
Pix2Pix
BycleGAN
DINO
Photo LLabel
B
Figure 11: Image translation performed the CelebAMask-HO dataset. Translation is performed the direction label + photo.
17

---

Published as a conference paper at ICLR 2021
Cityscapes
Examples of image-to-ime -mage translation from segmentation maps to photos for the Cityscapes dataset are shown in Figure 12
(2-way) Label Photoo
DINO((2-way)) Photo-LLbell
Labels
Photo
Pix2Pix
BycleGAN
SPADE
DINO
Figure 12: Image translation performed on the cityscapes dataset. Translation performed in the direction label + photo.
18

---

Published as a conference paper at ICLR 2021
A.6 HIDEO-TO-SPEECH TRANSLATION
This section presents examples of waveforms produced by the methods compared in Table 3 addition to the waveforms also present their corresponding spectrograms. The waveforms and spectrograms are shown in Figure [3] It is evident from the shape of the waveform that our method more accurately captures voiced sections in the audio. Furthermore, the spectrogram our method is closely resembles that of the ground truth although some high frequency components are not captured. The performance similar the Perceptual GAN proposed by VVougouuas et al. al. (2019) although our method relies on only an adversarial loss.
In
0000
1000
CycleGAN
6000
Magnitude
0.5
0
2000
1000
-1
0.5
1.5
2.5
3
Perceptual GAN Vougioukas (2019)
2030
8033
Magnitude
0.5
2030
-0.5
2030
1030
0.5
2
2.5
a330
Real Audio
Magnitude
0.5
5928
4038
3928
-0.5
-1
0.5
1.5
23
2
0038
Magnitude
0.5
5928
DINO
-0.5
1038
0
0.5
1.5
2
2.5
3
8320
Conditional WaveGAN
Magnitude
0.5
5338
4338
-0.5
-1
05
1.5
2.5
0.5
1.5
2
2.5
3
Time
Figure 13: Examples generated waveforms and their respective spectrograms. The real waveform and spectrogram is presented for comparison.
19


Reviews:
[{"review_id": "LG9-ry7s8se", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method for performing cross-domain or cross-modality translation models using a GAN-flavored framework where two models are trained to translate in both directions simultaneously.\n\n[As a caveat: I am not well-versed in this area of the literature]\n\nThe paper is well-written for the most part and the experimental results are promising. My main concern are the ethical implications of some of the lip-reading experiments which go unaddressed.\n\n**Pros**\n- Clear presentation (mostly, see remarks for some exceptions)\n- Good results as far as I can tell (although it is hard to interpret what all the various metrics mean, but it does seem like DINO is consistently better than the alternatives along most metrics)\n- Experiments are not limited to image to image translation but also to \"cross-modality translation\" (image to text)\n\n**Cons**\n- If I understand correctly, the video->speech task is essentially a lip-reading task. State-of-the-art lip reading raises a number of privacy related concerns, and I think the potential impact of this research should at least be acknowledged in the paper.\n\n\n**Remarks**\n- The proposed method bears some conceptual similarity with recent work in unsupervised machine translation (see eg. Artetxe et al. https://arxiv.org/abs/1710.11041, Lample et al. https://arxiv.org/abs/1711.00043, also Lample et al. 2019 https://openreview.net/pdf?id=H1g2NhC5KQ which is particularly relevant as it tackles \"style transfer\" for text). These similarities are worth mentioning in the paper.\n- I found the use of the term \"discriminator\" confusing, especially in the beginning in the paper. It makes sense in the usual GAN setup where the discriminator is an actual discriminative model, but it seems inappropriate in this case where the \"discriminator\" is a generative model.\n- Eq. 6 (the main objective) is incredibly confusing. First, (and this relates to my last remark) the notation D_disc, D_gen, G_disc, G_gen unnecessarily confounding. Consider using \"s -> t\" and \"t->s\" for source to target and vice-versa instead of G and D. Also, perhaps color-coding Eq.6 would make it easier to parse (or maybe just separate the different terms a bit more).", "extra_scores": {}, "extra_text": {}}, {"review_id": "H_Mlpd34jJ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a conditional energy-based GAN technique for translation between data domains.\n\nFirst, let me preface this review by noting that this paper is far outside of my area of expertise. I have tried to do my best in reviewing it, but I'd appreciate any clarification of mistaken points from the authors.\n\nOverall, the idea itself seems reasonable: in conditional GAN-based models, instead of using a discriminator that explicitly tries to predict whether the generated output is true or fake, the discriminator tries to maximize the reconstruction score of true outputs, and minimize it for fake outputs.\n\nHowever, there were many questions I had based on just reading the paper. I'm not sure whether this is due to my lack of background knowledge in this field, or because the writing itself is unclear (perhaps a bit of both):\n\n1. First big question: from my reading it seems that this is a *supervised* model, in that it needs $(x,y)$ pairs to calculate the objective in Equation (2). Is this correct? It doesn't seem to be explicitly stated anywhere.\n\n2. Given this, I was not sure if the baselines in Table 1 and 2 actually represent the state-of-the-art in this domain. Pix2pix seems to be from 2017, which seems to be quite old given the huge progress this field has made in the past 3 years. BiCycleGAN, according to my understanding, is an unsupervised method, which presumably will do much worse than supervised methods.\n\n3. Are the datasets in 4.1 standard and used in the literature? If so, what are some recent papers that evaluate on these datasets? If not, why use these datasets instead of others? While I'm not very familiar with the field, I do know that image style transfer is a big thing, and surely there are other datasets that people have evaluated on previously.\n\n4. The description of Equation (1) was a bit hard to follow, as the role of the discriminator was not made explicit. The \"margin loss\" was also not explained concretely.\n\n5. I was not able to understand the description of $\\gamma$ in Equation (3), please elaborate if possible. The gain value $\\lambda$ was also not easy to follow.\n\n6. It was mentioned that MirrorGAN is the most similar method. While there was an explanation that DINO is simpler than MirrorGAN, it would be nice to explain the implications of this. Does this just mean that DINO is a bit easier to implement? Or does it mean that it is fundamentally applicable to a wider variety of tasks? Also, why is there no empirical comparison with MirrorGAN?", "extra_scores": {}, "extra_text": {}}, {"review_id": "EnmRyVteYmm", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an adversarial framework DINO to train translation models from source to target and target to source. The basic idea is to replace generator and discriminator in the energy based GAN with two source-to-target generation models. The discriminator(reverse generator) and the generator competes in a minimax game to reconstruct the data. The framework is further extended with duplicate output heads for both discriminator and generator to enhance the training robustness. \nThe authors evaluated their framework on two tasks: image to image translation and silent-video to speech reconstruction. The DINO method impressive improvement in both tasks. \n\nStrong points:\n1. The proposed DINO framework is well motivated. The objectives in DINO are reasonable and novel.\n2. Experiments on both image to image translation and video to speech reconstruction verify the DINO method achieves significant improvement comparing with other translation methods. \n\nWeak points:\n1. Important details are omitted in image-to-image translation and video to speech reconstruction. It is unclear about the backbone network as well as the parameter setup. Therefore, it is impossible to reproduce the method. \n2. DINO and DINO(bidirectional) are not consistent winners. It is not explained or analyzed why DINO sometimes wins while DINO(bidirectional) wins otherwise. There is no recommendation for practical use either. \n3. The adaptive balancing seems reasonable. But it is not studied in the experiment whether it improves the training. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a novel method for general-purpose supervised domain transfer that trains both generator and discriminator to compete in a minimax game in order to reconstruct data. This setup is meant to address a common issue in conditional GAN setups: they often ignore conditioning information. Results are positive and span two very different tasks: image-to-image translation and silent-video-to-speech reconstruction. Overall reviewers were quite positive about this paper: they found the method to be novel and well-motivated, and after rebuttal, found experimental results to be sufficiently convincing. Several concerns were brought up: (a) lack of emphasis that the approach is in fact supervised, (b) need for comparisons with stronger or task-specific baselines, (c) lack of description of experimental details for reproducibility, and (d) lack of discussion of ethical implications. All of these concerns were satisfactorily addressed by authors in rebuttal and reviewers unanimously vote for acceptance. I agree, and recommend this paper be accepted. 
Decision: Accept (Poster)