TRIP: Refining Image-to-Image Translation via Rival Preferences

1 INTRODUCTION . Image-to-image ( I2I ) translation ( Isola et al. , 2017 ) aims to translate an input image into the desired ones with changes in some specific attributes . Current literature can be classified into two categories : binary translation ( Zhu et al. , 2017 ; Kim et al. , 2017 ) , e.g. , translating an image from “ not smiling ” to “ smiling ” ; fine-grained translation ( Lample et al. , 2017 ; He et al. , 2019 ; Liu et al. , 2018 ; Saquil et al. , 2018 ) , e.g. , generating a series of images with smooth changes from “ not smiling ” to “ smiling ” . In this work , we focus on the high-quality fine-grained I2I translation , namely , generate a series of realistic versions of the input image with smooth changes in the specific attributes ( See Fig . 1 ) . Note that the desired high-quality images in our context are two folds : first , the generated images look as realistic as training images ; second , the generated images are only modified in terms of the specific attributes . Relative attribute ( RA ) , referring to the preference of two images over the strength of the interested attribute , is widely used in the fine-grained I2I translation task due to their rich semantic information . Previous work Ranking Conditional Generative Adversarial Network ( RCGAN ) ( Saquil et al. , 2018 ) adopts two separate criteria for a high-quality fine-grained translation . Specifically , a ranker is adopted to distill the discrepancy from RAs regarding the targeted attribute , which then guides the generator to translate the input image into the desired one . Meanwhile , a discriminator ensures the generated images as realistic as the training images . However , the generated fine-grained images guided by the ranker are out of the real data distribution , which conflicts with the goal of the discriminator . Therefore , the generated images can not maintain smooth changes and suffer from low-quality issues . RelGAN ( Wu et al. , 2019 ) applied a unified discriminator for the high-quality fine-grained translation . The discriminator guides the generator to learn the distribution of triplets , which consist of pairs of images and their corresponding numerical labels ( i.e. , relative attributes ) . Further , RelGAN adopted the fine-grained RAs within the same framework to enable a smooth interpolation . However , the joint data distribution matching does not explicitly model the discrepancy from the RAs and fails to capture sufficient semantic information . The generated images fail to change smoothly over the interested attribute . In this paper , we propose a new adversarial ranking framework consisting of a ranker and a generator for high-quality fine-grained translation . In particular , the ranker explicitly learns to model the discrepancy from the relative attributes , which can guide the generator to produce the desired image from the input image . Meanwhile , the rival preference consisting of the generated image and the input image is constructed to evoke the adversarial training between the ranker and the generator . Specifically , the ranker can not differentiate the strength of the interested attribute between the generated image and the input image ; while the generator aims to achieve the agreement from the ranker that the generated image holds the desired difference compared to the input . Competition between the ranker and the generator drives both two modules to improve themselves until the generations exhibit desired preferences while possessing high fidelity . We summarize our contributions as follows : • We propose Translation via RIval Preference ( TRIP ) consisting of a ranker and a generator for a high-quality fine-grained translation . The rival preference is constructed to evoke the adversarial training between the ranker and the generator , which enhances the ability of the ranker and encourages a better generator . • Our tailor-designed ranker enforces a continuous change between the generated image and the input image , which promotes a better fine-grained control over the interested attribute . • Empirical results show that our TRIP achieves the state-of-art results on the fine-grained imageto-image translation task . Meanwhile , the input image can be manipulated linearly along the strength of the attribute . • We further extend TRIP to the fine-grained I2I translation of multiple attributes . A case study demonstrates the efficacy of our TRIP in terms of disentangling multiple attributes and manipulating them simultaneously . 2 RELATED WORKS . We mainly review the literature related to fine-grained I2I translation , especially smooth facial attribute transfer . We summarized them based on the type of generative models used . AE/VAE-based methods can provide a good latent representation of the input image . Some works ( Lample et al. , 2017 ; Liu et al. , 2018 ; Li et al. , 2020 ; Ding et al. , 2020 ) proposed to disentangle the attribute-dependent latent variable from the image representation but resorted to different disentanglement strategies . Then the fine-grained translation can be derived by smoothly manipulating the attribute variable of the input image . However , the reconstruction loss , which is used to ensure the image quality , can not guarantee a high fidelity of the hallucinated images . Flow-based Some works ( Kondo et al. , 2019 ) incorporates feature disentanglement mechanism into flow-based generative models . However , the designed multi-scale disentanglement requires large computation . And the reported results did not show satisfactory performance on smooth control . GAN-based GAN is a widely adopted framework for a high-quality image generation . Various methods applied GAN as a base for fine-grained I2I translation through relative attributes . The main differences lie in the strategies of incorporating the preference over the attributes into the image generation process . Saquil et al . ( 2018 ) adopted two critics consisting of a ranker , learning from the relative attributes , and a discriminator , ensuring the image quality . Then the combination of two critics is supposed to guide the generator to produce high-quality fine-grained images . However , the ranker would induce the generator to generate out-of-data-distribution images , which is opposite to the target of the discriminator , thereby resulting in poor-quality images . Wu et al . ( 2019 ) applied a unified discriminator , which learns the joint data distribution of the triplet constructed with a pair of images and a discrete numerical label ( i.e. , relative attribute ) . However , such a joint distribution modeling approach only models the discrete discrepancy of the RAs , which fails to generalize to the continuous labels very well . Rather than using RAs , He et al . ( 2019 ) directly modeled the attribute with binary classification , which can not capture detailed attribute information , and hence fail to make a smooth control over the attributes . Deng et al . ( 2020 ) embeded 3D priors into adversarial learning . However , it relies on available priors for attributes , which limits the practicality . Alharbi and Wonka ( 2020 ) proposed an unsupervised disentanglement method . It injects the structure noises to GAN for controlling specific parts of the generated images , which makes global or local features changed in a disentangled way . However , it is unclear how global or local features are related to facial attributes . Thus , it is difficult to change specific attributes . Our method is based on GAN . To ensure good control over the target attribute , the critic in GAN should transfer the signal about the subtle difference over the target attribute to the generator . Previous methods model it as two sequential processes . Namely , they capture the subtle difference over attribute using a classification model or a ranking model , and count on the learned attribute model to generalize learned attribute preference to the unseen generated images through interpolation . However , the learned attribute model never meets our expectation , since they haven ’ t seen the generated images at all during its training . As for our TRIP , we consider introducing the generated image into the training process of the attribute model , i.e. , the ranker . Since the supervision over the generated images is not accessible , we formulate the ranker into an adversarial ranking process using the constructed rival preference , following the adversarial training of vanilla GAN . Consequently , our ranker ( the attribute model ) can critic the generated image during its whole training process , and it no doubt can generalize to generated images to ensure sufficient fine-grained control over the target attribute . 3 TRIP FOR FINE-GRAINED IMAGE-TO-IMAGE TRANSLATION . In this section , we propose a new model , named TRanslation via Rival Preferences ( TRIP ) for high-quality fine-grained image-to-image ( I2I ) translation , which learns a mapping that translates an input image to a set of realistic output images by smoothly controlling the specific attributes . The whole structure of TRIP is shown in Fig . 2 , which consists of a generator and a ranker . The generator takes as input an image along with a continuous latent variable that controls the change of the attribute , and outputs the desired image ; while the ranker provides information in terms of image quality and the preference over the attribute , which guides the learning of the generator . We implement the generator with a standard encoder-decoder architecture following Wu et al . ( 2019 ) . In the following , we focus on describing the detailed design of the ranker and the principle behind it . 3.1 RANKER FOR RELATIVE ATTRIBUTES . Relative attributes ( RAs ) are assumed to be most representative and most valid to describe the information related to the relative emphasis of the attribute , owing to its simplicity and easy construction ( Parikh and Grauman , 2011 ; Saquil et al. , 2018 ) . For a pair of images ( x , y ) , RAs refer to their preference over the specific attribute : y x when y shows a greater strength than x on the target attribute and vice versa . Pairwise learning to rank is a widely-adopted technique to model the relative attributes ( Parikh and Grauman , 2011 ) . Given a pair of images ( x , y ) and its relative attribute , the pairwise learning to rank technique is formulated as a binary classification ( Cao et al. , 2006 ) , i.e. , R ( x , y ) = { 1 y x ; −1 y ≺ x , ( 1 ) where R ( x , y ) is the ranker ’ s prediction for the pair of images ( x , y ) . x y R +1/-1 Figure 3 : The ranker model . Further , the attribute discrepancy between RAs , distilled by the ranker , can then be used to guide the generator to translate the input image into the desired one . However , the ranker is trained on the real image pairs , which only focuses on the modeling of preference over the attribute and ignores image quality . To achieve the agreement with the ranker , the generator possibly produces unrealistic images , which conflicts with the goal of the discriminator . 3.2 RIVAL PREFERENCES ENHANCING THE RANKER . According to the above analysis , we consider incorporating the generated image pairs into the modeling of RAs , along with the real image pairs to reconcile the goal of the ranker and the discriminator . Meanwhile , the resultant ranker will not only generalize well to the generated pairs but also avoid providing untrustworthy feedback by discriminating the unrealistic images . Motivated by the adversarial training of GAN , we introduce an adversarial ranking process between a ranker and a generator to incorporate the generated pairs into the training of ranker . To be specific , • Ranker . Inspired by semi-supervised GAN ( Odena , 2016 ) , we assign a pseudo label to the generated pairs . In order to avoid a biased influence on the ranking decision over real image pairs , i.e. , positive ( +1 ) or negative ( -1 ) , the pseudo label is designed as zero . Note that the generated pair consists of a synthetic image and its input in order to connect the ranker prediction to the controlling latent variable . R ( x , ∆ ) = { +1 ∆ = y ∧ y x ; −1 ∆ = y ∧ y ≺ x ; 0 ∆ = ŷ . ( 2 ) where ŷ denotes the output of the generator given the input image x and v , i.e. , ŷ = G ( x , v ) . ∆ is a placeholder that can either be a real image y or be a generated image ŷ . • Generator . The goal of the generator is to achieve the consistency between the ranking prediction R ( x , ŷ ) and the corresponding latent variable v. When v > 0 , the ranker is supposed to believe that the generated image ŷ has a larger strength of the specific attribute than the input x , i.e. , R ( x , ŷ ) = +1 ; and vice versa . R ( x , ŷ ) = { +1 v > 0 ; −1 v < 0 . ( 3 ) We denominate the opposite goals between the ranker and the generator w.r.t . the generated pairs as rival preferences1 . An intuitive example of the rival preference is given in Fig . 4 for better understanding . The ranker is promoted in terms of the following aspects : ( 1 ) The function of the ranker on the real image pairs is not changed . The generated pairs are uniformly sampled regarding their latent variables . By assigning label zero , the ranking information implied within the pairs is neutral- ized to maintain the ranking performance on the real image pairs . ( 2 ) The ranker avoids providing biased ranking prediction for unrealistic image pairs . As we constrain the generated pairs at the decision boundary , i.e , R ( x , ŷ ) = 0 , the ranker is invariant to the features specified by the generated 1 “ rival ” means adversarial . We use it to distinguish it from adversarial training in the community . pairs ( Chapelle et al. , 2008 ) , suppressing the influence of the unrealistic features on the ranking decision . ( 3 ) The ranker can capture the exclusive difference over the specific attribute through the adversarial process . Since the ranker rejects to give effective feedback for unrealistic image pairs , only the realistic image pairs can attract the attention of the ranker . Therefore , the ranker only passes the effective information related to the target attribute to the generator . Then , we introduce a parallel head following the feature layer to ensure the image quality together with a rank head , shown in Fig . 2 . According to the above analysis , the ranker would not evoke conflict with the goal of the image quality . Therefore , we successfully reconcile the two goals of image quality and the extraction of the attribute difference . With a powerful ranker , the generator would “ win ” the adversarial game by producing the realistic pairs consistent with the latent variable . Remark 1 ( Assigning zero to similar real image pairs ) . It is natural to assign zero to pairs { ( x , y ) |y = x } , where = denotes that x and y have same strength in the interested attribute . They can improve the ranking prediction ( Zhou et al. , 2008 ) .

Summary: The authors proposed in this paper a supervised approach relying on given relative and quantitative attribute discrepancies. A UNet-like generator learns adversarially tends to generate realistic images while a "ranker" tends to predict the magnitude of the input parameter used to control the image manipulation. The controled parameter is defined implictly using images whose the discrepancy of the attribute of interest is known. This allows fine-grained manipulation of the attribute of interest. The results of the approach is illustrated on face datasets (CelebA-HQ and LFWA).