TRIP: Refining Image-to-Image Translation via Rival Preferences
Authors: Yinghua Yao
Yuangang Pan
Ivor Tsang
Xin Yao
Venue:  (2021)
Keywords: Fine-grained image-to-image translation
GAN
relative attributes
ranker

Abstract: We propose a new model to refine image-to-image translation via an adversarial ranking process. In particular, we simultaneously train two modules: a generator that translates an input image to the desired image with smooth subtle changes with respect to some specific attributes; and a ranker that ranks rival preferences consisting of the input image and the desired image. Rival preferences refer to the adversarial ranking process: (1) the ranker thinks no difference between the desired image and the input image in terms of the desired attributes; (2) the generator fools the ranker to believe that the desired image changes the attributes over the input image as desired. Real image preferences are introduced to guide the ranker to rank image pairs regarding the interested attributes only. With an effective ranker, the generator would “win” the adversarial game by producing high-quality images that present desired changes over the attributes compared to the input image. The experiments demonstrate that our TRIP can generate high-fidelity images which exhibit smooth changes with the strength of the attributes.

Under review as a conference paper at ICLR 2021
TRIP: REFINING IMAGE-TO-IMAGE TRANSLATION VIA RIVAL PREFERENCES
Anonymous authors Paper under double-blind review
ABSTRACT
We propose a new model to refine image-to-image translation via an adversarial ranking process. In particular, we simultaneously train two modules: a generator that translates an input image to the desired image with smooth subtle changes with respect to some specific attributes; and a ranker that ranks rival preferences consisting of the input image and the desired image. Rival preferences refer to the adversarial ranking process: (I) the ranker thinks no difference between the desired image and the input image in terms of the desired attributes; (2) the generator fools the ranker to believe that the desired image changes the attributes over the input image desired. Preferences over pairs of real images are introduced to guide the ranker to rank image pairs regarding the interested attributes only. With an effective ranker, the generator would "win" the adversarial game by producing high-quality images that present desired changes over the attributes compared to the input image. The experiments demonstrate that our TRIP can generate high-fidelity images which exhibit smooth changes with the strength of the attributes.
INTRODUCTION
Image-to-image (12I) translation (Isola et al. (017) aims to translate an input image into the desired ones with changes in some specific attributes. Current literature can be classified into two categories: binary translation (Zhu et al. [017]] Kim et al., 2017) e.g., translating an image from "not smiling" to "smiling"; fine- tee-grained translation (Lample al. [017] He al., 2019 [iu et al., 2018 (Saquil] et al. 2018), e.g., generating a series of images with smooth changes from "not smiling" to "smiling". In this work, we focus on the high-quality fine-grained 12T translation, namely, generate a series of realistic versions of the input image with smooth changes in the specific attributes (See Fig, [.. Note that the desired high-quality images in our context are two folds: first, the generated images look as realistic as training images; second, the generated images are only modified in terms of the specific attributes.
Relative attribute (RA), referring to the preference of two images over the strength of the interested attribute, is widely used in the fine-grained 12I translation task due to their rich semantic information. Previous work Ranking Conditional Generative Adversarial Network (RCGAN) (Saquil al., 2018) adopts two separate criteria for a high-quality fine-grained translation. Specifically, a ranker is adopted distill the discrepancy from RAs regarding the targeted attribute, which then guides the generator to translate the input image into the desired one. Meanwhile, a discriminator ensures the generated images as realistic as the training images. However, the generated fine-grained images guided by the ranker are out of the real data distribution, which conflicts with the goal of the discriminator. Therefore, the generated images cannot maintain smooth changes and suffer from low-quality issues. RelGAN (Wu et al. 2019) applied a unified discriminator for the high-quality fine-grained translation. The discriminator guides the generator to learn the distribution of triplets, which consist of pairs of images and their corresponding numerical labels (i.c.. relative attributes). Further, ReIGAN adopted the fine-grained RAs within the same framework to enable a smooth interpolation. However, the joint data distribution matching does not explicitly model the discrepancy from the RAs and fails to capture sufficient semantic information. The generated images fail to change smoothly over the interested attribute.
In this paper, we propose a new adversarial ranking framework consisting of a ranker and a generator for high-quality fine-grained translation. In particular, the ranker explicitly learns to model the

---

Under review as a conference paper at ICLR 2021
Less smiling
Inpur
More smiling
Figure 1: Fine-grained image-to-imant translation on the "smile" attribute (generated by our TRIP).
discrepancy from the relative attributes, which can guide the generator to produce the desired image from the imput image. Mcanwhile, the rival preference consisting of the generated image and the input image is constructed to evoke the adversarial training between the ranker and the generator. Specifically, the ranker cannot differentiate the strength of the interested attribute between the gener- ated image and the input image; while the generator aims to achieve the agreement from the ranker that the generated image holds the desired difference compared to the input. Competition between the ranker and the generator drives both two modules to improve themselves until the generations exhibit desired preferences while possessing high fidelity. We summarize our contributions as follows:
We propose Translation via RIval Preference (TRIP) consisting of ranker and a generator for high-quality fine-grained translation. The rival preference is constructed to evoke the adversarial training between the ranker and the generator, which enhances the ability of the ranker and encourages a better generator.
Our tailor-designed ranker enforces a continuous change between the generated image and the input image, which promotes a better fine-grained control over the interested attribute.
Empirical results show that our TRIP achieves the state-of-art results on the fine- gra-grained image- to-image translation task. Meanwhile, the input image can be manipulated linearly along the strength of the attribute.
We further extend TRIP to the fine-gr mne-grained 12I translation of multiple attributes. A case study demonstrates the efficacy of our TRIP in terms of disentangling multiple attributes and manipulat- ing them simultaneously.
RELATED WORKS
We mainly review the literature related fine-grainee 121 translation, especially smooth facial attribute transfer. We summarized them based on the type of generative models used.
AE/VAE-based methods can provide a good latent representation of the input image. Some works (Lample et al. 2017 Liu al. 2018 Li al. 2020 Ding et al. 2020) proposed to dis- entangle the serbbute-dependen latent varialle from the image representation but resorted to different disentanglement strategies. Then the fine- tne-grained translation can be derived by smoothly manipu- lating the attribute variable of the input image. However, the reconstruction loss, which is used to ensure the image quality, cannot guarantee high fidelity of the hallucinated images.
Flow-based Some works (Kondo et al.] 2019) incorporates feature disentanglement mechanism into flow-based generative models. However, the designed multi-scale disentanglement requires large computation. And the reported results did not show satisfactory performance on smooth control.
GAN-based GAN is widely adopted framework for a high-quality image generation. Various methods applied GAN as a base for fine-grained 12I translation through relative attributes. The main differences lie in the strategies of incorporating the preference over the attributes into the image generation process. [Saquil et al. (2018) adopted two critics consisting of a ranker, learning from the relative attributes, and a discriminator, ensuring the image quality. Then the combination of two critics is supposed to guide the generator to produce high-quality fine-grained images. However, the images, which is opposite to
ranker would induce the generator to generate out-of-datt-distributee the target of the discriminator, thereby resulting in poor-quality images. Wu et al. (011 applied unified discriminator, which learns the joint data distribution of the triplet constructed with a pair of images and discrete numerical label (i.e.. relative attribute). However, such a joint distribution modeling approach only models the discrete discrepancy of the RAs, which fails to generalize to the continuous labels very well. Rather than using RAs, He et al. 2019 directly modeled the attribute
2

---

Under review as a conference paper at ICLR 2021
Ranker
Rank Head
Generator
Concat
Down- Sampling
Up Samplin
Feature Layer
GAN
Heed
The network structure of TRIP. ! and ② denotes different image pairs, i.e., real image pairs and generated image pairs, corresponding to Fig. and Fig. respectively. R and D denotes the rank head and the GAN head, respectively.
Figure 2:
with binary classification, which cannot capture detailed attribute information, and hence fail make a smooth control over the attributes. Deng et al. (2020) embeded 3D priors into adversarial learning. However, it relies available priors for attributes, which limits the practicality. Alharbi and Wonka (2020) proposed an unsupervised disentanglement method. It injects the structure noises to GAN for controlling specific parts of the generated images, which makes global local features changed in disentangled way. However, it is unclear how global or local features are related to facial attributes. Thus, is difficult to change specific attributes.
Our method is based on GAN. To ensure good control over the target attribute, the critic in GAN should transfer the signal about the subtle difference over the target attribute the generator. Previous methods model it as two sequential processes. Namely, they capture the subtle difference over attribute using a classification model or a ranking model, and count on the learned attribute model generalize learned attribute preference to the unseen generated images through interpolation. However, the learned attribute model never meets our expectation, since they haven't seen the generated images at all during its training. As for our TRIP, consider introducing the generated image into the training process of the attribute model, i.e. the ranker. Since the supervision over the generated images is not accessible, formulate the ranker into adversarial ranking process using the constructed rival preference, following the adversarial training of vanilla GAN. Consequently, our ranker (the attribute model) critic the generated image during its whole training process, and it doubt generalize to generated images to ensure sufficient fine-grained control over the target attribute.
3 TRIP FOR FINE-GRAINED IMAGE-TO-IMAGE TRANSLATION
In this section, we propose a new model, named TRanslation via Rival Preferences (TRIP) for high-quality fine-grained image-to-image (I2I) translation, which learns mapping that translates an input image to a set of realistic output images by smoothly controlling the specific attributes.
The whole structure of TRIP is shown in Fig. 2 which consists of generator and ranker. The generator takes as input an image along with a continuous latent variable that controls the change of the attribute, and outputs the desired image; while the ranker provides information in terms of image quality and the preference over the attribute, which guides the learning of the generator. We implement the generator with standard encoder-decoder architecture following Wu al. (2019). In the following, we focus on describing the detailed design of the ranker and the principle behind it.
3.1 RANKER FOR RELATIVE ATTRIBUTES
Relative attributes (RAs) are assumed to be most representative and most valid to describe the information related to the relative emphasis the attribute, owing to its simplicity and casy construc- tion (Parikh and Grauman, 2011 Saquil et al. 2018). For pair of images (x,y), y), RAs refer their preference over the specific attribute: y > x when y shows a greater strength than x on the target attribute and vice versa.
Pairwise learning to rank is a Grauman 2011). Given a pair of images (x, y) and its relative attribute, the pairwise learning rank
technique to model the relative attributes Pariikh and
3

---

Under review as a conference paper at ICLR 2021
technique is formulated as binary classification (Cao et al. 2006), i...
1 y -1 x X,
R(x,y)
where R(x,y y) is the ranker's prediction for the pair of images (x,y). Further, the attribute discrepancy between RAs, distilled by the ranker, can then be used guide the generator to translate the input image into the desired one.
X
+1/11
However, the ranker is trained the real image pairs, which only focuses on the modeling of preference over the attribute and ignores image quality. To achieve the agreement with the ranker, the gener- ator possibly produces unrealistic images, which conflicts with the goal of the discriminator.
Figure 3: The ranker model.
3.2 RIVAL PREFERENCES ENIANNCNNG THE RANKER
According the above analysis, we consider incorporating the generated image pairs into the modeling of RAs, along with the real image pairs to reconcile the goal of the ranker and the discriminator. Meanwhile, the resultant ranker will not only generalize well to the generated pairs but also avoid providing untrustworthy feedback by discriminating the unrealistic images.
Motivated by the adversarial training of GAN, we introduce an adversarial ranking process between ranker and generator to incorporate the generated pairs into the training of ranker. To be specific,
Ranker, Inspired by semi-supervised GAN (Odena 2016), we assign a pseudo label to the generated pairs. In order to avoid biased influence the ranking decision over real image pairs, i.e., positive (+1) or negative (-1), the pseudo label is designed as zero. Note that the generated pair consists of synthetic image and its input in order connect the ranker prediction the controlling latent variable.
+1 =
vAyy xx A Ay x A
R(x, A)
-1
where ý denotes the output of the generator given the input image x and v, i.e., ÿ = G(x, v). A is a placeholder that can either be a real image y or generated image y.
Generator. The goal of the generator is achieve the consistency between the ranking prediction R(x,ý) and the corresponding latent variable v. When v > 0, the ranker is supposed to believe that the generated image has larger strength of the specific attribute than the input x, i.e..
R(x,ý) +1; and vice versa.
+1
O; O.
R(x, y)
(3)
We denominate the opposite goals between the ranker and the generator w.r.t. the generated pairs as rival prefer- ence An intuitive example of the rival preference is given in Fig. for better understanding.
X
R
G
The ranker is promoted in terms of the following aspects: (1) The function of the ranker on the real image pairs is not changed. The generated pairs uniformly sampled regarding their latent variables. By assigning label zero, the ranking information implied within the pairs is neutral- ized to maintain the ranking performance on the real image pairs. (2) The ranker avoids providing biased ranking prediction for unrealistic image pairs. As we constrain the generated pairs at the decision boundary, i.c, R(x, ý) O, the ranker is invariant to the features specified by the generated
v
V
Figure 4: Rival Preferences for the gen- erated pairs between the ranker and the generator. V denotes [-1, U (0. 1].
"rival" means adversarial. We distinguish from adversarial training in the community.
4

---

Under review as a conference paper at ICLR 2021
pairs (Chapelle et al. 2008), suppressing the influence of the unrealistic features the ranking decision. (3) The ranker can capture the exclusive difference over the specific attribute through the adversarial process. Since the ranker rejects to give effective feedback for unrealistic image pairs, only the realistic image pairs can attract the attention of the ranker. Therefore, the ranker only passes the effective information related to the target attribute the generator.
Then, we introduce a parallel head following the feature layer to ensure the image quality together with a rank head, shown in Fig. According the above analysis, the ranker would not evoke conflict with the goal of the image quality. Therefore, we successfully reconcile the two goals of image quality and the extraction of the attribute difference. With a powerful ranker, the generator would "win" the adversarial game by producing the realistic pairs consistent with the latent variable.
Remark 1 (Assigning zero to similar real image pairs). It is natural assign zero to pairs ylyy = x]. where denotes that x and have same strength in the interested allribute. They can improve the ranking prediction (Zhou el al. 2008).
3.3 LINEARIZING THE RANKING OUTPUT
Equation 3 models the relative attributes of the generated pairs binary classification, which fails to enable a fine-grann enn-grained translation since the subtle changes implied by the latent variable are not distinguished by the ranker. For example, given 211 1/2 > O, the ranker give same feedbacks for (x, V1) and (x. are both +1, which loses the discrimination between the two pairs. To achieve the fine-grained translation, we linearize the ranker's output for the generated pairs as to align the ranker prediction with the latent variable. We thus reformulate the binary classification as the regression:
R(x, y) V.
(4)
Note that the output of the ranker can reflect the difference in pair of images. Given two latent variables > 1/ > t1 > O, the ranking predictions for the pair generated from i2 should be larger than that from 1 i.e., 1 >>> R(x, V2) > R(x, v1) ) > O. The ranker's 's outputs for the generated pairs would be linearly correlated to the corresponding latent variable. Therefore, the generated output images can change smoothly over the input image according to the latent variable.
3.4 TRANSLATION VIA RIVAL PREFERENCES (TRIP)
In the following, we introduce the loss functions for the two parallel heads in the ranker. The overall network structure can be seen in Fig.
Loss of rank head R: we adopt the least square loss for the ranking predictions. The loss function for the ranker and the generator is defined as:
I yank = Exxxx..)))
(x - r)(] + (Epix)p()
[R G(x,))(())
(5a)
L = Ep(x)p(t) (R(x, G(x, v)) w)
(5b)
y xxx denotes the relative attribute. p(x, y, r) are the joint
where ÿ = u(x,v).
distribution of real image preferences. p(x) is the distribution of the training images. p(v) is a uniform distribution [-1, 1]. À is the weight factor that considers the rival preferences.
By optimizing Lann (equation 5a). the ranker is trained to predict correct labels for real image pairs and assign zero for generated pairs, i.e., equation 2 By optimizing Lianc Eqquation[5b), the generator is trained output the desire image ý. where the difference between y and is consistent with the latent variable U, i.e., equation
Loss of GAN head D: be consistent with the above rank head and also ensure stable training, a regular least square GAN's loss is adopted:
L gan Ep(x) (x())-11)) + Ep(xpp(o)
[DD v)) - 0]]
(6a)
I gan Ep(xct(()) [D(G(x,))) v)) 1)²
(6b)
5

---

Under review as a conference paper at ICLR 2021
where denotes the real image label while 0 denotes the fake image label.
Jointly training the rank head and the gan head, the gradients backpropagate through the shared feature layer to the generator. Thus our TRIP can conduct the high-quality fine-grained I2I translation.
3.5 EXTENDED TO THE MULTIPLE ATTRIBUTES
To generalize our TRIP to multiple (K) attributes, we use vectors v and with K dimension to denote the latent variable and the preference label, respectively. Each dimension controls the change of one of the interested attributes. In particular, the ranker consists one GAN head and parallel rank head. The overall loss function is summarized follows:
Lrank = B)(x...)
(R& (x, rz)²] + XEp())())) Σ
(Rt (x, G(x, -))
(7a)
Leank pp(x)p()
Σ (Re (x, G(x, v)) va) -v
(7b)
where Rg is the output of the k-th rank head. Va and rak are the k-th dimension of v and r, respectively.
EXPERIMENTS
In this section, we compare our TRIP with various baselines on the task of fine-grained image-to- image translation. We verify that our ranker can distinguish the subtle difference pair of images. Thus we propose to apply our ranker for evaluating the fine- tne-graininess of image pairs generated by various methods. We finally extend TRIP to the translation of multiple attributes.
Datasets. We conduct experiments on the high quality version of a subset from Celeb Faces Attributes Dataset (CelebA-HQ) (Karras et al. 2018 and Labeled Faces in the Wild with attributes (LFWA) (Liu et al. 2015). CelebA-HQ consists of 30K face images of celebrities, annotated with 40 binary attributes such as hair colors, gender and age. LFWA has 13, 143 images with 73 annotated binary attributes. We resize the images of two datasets to 256 x 256. The relative attributes are obtained for any two images x and y only based on the binary label of attributes. For instance, for "smiling" attribute, we construct the comparison label y is "not smiling", and vice versa. Therefore, we make a fair comparison with other baselines in terms of same supervision information.
y when the label of x is "smiling" while the
Implementation Details. As the translation is conducted on the unpaired setting, the cycle con- sistency loss Legele (Zhu et al. 2017) are usually introduced to keep the identity of faces when translation. An orthogonal loss L, and the gradient penalty loss Lg are added to stabilize the training following (Wu et al.) 2019). The weighting factor for Lyann Leyeles Lo and Lyp are Ag- Ae. Ao and Agp. respectively. Except Xg 0.5 for CelebA-HQ and Xg = 5 for LFWA, we set the same parameter for all datasets. Specifically, we set X 0.5. d 2.5, Agp = 150, Ao 10-6. We the Adam optimizer [23] with B1 0.5 and B2 = 0.999. The learning rate is set to le-5 for the ranker and 5e-5 for the generator. The batch size is set to 4. See appendix for details about the network architecture and the experiment setting.
Baselines. We compare TRIP with FN (Lample et al.) 2017), ReIGAN (Wu et al. RCGAN (Saquil al. 2018). We the released codes of FN, ReIGAN and RCGAN compare with AuGAN since RelGAN outperforms AtGAN, which is shown in (Wu et al. 2019).
2019) and
We did not
Evaluation Metries. Follow Wu et al. (2019), we use three metries quantitatively evaluate the performance of fine-grained translation. Standard deviation of structural similarity (SSIM) measures the fine- greiird translation. Frechet Inception Distance (FID) measures the visual quality. Accuracy of Attribute Swapping (AAS) evaluates the accuracy of the binary image translation. The swapping for the attribute is to translate an image, e.g., from "smiling" into "not smiling". The calculation can
be found in App.1 D

---

Under review as a conference paper at ICLR 2021
R
ReiGAN
R
R
R
RCGAN
R
S
R
S
S
TRIP
Input
0.8
0.6
0.4
0.2
0
0.2
0.4
v
0.6
V= 0.8
Figure 5: Comparison of fine- iin-grained facial attribute ("smile") translation on CelebA-HQ dataset.
Fine- Fine-grained (SSIM)
Image quality (FID)
Model
CelebA-HQ
LFWA
CelebA-HQ
LFWA
Smile Gender Mouth Cheekbones
Smiling Frown
Smile
Gender Mouth Cheekbones
Smiling
Frown
FN
0.0122 0.0036 0.0075
0.0039
0.0065 0.0005
41.48
48.66
42.79
43.15
12.58
11.37
RCGAN
0.0084 0.0138
0.01000
0.0079 0.0106
398.05
418.06
385.27
437.93
425.59
ReIGAN 0.0512 0.0924 0.0261
0.0510
0.0137
0.016
10.92
31.29
9.55
10.28
16.76
16.21
TRIP 0.0030 0.0077 0.0017
0.0028
0.0001
0.0005
10.19
26.47
7.18
9.50
22.25
23.65
Fine-grained performance (SSIM) and image quality (FID) of FN, RCGAN, RelGAN and TRIP on CelebA-HQ and LFWA. RCGAN fails to make a fine-grained translation w.r.t. the "mouth" attribute on CelebA-HQ dataset. So we did not collect result.
Table 1:
4.1 FINE-GRAINED IMAGE-TO- -O-GAGE TRANSLATION
We conduct fine-grained 12I translation on a single attribute. On CelebA-HQ dataset, we make the 12I translation in terms of "smile", "gender", "mouth open" and "high cheekbones" attributes. respectively. On LFWA dataset, we make the 12I translation in terms of "smile" and "Frown" attributes, respectively. We show that our TRIP achieves the best performance on the fine-grained 12I translation task comparing with various strong baselines.
Best visual results As shown in Fig. S (1) all GANs can translate the input image into "more smiling" when v > 0 or "less smiling" when V < O. The degree of changes is consistent with the numerical value of v. (2) Our GAN's generation shows the best visual quality, generating realistic output images that are different from the input images only in the specific attribute. In contrast, FN suffers from image distortion issues. RelGAN's generation not only changed the specific attribute "smile", but also other irrelevant attributes, "hair color". RCGAN exhibits extremely poor generation results.
Best fine-grained score We present the quantitative evaluation of the fine-grained translation in T Our TRIP achieves the lowest SSIM scores, consistent with the visual results. Note that trivial case to obtain a low SSIM is when the translation is failed. Namely, the generator would output the same value matter what the latent variable is. Therefore, we further apply AAS to evaluate the 12I translation in a binary manner. Most GANs achieve over 95% accuracy except for FN (See the App. Fig. [2). Under this condition, it guarantees that low SSIM indeed indicates the output images change smoothly with the latent variable.
Best image quality score Table presents the quantitative evaluation of the image quality. (1) Our TRIP achieves the best image quality with the lowest FID scores. (2) FN achieves the best FID on LFWA dataset. Because the FN achieves relatively low accuracy of the translation,
75% in Fig. I2 many generated images would be the same as the input image. It means that the statistics of the translated images are similar that of the input images, leading low FID. (3) RCGAN has the worst FID scores, consistent with the visual results in Fig.
4.2 PHYSICAL MEANING OF RANKER OUTPUT
From Fig. 5 and Table it shows that when conditioning on different latent variables, our TRIP can translate an input image into series of output images that exhibit the corresponding changes over
7

---

Under review as a conference paper at ICLR 2021
1.0
1.0
0.5
RCGAN
TRIP
Ronker oniput
0.5
ReGANN
TRIP
Ranker onlput
0.5
FN
TRIP
ReIGAN RCGAN
0.165
0.288
0:256
FN
0.157
TRIP
Banker
0.0
0.5
0.164
0.265
0.137
0.107
-0.5
0.5
0.5
0.
0.030
0.354
0.089
0.042
0.5
0.183
0.273
0.122
0.121
1. -0.5 0.0 0.5 1.0
-1.0
1.0
-0.5 0.0
0.5 Latent vanialle
1.0
1.0
-0.5 0.0
0.5
1.0
1.
0.172
0.290
0.222 0.159
Latemt variable
Latest variable
Figure 7: The first three subfigures plot the ranker's output for generated pairs in terms of different latent variables. The curve shows the mean of the output, while the shaded region depicts one standard deviation of the output. We summarize the standard deviation in the table for better understanding.
the attribute. We then evaluate the function of our ranker using these fine- fine-graing grained generated pairs. verifies that our ranker's 's output well-aligns to the relative change in the pair of images.
We further evaluate fine- ane-grained 12I translations w.r.t. the "smile" attribute on the test dataset of
CelebA-HQ (See Fig. (). The trained generator is applied to generate a set of G(x. v) by taking
as inputs an image from the test dataset and e = -1.0, -0.5, 0.0, 0.5, 1.0, respectively Then we collect the output of the ranker for each generated pair and plot the density in terms of different types of pairs, i.c., with different V.
As shown in Fig. 6 (I) for large v, the ranker would output a large prediction. It demonstrates that the ranker indeed generalizes to synthetic imaged pairs and dis- criminate the subtle change among each image pair. (2)
Discriniattrr output
0.6
0.4
The ranker capture the whole ordering instead of the
0.0
-022
exact value w.r.t. the latent variable. Because the ranker
-0.4
that assigns to the generated pairs inhibits the generator's 's loss optimizing to zero, although our generator's 's objective ensure the ranker output values are consistent with the latent variable. However, the adversarial training would help the ranker to achieve an equilibrium with the genera- tor when convergence, so that the ranker can maintain the whole ordering regarding the latent variable.
-0.6
-0.8
-0.5
Latent variable
Figure 6: The box plot of the ranker's output for generated pairs with different values of the latent variable.
4.3 LINEAR TENDENCY ON THE LATENT VARIABLE
As our ranker can reveal the relative changes in pairs of images, differences of the one-grained synthetic image pairs generated by different methods.
it evaluate the subtle
We generate the fine-graine pairs on the test dataset of CelebA-HQ w.r.t. the "smile" attribute. Each trained model produces a series of synthetic images by taking as input a real image and different latent variables. The range of the latent variable is from -1 to 1 with step 0. Then the ranker, pre-trained by our TRIP, is applied to evaluate the generated pairs and group them in terms of different conditioned latent variables for different models, respectively. In terms of each group. we calculate the mean and the standard deviation (std) for the outputs of the ranker (Fig 7
Fig. 7 shows that (1) the ranking output of our TRIP exhibits a linear trend with the lowest variance w.r.t. the latent variable. This demonstrates that TRIP can smoothly translate the input image to the desired image over the specific attribute along the latent variable. (2) The ranking output of RCGAN behaves like a tanh curve with sudden change when the latent variable is around zero. It means that RCGAN cannot smoothly control the attribute strength for the input image. In addition, RCGAN has the largest variance on the ranking output due to the low quality of the generated images, which introduces noises to the ranker's s prediction on the generated pairs. (3) ReIGAN manifests a three-step like curve, which indicates a failure of fine-grainee generation. This is mainly because of its specific design of the interpolation loss. (4) FN presents a linear tendency like our TRIP, which denotes that can make fine- mne-griined control over the attribute. However, the mean of the ranking output for the generated pairs is relatively low in FN, since it fails to translate some input images into the desired
When conditioning negative values of the latent variable, we the test samples with the "smiling" attribute. When conditioning on positive values, we use the test samples with the "not smiling" attribute.
8

---

Under review as a conference paper at ICLR 2021
output images. This is verified by its low translation accuracy (See the appendix Fig. 12 lower than 85%. In addition, FN also exhibits large variance of the ranking output due to the poor image quality.
4.4 EXTENDED TO MULTIPLE ATTRIBUTES
We conduct fine-grained 12I translation with two attributes "smile" and "male" on CelebA-HO -H to show that our model can generalize well the case with multiple attributes. We use the latent variable with two dimensions to control the change of "smile" and "male" attributes, respectively.
Less smiling
More smiling
mescullne
We show the generated outputs conditioning on different v Fig. g (1) Our GAN can disentangle multiple attributes. When conditioning on [-1, 0]/[1, [1, Oj, the output im- ages O.1,0/110 appear "less smiling" momr smiling" with no change in the "masculine" attribute.
When conditioning on v = [0, -0/ 1], the output im- ages Oo,- appear "less "aaucullieee"mmon" masculine" with no change in the "smiling" attribute. In addition, a fine-grained control over the strength of single attribute is still practical. (2) Our TRIP can manipulates the subtle changes of multiple attributes simultaneously. For exam- ple, when conditioning O appear "less smiling" and "more masculine", Our TRIP can make fine-gras ane-ngrieed translation on "smille" and "masculine".
Figure 8: Fine- Fine-grainn grained image-to-image translation with "smile" and male at- tributes. The middle is the input image. Others the generated output images conditioned on different v.
[1, 11.11, 1]. the output image
5 CONCLUSION
In this paper, we propose novel GAN for fine-grained image-to-image translation by modeling RAs, where the generated data model fake data region for the ranking model. We empirically show the efficacy of our GAN for the fine- mme-graieed translation on CelebA-HQ and LFWA dataset. Our proposed GAN can be deemed as a new form of semi-supervised GAN. The supervised pairwise ranking and the unsupervised generation target is incorporated into single model function. So one of the promise of this work can be extended to semi-supervised GAN area.
REFERENCES
Yazeed Alharbi and Peter Wonka. Disentangled image generation through structured noise injection. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 5134-5142, 2020.
Yunbo Cao, Jun Xu, Tie- Yan Liu, Hang Li, Yalou Huang, and Hsiao- Hon. Adapting ranking svm to document retrieval. Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pages 186-193, 2006.
Olivier Chapelle, Alekh Agarwal, Fabian H Sinz, and Bernhard Schölkopf. An analysis of inference with the universum. In Advances in neural information processing systems, pages 1369-1376.
2008.
Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen, and Xin Tong. Disentangled and controllable face image generation via 3d initative-contrsttvv learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 5154-5163, 2020.
Zheng Ding, Yifan Xu, Weijian Xu, Gaurav Parmar, Yang Yang, Max Welling, and Zhuowen Tu.
Guided variational autoencoder for disentanglement learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7920-7929, 2020.

---

Under review as a conference paper at ICLR 2021
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
Zhenliang He, Wangmeng Zuo, Meina Kan, Shiguang Shan, and Xilin Chen. Attgan: Facial attribute editing by only changing what you want. IEEE Transactions on Image Processing, 28(11): 5464-5478, 2019.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1 1134-1134, 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In 6th International Conference on Learning Representa- tions, ICLR 2018, Vancouver, BC, Canada, April 30 -My 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning dis- cover cross-domain relations with generative adversarial networks. In Proceedings of the 34th International Conference Machine Moarning-Voluuee 70, pages 1857-1865. JMLR. org, 2017.
Ruho Kondo, Keisuke Kawano, Satoshi Koide, and Takuro Kutsuna. Flow-based image-to-image translation with feature disentanglement. In Advances in Neural Information Processing Systems, pages 4168 4178, 2019.
Guillaume Lample, Neil Zeghidour, Nicolas Usunier, Antoine Bordes, Ludovic Denoyer, and Marc' Aurelio Ranzato. Fader networks: Manipulating images by sliding attributes. In Advances in Neural Information Processing Systems, pages 5967-5976, 2017.
Xiao Li, Chenghua Lin, Chaozheng Wang, and Frank Guerin. Latent space factorisation and manipulation via matrix subspace projection. International Conference on Machine Learning. 2020.
Alexander H Liu, Yen-Cheng Liu, Yu- Ying Yeh, and Yu-Chiang Frank Wang. A unified feature dis- entangler for multi-domain image translation and manipulation. In Advances neural information processing systems, pages 2590-2599, 2018
Ziwei Liu, Ping Luo, Xiaogang Wang. and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
Augustus Odena. Semi-superviser learning with generative adversarial networks. Workshop on Data-Efficient Machine Learning (ICML), 2016.
Devi Parikh and Kristen Grauman. Relative attributes. 2011 International Conference on Computer Vision, pages 503-510. IEEE, 2011.
Yassir Saquil, Kwang In Kim, and Peter M. Hall. Ranking egans: Subjective control over semantic image attributes. In British Machine Vision Conference 2018, BMVC 2018, Newcastle, UK, September 3-6, 2018, page 131. BMVA Press, 2018
Zhou Wang, Alan Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions image processing, 13(4):600-612, 2004.
Po- Wei Wu, Yu-Jing Lin, Che-Han Chang, Edward Y Chang, and Shih-Wei Liao. Relgan: Multi- domain imaee-t-iimage translation via relative attributes. In Proceedings of the IEEE International Conference on Computer Vision, pages
2019.
Ke Zhou, Gui-Rong Xue, Hongyuan Zha, and Yong Yu. Learning rank with ties. In Proceedings of the 31 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, page 275-282, 2008
Jun Yan Zhu, Taesung Park. Phillip Isola, and Alexei A Efros. Unpaired amage-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference computer vision, pages 2223-2232, 2017.
10

---

Under review as a conference paper at ICLR 2021
Ircc
(3X(4)
Bosalls
0.5
(4)
(4)
0.5
(9)
Figure 9: Ablation study. The experiments are the fine- ahe-grained Image-to- taage-to-image translation on the "smile" attribute of CelebA-HQ. In the results, the first column is the input image. The other columns are the generated images conditioned on the latent variable from -1 to with step 0.2. (3) and
(4) refers to equation and equation respectively. Trank refers to equation to equation
I Igan refers
Model
AAS
SSIM
FID
Lrank
9.73
1.51E-05
29
I Igan 0
98.47
0.0058
78.08
X
55.7
0.0011
8.55
CLS (3)
92.37
0,0163
11.33
TRIP 97.69
0.0030
10.19
Table Ablation study. Translation accuracy, (AAS), Fine- tineinraneed performance (SSIM) and image quality (FID) are reported, corresponding to the results in Fig. 12
A ABLATION STUDY
In Fig. and Table show ablation study of model. (1) Without Trank, the generated images exhibit no change over the input image. The generator fails to learn the mapping of the translation, which is reflected by a extremely low translation accuracy. (2) Without Lgan. the image quality degrades, achieving a high FID score. (3) Setting À 0, i.e. without considering the adversarial ranking, the performance of facial image manipulation collapses, obtaining a low translation accuracy. (4) When optimizing with equation 3 i.e., not linearing the ranking output for the generated pairs, the fine-grained control over the attributes fails, getting high SSIM score. (5) With our TRIP, the generated images present desired changes consistent with the latent variable and possess good quality.
B CONVERGENCE OF TRIP
We plot the training curve of the ranker and the generator, respectively, as shown in Fig. 10 demonstrates that the ranker and the generator are trained against each other until convergence.
We plot the distribution of the ranker's prediction for real image pairs and generated image pairs with different relative attributes (RAs) (+1/0/ 1) using the ranker in Fig. 11 (1) At the beginning of the training (Fig. Ila), the ranker gives similar predictions for real image pairs with different RAs. The same observations can also be found on the generated image pairs. (2) After 100 iterations (Fig. Tlb
the ranker learns to give the desired prediction for different kinds of pair, i.e., > 0 (averaged) for pairs with RA (+1), for pairs with RA (0) and -1 for pairs with RA (-1). (3) After 9, 900 iterations (Fig.
TIc TRIP converges. In terms of the real image pairs, the ranker output +1 for the pairs with
RA
0 for the pairs with RA (0) and -1 for the pairs with RA (-1) in the sense of average.
11

---

Under review as a conference paper at ICLR 2021
1.0
Ranker
Generator
0.8
Loss
0.4
0.2
0
20000
40000
60000
80000
100000
Step
Figure 10: The curve of training loss. The ranker and the generator are trained against each other until convergence. Ranker loss Lank + AgLgan Agp Generator loss Lrank + A gan Agp Logete
This verifies that ranker give precise ranking predictions for real image pairs. In terms of the generated pairs, the ranker outputs +0.5 for the pairs with RA (+1), 0 for the pairs with RA (0) and -05 for the pairs with RA (-1) 1) in the sense of average. This is convergence state due to rival preferences. We take pairs with RA (+1) as an example. The generated pairs with RA (+1) expected be assigned when optimizing the ranker and to be assigned +1 when optimizing the
generator. Therefore, the convergence state should be around 0.5. And so forth. This can explain why the ranker would output for the pairs with RA (0) and -0.5 for pairs with RA (-1) in the sense of average.
4
1
3
0
-1
2
-1
2-
0-
Discriminator output
Discriminator output
Diseriminator output
10
1
6
1
6
0
0
-1
-1
4
5-
2
2
0
0
Discriminator output
Discriminator output
Discriminator output (c) Iteration 9900
(a) Iteration
(b) Iteration 100
The density plot of the discriminator's output for pairs of image from real dataset (first row) and generation of TRIP (second row) with different relative attributes (+1/0/ - 1), respectively, during the training process.
Figure 11:
12

---

Under review as a conference paper at ICLR 2021
C EXPERIMENTAL SETTING
C.I. NETWORK STRUCTURE
Our generator will take image and random sampled relative attribute as input and output translated image. Our generator network is same as ReIGAN (Wu et al. 2019), which is composed of three convolutional layers for down-sampling, six residual blocks, and three transposed convolutional layers for up- up-sampling (shown in Table 3). Our proposed ranker will take pair of images (x, y) as inputs and output the classification score. is comprised of two functional components rank layer and GAN layer following feature layer (shown in Table 4 The rank layer and the GAN layer is for calculating I.Pank and I. respectively. The feature layer F is composed of six convolutional layers. The rank layer is composed of subtract layer, one convolutional layer, one flatten layer and one dense layer. The subtract layer operates on F(x) and F(y), i... F(y) F(x). The GAN layer composed of one flatten layer and one dense layer.
C.2 TRAINING
We split the dataset into training/test with a ratio 90/10. We pretrain our GAN only with Lgan to enable good reconstruction for the generator. By doing so, ease the training by sequencing the learning of our GAN. That is, we first make generation with good quality. Then when our GAN begins to train, the ranker can mainly focus on the relationship between the generated pairs and its corresponding conditional rather than handling the translation quality and the generation quality together. All the experiment results are obtained by a single rum.
Component
Input - Output Shape
Layer Information
Down-samplin
(h, wt, 3 + K) (h, w, 64)
(h, w. 64)
> {2.
w
2: 128)
7. 128)
[. T: 256)
Residual Blocks
A &
41 4 256)
7 {h 8 2,266
Residual Block: (mm,:XX))2250055555555555551111
1' 1 256)
(56)
Residual Block: Cmmm:(((y22265553333355555511111
T'T ]
ic 256)
- m. 256)
Residual Block: Cuw.:WWy=2566///55551:55:1)1111
1' 256)
256))
Residual Block: Comr:WW)222655333551111111111
w 256)
- [k4
4' 256)
Residual Block: CON.:1//22665333551155111111111
1' 4
256)
256)
Residual Block: Cow.:k1yy2266/33333155))/.....
Up-sampling
41 4 256)
7 7 128)
(h 1 128)
(h,,w,64 64)
C
(b, 64)
64)
+ (h,, w.
Con=<CV==3SS=7755511..5..5
Table 3:
Generator network architecture. We use switchable normalization, denoted as SN, in all layers except the last output layer. N2 is the number of filters. Sy is the filter size. S, is the stride size. Sp the padding size.
D EVALUATION
SSIM. We first apply the generator to produce set of fine-grained output images (xi,..., x11 by conditioning an input image and a set of latent variable values from -1 to 1 with step 0.2. We then compute the standard deviation the structural similarity (SSIM) (Wang et al. 2004) between xz-1 and z; as follows:
((SSTM (x;-1, | 1, ,1)))
We calculate SSIM for each image from the test dataset and average them to get the final score.
AAS. The accuracy is evaluated by a facial attribute classifier that uses the Resnet-18 18 architecture (He al. 2016). To obtain AAS, we first translate the test images with the trained GANs and then apply the classifier to evaluate the classification accuracy of the translated images coupled with its swapping attribute. Higher accuracy means that more images translated desired.
FID. It evaluated with 30K translated images on CelebA-HQ dataset and 13, 143 translated images on LFWA dataset.
13

---

Under review as a conference paper at ICLR 2021
E
MORE EXPERIMENT RESULTS
Translation accuracy (AAS)
100
85
80
70
Smile
Gender
Mouth
Checkbones
Smile
Frown
CekeAA-HQ
LFWA
FN ReIGAN
RCGAN
TRIP
Figure 12: Translation accuracy (AAS, higher is better) of FN, RCGAN, ReIGAN and TRIP on CelebA-HQ and LFWA. RCGAN fails make a fine- ane-grained translation w.r.t. the "mouth" attribute on CelebA-HQ dataset. So we did not collect its result.
F FINE-GRAINED 121 TRANSLATION ON NON-FACIAL DATASET
To further evaluate the effectiveness of our method, we conduct fine-grained 12I translation on shoes - edges from the UT 7appos50K dataset Isola et al. 2017). Fig. shows that our TRIP can make fine-grained translation from shoe images to edge images.
Component
Input + Output Shape
Layer Information
Feature Layer
x (h, 3) + 2 [ 64)
x (z+ 7. 64)
+
7: 128)
1
2 (b w 128) 128)
+2x (h
81 256)
K. 4.
2
2566)
+2 [T. 512)
Conv-(N Sonk4///.522.144455255551..L
2
15 T6. 512)
+2 [.g 32 1021)
2
32 321 1024)
+
64 2048)
Rank Layer
x
64 641 2048)
+ 64 2048)
Subtract
61 51 2048)
A "
- 57 01 2048)
w 2048)
2048)
Flatten
x 2048)
+ (K,)
Dense
67 6
GAN Layer
1 2048
x 2048)
Flatten
64
64
54
2018)
(1.)
Dense
Table 4: Ranker network architecture. LReLU is Leaky ReLU with negative slop of 0.01. K is the number attributes. N, the number of filters. Sy is the filter size. S, is the stride size. S, the padding size.
14

---

Under review as a conference paper at ICLR 2021
(a) FN
(b) ReIGAN
(c) RCGAN
15

---

Under review as a conference paper at ICLR 2021
(d) TRIP
Figure 13: The tra-ggai facial attribute ("male") translation on CelebA-HQ dataset applying various methods. The first column is the input image. The other columns are the generated images conditioned on the latent variable from - to with step 0.2. The presented results are randomly sampled from the test set.
(a) FN
(b) RelGAN
16

---

Under review as a conference paper at ICLR 2021
(c) RCGAN
(d) TRIP
Figure 14: The fine- fine-graine grained facial attribute ("male") translation CelebA-HQ dataset applying various methods. The first column is the input image. The other columns are the generated images conditioned the latent variable from -1 to with step 0.2. The presented results are randomly sampled from the test set.
17

---

Under review as a conference paper at ICLR 2021
(a) FN
(b) ReIGAN
(c) RCGAN
18

---

Under review as a conference paper at ICLR 2021
(d) TRIP
Figure 15: The fine- fine-graine grained facial attribute ("male") translation on CelebA-HO dataset applying various methods. The first column is the input image. The other columns are the generated images conditioned on the latent variable from to with step 0.2. The presented results are randomly sampled from the test set.
Bas
UBBR
Figure 16: Fine-grained image-to-image translation on the shoes - edges (generated by our TRIP)
19


Reviews:
[{"review_id": "xDYAIPavoPv", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThe authors proposed in this paper a supervised approach relying on given relative and quantitative attribute discrepancies. A UNet-like generator learns adversarially tends to generate realistic images while a \"ranker\" tends to predict the magnitude of the input parameter used to control the image manipulation. The controled parameter is defined implictly using images whose the discrepancy of the attribute of interest is known. This allows fine-grained manipulation of the attribute of interest. The results of the approach is illustrated on face datasets (CelebA-HQ and LFWA).\n\nReason for score\n\nThe general idea behind the proposed approach is interesting and somehow novel in this context, but the positioning of the paper is a bit confusining and unfair according to ranker-based approaches. On top of that the quality of the writting (clarity, misleading notations…), the organization of the model section especially and the provided (and missing) details make it hard to follow (see below). Overall I vote for rejecting this paper.\n\nPros\n1. Ranker-based approaches that require less supervision is essential to open image manipulation/translation to wider domain where its hard or costly to get precise labels while its generally most easier to relative information. But the proposed approach seem to be spuriously attached to this setting (see Cons 1.).\n2. The idea of controlling changes in images through relative information is good and effective and it worth it to diffuse this knowledge in the field. For instance, it can simplify the problem (especially the particular case of no modification boils down to having the generator behaving like the identity function when the condition is null).\n3.  Presented qualitative results are better compared to competition but I was wondering if the setting or the choosen example is fair enough (see question 2.)\n\n\nCons\n1. Since relative attributes are obtained from real absolute attributes, the approach is more supervised (at least in the presented experiments) that it seems at first glance. Indeed, in actual relative supervision scheme, the datasets are more \"quantitative\" and generally fairly binary (like in pairwise comparisons or with triplets). For instance, we can expect that the differences from Fig.7 (more linear trend of the ranker) is only due to this fact. If not, it has to be demonstrated. So to be more fair on the potential and the interest of the approach, the supervision signal should be downgraded, or the authors should assume an higher degree of supervision and compare their results to standard (non-ranked based) conditioning based on absolute values of parameters.\n2. Section 3.2: Since generated images have pseudo-label of 0, I don't understand why there passing through the ranker model during training rather than just being ignored (and removed from the rank loss). Besides, this seems to contradict the definition of $L_{rank}^G$ in equation 5b which tends to bring together $R(x,\\hat{y})$ to $v$. My belief is the confusion comes from the fact you introduce first a standard binary ranker and then \"linearized\" the output. The section 3 should be rewritten and reorganised for the sake of clarity.\n3. Since the whole optimized loss for D and G (and how there are trained: one after another or altogether ?) are not specified, it's not clear why $R(x,\\hat{y})$ is present both in equation 5a and 5b. My understanding is loss 5a is used when optimizing D and loss 5b when optimizing G, which may lead to some unconsistancies and does not allow to cleary understand which mix-max game is optimized by this process. Besides additional loss are lost in the experiments section (cycle, gradient penalty…).\n4. The authors claim that their approach can reach high-fidelity but all the proposed results still present strong artefacts even at a low resolution, even if better than competition, but high-fidelity may be too strong to recent work on image manipulation/translation.\n\n\nQuestions\n1. Since the role of the \"downsampling\" part of the GAN is partly to encode the input image (+ the condition), why not have shared a common encoding module for \"downsampling\" and \"Feature Layer\" (a siamese network architecture could be used to handle the 3 inputs in this case) ? It would certainly help to reduce the number of weights and ease the training.\n2. Can you clarify why your approach seems to only modify the targeted attribute (like in Fig. 5) compared to dAN which seem to derivate in your experimentations (whereas perfectly working examples are shown in RelGAN paper) ? Can you precise which component / principle could explain the difference (apart experimental biases or limitations) ?\n\n\nMajor comments\n1. There is a lot of redundancies between the introduction and the \"related works\" section (especially on RCGAN and RelGAN). This could be better organized for clarity and to save important place for additional material. Additionnaly, the related work section essentially focus on rank-based approach (which is good to position the paper thinly, but not enough to position it more broadly), but is a bit scarce on other technics. For instance, VAE or flow-based approaches could be cited in this section, plus any approach that seek to structure and control changes directly from manipulation of the latent representation.\n2. Fig2. vs Fig.3 are a bit misleading since the ranker model is acting on only 2 images while the \"rank head\" is taking 4. It is not well aligned with further explaination (for instance loss rank definition in section 3.4 that should be better reflected in Fig.2 and directly connected to $x$, $y$ and the output of the generator $\\hat{y}$.\n3. section 3.4 and 3.5: $Q$ notation does not really simplify equations but rather gives rise to some obfuscation. It's hard to understand your loss is only trying to bring together R(x,y) the prediction to r(x,y) the real order. Then, in equation 7a and 7b, $L_2$ norms could be used to simplify and clarify the equations.\n4. In equation 5a, it does not seem possible to sample on $r$, since in fact, it depends on $x$ and $y$ and should be written $r(x,y)$ for clarity.\n\n\nMinor comments\n1. The \"rival\" wording used here and there from the abstract of the paper is not really introduced and self-supported.\n2. In abstract: \"real image preferences\" was not clear for me before I see $y$ in Fig.2.\n3. English: \"evoke\" verb is used several times in the paper, but I guess the intended meaning by the authors was not the more common understood definition. You may use: \"provoke\", \"induce\", \"produce\", \"raise\" according to the context.\n4. In section 3.1, explaining how RCGAN uses its ranker its a bit confusing and should be better explained in the related work section.\n5. section 3.3: closing parenthesis missing 3 lines after equation 4.\n6. In equation 5a: notation error: it should be $L_{rank}^D$ rather than $L_{rank}^R$\n7. Table 1: \"UGGAN\" name is mentionned rather than TRIP\n8. section 4.1: typo: \"best FID On\" --> \"best FID on\"\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "bfHngJtXf3", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Summary]\nThis paper proposes a facial image manipulation method that incorporates ranker module using relative attributes and pairwise learning to rank, i.e. TRIP. The authors modify a typical discriminator of GAN into ranker with additional ranking head. They evaluate TRIP on CelebA-HQ and LFWA.\n\n[Strength]\n- Image manipulation is an interesting and practical topic.\n\n[Weakness]\n- Above all, I am not sure that how large can this work contributes. First, the authors argue that they address image-to-image (I2I) translation but the problem addressed is just facial image manipulation rather than I2I translation. In general, I2I term covers more wide topics: multidomain multimodal translation, style transfer, real-to-anime, etc. Second, this method was evaluated on facial image manipulation only. If the method can be applied to other domains (such as cars, birds, etc..), the contribution can be enhanced.\n- Even if the authors claim relative attribute-based methods are major, I cannot agree with this argument. Recent many facial image manipulation work uses other methods (e.g. reference images[Choi et al. 2020], structured noise [Alharbi & Wonka 2020], semantic mask [Lee et al. 2020]) for high-fidelity images, showing promising results. Also, the author need to explicitly argue the main advantages of the proposed method in related work. \n- Core related work were missed such as [Deng et al. 2020, Yan et al. 2019] except for the papers mention above. \n- It is better that Figure captions are self-contained to improve the readability.\n- The method part is not easy to follow. Some terms are used without definition (>, v). Eq(2) is not intuitive to understand. L^D should be L^R in 6(a). \n- Even if TRIP shows better results than RelGAN in terms of quantitative metrics, the covered attributes of TRIP presented in the result section looks smaller than those of RelGAN. \n- Considering all qualitative and quantitative results, I am not convinced that TRIP are superior than the baseline models.\n\n[Ref]\n- Choi et al. StarGAN v2: Diverse Image Synthesis for Multiple Domains. CVPR 2020.\n- Deng et al. Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning. CVPR 2020.\n- Alharbi & Wonka Disentangled Image Generation Through Structured Noise Injection. CVPR2020.\n- Lee et al. MaskGAN: Towards Diverse and Interactive Facial Image Manipulation. CVPR2020.\n- Yan et al. Joint Deep Learning of Facial Expression Synthesis and Recognition. IEEE T on Multimedia 2019.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8P46HAYl1zO", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to construct rival preference in the ranker to evoke the adversarial training between the ranker and the generator, leading to a better fine-grained control over the interested attribute in image-to-image translation task. \n\nWith tailor-designed loss functions, the ranker is promoted to pay attention on the effective information related to the target attribute and avoid providing biased prediction for unrealistic image pairs. Through the adversarial process between the generator and the ranker, the generator can improve the ability of generating desired attribute. Besides using the ranker to improve the ability of modeling desired attribute in the generator, a discriminator is adopted to improve the quality of generated images further. The design of proposed ranker is elegant and experimental results are promising.\n\nThis paper is well organized and clearly written.\n\nStrengths:\n1.\tProposed techniques are intuitive and well-motivated.\n2.\tBoth qualitative results and quantitative comparisons demonstrate the superiority of proposed method compared with other methods.\n\nWeaknesses:\n1.\tThe main contribution of this paper is introducing adversarial learning process between the generator and the ranker. The innovation of this paper is concerned.\n2.\tQuality of generated images by proposed method is limited. While good continuous control is achieved, the realism of generated results showed in paper and supplemental material is limited.\n3.\tVisual comparisons and ablation study are insufficient. \n\nComments/Questions:\n1.\tCould you elaborate more on why proposed method achieves better fine-grained control over the interested attribute? Was it crucial to change the formular of ranker’s loss function from classification to regression?\n2.\tCould you provide more visual comparisons between the proposed method and prior works?\n3.\tThere are also some other works focusing on the semantic face editing and they show the ability to achieve continuous control over different attributes, like [1]. Could you elaborate the difference between your work and these papers?\n4.\tStatements in Section 4.2 are somewhat redundant.\n\nMinor:\n1.\tMissing proper expression for the third face image in Figure 2.\n2.\tMissing close parenthesis at the bottom of Page 4.\n3.\tInconsistent statement and reference for Celeb Faces Attributes Dataset in experiment section.\n\n[1] Shen, Yujun and Gu, Jinjin and Tang, Xiaoou and Zhou, Bolei. “Interpreting the Latent Space of GANs for Semantic Face Editing”, In CVPR, 2020. https://dblp.org/rec/conf/cvpr/ShenGTZ20\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7GjmtVTe0Nc", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposes to improve the fine-grained image-to-image translation by utilizing an adversarial ranking framework. The new ranker helps the generator to have a better fine-grained control on the translation results. Experiment results exhibit that the proposed method achieves state-of-the-art results on several image translation tasks.\n\n\nMajor issues: \n-  Although the intuition of the adversarial ranker makes sense, it will be better to give some theoretical analysis about when and how the optimal state of generator and ranker will achieve. \n- Does the training scheme occur convergence problem? Because the model has three different networks and there has no explicit connection between ranker network and discriminator network. \n- The experiments are mainly conducted on face datasets. It remains some questions about the generalization of the model to other non-face image translation tasks.  \n- There lacks the ablation study analyzing the proposed ranker module.\n\nMinor issues\n- How about the computation time increased by the ranker module?\n- How do you choose the weighting factor for the networks?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers gave either borderline or negative scores; unfortunately, discussion was not lively, so scores remained the same. No reviewers voice strong support for acceptance, but acknowledge several merits of the work.
Decision: Reject