Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes to construct rival preference in the ranker to evoke the adversarial training between the ranker and the generator, leading to a better fine-grained control over the interested attribute in image-to-image translation task. 

With tailor-designed loss functions, the ranker is promoted to pay attention on the effective information related to the target attribute and avoid providing biased prediction for unrealistic image pairs. Through the adversarial process between the generator and the ranker, the generator can improve the ability of generating desired attribute. Besides using the ranker to improve the ability of modeling desired attribute in the generator, a discriminator is adopted to improve the quality of generated images further. The design of proposed ranker is elegant and experimental results are promising.

This paper is well organized and clearly written.

Strengths:
1.	Proposed techniques are intuitive and well-motivated.
2.	Both qualitative results and quantitative comparisons demonstrate the superiority of proposed method compared with other methods.

Weaknesses:
1.	The main contribution of this paper is introducing adversarial learning process between the generator and the ranker. The innovation of this paper is concerned.
2.	Quality of generated images by proposed method is limited. While good continuous control is achieved, the realism of generated results showed in paper and supplemental material is limited.
3.	Visual comparisons and ablation study are insufficient. 

Comments/Questions:
1.	Could you elaborate more on why proposed method achieves better fine-grained control over the interested attribute? Was it crucial to change the formular of ranker’s loss function from classification to regression?
2.	Could you provide more visual comparisons between the proposed method and prior works?
3.	There are also some other works focusing on the semantic face editing and they show the ability to achieve continuous control over different attributes, like [1]. Could you elaborate the difference between your work and these papers?
4.	Statements in Section 4.2 are somewhat redundant.

Minor:
1.	Missing proper expression for the third face image in Figure 2.
2.	Missing close parenthesis at the bottom of Page 4.
3.	Inconsistent statement and reference for Celeb Faces Attributes Dataset in experiment section.

[1] Shen, Yujun and Gu, Jinjin and Tang, Xiaoou and Zhou, Bolei. “Interpreting the Latent Space of GANs for Semantic Face Editing”, In CVPR, 2020. https://dblp.org/rec/conf/cvpr/ShenGTZ20

review_title: Incremental advances to a key module for fine-grained image-to-image translation.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: This paper proposes to improve the fine-grained image-to-image translation by utilizing an adversarial ranking framework. The new ranker helps the generator to have a better fine-grained control on the translation results. Experiment results exhibit that the proposed method achieves state-of-the-art results on several image translation tasks.


Major issues: 
-  Although the intuition of the adversarial ranker makes sense, it will be better to give some theoretical analysis about when and how the optimal state of generator and ranker will achieve. 
- Does the training scheme occur convergence problem? Because the model has three different networks and there has no explicit connection between ranker network and discriminator network. 
- The experiments are mainly conducted on face datasets. It remains some questions about the generalization of the model to other non-face image translation tasks.  
- There lacks the ablation study analyzing the proposed ranker module.

Minor issues
- How about the computation time increased by the ranker module?
- How do you choose the weighting factor for the networks?

review_title: A nice work if any theoretical analysis can be given 

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary

The authors proposed in this paper a supervised approach relying on given relative and quantitative attribute discrepancies. A UNet-like generator learns adversarially tends to generate realistic images while a "ranker" tends to predict the magnitude of the input parameter used to control the image manipulation. The controled parameter is defined implictly using images whose the discrepancy of the attribute of interest is known. This allows fine-grained manipulation of the attribute of interest. The results of the approach is illustrated on face datasets (CelebA-HQ and LFWA).

Reason for score

The general idea behind the proposed approach is interesting and somehow novel in this context, but the positioning of the paper is a bit confusining and unfair according to ranker-based approaches. On top of that the quality of the writting (clarity, misleading notations…), the organization of the model section especially and the provided (and missing) details make it hard to follow (see below). Overall I vote for rejecting this paper.

Pros
1. Ranker-based approaches that require less supervision is essential to open image manipulation/translation to wider domain where its hard or costly to get precise labels while its generally most easier to relative information. But the proposed approach seem to be spuriously attached to this setting (see Cons 1.).
2. The idea of controlling changes in images through relative information is good and effective and it worth it to diffuse this knowledge in the field. For instance, it can simplify the problem (especially the particular case of no modification boils down to having the generator behaving like the identity function when the condition is null).
3.  Presented qualitative results are better compared to competition but I was wondering if the setting or the choosen example is fair enough (see question 2.)


Cons
1. Since relative attributes are obtained from real absolute attributes, the approach is more supervised (at least in the presented experiments) that it seems at first glance. Indeed, in actual relative supervision scheme, the datasets are more "quantitative" and generally fairly binary (like in pairwise comparisons or with triplets). For instance, we can expect that the differences from Fig.7 (more linear trend of the ranker) is only due to this fact. If not, it has to be demonstrated. So to be more fair on the potential and the interest of the approach, the supervision signal should be downgraded, or the authors should assume an higher degree of supervision and compare their results to standard (non-ranked based) conditioning based on absolute values of parameters.
2. Section 3.2: Since generated images have pseudo-label of 0, I don't understand why there passing through the ranker model during training rather than just being ignored (and removed from the rank loss). Besides, this seems to contradict the definition of $L_{rank}^G$ in equation 5b which tends to bring together $R(x,\hat{y})$ to $v$. My belief is the confusion comes from the fact you introduce first a standard binary ranker and then "linearized" the output. The section 3 should be rewritten and reorganised for the sake of clarity.
3. Since the whole optimized loss for D and G (and how there are trained: one after another or altogether ?) are not specified, it's not clear why $R(x,\hat{y})$ is present both in equation 5a and 5b. My understanding is loss 5a is used when optimizing D and loss 5b when optimizing G, which may lead to some unconsistancies and does not allow to cleary understand which mix-max game is optimized by this process. Besides additional loss are lost in the experiments section (cycle, gradient penalty…).
4. The authors claim that their approach can reach high-fidelity but all the proposed results still present strong artefacts even at a low resolution, even if better than competition, but high-fidelity may be too strong to recent work on image manipulation/translation.


Questions
1. Since the role of the "downsampling" part of the GAN is partly to encode the input image (+ the condition), why not have shared a common encoding module for "downsampling" and "Feature Layer" (a siamese network architecture could be used to handle the 3 inputs in this case) ? It would certainly help to reduce the number of weights and ease the training.
2. Can you clarify why your approach seems to only modify the targeted attribute (like in Fig. 5) compared to dAN which seem to derivate in your experimentations (whereas perfectly working examples are shown in RelGAN paper) ? Can you precise which component / principle could explain the difference (apart experimental biases or limitations) ?


Major comments
1. There is a lot of redundancies between the introduction and the "related works" section (especially on RCGAN and RelGAN). This could be better organized for clarity and to save important place for additional material. Additionnaly, the related work section essentially focus on rank-based approach (which is good to position the paper thinly, but not enough to position it more broadly), but is a bit scarce on other technics. For instance, VAE or flow-based approaches could be cited in this section, plus any approach that seek to structure and control changes directly from manipulation of the latent representation.
2. Fig2. vs Fig.3 are a bit misleading since the ranker model is acting on only 2 images while the "rank head" is taking 4. It is not well aligned with further explaination (for instance loss rank definition in section 3.4 that should be better reflected in Fig.2 and directly connected to $x$, $y$ and the output of the generator $\hat{y}$.
3. section 3.4 and 3.5: $Q$ notation does not really simplify equations but rather gives rise to some obfuscation. It's hard to understand your loss is only trying to bring together R(x,y) the prediction to r(x,y) the real order. Then, in equation 7a and 7b, $L_2$ norms could be used to simplify and clarify the equations.
4. In equation 5a, it does not seem possible to sample on $r$, since in fact, it depends on $x$ and $y$ and should be written $r(x,y)$ for clarity.


Minor comments
1. The "rival" wording used here and there from the abstract of the paper is not really introduced and self-supported.
2. In abstract: "real image preferences" was not clear for me before I see $y$ in Fig.2.
3. English: "evoke" verb is used several times in the paper, but I guess the intended meaning by the authors was not the more common understood definition. You may use: "provoke", "induce", "produce", "raise" according to the context.
4. In section 3.1, explaining how RCGAN uses its ranker its a bit confusing and should be better explained in the related work section.
5. section 3.3: closing parenthesis missing 3 lines after equation 4.
6. In equation 5a: notation error: it should be $L_{rank}^D$ rather than $L_{rank}^R$
7. Table 1: "UGGAN" name is mentionned rather than TRIP
8. section 4.1: typo: "best FID On" --> "best FID on"

review_title: This paper deals with image manipulation though relative effect of a given attribute using a generator and a ranker.

final_score: rating: 4: Ok but not good enough - rejection
review_content: [Summary]
This paper proposes a facial image manipulation method that incorporates ranker module using relative attributes and pairwise learning to rank, i.e. TRIP. The authors modify a typical discriminator of GAN into ranker with additional ranking head. They evaluate TRIP on CelebA-HQ and LFWA.

[Strength]
- Image manipulation is an interesting and practical topic.

[Weakness]
- Above all, I am not sure that how large can this work contributes. First, the authors argue that they address image-to-image (I2I) translation but the problem addressed is just facial image manipulation rather than I2I translation. In general, I2I term covers more wide topics: multidomain multimodal translation, style transfer, real-to-anime, etc. Second, this method was evaluated on facial image manipulation only. If the method can be applied to other domains (such as cars, birds, etc..), the contribution can be enhanced.
- Even if the authors claim relative attribute-based methods are major, I cannot agree with this argument. Recent many facial image manipulation work uses other methods (e.g. reference images[Choi et al. 2020], structured noise [Alharbi & Wonka 2020], semantic mask [Lee et al. 2020]) for high-fidelity images, showing promising results. Also, the author need to explicitly argue the main advantages of the proposed method in related work. 
- Core related work were missed such as [Deng et al. 2020, Yan et al. 2019] except for the papers mention above. 
- It is better that Figure captions are self-contained to improve the readability.
- The method part is not easy to follow. Some terms are used without definition (>, v). Eq(2) is not intuitive to understand. L^D should be L^R in 6(a). 
- Even if TRIP shows better results than RelGAN in terms of quantitative metrics, the covered attributes of TRIP presented in the result section looks smaller than those of RelGAN. 
- Considering all qualitative and quantitative results, I am not convinced that TRIP are superior than the baseline models.

[Ref]
- Choi et al. StarGAN v2: Diverse Image Synthesis for Multiple Domains. CVPR 2020.
- Deng et al. Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning. CVPR 2020.
- Alharbi & Wonka Disentangled Image Generation Through Structured Noise Injection. CVPR2020.
- Lee et al. MaskGAN: Towards Diverse and Interactive Facial Image Manipulation. CVPR2020.
- Yan et al. Joint Deep Learning of Facial Expression Synthesis and Recognition. IEEE T on Multimedia 2019.

review_title: Interesting but limited contribution

Meta-review: All reviewers gave either borderline or negative scores; unfortunately, discussion was not lively, so scores remained the same. No reviewers voice strong support for acceptance, but acknowledge several merits of the work.
Decision: Reject