Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents an approach for image to image translation by introducing extra layers into the generator, which can be trained in an unsupervised way. The paper is generally easy to follow. I have the following concerns: the novelty is quite marginal since the backbone network and the training process are well developed before and the technique of employing more layers to the generator seems like simply extending the network. Please consider to use different notations in Section 3 to denote image from different domains. Also, please use the same subscript for G_{AB} and G_{\mathcal{BA}}.
review_title: reviews for Powers of layers for image-to-image translation

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes an unpaired image-to-image translation method which applies a pre-trained auto-encoder and a latent feature transformer (single block) to perform iterative image transformation. A progressive training and warm-up strategy is used to settle the numerical exponentiation effects caused by powers of layers. In the testing phase, the discriminator is also used to adjust the inference time.

Pros: 
1) Compared with the vanilla CycleGAN, the proposed PoL has significantly fewer parameters and similar performance.
2) The flexibility offered by the common embedding space allows the modulation of the transformation strength, or to compose several transformations.
3) The discriminator is used to adjust the inference time and find the optimal number of iterations in the testing phase.

Cons:
1) CycleGAN is the only competitor in most comparative experiments, which is not sufficient. Besides, CycleGAN is not a good competitor for image restoration tasks (debluring, denoising, etc.), so the potential of the proposed PoL is questionable. Additional comparison results generated by other general image restoration methods [1, 2] should be reported. 
2) Is CycleGAN also pre-trained on the same dataset as PoL for fair comparison?
3) Although progressive training contributes to more natural intermediate outputs, the final output is not satisfactory, for example the unnatural patterns on zebras in Fig. 4.
4) More effective transformation modulation is a major advantage of the proposed PoL, but the provided experiments did not demonstrate this well. I think it would be more appropriate to put the results obtained along the iterations of the recurrent block in the main text instead of in the appendix.
5) Why the proposed embedding transformer “is similar to the feed-forward network used in transformers [3]”? It seems that it is just a simple residual convolution module with expansion factor K to adjust the model’s capacity, which is not related to transformer or self-attention.
6) What are the significant advantages of PoL compared with traditional RNNs (LSTM, GRU)? Is it possible to directly replace PoL with RNNs to achieve close results?

[1] Neural Sparse Representation for Image Restoration. NeurIPS, 2020.
[2] Learning Invariant Representation for Unsupervised Image Restoration. CVPR, 2020.
[3] Attention is all you need. NeurIPS, 2017.
review_title: Insufficient comparison and experiment settings

final_score: rating: 5: Marginally below acceptance threshold
review_content: 1. Summary. The submitted paper proposes to use a recurrent residual block for the task of unpaired img2img translation. A number of strategies of how many times to apply this block are suggested. 

1. Decision. I do really like the direction of weight sharing in img2img models and this work is a pretty nice case. This approach helps to decrease the number of weights, and, if done properly, does not harm quality. The results in deblurring/denoising look rather interesting.

Cons. 

However, the downside of the presented recurrent block is the increased computation time at the inference step, as far as I can judge. This is especially crucial when the discriminator is involved as the stopping criterion. Could you provide a comparison of the inference speed (FPS or FLOPS or any other measure) between CycleGAN/NiceGAN and PoL?

Second, I believe this approach could be also put into the context of the adaptive computation time research field [1,2,3,4]. This may help to determine the number of layers to apply.

Third, the proposed block may be straightforwardly generalized to multi-domain img2img translation and showcased on more interesting and recent datasets against stronger baselines like MUNIT [5], FUNIT[6], etc. This could make img2img part of your experiments more solid, I suppose.

All-in-all, to my mind there is great room for improvement for your submission to demonstrate the real power of PoL. Therefore, now I tend to rate the submission a bit below the threshold.

[1] https://openreview.net/forum?id=r1W1OxAF
[2] https://openreview.net/forum?id=SkZq3vyDf
[3] https://openaccess.thecvf.com/content_cvpr_2017/html/Figurnov_Spatially_Adaptive_Computation_CVPR_2017_paper.html
[4] https://openreview.net/forum?id=HyzdRiR9Y7
[5] https://link.springer.com/chapter/10.1007/978-3-030-01219-9_11
[6] https://openaccess.thecvf.com/content_ICCV_2019/html/Liu_Few-Shot_Unsupervised_Image-to-Image_Translation_ICCV_2019_paper.html
review_title: A good point to reduce the number of weights but the computation time is questionable

final_score: rating: 3: Clear rejection
review_content: The paper proposes a method for unsupervised image translation between unpaired domains of images. The main idea is to develop an iterative transformation module that operates in the embedding space.

Overall I have the following concerns about the paper:

The motivation for this architecture is unclear. The introduction motivates this model with fractals and iterated function spaces, but that seems to have nothing to do with the types of applications shown here. What do IFSs have to do with denoising and pictures of zebras? This iterated refinement strategy seems more similar to iterative refinement/projection algorithms like conjugate gradient and Richardson-Lucy deconvolution, which are relevant to low-level signal processing operations like denoising/deblurring, but not zebra synthesis.  The paper includes as motivation the idea that applying that different levels of transformation can be achieved by choosing different numbers of iterations, but the application of this is shown only for denoising (Table 1). 

No comparison is provided to the state-of-the-art in unpaired image translation: 
Contrastive Learning for Unpaired Image-to-Image Translation
Taesung Park, Alexei Efros, Richard Zhang, Jun-Yan Zhu
ECCV 2020

Visually, the results are not convincing. Not many results are shown, and most do not look better than those from CycleGAN.  The results may be cherry-picked, since there was no statement as to how these results were chosen.  There is simply not enough visual evidence that the method has evidence over previous work. Additionally, quantitatve comparisons do not give a compelling outcome, but I would put more weight on visual comparisons anyway.

For the task of denoising, it is unclear why one would want to use a general-purpose unpaired translation method; supervised methods ought to be much effective here, and there is an enormous literature of related work that is not cited or compared with here. If one is to use denoising as a motivation application (rather than a toy example), then much more rigor is required. 

review_title: Review

Meta-review: All the reviewers shared the concerns about the novelty and the quality of the results. Comparisons with some SOTA results are missing, and the inclusion of deblurring/denosing tasks is not convincing. The authors carefully addressed these issues in the rebuttal but the reviewers didn’t change their mind afterwards. After carefully examining the results in the paper, the AC agrees with the reviewers that the improvement on image quality, if any, seems to be too small to warrant a publication. 
Decision: Reject