Powers of layers for image-to-image translation
Authors: Hugo Touvron
Matthijs Douze
Matthieu Cord
Herve Jegou
Venue:  (2021)

Abstract: We propose a simple architecture to address unpaired image-to-image  translation tasks: style or class transfer, denoising, deblurring, deblocking, etc. 
We start from an image autoencoder architecture with fixed weights. 
For each task we learn a residual block operating in the latent space, which is iteratively called until the target domain is reached. 
A specific training schedule is required to alleviate the exponentiation effect of the iterations. 
At test time, it offers several advantages: the number of weight parameters is limited and the compositional design allows one to modulate the strength of the transformation with the number of iterations. 
This is useful, for instance, when the type or amount of noise to suppress is not known in advance.  
Experimentally, we show that the performance of our model is comparable or better than CycleGAN and Nice-GAN with fewer parameters.

Under review as a conference paper at ICLR 2021
POWERS OF LAYERS FOR IMAGE-TO-IMAGE TRANSLATION
Anonymous authors Paper under double-blind review
ABSTRACT
We propose a simple architecture to address unpaired image-to-image translation tasks: style or class transfer, denoising, deblurring, deblocking, etc. We start from an image autoencoder architecture with fixed weights. For each task we learn a residual block operating in the latent space, which is applied iteratively until the target domain is reached. A specific training schedule is required alleviate the exponentiation effect of the iterations. At test time, it offers several advantages: the number of weight parameters is limited and the strength of the transformation can be modulated simply with the number of iterations. This is useful, for instance, when the type or amount of noise to suppress is not known advance. Experimentally, show that the performance of our model is comparable or better than CycleGAN and Nice-GAN with fewer parameters.
INTRODUCTION
Neural networks define arbitrarily complex functions involved in discriminative generative tasks by stacking layers, as supported by the universal approximation theorem (Horii et al.] 1989; Montufar 2014). More precisely, the theorem states that stacking a number of basic blocks can approximate any function with arbitrary precision, provided it has enough hidden units, with mild conditions on the non-linear basic blocks. Studies on non-linear complex holomorphic functions involved in escape-time fractals showed that iterating simple non-linear functions can also construct arbitrarily complex landscapes (Barnsley et al., 1988). These functions are complex in the sense that their iso-surfaces are made arbitrarily large by increasing the number of iterations. Yet there is no control on the actual shape of the resulting function. This why generative fractals remain mathematical curiosities or at best tools to construct intriguing landscapes.
In this paper, our objective to combine the expressive power of both constructions, and we study the optimization of function that iterates a single building block in the latent space of an auto-encoder. We focus on image translation tasks, that can be trained from either paired or unpaired data. In the paired case, pairs of corresponding input and output images provided during training. It offers a direct supervision, so the best results are usually obtained with these methods (Chen et al. 2017: Wang et al. 2018 Park al. 2019). We focus unpaired translation: only two corpora of images are provided, one for the input domain A and the other for the output domain B. Therefore do not have access to any parallel data (Conneau et al., 2017), which is a realistic scenario in many applications, e.g., image restoration. We train a function fAB : A + B, such that the output 6" Fa) for A is indiscernible from images of B.
Our transformation is performed by a single residual block that is composed a variable number of times. We obtain this compositional property thanks progressive learning scheme that ensures that the output is valid for large range of iterations. As a result, we can modulate the strength of the transformation by varying the number of times the transformation composed. This is of particular interest in image translation tasks such as denoising, where the noise level is unknown at training time, and style transfer, where the user may want to select the best rendering. This "Powers of layers" (PoL) mechanism is illustrated in Figure | in the category transfer context (horse to zebra).
Our architecture is very simple and only the weights the residual block differ depending on the task, which makes it suitable to address a large number of tasks with a limited number of parameters. This proposal is in sharp contrast with the trend of current state-of- of-the-art works to specialize the architecture and to increase its complexity and number of parameters (Fu et al. 2019 VViazovetskyi jet al.] 2020) Choi et al.) 2020). Despite its simplicity, our proof of concept exhibits similar or better performance than vanilla CycleGAN architecture, all things being equal otherwise, for the

---

Under review as a conference paper at ICLR 2021
reS
Encoder
Decoder
Stopping
criverion
Discriminator
Illustration of Powers of layers for a category transfer task. The encoder and decoder directly borrowed from a vanilla auto-encoder and are not learnable. At inference time, we apply variable number of compositions, producing different images depending compose the residual block the embedding space. Depending on the task, we either modulate the transformation and choose the result, let discriminator determine when stop iterating.
Figure 1:
how many times
original set of image-to-image translation tasks proposed in their papers, as well as for denoising, deblurring and deblocking. With significantly fewer parameters and versatile architecture, we report competitive results confirmed by objective and psycho-visual metrics, illustrated by visualizations. We will provide the implementation for the sake of reproducibility.
RELATED WORK
Generative adversarial networks (GANs) Goodfellow et al. 2014 is a framework where two networks, generator and discriminator, learned together in zero-sum game fashion. The gen- erator learns to produce more and more realistic images w.r.t. the training dataset. The discriminator learns separate between real data and increasingly realistic generated images. GANs used in many tasks such as domain adaptation (Bousmalis et al. 2016). style transfer (Karras et al.) (019b), inpainting (Pathak et al. 2016) and talking head generation (Zakharov et al. 2019).
Unpaired image-to-image translation considers the tasks of transforming an image from domain A into an image a domain B3 The training set comprises sample of images from domains A and B, but no pairs of corresponding images. A classical approach is to train two generators (A4 - B and - .A) and two discriminators, one for each domain. When there a shared latent space between the domains, a possible choice is to use variational auto encoder like in CoGAN Liu & Tuzel (2016). CycleGAN Zhu et al. (2017). DualGAN Yi et al. (217)) and subsequent works (Luu et 2019 Fu
al. 2019 Choi et al.] (020) augment the adversarial loss induced by the discriminators with cycle consistency constraint to preserve semantic information throughout the domain changes. All these variants have architectures roughly similar to CycleGAN: encoder, a decoder and residual blocks operating on the latent space. They also incorporate elements of other networks such as StyleGAN Karras et al. 2019a). In work, build upon simplified form of the CycleGAN architecture that generalizes over tasks easily. We applied also our method to NICE-GAN Chen al. (222). A concurrent work, GANHopper by Lira al. (2020) proposes iterate CycleGAN generators in order to perform the transformation. However, their method has many difference single residual block, their encoder and decoder
with ours: They iterate full generators and are not fixed, their number of iteration is fixed and they have to use additional discriminators to on intermediate transformation states. Other works ZZhag et al., 2019 Li et al. 2020) use recurrent networks to perform transformations but this is done in a paired context, which results in very different methods from ours and GANHopper.
Transformation modulation is an interpolation between two image domains. It is byproduct of some approaches:For instance, linear interpolation in latent space (Brock et al.) 2018 Radlord et al. 2015) morphs between two images. Nevertheless, one important limitation that the starting and ending points must both be known, which is not the case in unpaired learning. Other approaches such the Fader networks (Lample et al. 2017) ) StyleGan2 (Viazovetskyi et al. 2020) act scalar boolean attributes that are disentangled in the latent space (eg., age for face images, wear glasses or not, etc). Nevertheless, this results in complex models, for which dataset size and the variability of
2

---

Under review as a conference paper at ICLR 2021
images strongly impacts the performance: they fail to modulate the transform with small datasets or with large variabilities. A comparison of PoL with the Fader network is provided in Appendix C and shows that our approach is more effective.
Progressive learning and inference time modulation are performed in multi-scale methods such as SinGAN (Rott Shaham al., 2019) and ProgressiveGANN (Karras al., 2017). Progressive learning obtains excellent results for high resolution images where it is more difficult to use classical approaches. The training is performed in several steps during which the size of both the images and the network are increased. The inference time of some architectures can be modulated by stopping the forward pass at some layer (Huang et al., 2016 Wu al., 2017; Veit & Belongie 2017). This differs from our approach, where the number of residual block compositions ("powers" can be chosen, to shorten the inference. by-product a reduction of the number of network parameters.
Weight sharing is way of looking at our method, because the same layer is applied several times within the same network. Recurrent Neural Networks (RNN) are the most classical example weight sharing in recursive architecture. Besides RNNs, weight sharing is mainly used for model compression, sequential data and ordinary differential equations (ODE) (Gao et al. 2019 Han et al. 2013, Chen et al.) 2018). A. few works (Jastrzebski et al. 2017 Zhang& & Jung 2018) apply weight sharing to unfold a ResNet and evaluate its performance in classification tasks. The optimization is inherently difficult, so they use independent batch normalization for each shared layer. With PoL we observe the same optimization issues, that we solve by a progressive training strategy, see Section 3.3 Recent work (Jeon et al. 2020) interested in the composition of the same block by considering the parallel with the fixed point theorem, nevertheless their application remains to rather simple problems compared to our unpaired image-to-image translation tasks.
POWER OF LAYERS
We adopt the same context CycleGANand focus unpaired image-to-image translation: the objective is to transform an image from domain A into an image from domain B. In our case the domains can be noise levels. painting styles, blur, JPEG artifacts, or simply object classes that appear in the image. The training is unpaired: we do not have pairs of corresponding images at training time. CycleGAN is simple, adaptable to different tasks and allows direct comparison in Sections and
We learn two generators and two discriminators. The generator GAB I + transforms an element of A into an element of B, and GBA goes the other way round, 7 being the fixed-resolution image space. The discriminators A I - [o, 1] (resp. D&) predicts whether an element belongs to domain A (resp B). We use the same losses as commonly used in unpaired image-to-image translation:
LTotal - AAdvLAdv + ACye AtaCid,
(1)
where Laav
CCav(G (GAB, DB) = Eo-s(log Db + AooaaAllo((( - DD(GABB()))
bll,1 + (l (C GBA) Es A
alli].
LIa(GAn, GBA) = Em-B| Caaess|[Gniie (n) - bl|g] + Ee- alla)
The Adversarial loss LAAv(G (GAv(GAR AR, DR) verifies that the generated images are in the correct domain. The Cycle Consistency loss (GB, GBA) ensures round-trip through the two generators recon- structs the initial image, and the identity loss (ia GBA) penalizes the generators transforming images that are already their target domain. We keep the same linear combination coefficients as in CycleGAN:AAdv = 1, ACye 10, ALd = 5.
3.1 NETWORK ARCHITECTURE
We start from the CycleGAN architecture. The encoder and decoder consist of layers and a residual block. The embedding space & of our model is 256 x 64 64: its spatial resolution is 1/4 the input image resolution of 256 x 256 and has 256 channels. All translation operations take place in the fixed embedding space E. The encoder Enc I - & produces the embedding and consists of two convolutions. The decoder Dec : E + turns the embedding back to image space and consists of two transposed convolutions.
Pre-training of the auto-encoder. We train the encoder and decoder of our model on a recon- struction task using an &g reconstruction loss in pixel space. We the Adam optimizer. Our
3

---

Under review as a conference paper at ICLR 2021
data-augen -ugammntationn consists of image resizing, a random crop and random horizontal flip. Both the encoder and decoder weights are fixed for all the other tasks, only the residual block is adapted (and the discriminator in case we use it for the stopping criterion).
There are two choices to train the auto-encoder: (1) train with 6M unlabeled images randomly drawn from the YFCC100M dataset (Thomee al.) 2016) during
epoch, which makes it independent of translation task. For simplicity, we choose the first option and use single encoder and decoder for all the tasks presented in this article. For the NiceGAN experiments, the encoder and decoder are trained jointly with the discriminator, so the auxiliary dataset YFCC100M not used.
the translation training dataset; (2) train on the dataset of our unpaired image-to-imal
The embedding transformer single block. The transformation between domains is based on residual block fAB. similar the feed-forward network used in transformers (Vaswani et al. 2017).
It writes:
f (z) x (C)) Ve A.
There is dimensionality expansion factor K between the two convolutions in the residual block (see Figure D. Adjusting K changes the model's capacity, We adopt the now standard choice of the original transformer paper (K 4). The full generator writes
GAB (3) = Dec([AssEnn((z)) Vo E A.
The other direction, with fGA and reSBA, defined accordingly.
Powers of layers. We start from the architecture above and augment its representation capacity. There two standard ways of doing this: (1) augmenting the capacity of the LAs block by increasing K: (2) increasing the depth of the network by chaining several instances of fAs. since the intermediate representations are compatible.
contrast to these fixed architectures, PoL iterates the fAB block n 2 times, which amounts to sharing the weights of deeper network:
G (B) (ec((("((En((z))) Vr E A.
3.2 OPTIMIZATION IN RESIDUALS BLOCKS WEIGHT SHARING CONTEXT
In the following, drop the AB suffix from fAB, since powers of layers operates in the same way on fAB and fSA- Thus, f >> E is f(x) = x + res( The parameters of f are collected in vector The embedding E & is 3D activation map, but for the sake of the mathematical derivation we linearize it to vector. The partial derivatives of f are öf ares + Id and 0f dres We compose the / function times as
Du
05" dx (z) = II
(f(+)) and dr i=n-1
0
as
05 dw
(z) = II
as dr
(F(x))
05 (z).
(5)
Dw
i=n-1
The stability of the SGD optimization depends on the magnitude and conditioning of the matrix:
Mn II (=1-1
(f(2)) of Oz
0f
II (=--1 òr (f((z)) (Id)))+d)
(ores
Ôres
(6)
which is sensitive to initialization during the first optimization epochs. Indeed, the length of the SGD steps on w depends on the eigenvalues of M,. When simplifying the basic residual block to linear transformation L E pdddd (i.e., ignoring the normalization and the ReLU non-linearity),
have M, = (L+Id)"~1 The eigenvalues of M, are (A; +1="1 where λı.... Au are the eigenvalues of L. At initialization, the components of L are sampled from random uniform distribution. To reduce the magnitude of X; one option is to make the entries of L small. However, to decrease (A; + 1)a-1 sufficiently, A; must be so small that it introduces floating-pon mating cancellations when the residual block is added back to the shortcut connection. This is why we prefer to adjust n, detailed next.
3.3 PROGRESSIVE TRAINING
We adopt a progressive learning schedule in a "warm up" phase: we start the optimization with single block and add one iteration at every epoch until we reach the required blocks. This is
4

---

Under review as a conference paper at ICLR 2021
Gaussian noise (std=30)
Gaussian blur (crw4)
Table 1: Denoising: PSNR Urban-100) Huang et al. 2015). Comparison between Power of layers (POL) and independent (ind) blocks for different maximum number of composition / residual blocks. The best value for each column is in Bold. We could not more than 16 independent blocks in memory in our experiments. Appendix reports standard deviations and more results.
Fir
POL
ind
POL
23.3
23.3
18.6
18.6
4
24.4
23.2
19.2
19.2
23.9
22.3
19.3
12
23.9
22.5
19.7
18.8
23.9
18
24.2
19.0
30
23.5
19.0
possible because the blocks operate in the same embedding space & and because their weights shared, so all blocks are still in the same learning schedule. This approach avoids the numerical exponentiation effects at initialization mentioned in the previous paragraph. After the warm-up phase, the gradient descent starts to converge, so these numerical effects become unlikely. In addition, this approach allows the discriminator to improve progressively during the training. For example, in the case of the transformation horse - zebra, a slightly whitened horse fools the discriminator at the beginning of the training, but texture with stronger stripes is required later on.
Training for modulation. If the network is trained with fixed number of compositions, the intermediate states do not correspond to modulations of the transformation that "look right" (see Appendix A Therefore, in addition to this scheduled number of iterations during the first epochs of warm-up, we also randomize the number of iterations in subsequent epochs. This forces the generator to also produce acceptable intermediate states, and enables modulating the transform.
Stopping criterion at inference time. Each image of domain A is more or less close to domain B. For example, when denoising, the noise level can vary the denoising strength should adapt to the input. Similarly for horse->zebra: a white horse is closer a zebra than a brown horse. Therefore, at inference time, we select n that best deceives the discriminator, thus effectively adapting the processing to the relative distance to the target domain.
adjust the number of compositions as well. In particular, for each test image,
ANALYSIS
In this section we study the impact of the main training and design choices and on the performance of powers of layers. Appendices and provide complementary analysis for training- and inference- time choices, respectively. For this preliminary analysis, we focus on denoising tasks for which the performance is easily measurable. We add three types of noise to images: Gaussian noise, Gaussian blur and JPEG artifacts. The noise intensity is quantified by the noise standard deviation, the blur radius and the JPEG quality factor, respectively. We generate transformed images and measure how well our method recovers the initial image. This provides a more reliable accuracy measure than a purely unpaired evaluation, that will be carried in the experimental section.
Note that, in the literature, these tasks are best addressed by providing (original, noisy) pairs if images. Our objective is to remain in completely unpaired setting during the training phase. It corresponds to the case where parallel data is not available (like for the restoration of ancient movies), and also better reflects the situation where the noise strength is not known in advance. Therefore, the original image is solely employed to measure the performance.
Experimental protocol. To train, we sample 800 domain images from the high-resolution Div2K dataset (Agustsson & Timolte, 2017). In the baseline training procedure, the warm-up phase starts from a single block and increases the number of compositions every epoch, until we reach epoch Mer. Then we keep the number of compositions fixed.
We test on the Urban-100 (Huang et al. 2015) dataset. Unless specified otherwise, we set the number of compositions The Tüte and measure the Peak Signal to Noise Ratio (PSNR) of our model on the dataset images, degraded with the same intensity as at training time. For the JPEG case we use the Naturalness Image Quality Evaluator metric (NIQE, lower=better) instead, because it is more sensitive to JPEG artifacts (Mittal et al. 2013). NIQE is a perceptual metric that does not take the unpaired metric).
original image into account (i.e. for JPEG emphdo
Block composition or independent successive blocks? Table MaaleToooe compares PoL's composition of the same block versus using independent blocks with distinct weights. In spite of the much larger
5

---

Under review as a conference paper at ICLR 2021
Discriminator Natural images
Diseriminator of Noisy images
Figure 2: Response of the discriminators as function of the number of composi- tions for the transformation Gaussian noise- im- age. We plot the average re- sponse and the standard de- viation over examples (gray). Higher=the discriminator clas- sifies the image into its target domain. Top: training with a fixed number of compositions, =30 Bottom: training with randomised Thr E [20, 30].
Fixed
30
20
25
35 compositions (mm)
20
25
30
46
compositions (nt))
Random Mar [20,
30
30 Number compositions (nt))
Number compositions (mm))
capacity offered by independent blocks, the noise reduction operated by Power of layers is stronger. Our interpretation is that the model is easier to train.
Analysis of the progressive training strategy. Table also evaluates the impact of the maximum number of compositions 7tr- Having several compositions clearly helps. Since we choose the number of compositions The at inference time (see next paragraph), it may relevant to vary Thr training time to minimize the discrepancy between the train- and test-time settings. For this, we tried different intervals to randomly draw the maximum number compositions for each epoch, after the warm-up phase. If Aee is fixed, the optimal choice is Thr Tite However, if we use adaptive Ate the best range The [20, 30], and the adaptive case with randomised training gives the best performance for denoising and debluring. Appendix reports results obtained with different f4r ranges.
Stopping criterion. We consider two cases: either we use fixed Nite. or we use the discriminator to evaluate the transformation quality: selects the value aie maximizing the target discriminator error for given image. Figure 2 shows that setting a fixed far causes the discriminator to select 7te Bar the best iteration at inference time. By selecting the best The for each image obtain average a PSNR improvement of +1.+1.. +1.36dB for a Gaussian noise of standard deviation 30, compared to fixing nee. In Appendix B we compare it with the best possible stopping criterion: an oracle that selects directly on the PSNR. Our adaptive strategy significantly tightens the gap to this upper bound.
Comparison with CycleGAN. We use CycleGAN as a baseline. The differences between Cycle- GAN and powers of layers are (1) we use single encoder and decoder trained in advance and common to all tasks: (2) CycleGAN has 9 residual blocks, PoL iterates single residual block an arbitrary number of times. The inference time of Pol. depends on the number of compositions The but the number of parameters does not:
encoder decoder residual block 1x 1.7M
discriminators
total
PoL CycleGAN
2 x K 1.1M
2x2.7M 2x2.7M
15.9M
2x 11.4M
28.2M
Figure compares the performance obtained by the two methods on denoising tasks with varying noise intensities. PoL. gives better results than CycleGAN in terms of objective metrics, and overall the images produced by our method look as realistic and/or accurate.
Training with few images. Figure also compares our method with CycleGAN when training in data-starving scenario. Whatever the number of training images, PoL. outperforms CycleGAN, but gap is particularly important with very few images. This expected for two reasons. Firstly, approach has fewer parameters to learn than CycleGAN. Secondly, it only requires to learn the transformation because the encoder and decoder are pre-learned as a vanilla auto-encoder, while CycleGAN needs to learn how to encode and decode images.
Parametrization: remarks. Beyond the settings inherited from CycleGAN, the main training parameters of Powers of layers are the maximum number of compositions 74r and the range from which they are randomly sampled. The number compositions at inference time nu also important but the discriminator criterion can be used to set it automatically.

---

Under review as a conference paper at ICLR 2021
Ingreti intemity applial Uthan- imegra
Ingaet training
PSNR (bighee-bene))
NIQE (lowerbetter)
PSNR
NIÇE
Noisy
Signia
Deblucing
JPEG
(sid=30)
(amé)
(qual
(sid)
images
images
quality
images
20.4
22.1
9.01
7.9
14.5
7.9
8.0
7.6
8.9
7.5
7.3
127
21.0
20.9
16.1
16.1
16.2
9.0
7.0
21.0
23.8
18.2
19.
7.5
7.0
100
10.4
20.0
19.4
24
12.9
153
8.9
7.1
21.8
23.6
18.2
19.0
6.9
Debloking
Denoising Image with Soisef 10
Debluing
Original Image
Image IPEG (gulliy=25)
CycleGAN
Image with Blar (m=i))
CycleGAN
CydleGAN
(diseiiminatr))
Pvl.
Pol. (divcriminstor)
Figure 3: Top: Comparison between Powers of layers (PoL) and CycleGAN (CLG) to denoise
images of 1ooanng et al. (2013). We provide standard deviation and additional results in
Appendix Bottom: Visual comparison between our method (manual and discriminator choice) and CycleGAN for deblocking, denoising and debluring.
summer-
Monet-
orange tapple
CycleGAN
POL/ CyeleGAN
POLS
Origital imnge
Summer Winter
48.8
46.1
40.5
41.9
Summer Winter
48.4
44.4
39.5
Horse Zebra
89.7
53.0
4488
45.7
Horse + Zobra
110.5
112.5
69.1
62.2
Van Gogh Picture
163.4
134.4
101.4
102.8
Pineess layers CyllCAN
Gogh Picture
Ceranne -> Picrare
127.4
138.8
91.1
92.2
Cozanne Picrare
145.5
147.6
126.1
122.9
Picture
Monel Picture
6188
55.4
Apple Orange
88.9
83.2
112.7
114.6
Orange
116.7
113.2
84.8
80.7
Figure 4: Left: Frechet Inception Distance (FID) for image-to-image translation tasks (lower is better). We compare CycleGAN and NiceGAN with POL/CycleGAN and POL/NiceGAN, where the respective translation functions are replaced with POL blocks. Right: example results.
5 EXPERIMENTS
We now run experiments on two image generation applications. We refer to Section 3.1 for the architecture and training protocol. In Appendix C we also give a comparison with the Fader network for the capacity to modulate a transformation, and more visual examples in Appendix D
Unpaired image-to-iman translation. We report results for of the unpaired image-to-image translation tasks introduced in the CycleGAN (Zhu et al. 2017) paper (the two remaining ones lead to the same conclusions) and we used the datasets from their website. We compare the Frechet Inception Distance (FID) of CycleGAN, NiceGAN and our approach applied to this method in Figure 4 The FID measures the similarity between two datasets images, we use it to compare the target dataset with the transformed dataset. It is noisy measure for which only large deviations significant. Yet the results and visualization show that our method has results comparable to those of CycleGAN and NiceGAN, achieved with much fewer parameters.
High resolution experiments. PoL is fully convolutional architecture, therefore it is technically possible to apply models trained in low resolution to high resolution images. However, the results
7

---

Under review as a conference paper at ICLR 2021
Photo Van Gongh
Photo Monet
Horse - Zehra
Toined
Different visual results with high resolution images. See the original images in Appendix D Right: Comparisons of the generations obtained by models trained either with high-resolution low-resolution images, applied to high-resolution test image.
Figure 5: Left:
Optimal number of compositions
25
20
15-
30.0
27.5
25.0
22.5
20.0
17.5
15.0
12.5
Noisy image initial PSNR (dB)
Figure 6: Evolution of the optimal number of composition according the noise. The maximum number of compositions used for training is 30 and the Gaussian Noise standard deviation is 30.
are not always convincing, as shown in Figure 5 (right) where the model trained on low resolution images does not create stripes the "right" scale zebras. To circumvent this problem, CycleGAN trains on patches taken from high resolution images. This works for transformations affecting the whole image (painting<<photo)) but this is applicable in the where only part of the image is affected (horse -zebra. In contrast, our proposal can adapt the memory used by changing its number of compositions, so we can apply
very large images without running out of memory. Figure (left) depicts results obtained with our method trained on high resolution images.
Adjusting Thee at inference time. Each image is more or less distant from the target domain, we explore adapting the transformation to each image rather than applying fixed transformation. For example, depending on the amount of noise, we may want adjust the strength of the denoising. By modulating nte we can adapt the transformation to each image. Figure shows that the more noisy the input image is, the more we should compose best denoise with Powers-of-Laaers.
In the same way, Figure shows the progressive transformations obtained on different unpaired image-to-image translation tasks. It shows that progressive transformations are realistic for most tasks.
8

---

Under review as a conference paper at ICLR 2021
Domain Composition
5
10
15
20
30
35
40
Horse Zebra
Zobra Horso
Summer + Winter
Winter - Summer
Photo
Monet
photo
Figure 7:
Illustration of the different results obtained along the iterations of the recurrent block, during the transformation of horse into zebra. The first image is the original image, then each image corresponds to 5 additional compositions of our method. Since our method was learned with maximum of 30 compositions the last two images extrapolations.
Combining transformation. The different blocks associated with different transformations operate in the same embedding space for different tasks. Hence can compose transformations, each being realized by one residual block. We train Transform #1 in the usual way, then freeze its residual block. Transform #2 is trained the output of #1. Visual results are in Figure The composition in the embedding space gives better results than decoding/encoding to image space mid-way.
CONCLUSION
Powers of layers iterates a residual block to learns a complex transformation with no direct supervision. On various tasks, It gives similar performance CycleGAN and NiceGAN with fewer parameters. The flexibility offered by the common embedding space allows the modulation of the transformation strength, or compose several transformations. While in most examples the discriminator is only used for training, Powers of layers can also exploit it to adjust the transformation to the input image inference time.
Composition embedding space +denoising
Composition in image space +denoising
Original
+1o zebra
+to zebra
Figure Composition of transformations in the embedding/image space.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2017.
Michael F Barnsley, Robert L Devaney, Benoit B Mandelbrot, Heinz- Otto Peitgen, Dietmar Saupe, Richard F Voss, Yuval Fisher, and Michael McGuire. The science of fractal images. Springer, 1988.
Konstantinos Bousmalis, George Trigeorgis. Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. In Advances Neural Information Processing Systems, 2016.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. International Conference on Learning Representations, 2018.
Qifeng Chen, Jia Xu, and Vladlen Koltun. Fast image processing with fully-convolutionall networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2497-2506, 2017.
Runfa Chen, Wenbing Huang, Binghui Huang. Fuchun Sun, and Bin Fang. Reusing discriminators for encoding: Towards unsupervised image-to-image translation. arXiv preprint a3X1v:2003.00223, 2020.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems, 2018.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. Conference on Computer Vision and Pattern Recognition, 2020.
Alexis Conneau, Guillaume Lample. Marc' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. arXiv preprint arXiv:/710.040077 2017.
Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, Kun Zhang, and Dacheng Tao. Geometry-consister generative adversarial networks for one-sided unsupervised domain mapping. Conference on Computer Vision and Pattern Recognition, 2019.
Shangqian Gao, Cheng Deng, and Heng Huang. Cross domain model compression by structurally weight sharing. Conference Computer Vision and Pattern Recognition, 2019.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, 2014.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. CoRR, 2015.
Kurt Hornik, Maxwell Stinchcombe, Halbert White, et al. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359-366, 1989.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In European Conference on Computer Vision, 2016.
Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-exemplars. In Conference on Computer Vision and Pattern Recognition, 2015.
Stanislaw Jastrzebski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Ben- gio. Residual connections encourage iterative inference. International Conference on Learning Representations, 2017.
Younahan Jeon, Minsik Lee, and Jin Young Choi. Differentiable fixed soedt iteration layer. arXiv preprint arXiv:2002.0288 2020.
Tero Karras, Timo Aila. Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In International Conference on Learning Representations, 2017.
10

---

Under review as a conference paper at ICLR 2021
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Conference on Computer Vision and Pattern Recognition, 2019a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. arXiv preprint
2019b.
Guillaume Lample, Neil Zeghidour, Nicolas Usunier, Antoine Bordes, Ludovic Denoyer, et al. Fader networks: Manipulating images by sliding attributes. In Advances in Neural Information Processing Systems, 2017.
Jingyuan Li, Ning Wang, Lefei Zhang, B. Du, and Dacheng Tao. Recurrent feature reasoning for image inpainting. Conference on Computer Vision and Pattern Recognition, 2020.
Wallace P. Lira, Johannes Merz, D. Ritchie, D. Cohen-Or, and Hao Zhang. Ganhopper: Multi-hop gan for unsupervised maage-to-image translation. European Conference Computer Vision, 2020.
Ming- Yu Liu and Oncel Tuzel. Coupled generative adversarial networks. In Advances in Neural Information Processing Systems, 2016.
Ming- Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. Few-shot unsupervised image-to-image translation. In International Conference Computer Vision, 2019.
Anish Mittal, Rajiv Soundararajan, and Alan C. Bovik. Making a "completely blind" image quality analyzer. IEEE Signal Processing Letters, 2013.
Guido F Montúfar. Universal approximation depth and errors of narrow belief networks with discrete units. Neural computation, 26(7):138-11477, 2014.
Taesung Park, Ming-Yu Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu, Semantic image synthesis with adatially-adaptive normalization. In Conference Computer Vision and Pattern Recognition, 2019.
Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Context encoders: Feature learning by inpainting. Conference on Computer Vision and Pattern Recognition, 2016.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep con- volutional generative adversarial networks. International Conference on Learning Representations, 2015.
Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. Singan: Learning a generative model from a single natural image. In International Conference on Computer Vision, 2019.
Bart Thomee, David A. Shamma. Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Lijia Li. Yfcc100m: the new data multimedia research. Commun. ACM, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
Andreas Veit and Serge J. Belongie. Convolutional networks with adaptive inference graphs. In European Conference on Computer Vision, 2017.
Yuri Viazovetskyi, Vladimir Ivashkin, and Evgeny Kashin. Stylegan2 distillation for feed-forward image manipulation, 2020.
Ting- Chun Wang, Ming- Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Conference on Computer Vision and Pattern Recognition, 2018
Zuxuan WI, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S. Davis, Kristen Grauman, and Rogério Schmidt Feris. Blockdrop: Dynamic inference paths in residual networks. Conference Computer Vision and Pattern Recognition, 2017.

---

Under review as a conference paper at ICLR 2021
Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image-to-imas translation. International Conference on Computer Vision, pp. 1868-2876, 2017.
Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor S. Lempitsky. Few-shot adversarial learning of realistic neural talking head models. arXiv preprint arXiv:1905.088233, 2019.
Xiaoshuai Zhang, Yiping Lu, Jiaying Liu, and Bin Dong. Dynamically unfolding recurrent restorer: A moving endpoint control method for image restoration. arXiv preprint arXiv:/805.07709, 2019.
Zhendong Zhang and Cheolkon Jung. Recurrent convolutions: A model compression point of view. NIPS Workshops: Compact Deep Neural Network Representation with Industrial Applications, 2018.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In International Conference on Computer Vision, 2017.
12


Reviews:
[{"review_id": "G-sg4zguXo6", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. Summary. The submitted paper proposes to use a recurrent residual block for the task of unpaired img2img translation. A number of strategies of how many times to apply this block are suggested. \n\n1. Decision. I do really like the direction of weight sharing in img2img models and this work is a pretty nice case. This approach helps to decrease the number of weights, and, if done properly, does not harm quality. The results in deblurring/denoising look rather interesting.\n\nCons. \n\nHowever, the downside of the presented recurrent block is the increased computation time at the inference step, as far as I can judge. This is especially crucial when the discriminator is involved as the stopping criterion. Could you provide a comparison of the inference speed (FPS or FLOPS or any other measure) between CycleGAN/NiceGAN and PoL?\n\nSecond, I believe this approach could be also put into the context of the adaptive computation time research field [1,2,3,4]. This may help to determine the number of layers to apply.\n\nThird, the proposed block may be straightforwardly generalized to multi-domain img2img translation and showcased on more interesting and recent datasets against stronger baselines like MUNIT [5], FUNIT[6], etc. This could make img2img part of your experiments more solid, I suppose.\n\nAll-in-all, to my mind there is great room for improvement for your submission to demonstrate the real power of PoL. Therefore, now I tend to rate the submission a bit below the threshold.\n\n[1] https://openreview.net/forum?id=r1W1OxAF\n[2] https://openreview.net/forum?id=SkZq3vyDf\n[3] https://openaccess.thecvf.com/content_cvpr_2017/html/Figurnov_Spatially_Adaptive_Computation_CVPR_2017_paper.html\n[4] https://openreview.net/forum?id=HyzdRiR9Y7\n[5] https://link.springer.com/chapter/10.1007/978-3-030-01219-9_11\n[6] https://openaccess.thecvf.com/content_ICCV_2019/html/Liu_Few-Shot_Unsupervised_Image-to-Image_Translation_ICCV_2019_paper.html", "extra_scores": {}, "extra_text": {}}, {"review_id": "7kOHFIjWEfs", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an approach for image to image translation by introducing extra layers into the generator, which can be trained in an unsupervised way. The paper is generally easy to follow. I have the following concerns: the novelty is quite marginal since the backbone network and the training process are well developed before and the technique of employing more layers to the generator seems like simply extending the network. Please consider to use different notations in Section 3 to denote image from different domains. Also, please use the same subscript for G_{AB} and G_{\\mathcal{BA}}.", "extra_scores": {}, "extra_text": {}}, {"review_id": "5mAkQKsIMN", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method for unsupervised image translation between unpaired domains of images. The main idea is to develop an iterative transformation module that operates in the embedding space.\n\nOverall I have the following concerns about the paper:\n\nThe motivation for this architecture is unclear. The introduction motivates this model with fractals and iterated function spaces, but that seems to have nothing to do with the types of applications shown here. What do IFSs have to do with denoising and pictures of zebras? This iterated refinement strategy seems more similar to iterative refinement/projection algorithms like conjugate gradient and Richardson-Lucy deconvolution, which are relevant to low-level signal processing operations like denoising/deblurring, but not zebra synthesis.  The paper includes as motivation the idea that applying that different levels of transformation can be achieved by choosing different numbers of iterations, but the application of this is shown only for denoising (Table 1). \n\nNo comparison is provided to the state-of-the-art in unpaired image translation: \nContrastive Learning for Unpaired Image-to-Image Translation\nTaesung Park, Alexei Efros, Richard Zhang, Jun-Yan Zhu\nECCV 2020\n\nVisually, the results are not convincing. Not many results are shown, and most do not look better than those from CycleGAN.  The results may be cherry-picked, since there was no statement as to how these results were chosen.  There is simply not enough visual evidence that the method has evidence over previous work. Additionally, quantitatve comparisons do not give a compelling outcome, but I would put more weight on visual comparisons anyway.\n\nFor the task of denoising, it is unclear why one would want to use a general-purpose unpaired translation method; supervised methods ought to be much effective here, and there is an enormous literature of related work that is not cited or compared with here. If one is to use denoising as a motivation application (rather than a toy example), then much more rigor is required. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4BhKUFuipDC", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an unpaired image-to-image translation method which applies a pre-trained auto-encoder and a latent feature transformer (single block) to perform iterative image transformation. A progressive training and warm-up strategy is used to settle the numerical exponentiation effects caused by powers of layers. In the testing phase, the discriminator is also used to adjust the inference time.\n\nPros: \n1) Compared with the vanilla CycleGAN, the proposed PoL has significantly fewer parameters and similar performance.\n2) The flexibility offered by the common embedding space allows the modulation of the transformation strength, or to compose several transformations.\n3) The discriminator is used to adjust the inference time and find the optimal number of iterations in the testing phase.\n\nCons:\n1) CycleGAN is the only competitor in most comparative experiments, which is not sufficient. Besides, CycleGAN is not a good competitor for image restoration tasks (debluring, denoising, etc.), so the potential of the proposed PoL is questionable. Additional comparison results generated by other general image restoration methods [1, 2] should be reported. \n2) Is CycleGAN also pre-trained on the same dataset as PoL for fair comparison?\n3) Although progressive training contributes to more natural intermediate outputs, the final output is not satisfactory, for example the unnatural patterns on zebras in Fig. 4.\n4) More effective transformation modulation is a major advantage of the proposed PoL, but the provided experiments did not demonstrate this well. I think it would be more appropriate to put the results obtained along the iterations of the recurrent block in the main text instead of in the appendix.\n5) Why the proposed embedding transformer “is similar to the feed-forward network used in transformers [3]”? It seems that it is just a simple residual convolution module with expansion factor K to adjust the model’s capacity, which is not related to transformer or self-attention.\n6) What are the significant advantages of PoL compared with traditional RNNs (LSTM, GRU)? Is it possible to directly replace PoL with RNNs to achieve close results?\n\n[1] Neural Sparse Representation for Image Restoration. NeurIPS, 2020.\n[2] Learning Invariant Representation for Unsupervised Image Restoration. CVPR, 2020.\n[3] Attention is all you need. NeurIPS, 2017.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers shared the concerns about the novelty and the quality of the results. Comparisons with some SOTA results are missing, and the inclusion of deblurring/denosing tasks is not convincing. The authors carefully addressed these issues in the rebuttal but the reviewers didn’t change their mind afterwards. After carefully examining the results in the paper, the AC agrees with the reviewers that the improvement on image quality, if any, seems to be too small to warrant a publication. 
Decision: Reject