Adjustable Real-time Style Transfer
Authors: Mohammad Babaeizadeh
Golnaz Ghiasi
Venue:  (2020)
Keywords: Image Style Transfer
Deep Learning

Abstract: Artistic style transfer is the problem of synthesizing an image with content similar to a given image and style similar to another. Although recent feed-forward neural networks can generate stylized images in real-time, these models produce a single stylization given a pair of style/content images, and the user doesn't have control over the synthesized output. Moreover, the style transfer depends on the hyper-parameters of the model with varying ``optimum" for different input images. Therefore, if the stylized output is not appealing to the user, she/he has to try multiple models or retrain one with different hyper-parameters to get a favorite stylization. In this paper, we address these issues by proposing a novel method which allows adjustment of crucial hyper-parameters, after the training and in real-time, through a set of manually adjustable parameters. These parameters enable the user to modify the synthesized outputs from the same pair of style/content images, in search of a favorite stylized image. Our quantitative and qualitative experiments indicate how adjusting these parameters is comparable to retraining the model with different hyper-parameters. We also demonstrate how these parameters can be randomized to generate results which are diverse but still very similar in style and content.

Published as a conference paper at ICLR 2020
ADJUSTABLE REAL-TIME STYLE TRANSFER
Mohammad Babacizadeh' and Golnaz Ghiasi!
!Google Brain
ABSTRACT
Artistic style transfer is the problem of synthesizing an image with content sim- ilar given image and style similar to another. Although recent feed-forward neural networks can generate stylized images in real-time, these models produce a single stylization given pair of style/content images, and the user doesn't 't have control over the synthesized output. Moreover, the style transfer depends on the hyper-parameters of the model with varying "optimum" for different input im- ages. Therefore, if the stylized output is not appealing to the user, she/he has to try multiple models or retrain one with different hyper-parameters to get a favorite stylization. In this paper, we address these issues by proposing novel method which allows adjustment of crucial papper-prramterss. after the training and in real-time, through a set of manually adjustable parameters. These parameters able the user to modify the synthesized outputs from the same pair of style/content images, in search of a favorite stylized image. Our quantitative and qualitative ex- periments indicate how adjusting these parameters is comparable to retraining the model with different syper-parameters We also demonstrate how these parame- ters can be randomized to generate results which are diverse but still very similar in style and content. The source-code of the project is available the project website:
INTRODUCTION
Style transfer is a long-standing problem in computer vision with the goal of synthesizing new images by combining the content of one image with the style of another (Efros & Freeman, 2001 Hertzmann] 1998; Ashikhmin 2001]. Recently, neural style transfer techniques (Gatys et al. 2015
Johnson al. 2016:] Ghiasi al., 2017 Li al. 2018: 2017b) showed that the correlation between the features extracted from the trained deep neural networks is quite effective on capturing the visual styles and content that can be used for generating images similar in style and content. However, since the definition of similarity is inherently vague, the objective of style transfer is not well defined (Dumoulin et al., (017) and one can imagine multiple stylized images from the same pair of content/style images.
Existing real-time style transfer methods generate only one stylization for a given content/style pair and while the stylizations of different methods usually look distinct (Sanakoyeu et al. 2018 Huang & Belongic 2017). is not possible to say that one stylization is better in all contexts since people react differently to images based their background and situation. Hence, get favored stylizations users must try different methods that is not satisfactory. It is more desirable to have a single model which can generate diverse results, but still similar in style and content, in real-time, by adjusting some input parameters.
One other issue with the current methods is their high sensitivity to the hyper-parameters. More specifically, current real-time style transfer methods minimize a weighted sum of losses from dif- ferent layers of a pre-trained image classification model (Johnson et al. 2016 Huang & Belongie, 2017) (check Sec 3 for details) and different weight sets can result into very different styles (Fig- 6p. However, one can only observe the effect of these weights in the final stylization by retraining the model with the new set of weights. Considering the fact that the "optimal" set of weights can be different for any pair of style/content (Figure and also the fact that this "optimal" truly doesn't 't exist (since the goodness of the output is personal choice) retraining the models over and over until the desired result is generated is not practical.

---

Published as a conference paper at ICLR 2020
(Fixed) Contentt
The primary goal of this paper is to address these issues by providing a novel mechanism which allows for adjustment of the stylized im- age, in real-time and after training. To achieve this,
Style
auxiliary network which accepts additional parameters as inputs and changes the style transfer process by adjusting the weights between multiple losses. We show that chang- ing these parameters at inference time results to stylizations similar to the ones achievable by retraining the model with different hyper- parameters. We also show that random se- lection of these parameters at run-time can generate random stylization. These solu- tions, enable the end user to be in full con- trol of how the stylized image is being formed as well as having the capability of generat- ing multiple stochastic stylized images from fixed pair of style/content. The stochastic na- ture of our proposed method is most appar- ent when viewing the transition between ran- dom generations. Therefore, we highly en- courage the reader to check the project website http::/gooo.g/PP gl/PVWQ9K.
Figure I: Adjusting the output of the synthesized styl- ized images real-time without retraining. Each col- umn shows a different stylized image for the same con- tent and style image. Note how each row still resembles the same content and style while different in details.
RELATED WORK
The strength of deep networks in style transfer first demonstrated by Gatys al. (Gatys al., 2016). While this method generates impressive results, is too slow for real-time applications due its optimization loop. Follow up works speed up this process by training feed-forwart networks that can transfer style of a single style image (Johnson et al.] 2016 Ulyanov et al. 2016) or mul- tiple styles (Dumoulin al., 2017). Other works introduced real- time methods to transfer style of arbitrary style image to an arbitrary content image (Ghiasi et al. 2017 Huang & Belongie (2017). Although, these methods can generate stylization for the arbitrary inputs, they can only produce one stylization for a single pair of content/style images. In the case that the user does not like the result, is not possible to get different result without retraining the network for different set of hyper- parameters. Our goal in this paper is to train single network that user can get different stylization without retraining the network.
Generating diverse results have been studied in multiple domains such as colorizations (Desh- pande et al. 2017; Cao et al., 2017), image synthesis (Chen & Koltun| 2017). video predic- tion (Babaeizadeh et al. 2017: Lee al. 2018), and domain transfer (Huang et al. 2018 Zhang 2018 Domain transfer is the most similar problem to the style transfer. Although we can gen- multiple outputs from given input image (Huang et al. 2018 Zhu al.] (017), we need
collection of target or style images for training. Therefore we cannot use it when we do not have a collection of similar styles. For instance, when we want generate multiple stylizations for the Stary Night painting, it is hard to find different similar paintings.
Style loss function is crucial part of style transfer which affects the output stylization significantly. The most common style loss is Gram matrix which computes the second-order statistics of the feature activations (Gatys et al. 2016). however many alternative losses have been introduced to measure distances between feature statistics of the style and stylized images such as correlation alignment loss (Peng & Saenko] 2018), histogram loss (Risser et al.] 2017). and MMD loss (Li al. 2017a). More recent work (Liu et al. 2017) has used depth similarity of style and stylized images as a part of the loss. We demonstrate the success of our method using only Gram matrix; however, our approach can be expanded to utilize other losses as well.
To the best of our knowledge, the only previous work which generates multiple stylizations is (Ulyanov al. 2017) which utilized Julesz ensemble to explicitly encourage diversity in styliza- tions. However their results are quite similar in style and only differ in minor details. A qualitative comparison in Figures 14 show that our proposed method is more effective in diverse stylization.
2

---

Published as a conference paper at ICLR 2020
a c
Loss Calculation
0.
Style
Σ ((li Σ Mot
Adjustable Loss Weights
14
13
12
1
ra
Yas Ba
T
P
Content
Stylized
Figure 2: Architecture of the proposed model. The loss adjustment parameters a and a is passed to the network A which will predict activation normalizers "a and 3a that normalize activation of main stylizing network T. The stylized image is passed intermediate representation is used to calculate the style loss L, and content loss L.. Then the loss from each layer is multiplied by the corresponding input adjustment parameter. Models A and I trained jointly by minimizing this weighted sum. At generation time, values for a and a can be adjusted manually or sampled randomly.
trained image classifier where its
BACKGROUND
3.1 STYLE TRANSFER USING DEEP NETWORKS
Style transfer can be formulated as generating stylized image p which its content similar given content image and its style is close to another given style image s. The similarity in style can be vaguely defined as sharing the same spatial statistics in low-level features, while similarity in content is roughly having close Euclidean distance in high-level features (Ghiasi et al.) 2017)- These features are typically extracted from pre-trained image classification network, commonly VGG-19 (Simonyan & Zisserman, 2014). The main idea here is that the features obtained by the im- age classifier contain information about the content of the input image while the correlation between these features represents its style.
In order to increase the similarity between two images, Gatys et al. (Gatys et al. following distances between their extracted features:
2016 minimize the
(p) = || (p) o' (s(s)] 1 c.(p) = -|G (p)) - G(o (s))
(1)
where of (x) activation
pre-trained classification network at layer given the input image x. while (p) and C (p) are content and style loss at layer I respectively. G(ø (p)) denotes the Gram matrix associated with ø' (p).
The total loss is calculated as the weighted sum of losses across set of content layers C' and style layers S:
Cc = Σ w.....p). C.(p) = Σ wf
LEC
IES
where w/ w hhper--paramenn parameters adjust the contribution of each layer to the loss. Layers be shared between C and S. These hyper-parameters have to be manually fine tuned through try and error and usually vary for different style images (Figure Finally, the objective of style transfer can be defined as:
min (c)(p) + (p))
This objective be minimized by iterative gradient- gradient-based optimization methods starting from an initial p which usually is random noise or the content image itself.
3.2 REAL-TIME STYLE TRANSFER
Solving the objective in Equation using an iterative method can be very slow and has to be repeated for any given pair of style/content image. A much faster method is to directly train deep network which maps a given content image с to a stylized image (Johnson et al. 2016). is usually a convolutional network (parameterized by 0) with residual connections between down-
3

---

Published as a conference paper at ICLR 2020
Content
Style
1.0
Figure 3: Effect of adjusting the input parameters as on stylization. Each row shows the stylized output when single a₂ increased gradually from zero to one while other are fixed to zero Notice how the details of each stylization different specially the last column where the value is maximum. Also note how deeper layers use bigger features of style image to stylize the content.
sampling and up-sampling layers (Ruder et al. 2018) and is trained on many content images using Equation as the loss function:
min (Co(T(c)) (T(()))
The style image is assumed to be fixed and therefore a different network should be trained per style image. However, for fixed style image, this method generate stylized images in real- time (Johnson et al., (2166). Recent methods (Dumoulin et al.] 2017) Ghiasi et al.) [2017] Huang & Belongie, 2017) introduced real-time style transfer methods for multiple styles. But, these methods still generate only one stylization for a pair of style and content images.
PROPOSED METHOD
This paper addresses the following issues in real-time feed-forward style transfer methods:
Style
1e-2
1. The output of these models is sensitive to
the hyper- paprampprreeterss w and w and different weights significantly affect the generated styl- ized image as demonstrated in Figure 6 More- over, the "optimal" weights vary from one style image another (Figure 4 and finding a good set of weights should be repeated for each style image. Note that for each set w! and w the model has to be retrained that limits the practi- cality of style transfer models.
2. Current methods generate single stylized image given a content/style pair. While the styl- izations of different methods usually look very distinct (Sanakoyeu et al. 2018), it is not pos- sible say which stylization better for every context since it is matter of personal taste. To get a favored stylization, users may need try different methods or train network with differ- ent hyper-parameteer which is not satisfactory and, idcally, the user should have the capability of getting different stylizations in real-time.
Figure 4: Effect of adjusting the style weight in style transfer network from (Johnson et al. 2016). Each column demonstrates the result of separate training with all w² set to the printed value. As can be seen, the "optimal" weight different from one style image to another and there can be multiple "good" stylizations depending on ones' personal choice. Check supplementary materials for more examples.
We address these issues by conditioning the generated stylized image on additional input parameters where each parameter controls the share of the loss from a corresponding layer. This solves the problem (1) since one adjust the contribution of each layer to adjust the final stylized result after the training and in real-time. Secondly, we address the problem (2) by randomizing these parameters which result in different stylizations.
4

---

Published as a conference paper at ICLR 2020
Rondonezzed
Purameter
Mondomstoo Content Noize
Botts
Figure 5: Effect of randomizing a and additive Gaussian noise on stylization. Top: randomizing a results to different stylizations while the style features appear in the same spatial position (e.g.. look the swirl effect on the left eye). Middle: the effect adding random noise to the content image in moving these features with fixed a. Bottom: combination of this two randomization techniques can generate highly versatile outputs. Notice how each image this row differs in both style and the spatial position of style elements.
4.1 STYLE TRANSFER WITH ADJUSTABLE LOSS
We enable the users to adjust whng without retraining the model by replacing them with input parameters and conditioning the generated style images on these parameters:
p p= (c, a,,,,,, ae and a, are vectors of parameters where each element corresponds to a different layer in content
layers C' and style layers S respectively. a and a' replace the hyper-paramens parameters w/ and wa in the
objective Equation 2
(.(p)
Σ (p.c.(p) IEC
(.) -o.cc( LES
(5)
To learn the effect of ar and a₂ on the objective, we use a technique called conditional instance normalization (Ulyanov et al.). This method transforms the activations of layer x in the feed- forward network to a normalized activation = conditioned on extra inputs a (az, a.):
2 %ac 1 1) Sa
where y and are mean and standard deviation of activations at layer across spatial axes (Ghiasi jet al., 2017) and Tas 3a are the learned mean and standard deviation of this transformation. These parameters can be approximated using second neural network which will be trained end-to-end with
Ya Pa A(ac,, a.)
(7)
Since L can be very different in scale, one loss term may dominate the others which will fail the training. To balance the losses, we normalize them using their exponential moving average as a normalizing factor, i.e. each L will be normalized to:
c'(p) Eieous (2( * (')
(8)
(P(p)
where (p) is the exponential moving average of (p).
5 EXPERIMENTS
In this section, first study the effect of adjusting the input parameters in method. Then demonstrate that we can use our method to generate random stylizations and finally, we compare method with few baselines in terms of generating random stylizations.
5

---

Published as a conference paper at ICLR 2020
cOnDd cont3 cOMD2
come3 come2 3
cORD4_5 3
COMD3 3
comu2 3
comu3 3
comu2 3
Content
base
Style
ours
Style
Figure 6: Qualitative comparison between the base model from (Johnson et al. 2016) with our proposed method. For the base model, each column has been retrained with all the weights set to zero except for the mentioned layers which has been set to 1c-3. For our model, the respective parameters a has been adjusted. Note how close the stylizations and how the combination of layers stays the same both models.
5.1 IMPLEMENTATION DETAILS
We implemented A multilayer fully connected neural network. We used the same architecture as (Johnson et al. 2016) Dumoulin et al., (017)] Ghiasi et al., 2017) for 7 and only increased number of residual blocks by (look supplementary materials for details) which improved stylization results. We trained T' and A jointly by sampling random values for a from (0, 1). We trained our model ImageNet (Deng et al. 2009) as content images while using paintings from Kaggle Painter by Numbers (Kaggle) and textures from Descibable Texture Dataset (Cimpoi et al., 2014) as style images. We selected random images form ImageNet test set, MS-COCO (Lin et 2014) and faces from CelebA dataset (Liu et al. 2018) as our content test images. Similar to (Ghiasi et al. 2017 Dumouuin al., (017), we used the last feature set of cona3 as content layer feature set of conv2, cone3 and cone4 layers from VGG-19 network as style layers S. Since there only one content layer, we fix a 1. Our implementation can process 47.5 fps on a NVIDIA GeForce 1080, compared to 52.0 for the base model without A.
used last
5.2 EFFECT OF ADJUSTING THE INPUT PARAMETERS
The primary goal of introducing the adjustable parameters was modify the loss of each separate layer manually. Qualitatively, this is demonstrable by increasing one of the input parameters from zero to one while fixing the rest of them to zero. Figure 3 shows one example of such transition. Each row in this figure is corresponding to different style layer, and therefore the stylizations at each row would be different. Notice how deeper layers stylize the image with bigger stylization elements from the style image but all of them still apply the coloring. We also visualize the effect of increasing two the input parameters at the same time Figure 9 However, these transitions are best demonstrated interactively which is accessible at the project website https://goool/P/WW//990
To quantitatively demonstrate the change in losses with adjustment of the input parameters, we rerun the same experiment of assigning fixed value to all of the input parameters while gradually increasing one of them from zero to one, this time across 100 different content images. Then we calculate the median loss at each style loss layer S. As can be seen in Figure (top), increasing

---

Published as a conference paper at ICLR 2020
Zun- time Noise Truin WainWWastte tume Matic StyleNer
Other parameters are flxed ato.0
Other paramelers are feod 1.
12
-4
04
02
Content
08:
2.
0.0
Style
62
2.0
8.0
Look
Figure 8: Diversity comparison of our method and baselines. First row shows results for base- line that adds random noises to the style param- eters run-time. While we get diverse styliza- tions, the results are not similar to the input style image. Second row contains results for base- line that adds random noises to the style param- eters both training time and run-time. Model is robust to the noise and it does not generate di- verse results. Third row shows stylization results of StyleNet (Ulyanov et al. 2017). Our method generates diverse stylizations while StyleNet sults mostly differ minor details. More exam- ples can be seen at Figure 14
Figure 7: Effect of adjusting the input parame- a. style loss from different layers single style image of Figure (top) or 25 different style images (bottom). input parameters a², has been increased from zero to one while others fixed at to zero (left) and one (right). Then the style loss has been cal- culated across different content images. can be seen, increasing a decreases the loss of the corresponding layer. Note that the losses is normalized in each layer for better visualization.
each curve, one the
af decreases the measured loss corresponding to that parameter. To show the generalization of our method across style images, we trained 25 models with different style images and then measured median of the loss at any of the S layers for 100 different content images (Figure (bottoom) The similarly exhibited trends show that the model can generate stylizations conditioned on the input parameters.
Finally, we verify that modifying the input parameters as generates visually similar stylizations to the retrained base model with different loss weights w To do so, we train the base model (Johnson et al. 2016) multiple times with different w. and compare the generated results with the output of our model when VT E S, a² = w (Figure 6). Note how the proposed stylizations in test time and without retraining match the output of the base model.
5.3 GENERATING RANDOMIZED STYLIZATIONS
One application of our proposed method is to generate multiple stylizations given a fixed pair of content/style image. To do so, we randomize a generate randomized stylization (top row of Figure 5). Changing values of a usually do not randomize the position of the "elements" of the style. We can enforce this kind of randomness by adding some noise with the small magnitude to the content image. For this purpose, we multiply the content image with a mask which is computed by applying an inverse Gaussian filter on a white image with a handful (< 10) random zeros. This masking can shadow sensitive parts of the image which will change the spatial locations of the "elements" of style. Middle row in Figure demonstrates the effect of this randomization, Finally, combine these two randomizations to maximizes the diversity of the output which is shown in the bottom row of Figure? 5] More randomized stylizations can be seen Figure [0 and at htps://gooog//PVWW0KK
7

---

Published as a conference paper at ICLR 2020
5.4 COMPARISON WITH OTHER METHODS
To the best of our knowledge, generating diverse stylizations at real-time is only have been studied at (Uyaanov al., 2017) before. In this section, we qualitatively compare our method with this baseline. Also, we compare our method with a simple baseline where we add noise to the style parameters.
The simplest baseline for getting diverse stylizations is to add noises to some parameters or the inputs of the style-transfer network. the last section, we demonstrate that we can move the locations of elements of style by adding noise to the content input image. To answer the question that if we can get different stylizations by adding noise to the style input of the network, we train (Dumoulin) et al. 2017) which uses conditional instance normalization for transferring style. We train this model with only one style image and to get different stylizations, we add random noise to the style run-time. The stylization results for this bascline are shown on the top row of Figure g While we get different stylizations by adding random noises, the stylizations no longer similar the input style image.
parameters (% and 3a parameters of equation
To enforce similar stylizations, we trained the same baseline while adding random noises at the training phase. As it can be seen in the second of Figure 8 adding noise the training time makes the model robust to the noise and the stylization results are similar. This indicates that a loss term that encourages diversity is necessary.
We also compare the results of our model with StyleNet (Ulyanov et al., 2017). As visible in Figures [] although StyleNet's stylizations are different, they vary minor details and all carry the same level of stylization elements. In contrast, our model synthesizes stylized images with varying levels of stylization and more randomization.
CONCLUSION
Our main contribution in this paper is a novel method which allows adjustment of each loss layer's contribution in wred-fo style transfer networks, in real-time and after training. This capability allows the users to adjust the stylized output to find the favorite stylization by changing input param- eters and without retraining the stylization model. We also show how randomizing these parameters plus some noise added to the content image can result in very different stylizations from the same pair of style/content image.
Our method can be expanded in numerous ways e.g. applying it to multi-style transfer methods such as (Dumoulin al. 2017 Ghiasi al. 2017), applying the same parametrization technique to randomize the correlation loss between the features of each layer and finally using different loss functions and pre-trained networks for computing the loss randomize the outputs even further. One other interesting future direction is to apply the same "loss adjustment after training" technique for other classic computer vision and deep learning tasks. Style transfer is not the only task in which modifying the hyper-parameters can greatly affect the predicted results and it would be rather interesting to try this method for adjusting the hyper-parameter in similar problems.
REFERENCES
Michael Ashikhmin. Synthesizing natural textures. In Proceedings of the 2007 symposium on Interactive 3D graphics, pp. 217-226. ACM, 2001.
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction. arXiv preprint arxiv:1710.11252, 2017.
Yun Cao, Zhiming Zhou, Weinan Zhang, and Yong Yu. Unsupervised diverse colorization via generative adversarial networks. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 2017.
Qifeng Chen and Vladlen Koltun. Photographic image synthesis with cascaded refinement networks. In ICCV, 2017.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEFE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
8

---

Published as a conference paper at ICLR 2020
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. Computer Vision and Pattern Recognition, 2009, CVPR 2009, IEEE Conference on, pp. 248-255. 255, lece, 2009.
Aditya Deshpande, Jiajun Lu, Mao-Chuang Ych, Min Jin Chong, and David Forsyth. Learning diverse image colorization, CVPR, 2017.
Vincent Dumoulin, Jonathon Shiens, and Manjunath Kudlur. learned representation for artistic style. Proc. of ICLR, 2017.
Alexei A Efros and William T Freeman. Image quilting for texture synthesis and transfer. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pp. 341-346. ACM, 2001.
Leon Gatys, Alexander S Eeker, and Matthias Bethge. Texture synthesis using convolutional neural networks.
In Advances Neural Information Processing Systems, pp. 262-270, 2015.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In CVPR, pp. 2414-2423, 2016.
Golnaz Ghiasi, Honglak Lee, Manjunath Kudlur, Vincent Dumoulin, and Jonathon Shlens. Exploring the structure of real-time, arbitrary neural artistic stylization network. arXiv preprint arXiv:1705.06833 2017.
Aaron Hertzmann. Painterly rendering with curved brush strokes of multiple sizes. In Proceedings of the 25th annual conference on Computer graphics and interoctive techniques, pp. 453-460. ACM, 1998.
Xun Huang and Serge J Bellongie. Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV, 2017.
Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to-inage transla- tion. arXiv preprint 1804::8804,0493 2018.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. In European Conference Computer Vision, pp. 694-711. Springer, 2016.
Kaggle. Kaggle Painter by numbers kernel description. 2016.
Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprini WVXiv:1804.01522 2018.
Yanghao Li. Naiyan Wang, Jiaying Liu, and Xiaodi Hou. Demystifying neural style transfer. arXiv preprint arXiv:1701.01033 2017a.
Yijun Li, Chen Fang, Jimei Yang. Zhaowen Wang, Xin and Ming-Hsuan Hsuan Yang. Universal style transfer via feature transforms. In Advances in Neural Information Processing Systems, pp. 386-396, 2017b.
Yijun Li, Ming-Yi Yu Liu, Xueting Li, Ming-Hsuan Yang, and Jan Kautz. closed-form solution to photorcalistic image stylization. arXiv preprint arXiv: 1802.06404 2018.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft enco: Common objects context. European conference on computer vision, pp. 740-755. Springer, 2014.
ChonChhng Liu, Ming-Ming Cheng, Yu-Kun Lai, and Paul L Rosin. Depth- aware neural style transfer. In Proceedings of the Symposium on Non-Phoooreallstic Animation and Rendering. 2017.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) datasct. Retrieved August 15:2018, 2018
Xingchao Peng and Kate Saenko. Synthetic real adaptation with generative correlation alignment networks. In WACV, 2018
Eric Risser, Pierre Wilmot, and Connelly Barnes. Stable and controllable neural texture synthesis and style transfer using histogram losses. arXiv preprint arXiv:1701.089/0 2017.
Manuel Ruder, Alexey Dosovitskiy, and Thomas Brox. Artistic style transfer for vidcos and spherical images. International Journal of Computer Vision, pp. 1-21, 2018.
Artsiom Sanakoyeu, Dmytro Kotovenko, Sabine Lang, and Björn Ommer. A style-aware content loss for real- time hd style transfer. arXiv preprint 1807. 10201, 2018.

---

Published as a conference paper at ICLR 2020
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint (eeei4449.55555 2014.
Ulyanov, A Vedaldi, and VS Lempitsky. Instance normalization: the missing ingredient for fast stylization. corr abs/1607.0 (2016).
Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, and Victor Lempitsky. Texture networks: Feed-forward synthesis of textures and stylized images. In ICML, pp. 1349-1333 2016.
Dmitry Ulyanov, Andrea Vedaldi, and Victor S Lempitsky. Improved texture networks: Maximizing quality and diversity ferwarorwwrd stylization and texture synthesis. In CVPR. 2017.
Yongqi Zhang. Xogan: One-to-many unsupervised image-to-image translation. arXiv:1805.077 072777 2018.
arXiv preprint
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei Efros. Unpaired image-to-iman translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference on computer vision, pp. 2023-2232,222 2017.
10

---

Published as a conference paper at ICLR 2020
conv__3
-.0
conv4_3 3
1.0
C0TT23 3
10
0.0
0.0
conu4_2
3
Figure 9: More results for adjusting the input parameters in real-time and after training. In each block the style/content pair is fixed while the parameters corresponding to conv3 and cone4 in- creases vertically and horizontally from zero one. Notice how the details are different from one layer to another and how the combination of layers may result to more favored stylizations. For an interactive presentation please visit ht9s://
11

---

Published as a conference paper at ICLR 2020
Figure 10: More results of stochastic stylization from the same pair of content/style. Each block represents randomized stylized outputs given the fix style/content image demonstrated at the top. Notice how stylized images vary in style granularity, the spatial position of style elements while maintaining similarity the original style and content image. For more results please visit https///go.gl/PVWOOK
12

---

Published as a conference paper at ICLR 2020
Content
Style
0.0
1.0
Content
Style
0.0
1.0
23000
Content
E SHANCO
Style
C
0.0
1.0
Content
Style
0.0
1.0
Figure 11: More examples for effect of adjusting the input parameters in real-time. Each row shows the stylized output when single o increased gradually from zero to one while other a, are fixed to zero. Notice how the details of each stylization is different specially at the last column where the weight maximum, Also how deeper layers use bigger features of style image to stylize the content.
13

---

Published as a conference paper at ICLR 2020
Style
3e-2
1e-2
3e-3
le-3
3e-4
1e-4
Figure 12: More examples for effect of adjusting the style weight in style transfer network from (Johnson et al. 2016). Each column demonstrates the result of a separate training. As can be seen, the "optimal" weight is different from one style image to another and there be more than one "good" stylization depending on ones personal choice.
14

---

Published as a conference paper at ICLR 2020
conD2 3 COMI3
cone2_ 3 cono4 3
conv3_3 conD4
conD2
COM1:
conv4 3
Figure 13: Results of combining losses from different layers at generation time by adjusting their corresponding parameters. The first column is the style image which fixed for each row. The content image is the same for all of the outputs. The corresponding parameter for each one of the losses zero except for the one(s) mentioned in the title of each column. Notice how each layer enforces different type of stylization and how the combinations vary well. Also note how single combination layers cannot be the "optimal" stylization for any style image and one may prefer the results from another column.
15

---

Published as a conference paper at ICLR 2020
Ours
Ours
StyleNet
StyleNet
Ours
Ours
StyleNet
StyleNet
Figure 14: Diversity comparison of method with StyleNet (Ulynnov et al. 2017). Our method
generates diverse stylizations while StyleNet results mostly differ in minor details.
16

---

Published as a conference paper at ICLR 2020
Operation
input dimensions
output dimensions
input parameters a
1000
10x Dense
1000
1000
Dense
1000
2( Pa
Optimizer
Adam (a = 0.001, 31 0.9, 82 = 0.999)
Training iterations
200K
Batch size
8
Weight initialization Isotropic gaussian (μ1 = = 0.01)
Table 1: Network architecture and hyper-parameters of A.
Operation
Kernel size
Stride
Feature maps
Padding
Nonlinearity
Network - 256 256 3 input
Convolution
9
1
32
SAME SAME SAME
ReLU
Convolution
3
2
64
ReLU
Convolution
2
ReLU
Residual block
128
Residual block
128
Residual block
128
Residual block
128
Residual block
Residual block
128
Residual block
128
Upsampling
64
Upsampling
Convolution
9
1
3
SAME
Sigmoid
Residual block
C feature maps
Convolution
3
1
C
SAME SAME
ReLU Linear
Convolution
3
C
Add input and output
Upsampling
C' feature maps
Nearest-neighbor interpolation, factor
Convolution
3
1
C
SAME
ReLU
Normalization
Conditional instance normalization after every convolution Adam (a 0.001, 31 0.9, 32 0.999)
Optimizer
Training iterations
200K
Batch size
8
Weight initialization Isotropic gaussian (μ = 0, 0.01)
Table 2: Network architecture and hyper-parameters of T.
17


Reviews:
[{"review_id": "HJgiTy_GsB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a generative model for image style transfer in real time. In particular, comparing to the existing work, the proposed method is able to generate a series of transferred images instead of one, and more importantly, users can adjust different parameters without re-training the network to control over the synthesized output. The proposed method was evaluated on publicly-available datasets, and achieved convincing experimental results.\n\nPros:\n- The problems that the paper tried to solve is interesting and important. I think the proposed model could make some contributions to related research communities.\n- The idea of learning and incorporating \"adjustable loss weights\" is interesting and reasonable.\n- The paper is well written and easy to understand.\n- Experimental results appear to be promising.\n\nCons:\n- The \"adjust loss weights\" are able to control the synthesized output, but does each of them have a particular meaning? For example, if I want to generate other outputs more realistic, colorful, or in other styles, is it possible for the users to (roughly) know which parameter should be adjust, and how?\n- Following the above question, I suggest the authors to provide more qualitative results with different \"loss weights\", and give some detailed explanations.\n- Some typos and missing links could be corrected.\n\nI think the proposed approach is reasonable and and experimental results are convincing. I'm not an expert in this area, and I'm not sure about the method's novelty. If other reviewers find other existing work that proposed very similar ideas, then this would have an impact on my recommendation.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByeD9kIN5S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an approach for style transfer with controlable parameters. The controllable parameters correspond to the weights associated to \"style losses\" or ordinary style transfer models (distance between gram matrices of generated vs style image at specific layers of a network). The authors propose to learn a single architecture that takes these parameters as input to generate an image that resembles what would be generated by optimizing directly on these parameters. Examples of transfer and of the effect of these parameters are given. A quantitative evaluation shows that the effect of changing the parameters of the new network has the effect of reducing the loss at the desired layers.\n\nOverall, I found the idea of learning controllable parameters interesting. The technical contribution is to show that learning the weights of instance normalization as a function of the hyperparameters actually is satisfactory. To be honest, I am not sure to understand why playing on instance normalization weights turns to be effective for learning an equivalent of a reduction of per-layer loss. I would have liked more motivation and intuition behind it.\n\nThe paper seems overall a bit incremental with respect to prior work on style transfer. While adjustable parameters could have application in more general tasks of image generation (for instance, in the line of work on disentangled representations), it is unclear how the specific approach of learning instance normalization weights extends beyond style transfer. As such, my feeling is that the paper is borderline.\n\nother comments:\n- I found Figure 7 important because it guarantees learning has the desired effect. However, there is a bit a lack of baseline/topline: how do the true losses decrease when their weights increase?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xU8is7qS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n + Interesting problem\n + Good results\nAreas of improvement:\n - Structure of paper\n - Main technical section (4.1) very hard to understand.\n\nThe paper works on a very interesting and important problem. To the best of my knowledge there is no real time style transfer method that can adjust certain style parameters after it has been trained. The paper addresses this by parametrizing the normalization layers in the style transfer model, and training with a parametrized loss function. The results look compelling both in the paper, and the supplemental webpage.\n\nThe main area of improvement is the structure of the paper itself. The paper spends 4 pages on introduction, motivation and background. This unfortunately only leaves less than half a page for the main technical section. This main technical section is very dense, technical choices poorly motivated, and the section is impossible to understand in a single read (I had to read it 3-4 times). I'd highly encourage the authors to consolidate some of the early parts of the paper and greatly expand the technical section. For example, the first two paragraphs in section 4 can easily go into the introduction to avoid having to motivate the method twice. The main technical section should be motivated better. I had to constantly jump forth and back between background and technical section to understand where the technical changes are applied. It would help to have a more consistent story.\n\n\nMinor:\n * Figure reference bottom page 2 broken (??)", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper offers an innovative approach to adjusting style transfer parameters.
The reviewers were consistent, and all recommend acceptance.  I concur. 

Decision: Accept (Poster)