Adjustable Real-time Style Transfer

1 INTRODUCTION . Style transfer is a long-standing problem in computer vision with the goal of synthesizing new images by combining the content of one image with the style of another ( Efros & Freeman , 2001 ; Hertzmann , 1998 ; Ashikhmin , 2001 ) . Recently , neural style transfer techniques ( Gatys et al. , 2015 ; 2016 ; Johnson et al. , 2016 ; Ghiasi et al. , 2017 ; Li et al. , 2018 ; 2017b ) showed that the correlation between the features extracted from the trained deep neural networks is quite effective on capturing the visual styles and content that can be used for generating images similar in style and content . However , since the definition of similarity is inherently vague , the objective of style transfer is not well defined ( Dumoulin et al. , 2017 ) and one can imagine multiple stylized images from the same pair of content/style images . Existing real-time style transfer methods generate only one stylization for a given content/style pair and while the stylizations of different methods usually look distinct ( Sanakoyeu et al. , 2018 ; Huang & Belongie , 2017 ) , it is not possible to say that one stylization is better in all contexts since people react differently to images based on their background and situation . Hence , to get favored stylizations users must try different methods that is not satisfactory . It is more desirable to have a single model which can generate diverse results , but still similar in style and content , in real-time , by adjusting some input parameters . One other issue with the current methods is their high sensitivity to the hyper-parameters . More specifically , current real-time style transfer methods minimize a weighted sum of losses from different layers of a pre-trained image classification model ( Johnson et al. , 2016 ; Huang & Belongie , 2017 ) ( check Sec 3 for details ) and different weight sets can result into very different styles ( Figure 6 ) . However , one can only observe the effect of these weights in the final stylization by retraining the model with the new set of weights . Considering the fact that the ” optimal ” set of weights can be different for any pair of style/content ( Figure 4 ) and also the fact that this ” optimal ” truly doesn ’ t exist ( since the goodness of the output is a personal choice ) retraining the models over and over until the desired result is generated is not practical . The primary goal of this paper is to address these issues by providing a novel mechanism which allows for adjustment of the stylized image , in real-time and after training . To achieve this , we use an auxiliary network which accepts additional parameters as inputs and changes the style transfer process by adjusting the weights between multiple losses . We show that changing these parameters at inference time results to stylizations similar to the ones achievable by retraining the model with different hyperparameters . We also show that a random selection of these parameters at run-time can generate a random stylization . These solutions , enable the end user to be in full control of how the stylized image is being formed as well as having the capability of generating multiple stochastic stylized images from a fixed pair of style/content . The stochastic nature of our proposed method is most apparent when viewing the transition between random generations . Therefore , we highly encourage the reader to check the project website https : //goo.gl/PVWQ9K . 2 RELATED WORK . The strength of deep networks in style transfer was first demonstrated by Gatys et al . ( Gatys et al. , 2016 ) . While this method generates impressive results , it is too slow for real-time applications due to its optimization loop . Follow up works speed up this process by training feed-forward networks that can transfer style of a single style image ( Johnson et al. , 2016 ; Ulyanov et al. , 2016 ) or multiple styles ( Dumoulin et al. , 2017 ) . Other works introduced real-time methods to transfer style of arbitrary style image to an arbitrary content image ( Ghiasi et al. , 2017 ; Huang & Belongie , 2017 ) . Although , these methods can generate stylization for the arbitrary inputs , they can only produce one stylization for a single pair of content/style images . In the case that the user does not like the result , it is not possible to get a different result without retraining the network for a different set of hyperparameters . Our goal in this paper is to train a single network that user can get different stylization without retraining the network . Generating diverse results have been studied in multiple domains such as colorizations ( Deshpande et al. , 2017 ; Cao et al. , 2017 ) , image synthesis ( Chen & Koltun , 2017 ) , video prediction ( Babaeizadeh et al. , 2017 ; Lee et al. , 2018 ) , and domain transfer ( Huang et al. , 2018 ; Zhang , 2018 ) . Domain transfer is the most similar problem to the style transfer . Although we can generate multiple outputs from a given input image ( Huang et al. , 2018 ; Zhu et al. , 2017 ) , we need a collection of target or style images for training . Therefore we can not use it when we do not have a collection of similar styles . For instance , when we want to generate multiple stylizations for the Stary Night painting , it is hard to find different similar paintings . Style loss function is a crucial part of style transfer which affects the output stylization significantly . The most common style loss is Gram matrix which computes the second-order statistics of the feature activations ( Gatys et al. , 2016 ) , however many alternative losses have been introduced to measure distances between feature statistics of the style and stylized images such as correlation alignment loss ( Peng & Saenko , 2018 ) , histogram loss ( Risser et al. , 2017 ) , and MMD loss ( Li et al. , 2017a ) . More recent work ( Liu et al. , 2017 ) has used depth similarity of style and stylized images as a part of the loss . We demonstrate the success of our method using only Gram matrix ; however , our approach can be expanded to utilize other losses as well . To the best of our knowledge , the only previous work which generates multiple stylizations is ( Ulyanov et al. , 2017 ) which utilized Julesz ensemble to explicitly encourage diversity in stylizations . However their results are quite similar in style and only differ in minor details . A qualitative comparison in Figures 8,14 show that our proposed method is more effective in diverse stylization . 3 BACKGROUND . 3.1 STYLE TRANSFER USING DEEP NETWORKS . Style transfer can be formulated as generating a stylized image p which its content is similar to a given content image c and its style is close to another given style image s. The similarity in style can be vaguely defined as sharing the same spatial statistics in low-level features , while similarity in content is roughly having a close Euclidean distance in high-level features ( Ghiasi et al. , 2017 ) . These features are typically extracted from a pre-trained image classification network , commonly VGG-19 ( Simonyan & Zisserman , 2014 ) . The main idea here is that the features obtained by the image classifier contain information about the content of the input image while the correlation between these features represents its style . In order to increase the similarity between two images , Gatys et al . ( Gatys et al. , 2016 ) minimize the following distances between their extracted features : Llc ( p ) = ∣∣∣∣φl ( p ) − φl ( s ) ∣∣∣∣2 2 , Lls ( p ) = ∣∣∣∣G ( φl ( p ) ) −G ( φl ( s ) ) ∣∣∣∣2 F ( 1 ) where φl ( x ) is activation of a pre-trained classification network at layer l given the input image x , while Llc ( p ) and Lls ( p ) are content and style loss at layer l respectively . G ( φl ( p ) ) denotes the Gram matrix associated with φl ( p ) . The total loss is calculated as the weighted sum of losses across a set of content layers C and style layers S : Lc ( p ) = ∑ l∈C wlcLlc ( p ) , Ls ( p ) = ∑ l∈S wlsLls ( p ) ( 2 ) where wlc , w l s are hyper-parameters to adjust the contribution of each layer to the loss . Layers can be shared between C and S . These hyper-parameters have to be manually fine tuned through try and error and usually vary for different style images ( Figure 4 ) . Finally , the objective of style transfer can be defined as : min p ( Lc ( p ) + Ls ( p ) ) ( 3 ) This objective can be minimized by iterative gradient-based optimization methods starting from an initial p which usually is random noise or the content image itself . 3.2 REAL-TIME STYLE TRANSFER . Solving the objective in Equation 3 using an iterative method can be very slow and has to be repeated for any given pair of style/content image . A much faster method is to directly train a deep network T which maps a given content image c to a stylized image p ( Johnson et al. , 2016 ) . T is usually a feed-forward convolutional network ( parameterized by θ ) with residual connections between down- sampling and up-sampling layers ( Ruder et al. , 2018 ) and is trained on many content images using Equation 3 as the loss function : min θ ( Lc ( T ( c ) ) + Ls ( T ( c ) ) ) ( 4 ) The style image is assumed to be fixed and therefore a different network should be trained per style image . However , for a fixed style image , this method can generate stylized images in realtime ( Johnson et al. , 2016 ) . Recent methods ( Dumoulin et al. , 2017 ; Ghiasi et al. , 2017 ; Huang & Belongie , 2017 ) introduced real-time style transfer methods for multiple styles . But , these methods still generate only one stylization for a pair of style and content images . 4 PROPOSED METHOD This paper addresses the following issues in real-time feed-forward style transfer methods : 1 . The output of these models is sensitive to the hyper-parameters wlc and w l s and different weights significantly affect the generated stylized image as demonstrated in Figure 6 . Moreover , the ” optimal ” weights vary from one style image to another ( Figure 4 ) and finding a good set of weights should be repeated for each style image . Note that for each set of wlc and w l s the model has to be retrained that limits the practicality of style transfer models . 2 . Current methods generate a single stylized image given a content/style pair . While the stylizations of different methods usually look very distinct ( Sanakoyeu et al. , 2018 ) , it is not possible to say which stylization is better for every context since it is a matter of personal taste . To get a favored stylization , users may need to try different methods or train a network with different hyper-parameters which is not satisfactory and , ideally , the user should have the capability of getting different stylizations in real-time . We address these issues by conditioning the generated stylized image on additional input parameters where each parameter controls the share of the loss from a corresponding layer . This solves the problem ( 1 ) since one can adjust the contribution of each layer to adjust the final stylized result after the training and in real-time . Secondly , we address the problem ( 2 ) by randomizing these parameters which result in different stylizations .

Summary: The paper presents an approach for style transfer with controlable parameters. The controllable parameters correspond to the weights associated to "style losses" or ordinary style transfer models (distance between gram matrices of generated vs style image at specific layers of a network). The authors propose to learn a single architecture that takes these parameters as input to generate an image that resembles what would be generated by optimizing directly on these parameters. Examples of transfer and of the effect of these parameters are given. A quantitative evaluation shows that the effect of changing the parameters of the new network has the effect of reducing the loss at the desired layers.