Cascade Style Transfer

1 INTRODUCTION . Given the content and style images , the goal of style transfer is to synthesize an image that preserves some notion of the content but carries characteristics of the style . Recently , the seminal work of Gatys et al . ( 2015 ) firstly captured the style of artistic images and transferred it to other images using Convolutional Neural Networks ( CNNs ) . Since then , various Neural Style Transfer ( NST ) Jing et al . ( 2017 ) methods have been advanced and obtained visually pleasing results . Despite the recent rapid progress , these existing works often limited to one or few specific domains ( in this paper , we mainly focus on three domains : artistic , semantic and photo-realistic ) . For instance , Li et al . ( 2017b ) ; Huang & Belongie ( 2017 ) ; Gatys et al . ( 2016 ) can transfer the artistic styles well , but they perform poorly on the style transfer of photographs and corresponding semantics . Luan et al . ( 2017 ) ; Li et al . ( 2018 ) specialize in photo-realistic style transfer , and Li & Wand ( 2016 ) ; Champandard ( 2016 ) mainly target semantic style transfer . Fortunately , there are some multi-domain approaches which can perform well on multiple domains , e.g. , Liao et al . ( 2017 ) can perform well on semantic and photo-realistic style transfer , Gu et al . ( 2018 ) is suitable for artistic and semantic style transfer , and Li et al . ( 2019 ) can adapt to artistic and photo-realistic style transfer . Nevertheless , they still have some limitations , and the quality could be further improved ( see Fig . 1 ( a ) ) . That is to say , nowadays , it is still inconvenient to users that have to choose the appropriate methods for specific domains . In this sense , finding a common approach which could perform well in all style transfer domains is extremely hard but significant . As a coin has two sides , every existing NST method has both advantages and shortcomings . Fig . 1 ( b ) shows some typical examples , we can observe that using Gram matrices Gatys et al . ( 2016 ) to transfer the artistic styles performs well on global color , but fails to capture enough local patterns ( e.g. , circles and droplets ) . Patch-based method Li & Wand ( 2016 ) can alleviate this problem , but may cause insufficient color . Is there a way to combine the advantages of both and overcome their shortcomings ? Obviously , redesigning a new algorithm is difficult , why not use some simpler ways , such as combining existing methods directly through some general architectures ? Based on the above analyses , we propose Cascade Style Transfer ( CST ) mainly for two targets , i.e. , higher quality and higher flexibility , and design two architectures , i.e. , the Serial Style Transfer ( SST ) and the Parallel Style Transfer ( PST ) for these targets . In this work , we first revisit and demonstrate the impact of different initialization strategies on style transfer , and inspired by this , design our SST for higher quality domain-specific style transfer . Moreover , we develop upon this and further propose our PST for more flexible style transfer , this could guide us to create domainindependent approaches . As far as we know , this is the first paper to propose domain-independent style transfer ( note that this kind of approach is flexible for arbitrary images in arbitrary domains , while existing so-called arbitrary style transfer methods are only flexible for arbitrary images in specific domains ) , and also the first attempt to combine multiple existing approaches directly to improve the quality and flexibility of style transfer . The main contributions of our work are : •We revisit the initialization of style transfer , and demonstrate that initialization can play an important role in improving the quality of style transfer . • We propose a serial architecture for cascade style transfer , it is simple yet effective , which could help improve the quality of domain-specific style transfer . • We first propose domain-independent style transfer , and design a parallel architecture to help improve the quality and flexibility of style transfer . 2 RELATED WORK . Our cascade style transfer can be related to the most style transfer methods . In this paper , we mainly focus on the NST methods in three major domains : artistic , semantic and photo-realistic . Artistic style transfer . This domain is dedicated to transferring the global artistic styles ( e.g. , abstract , painterly or sketch ) . The most representative work is Gatys et al . ( 2015 ) . This method could produce amazing results but suffers from a slow iterative optimization procedure . To address it , Johnson et al . ( 2016 ) and Ulyanov et al . ( 2016 ; 2017 ) trained feed-forward generative networks for fast artistic style transfer . But one limit is that each model is trained to transfer exactly one fixed style . Some methods Dumoulin et al . ( 2017 ) ; Zhang & Dana ( 2018 ) ; Li et al . ( 2017a ) ; Chen et al . ( 2017 ) further incorporated multiple styles into one single model , but they are still limited to a fixed number of pre-trained styles . Recently , several methods Li et al . ( 2017b ) ; Huang & Belongie ( 2017 ) ; Li et al . ( 2019 ) were proposed to allow artistic style transfer for arbitrary images . Semantic style transfer . Transferring the styles between the corresponding semantic regions of the style and content images is referred to as semantic style transfer . The most representative work is Li & Wand ( 2016 ) . They combined Markov Random Fields ( MRFs ) and CNNs to match the most similar local neural patches of the style and content images . Later , Champandard ( 2016 ) incorporated the segmentation masks for stricter semantic constraints . Recently , Liao et al . ( 2017 ) proposed Deep Image Analogy for accurate semantic-level patch match . Gu et al . ( 2018 ) used Deep Feature Reshuffle to consider both global and local information . Mechrez et al . ( 2018 ) proposed an alternative contextual loss for segmentation-free semantic style transfer . Moreover , some feedforward methods Chen & Schmidt ( 2016 ) ; Lu et al . ( 2017 ) ; Sheng et al . ( 2018 ) ; Park & Lee ( 2019 ) ; Yao et al . ( 2019 ) were also proposed for fast and real-time semantic style transfer . Photo-realistic style transfer . Photo-realistic style transfer seeks to transfer the style of a reference style photo onto other pictures . The greatest characteristic is that both the global structures and detailed contours in the content images should be preserved during the process . Traditional methods based on Global Reinhard et al . ( 2001 ) ; Pitie et al . ( 2005 ) and Local Laffont et al . ( 2014 ) ; Shih et al . ( 2014 ) are slow in practice and limited in specific scenarios ( e.g. , outdoor scenes or headshot portraits ) . Recently , Luan et al . ( 2017 ) incorporated a new loss term to the optimization objective of Gatys et al . ( 2015 ) to improve the photorealism of stylization outputs . Li et al . ( 2018 ) introduced a closed-form solution consisting of a stylization and a smoothing step for faster speed . More recently , Li et al . ( 2019 ) have also shown an effective approach for fast photo-realistic style transfer . Despite the fact that current NST methods have shown good performance in specific domains , there are few studies on how to combine them directly to improve the quality and flexibility . In our work , we select Gatys et al . ( 2016 ) ; Li et al . ( 2017b ) ; Huang & Belongie ( 2017 ) ; Li et al . ( 2019 ; 2018 ) ; Luan et al . ( 2017 ) ; Sheng et al . ( 2018 ) ; Liao et al . ( 2017 ) ; Gu et al . ( 2018 ) ; Li & Wand ( 2016 ) ; Champandard ( 2016 ) for our studies mainly because of their representativeness in specific domains . 3 REVISIT INITIALIZATION IN STYLE TRANSFER . Initialization is the first yet important step in almost all style transfer algorithms . Although some papers Gatys et al . ( 2016 ) ; Nikulin & Novak ( 2016 ) have discussed the impact of the initialization on style transfer , they only use white noise , the content image or the style image , and the evaluation criteria are only based on the qualitative aspect . Here , back to the most original NST algorithms , we revisit and compare more initialization strategies from both qualitative and quantitative aspects . Our experiments are based on two most original NST methods Gatys et al . ( 2016 ) ; Li & Wand ( 2016 ) . These methods always initialize with white noise or the content image and iteratively optimize pixels to match the content features of the content image and style features of the style image . Besides white noise , the content and style images , we try and compare four other initialization strategies : salt and pepper noise , poisson noise , the stylized results of other methods ( SROOM ) , and moreover , replacing the content images with SROOM and then initializing with them ( RC-SROOM ) . Qualitative Comparisons : Fig . 2 demonstrates the qualitative comparisons . As we can see , initializing with the content image produces results with insufficient color ( column a ) . Results generated from the style image introduce some undesired color ( column b ) . Using salt and pepper noise produces over-bright results , and using poisson noise produces darker results ( column d and e ) . By contrast , white noise initialization yields satisfying results ( column c ) , but the results of SROOM performs better on overall effect ( column f ) . Remarkably , using RC-SROOM ( column g ) can dramatically improve the effect and absorb the merits ( as shown and discussed in Fig . 1 ( b ) ) of both methods . More results can be found in appendix . Quantitative Comparisons : Fig . 3 shows the quantitative comparisons . We evaluate these initialization strategies on method Gatys et al . ( 2016 ) , the optimization is conducted by Adam method , and stopped at 1000 iterations . We can see that initializing with images ( i.e. , content , style , SROOM and RC-SROOM ) makes loss fall faster , while initializing with noise ( i.e. , white noise , salt and pepper noise , and poisson noise ) decreases loss more steadily . It is worth noting that using RC-SROOM achieves much lower total , content and style loss than all other strategies . Conclusion : In this section , we have demonstrated that initialization plays an important role in improving the quality of style transfer . Compared with other initialization strategies , RC-SROOM has outstanding performance . More importantly , this could help produce higher quality style transfer results that absorb the merits of multiple methods . Inspired by this , we propose cascade style transfer , which will be presented in latter sections , and in turn verify this conclusion .

Summary: This paper proposes two ways to aggregate existing style transfer methods and shows improvements on quality and flexibility. However, the proposed method does not solve the limitations of any previous methods. Instead, it is as simple as an easy combination: the proposed SST is a sequential combination and the proposed PST has no difference with running each single method separately. To me this is more like an engineering effort rather than a research work.