Cascade Style Transfer
Authors: Zhizhong Wang
Lei Zhao
Qihang Mo
Sihuan Lin
Zhiwen Zuo
Wei Xing
Dongming Lu
Venue:  (2020)
Keywords: style transfer
cascade
quality
flexibility
domain-independent
serial
parallel

Abstract: Recent studies have made tremendous progress in style transfer for specific domains, e.g., artistic, semantic and photo-realistic. However, existing approaches have limited flexibility in extending to other domains, as different style representations are often specific to particular domains. This also limits the stylistic quality. To address these limitations, we propose Cascade Style Transfer, a simple yet effective framework that can improve the quality and flexibility of style transfer by combining multiple existing approaches directly. Our cascade framework contains two architectures, i.e., Serial Style Transfer (SST) and Parallel Style Transfer (PST). The SST takes the stylized output of one method as the input content of the others. This could help improve the stylistic quality. The PST uses a shared backbone and a loss module to optimize the loss functions of different methods in parallel. This could help improve the quality and flexibility, and guide us to find domain-independent approaches. Our experiments are conducted on three major style transfer domains: artistic, semantic and photo-realistic. In all these domains, our methods have shown superiority over the state-of-the-art methods.

Under review as a conference paper at ICLR 2020
CASCADE STYLE TRANSFER
Anonymous authors
Paper under double-blind review
ABSTRACT
Recent studies have made tremendous progress in style transfer for specific do- mains, e.g., artistic, semantic and photo-realistic. However, existing approaches have limited flexibility in extending to other domains, as different style representa- tions are often specific to particular domains. This also limits the stylistic quality, To address these limitations, we propose Cascade Style Transfer, a simple yet effective framework that can improve the quality and flexibility of style transfer by combining multiple existing approaches directly. Our cascade framework con- tains two architectures, i.e., Serial Style Transfer (SST) and Parallel Style Transfer (PST). The SST takes the stylized output of one method as the input content of the others. This could help improve the stylistic quality. The PST uses a shared backbone and loss module to optimize the loss functions of different methods in parallel. This could help improve the quality and flcxibility, and guide us to find domain-independee approaches. Our experiments are conducted three major style transfer domains: artistic, semantic and photo-realistic. In all these domains, our methods have shown superiority over the state-of-the-art methods.
INTRODUCTION
Given the content and style images, the goal of style transfer is to synthesize an image that pre- serves some notion of the content but carries characteristics of the style. Recently, the seminal work of Gatys et al. (2015) firstly captured the style of artistic images and transferred it to other images using Convolutional Neural Networks (CNNs). Since then, various Neural Style Transfer (NST) Jing et al. (017) methods have been advanced and obtained visually pleasing results.
Despite the recent rapid progress, these existing works often limited to one or few specific domains (in this paper, we mainly focus on three domains: artistic, semantic and photo-realistic. For in- stance, Li et al. (2017b); Huaan & Belongie (017);; Gatys et al. (2016) can transfer the artistic styles well, but they perform poorly on the style transfer of photographs and corresponding seman- tics. Luan et al. (2017): Li et al. (2018) specialize in photo-realistic style transfer, and Li & Wand (016)); Champandard (2066) mainly target semantic style transfer. Fortunately. there multi-domain approaches which can perform well on multiple domains, e.g., Liao et al.] (2017) can perform well on semantic and photo-realistic style transfer, (u et al. (2018) is suitable for artistic and semantic style transfer, Li et al. (2019) can adapt to artistic and photo-realistic style transfer. Nevertheless, they still have some limitations, and the quality could be further improved (see Fig. I (a)). That is to say, nowadays, it is still inconvenient to users that have to choose the appropriate methods for specific domains. In this sense, finding a common approach which could perform well in all style transfer domains is extremely hard but significant.
some
As coin has two sides, every existing NST method has both advantages and shortcomings. Fig. (b) shows some typical examples, we can observe that using Gram matrices Gatys et al. (2016) to transfer the artistic styles performs well on global color, but fails to capture enough local patterns (e.g., circles and droplets). Patch-based method Li & Wand (2016) can alleviate this problem, but may cause insufficient color. Is there a way to combine the advantages of both and overcome their shortcomings? Obviously, redesigning new algorithm is difficult, why not some simpler ways, such as combining existing methods directly through some general architectures?
Based on the above analyses, we propose Cascade Style Transfer (CST) mainly for two targets, ie., higher quality and higher flexibility, and design two architectures, ie.. the Serial Style Transfer (SST) and the Parallel Style Transfer (PST) for these targets. In this work, we first revisit and

---

Under review as a conference paper at ICLR 2020
Content
Siyle
Lisn (2017)
dt (2008)
Li (28199
Content
Style
Gotys et al. (2016) Li and Wand (2016)
(a)
(b)
Figure 1:
(a) Examples of multi-domain methods. The rows from top to bottom show artistic, photo-
realistic and semantic style transfer, respectively. (b) Typical examples of artistic style transfer.
demonstrate the impact of different initialization strategies on style transfer, and inspired by this, design our SST for higher quality domain-specific style transfer. Moreover, we develop upon this and further propose our PST for more flexible style transfer, this could guide us to create domain- independent approaches. As far as know, this is the first paper propose domain-indepente style transfer (note that this kind of approach is flexible for arbitrary images in arbitrary domains, while existing so-called arbitrary style transfer methods specific domains), and also the first attempt to combine multiple existing approaches directly to improve the quality and flexibility of style transfer. The main contributions of our work are:
only flexible for arbitrary images in
We revisit the initialization of style transfer, and demonstrate that initialization can play an impor- tant role in improving the quality of style transfer.
We propose a serial architecture for cascade style transfer, is simple yet effective, which could help improve the quality of domain-specific style transfer.
We first propose domain-independent style transfer, and design a parallel architecture to help improve the quality and flexibility of style transfer.
RELATED WORK
Our cascade style transfer can be related to the most style transfer methods. In this paper, we mainly focus on the NST methods in three major domains: artistic, semantic and photo-realistic.
Artistic style transfer. This domain dedicated to transferring the global artistic styles (c.g.. abstract, painterly sketch). The most representative work is Gatys et al. (2015). This method could produce amazing results but suffers from a slow iterative optimization procedure. To address it. Johnson al. (2016) and [lyanov et al. (2016) (017) trained feed-forward generative networks for fast artistic style transfer. But one limit is that each model is trained to transfer exactly one fixed style. Some methods Dumoulin et al. (2017). Zhang & Dana (2018): Li al. (2017a); Chen al. (017)) further incorporated multiple styles into one single model, but they are still limited to fixed number of pre-trained styles. Recently, several methods et al. (2017)); (Hang & Belongie (2017)
Li al. (019) were proposed to allow artistic style transfer for arbitrary images.
Semantic style transfer. Transferring the styles between the corresponding semantic regions of the style and content images is referred to as semantic style transfer. The most representative work Li & Wand (016)). They combined Markov Random Fields (MRFs) and CNNs to match the most similar local neural patches of the style and content images. Later, Champandard] (2016)
incorporated the segmentation masks for stricter semantic constraints. Recently, |iao al. (017))
proposed Deep Image Analogy for accurate semantic-level patch match. Gu et al. (2018) used Deep Feature Reshuffle to consider both global and local information. Mechrez et al. (2018) proposed alternative contextual loss for segmentation-free semantic style transfer. Moreover, some feed-
forward methods Chen & Schmidt (2016); Lu et al. (2017); [Sheng et al. (2018): 8) Yao al. (2019) were also proposed for fast and real- time semantic style transfer.
Park & Lee (201);;
Photo-realistic style transfer. Photo-realistic style transfer seeks to transfer the style of a reference style photo onto other pictures. The greatest characteristic is that both the global structures and
2

---

Under review as a conference paper at ICLR 2020
detailed contours in the content images should be preserved during the process. Traditional methods based on Global Reinhard et al. (200T); Pitie et al. (2005) and Local Laffont et al. (2014): y: Shih
ct al.
2014) are slow in practice and limited in specific scenarios (e.g., outdoor scenes or headshot portraits). Recently, Luan et al. 2017) incorporated a new loss term to the optimization objective of Gatys et al. (2015) improve the photorealism of stylization outputs. [ et al. (2018) introduced
closed-form solution consisting of stylization and a smoothing step for faster speed. More recently,
Li et al. (0199) have also shown an effective approach for fast photo-realistic style transfer.
Despite the fact that current NST methods have shown good performance in specific domains, there are few studies on how to combine them directly to improve the quality and flexibility. In our work, we select Gatys et al. (2016); [i ct al. (2017b); HHunn & Belongie| (2017); Li et (2019) 2018);
Luan et al. (2017 Sheng et al. (2018); Liao et al. (2017); Gu et al. (0018): Li & Wand (2016);
Champandard (2016) for our studies mainly because of their representativeness in specific domains.
3 REVISIT INITIALIZATION IN STYLE TRANSFER
Initialization is the first yet important step in almost all style transfer algorithms. Although some papers Gatys et al. 2016); Nikulin & Novak 2016) have discussed the impact of the initialization on style transfer, they only use white noise, the content image or the style image, and the evaluation criteria are only based on the qualitative aspect. Here, back to the most original NST algorithms, revisit and compare more initialization strategies from both qualitative and quantitative aspects.
Our experiments are based on two most original NST methods Garys et al.) (016); Li & Wand (016). These methods always initialize with white noise or the content image and iteratively opti- mize pixels to match the content features the content image and style features of the style image. Besides white noise, the content and style images, we try and compare four other initialization strategies: salt and pepper noise, poisson noise, the stylized results of other methods (SROOM), and moreover, replacing the content images with SROOM and then initializing with them (RC-SROOM).
Qualitative Comparisons: Fig. demonstrates the qualitative comparisons. As we can see, initial- izing with the content image produces results with insufficient color (column a). Results generated from the style image introduce some undesired color (column b). Using salt and pepper noise pro- duces over-bright results, and using poisson noise produces darker results (column d and e). By contrast, white noise initialization yields satisfying results (column c). but the results of SROOM performs better on overall effect (column f). Remarkably, using RC-SROOM (column g) can dra- matically improve the effect and absorb the merits (as shown and discussed in Fig. (b)) of both methods. More results can be found in appendix.
Quantitative Comparisons: Fig. 3 shows the quantitative comparisons. We evaluate these initial- ization strategies on method stopped at 1000 iterations. We can see that initializing with images (i.e., content, style, SROOM and RC-SROOM) makes loss fall faster, while initializing with noise (i.e., white noise, salt and pepper noise, and poisson noise) decreases loss more steadily. It is worth noting that using RC-SROOM achieves much lower total, content and style loss than all other strategies.
al. (2066), the optimization is conducted by Adam method, and
In this section, we have demonstrated that initialization plays an important role in im- proving the quality of style transfer. Compared with other initialization strategies, RC-SROOM has outstanding performance. More importantly, this could help produce higher quality style transfer re- sults that absorb the merits of multiple methods. Inspired by this, we propose cascade style transfer, which will be presented in latter sections, and in turn verify this conclusion.
Conclusion:
4 CASCADE STYLE TRANSFER
In this paper, we define cascade style transfer as combinations of different NST methods. It contains two architectures: the serial style transfer and the parallel style transfer.
4.1 SERIAL STYLE TRANSFER (SST)
As shown in Fig. (a), serial style transfer serially connects multiple style transfer methods. Let č 5 and F. be the content image, the style image and the stylized result of method i. The style transfer
3

---

Under review as a conference paper at ICLR 2020
Images
(a)
(b) Style
(c)
(4) Pepper (e)
(9) (g)
Figure 2: Qualitative comparisons of different initialization strategies. The first column shows the
content image (top) and the style image (bottom). The other columns show the initialization images (top) and the corresponding style transfer results (bottom) of method Gatys et al. (206)). In column (f), we initialize with default stylized result of method Li Wand (2016) (SROOM), column (g), we replace the content image with SROOM and then initialize with it (RC-SROOM).
Compare different initialization
content loss difforent inftolications
initializations
ontert
oetert
contant
dgyll
salt and pooper
saft and pespor
and paoper
SROOM
0.0
SROOM
SKDOM
SPOCOR
SROOH
04
0.4
04
02
Becations
Figure 3: Quantitative comparisons of different initialization strategies on total (left), content (cen- ter) and style (right) loss.
process of method i is denoted as fi- Specifically, for the first method, we use Ji f1 8, to denote transferring the style of 3 to b' method 1 with the default initialization settings. For others, we use fi 5) to denote initializing with ē;-1 and then transferring the style of 3 to -;-1 by method Our serial style transfer can be formulated as
f. 3,)) if (((
otherwise.
4.2 PARALLEL STYLE TRANSFER (PST)
As far as we know, current NST methods are mainly conducted in two different ways. One is based on VGG|Simonyan & Zisserman 2014), iteratively optimizing the pixels of input images. The other is training a feed-forward network to directly generate the stylized results. Here, to demonstrate our PST more intuitively, we design a simple parallel architecture based on the former way.
As shown in Fig. (b), our PST contains two important parts. One is the shared backbone (c.g.. VGG-19), is mainly used for feature extraction and error back-propagation. The other the loss module, it is used to combine loss functions of different methods.
Let L. denote the loss function of method i, "ø" denote a linear combination operation between different loss functions. We give hyperparameter to weight every loss function L, The total loss defined as follows:
Ci
ii otherwise.
Lrotat (i)
Lrotal $ W/C,
We compute its gradients with respect to the pixel values and use them to iteratively update the input images. In this way, all methods can be optimized in parallel with the total loss.
To verify the effectiveness our PST, we design a specific parallel scheme ParallelNer based on four popular domain-specific style transfer methods Chaimandaadd (2016); [. & Wand 2016); Gatys
4

---

Under review as a conference paper at ICLR 2020
Optional
content
Style Transfer (Method
content
Style Transfer (Method
Style Transfer (Methed n-1)
content
Style Transfier
(Method
Result
style
style
style
(a) Serial Style Transfer (SST)
Mothod
Loss Module
L2
content
style
Shared Backbone (eg. VGG-19)
Oplional
Leotal (n)
@h--1
4-1
Mechaa m-1
Whn
Result
(b) Parallel Style Transfer (PST)
Figure 4: Our proposed cascade style transfer, which contains architectures: serial style transfer, b) parallel style transfer.
6 Luan al. (2017). The principles for selecting the appropriate approaches will be discussed in later sections. The detailed combination procedure is as follows:
For method Gatys et al. (2016), we capture the vectorized features Fe[ and Fe[s] of layers ê E {convk 1 k for the generated image I and the style image 5, and compute the Gram matrices:
Gel- Fe
here the Gram matrix is defined the inner product between the vectorized features. In each layer, there are N₂ filters each with a vectorized feature of size Mp. We define our style loss L₁ as follows:
L1
G
2
(2N
For method Champandard (016) and Li & Wand 2016). reuse the features [a[ and Fa[s] /k=3- and then concatenate them with the segmentation masks Cn and 8.
of layers E [convk 1 of the content image and the style image at the same resolutions, respectively:
Fi.m
T
Fx [
A H(e., Ml.
(5)
Fn.m
Fn
5m, h).
where ö[. fi] denotes resizing the masks to the same resolution as the output of layer h. "0" denotes the channel concatenation, the hyperparameter is given to weight the semantic awareness.
Then we extract set x 3 neural patches for both Fr.m [X] and [5)m denoted by { Jee, and {4) (5)) ())enn,, where n, and n, are the number of extracted patches. For each patch o (7), we determine a flosest-matching style patch (c (5) based on the following measure:
( (8())
CM(1) = arg max
(6)
+
(5)
j=l....n,
Finally, our style loss L2 is defined as follows:
L2
(
+ C
(7)
1=1
For method Luan et al. (201)), ) we directly use their photorealism regularization term Lx:
3
La =
Vala MrVo
(8)
5

---

Under review as a conference paper at ICLR 2020
where M2 is the Matting Laplacian Matrix Levin et al. (2008), which is used to express a locally affine combination of the input RGB channels and only depends on the input image I. V2[2]] denotes the vectorized version of the output image i' in channel c.
At last, these methods use the same content loss L. to preserve the structure of the content image. F[dll;
Le
||
where F[ and F(c] denote the features extracted from layer conv4 2.
Total loss: In our loss module, the total loss is the simple linear combination of the above loss: Leotal = aL. + w1 + w2L2 + wyLy + vLrv,
(10)
where LTV refers to the total variation regularization loss Aly & Dubois (2005).
EXPERIMENTS
For SST, we select some of the most representative methods in specific domains, and the pub- lished implementation with default stylization settings for each method.
For PST (we specifically refer our ParalleiNer), the hyperparameters a, Wi, w2 and p are fixed at 10, 0.1, 10 and I, respectively. À is set to 10 for cases with segmentation masks and 0 for others. wa set to 10 for photo-realistic style transfer, and for others. The optimization is conducted by L- BFGS Zhu et al. (1997), and stopped at 500 iterations. The initialization is the content image.
5.1 ABLATION STUDIES
How select the appropriate methods? To find some empirical principles, we conduct user study (Table I on some representative NST methods in terms of content fidelity, stylization global color and local texture patterns, since these three aspects are our main considerations in selecting the methods. To alleviate the burden of subjects, we show them 150 synthesized images of each method, and ask them to select one overall level for each aspect. We collect totally 990 votes from 30 subjects, each method has 30 votes in each aspect. The levels with the most votes are chosen.
For SST, which methods could be combined and which order will return the optimal solutions? Take artistic style transfer as the example, Fig. 5 shows the comparisons of different serial schemes. The first column shows a reasonable serial scheme, since the method (b) help enrich the global color, and method (c) improves the local styles. The second column shows an unreasonable one, as the method (d) introduces some content distortions, and this directly affects the subsequent outputs. The third and fourth column show the effects of node order. We can find in the third column that method (b) and (a) progressively enrich the global color of (c), but do not change the local patterns. See the fourth column, unfortunately, changing the order of (d) cannot avoid the content distortions, but since the artistic style transfer does not require high content fidelity, the result is still acceptable. The last column shows some invalid nodes, method (b) and (e) have subtle effect on the result of (d). Now, look at these in conjunction with Table we can conclude the following:
For the aspects of global color and local patterns, the same level can promote each other, and the higher level can improve the performance of the lower level, while the lower level has little effect on the higher level. The results are determined by the order of methods with different levels.
For the aspect of content fidelity, the higher level is only conducive to content preservation, while the distortions produced by the lower level are irreversible and sequence-independen..
The selection is specific for particular domains, e.g., for artistic semantic style transfer, the global color and local patterns are prior to content fidelity, while for photo-realistic style transfer, content fidelity is the primary concern.
These conclusions are also applicable to semantic and photo-realistic style transfer. More results can be found in appendix. We believe that these would help users design useful serial schemes to obtain desired results, and inspire future works in neural style transfer.
For PST, how many and which methods could be combined? As introduced in Section I our PST is proposed to improve the quality and flexibility for different domains. Generally, any approach that satisfies the following two conditions be incorporated into PST:

---

Under review as a conference paper at ICLR 2020
Table I: User study representative NST methods in terms of the content fidelity, global color and local patterns. The performance has three levels: H (High), M (Medium), L (Low).
content fidelity
global color
local patterns
content fidelity
global color
local patterss
method
method
Shene ctal. 2018
M
M
M
Lnan of al
2017
H
H
M
L
MIS
M
M
L
Wand 2016
M
L
H
Gatys ot al
2016
M
Champsndann
M
L
H
Hunng
M
M
L
Go ot al
2018
M
M
H
et al
2017
M
M
H
Figure 5: Comparisons of different serial schemes of some NST methods, i.e. (a) Sheng et al. (2018), (b) Li et (2019), (c) & Wand (2016), (d) al. (011b)). (e) Huang & (211 Each column shows the intermediate outputs and corresponding close-ups of serial scheme.
Sharing the same backbone (e.g., VGG-19) and similar process flow (e.g., backward optimization).
Having the capacity solve the problem of different style transfer domain.
How to get the optimal hyperparameters for PST? Our PST introduces some hyperparameters, adapt the model to different domains by adjusting them, but the fine-tuning may be in- tractable. Fortunately, we find that for most methods, the optimal hyperparameters before and after combinations almost the same, e.g., the optimal values of À and wy of our Paralle[Ner same as those of the original methods. And for others, we just need to make a few adjustments according to the original methods. The optimal hyperparameters for some special ones, fine-tuning may produce better results. Ablation studies about different loss terms and their hyperparameters of our ParallelNet can be found in appendix.
the
generalized for most cases, but
5.2 EVALUATION AND COMPARISONS
Qualitative Comparisons: Fig. shows some qualitative comparisons on artistic, semantic and photo-realistic style transfer. In the top row, we select (Sheng al.] (2018), Li et al. (2019) and Li & Wand (2016) as three sequential nodes of our SST(a). Compared with them that only transter the global color or local textures, our SST(a) and PST can consider both aspects simultaneously. In the middle row, we select [hampandard] (2016). Gu et al. (2018) and Lia et al. (2017) three sequential nodes of our SST(b). We observe that these methods often produce either insufficiently stylized results or abnormal artifacts. By contrast, our SST(b) and PST produce much more satisfy- ing results. the bottom row, since the content fidelity level Li et al. (2018) in Table D is only M (we can also observe some undesired shadows in column 3) and this detrimental
SST in such high fidelity task, we only select [i et al. (2019) and Luan et al. (2017) as sequential nodes of (ST(c). As see, the result of method Li al. 2019 may lose some striking styles, e.g.,

---

Under review as a conference paper at ICLR 2020
Content
Style
Sheng et al. (2018)
Li al. (2019)
Li and Wand (2016)
Our SST (a)
Our PST
Content
Style
Champandard (2016)
Gu et al. (2018)
Lino et al. (2017)
Our SST (b)
Oar PST
Content
Style
Lict al.
Li al. (2019)
Luan (2017)
Our SST (c)
Our PST
Figure 6: Qualitative comparisons in artistic (top), semantic (middle) and photo-realistic (bottom) style transfer. More results can be found in appendix.
Table 2: Flexibility evaluation of different NST methods.
semantic
(mazk))
anistic
(So mask)
(mask)
realistic
Shang 2018
2016
X
X
ZVVA
Lieral 2019
X
Huang & Relongic 2017
X
X
X
VAV
V
X
2017
X
X
Clanjjandrd
2016
AANAA
Our SST(a)
X
Our SST(b)
X
V
Lioo el all 7017
Our SST(c)
Luna
X
PST
V
V
V
Li ct al.
2015
X
V
the light at the bottom of the ship. The method Luan al. (2017) may introduce some unacceptable artifacts, e.g., the abnormal hull. These problems do not occur in the results of our SST(c) and PST.
Quantitative Comparisons: We conduct several user studies on different style transfer domains. The results can be found in appendix. In all domains, our SSTs and PST show their superiority. And we also compare the flexibility of different methods in different style transfer domains Table 2 Since our SSTs target the higher quality for specific domains, they do not contribute to more Hexibiliity By contrast, our PST is the only one that can flexibly adapt all the mentioned domains.
Speed and Memory Analysis: Obviously, the time and memory requirements of our SSTs are simply the sums of those of its nodes. For PST, since the combined methods all share the same backbone, and the intermediate features can be stored and reused, the increments of time and mem- ory are slight. The speed our ParallelNet comparable to that method Gatys et (0116).
CONCLUSION AND FUTURE DIRECTIONS
In this paper, we propose cascade style transfer to combine multiple approaches in serial in par- allel without modifying any algorithm. Experiments have verified that our methods can improve the stylistic quality and flexibility over previous state-of-the-art methods in artistic, semantic and photo-realistic style transfer domains. In the future, more effective and efficient schemes could be designed, and the combination of SST and PST is also an interesting direction worthy of further methods can be regarded data-flow graphs of neural image editing oper-
studies. Moreover, ators from high-level perspective, it is another neat idea for interactive computer graphics system and will probably have considerable impact future work and architectures in this domain.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Hussein A Aly and Eric Dubois. Image up-sampling using total-variation regularization with new observation model. IEEE Transactions on Image Processing, 14(10):1647-1633 2005.
Alex J Champandard. Semantic style transfer and turning two-bit doodles into fine artworks. arXiv preprint 1.X3vv6603301660 2016.
Dongdong Chen, Lu Yuan, Jing Liao, Nenghai Yu, and Gang Hua. Stylebank: An explicit represen- tation for neural image style transfer. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1897-1906, 2017.
Tian Qi Chen and Mark Schmidt. Fast patch-based style transfer of arbitrary style. arXiv preprint arXiv: 0433.0 04337, 2016.
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. Proc. of ICLR, 2017.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. A neural algorithm of artistic style. arXiv preprint arXiv: 06576, 2015.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pp. 2414-2423, 2016.
Shuyang Gu, Congliang Chen, Jing Liao, and Lu Yuan. Arbitrary style transfer with deep feature reshuffle. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 8222-8231, 2018.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance nor- malization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1501- 1510, 2017.
Yongcheng Jing, Yezhou Yang, Zunlei Feng, Jingwen Ye, Mingli Song, Yongcheng Jing, Yezhou Yang, Zunlei Feng. Jingwen Ye, and Mingli Song. Neural style transfer: A review. arXiv preprint arXiv:1705.04055, 2017.
Justin Johnson, Alexandre Alahi, and Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference Computer Vision, pp. 694-711. Springer, 2016.
Pierre-Yve Laffont, Zhile Ren, Xiaofeng Tao, Chao Qian, and James Hays. Transient attributes for high-level understanding and editing of outdoor scenes. ACM Transactions Graphics (TOG), 33(4):149, 2014.
Anat Levin, Dani Lischinski, and Yair Weiss. A closed-form solution to natural image matting. IEEE transactions on pattern analysis and machine intelligence, 30(2):228-24) 242, 2008
Chuan Li and Michael Wand. Combining markov random fields and convolutional neural networks for image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2479-2486, 2016.
Xueting Li, Sifei Liu, Jan Kautz, and Ming- Hsuan Yang. Learning linear transformations for fast arbitrary style transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019,
Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming- Yang. Diversified tex- ture synthesis with feed-forward networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017a.
Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang. Universal style transfer via feature transforms. In Advances in Neural Information Processing Systems, pp. 386- 396, 2017b. 7b.

---

Under review as a conference paper at ICLR 2020
Yijun Li, Ming- Yu Liu, Xueting Li, Ming- Hiuan Yang, and Jan Kautz. A closed-form solution to photorealistic image stylization. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 453-468. 2018.
Jing Liao, Yuan Yao, Lu Yuan, Gang Hua, and Sing Bing Kang. Visual attribute transfer through deep image analogy. ACM Trans. Graph., 2017.
Ming Lu, Hao Zhao, Anbang Yao, Feng Xu, Yurong Chen, and Li Zhang. Decoder network over lightweight reconstructed feature for fast semantic style transfer. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2469-2477, 2017.
Fujun Luan, Sylvain Paris, Eli Shechtman, and Kavita Bala. Deep photo style transfer. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4990-4998, 2017.
Roey Mechrez, Itamar Talmi, and Lihi Zelnik-Manor. The contextual loss for image transformation with non-aligned data. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 768-783, 2018.
Yaroslav Nikulin and Roman Novak. Exploring the neural algorithm of artistic style. arXiv preprint arXiv:1602.071188 2016.
Dae Young Park and Kwang Hee Lee. Arbitrary style transfer with style-attentional networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2019.
Francois Pitie, Anil C Kokaram, and Rozenn Dahyot. N-dimensional probability density function transfer and its application to color transfer. In Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1, volume 2, pp. 1434- 1439. IEEE, 2005.
Erik Reinhard, Michael Adhikhmin, Bruce Gooch, and Peter Shirley. Color transfer between images. IEEE Computer graphics and applications, 21(5):3441, 2001.
Lu Sheng, Ziyi Lin, Jing Shao, and Xiaogang Wang. Avatar-net: Multi-scale zero-shot style transfer by feature decoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8242-8250, 2018.
YiChang Shih, Sylvain Paris, Connelly Barnes, William T Freeman, and Frédo Durand. Style trans- fer for headshot portraits. ACM Transactions on Graphies (TOG), 33(4):148, 2014.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.15555 2014.
Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, and Victor S Lempitsky. Texture networks: Feed-forward synthesis of textures and stylized images. In ICML, pp. 1349-1357, 2016.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis. In Proceedings of the IECE Conference on Computer Vision and Puttern Recognition, volume 1, pp. 6, 2017.
Yuan Yao, Jianqiang Ren, Xuansong Xie, Weidong Liu, Yong-Jin Liu, and Jun Wang. Attention- aware multi-stroke style transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Hang Zhang and Kristin Dana. Multi-style generative network for real-time transfer. European Conference Computer Vision, pp. 349-369955555 -633-33556060 Springer, 2018.
Ciyou Zhu, Richard H Byrd, Peihuang Lu, and Jorge Nocedal. Algorithm 778: L-bfgs-b: Fortran subroutines for large-scale hound-constrained optimization. ACM Transactions on Mathematical Software (TOMS), 23(4):500-560, 1997.
10

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
A.1 MORE RESULTS ABOUT INITIALIZATION
A.1.1 GATYS AL. (2016)
Fig. shows more results of different initialization strategies on method Gatys et al. (0116). As mentioned in the paper, the first column shows the content image (top) and the style image (bottom). The other columns show the initialization images (top) and the corresponding style transfer results (bottom) of method Gatys et al. (2016). In column (f), we initialize with the default stylized results of Le & Wand (2016) (SROOM). In column (g), we replace the content image with SROOM and then initialize with (RC-SROOM).
Images
(4) Content
(b) Style
(c) White Noise
(4) Salt and Pepper
(c) Poisson Noise
(1 SROOM
(g) RC-SROOM
Figure 7: More results of different initialization strategies on method Gatys et al. (2016)

---

Under review as a conference paper at ICLR 2020
A.1.2
Li & WAND (0066)
Fig. shows more results of different initialization strategies on method Li Wand (2016). The first column shows the content image (top) and the style image (bottom). The other columns show the initialization images (top) and the corresponding style transfer results (bottom) method Li & Wand (216)) In column (f), we initialize with the default stylized results of method Gatys et al. (2016) (SROOM). In column (g), we replace the content image with SROOM and then initialize with it (RC-SROOM).
Images
(x) Content
(b) Style
White Noise
(d) Salt and Pepper
(6) Poisson Noise
SROOM
(g) RC-SROOM
Figure 8: More results of different initialization strategies method Li & Wand 2016
12

---

Under review as a conference paper at ICLR 2020
A.2 MORE SERIAL SCHEMES FOR SST
A.2.1 ARTISTIC STYLE TRANSFER
Fig. shows more comparisons of different serial schemes in artistic style transfer domain. As men- tioned in the paper, each column represents a serial scheme, and each row shows the intermediate output. We select (a) Sheng al. (2018). (b) Li et al. (2019), (c) Li & Wand (2016), (d) Li et
2017b), (e) Huang & Belongie (2017) for the nodes of different serial schemes.
(a)
(d)
(b)
(d)
(c)
cc)
a
(b)
(c)
Figure 9: Comparisons of different serial schemes in artistic style transfer domain.
A.2.2 SEMANTIC STYLE TRANSFER
Fig. T0 shows some comparisons of different serial schemes in semantic style transfer domain. Each column represents serial scheme, and each row shows the intermediate output. We select (a)
Champandard (2016), (b) Gu et al. (2018), (c) Liao al. (2017). (d) Li et al. (2017b), (e) Sheng
ct al. (2018), (f) Li al. 2019) for the nodes of different serial schemes.
aa)
(b)
(c)
a
(b)
(b)
(d)
(b)
(d)
(c)
b
(I)
Figure 10: Comparisons of different serial schemes in semantic style transfer domain.
13

---

Under review as a conference paper at ICLR 2020
A.2.3 PHOTO-REALISTIC STYLE TRANSFER
Fig. 11 shows some comparisons of different serial schemes in photo-realistic style transfer domain.
Each column represents serial scheme, and each row shows the intermediate output. We select
(a) Li et (2019). (b) Luan et al. (2017), (c) Li et al. (2018), (d) Liao al. (2017). (e) Huang &
Belongie (2017) for the nodes of different serial schemes.
(a)
(b)
(b)
Figure 11: Comparisons of different serial schemes in photo-realistic style transfer domain.
Content
@1 0.01
W11 0.1
@== 1
w1= 10
Gatys et al. (2016)
Style
W2
W2
@2 20
@2 50
Li and Wand (2016)
Figure 12: Varying weight wi of loss L, and W2 of loss La for artistic style transfer. The top row shows the results obtained by fixing w2, w' and at 10, 0 and 0, respectively, and increasing from 0.01 10. The bottom row shows the results obtained by fixing w1, w/ and 0.1, 0 and O, respectively, and increasing We from to 50.
A.3 ABLATION STUDIES OF PARALLELNET
We study the effects of different loss terms of proposed ParalleiNer, including style loss Li (in (4) Le (in (5) and Eq.7)) and photorealism regularization loss L3 (in Eq. (8)) Generally, the weight a of L. is fixed 10, and the weight μ of Lrv fixed to Fig. 12 shows the results of artistic style transfer by varying weight w/ of loss L1 and Wa of loss Lg while fixing the other weights. As the last column shows, the original method of Gatys et al.) (2016) transfers the global color and rough textures of the style image, but does not transfer the local intricate patterns. The original method oi & Wand (2016) transfers much more local style patterns, but as shows, this method may generate insufficiently stylized result when huge difference exists between the content and style image. By combining the characteristics of these two methods, our ParalleiNer can transfer both global color and local textures of the style images. As the top row shows, increasing the weight wi of loss L, transfers more global color and rough textures. And as the bottom row shows, increasing the weight w2 of loss La retains more local style patterns. However, since the trade-off
14

---

Under review as a conference paper at ICLR 2020
Image (a)
W2 0
02 10
@2 50
W2 100
Image (b)
2=
À=
λ= 10
2= 50
Figure 13: Varying weight we and À of loss L2 for semantic style transfer. The top row shows the results obtained by fixing wi, wz and X at 0.1, 0 and O, respectively, and increasing w2 from 0 to 100. The bottom row shows the results obtained by fixing W1, W2 and Wa at 0.1, 10 and 0, respectively, and increasing À from 0 to 50.
Content and Style
6dg
43g 102
Qgg 10
Bdg 10
4bg 10
Figure 14: Varying weight w/ of loss L3 for photo-realistic style transfer. The values of W1, w2 and À are fixed at 0.1, 10 and 10, respectively.
between the content loss and the style loss, distortions the content structure are inevitable the values of w1 and w2 too high. In our work, for artistic style transfer, wi and we set to 0.1 and 10 by default, respectively.
Fig. 13 shows the results of semantic style transfer by varying weight we and À of loss Lz. The top row shows transfer of painted style (a) onto a photo (b), this easy and does not require the constraints of segmentation masks, hand, according to the aforementioned practice, we set the value of w1 and Wa to 0.1 and 0 by default, respectively. As the top row shows, increasing W2 refines more detailed information of the corresponding semantics, but this may change the original structure of the content image. To avoid this, the value of W2 should not be too high, so in our work, these, the bottom row shows transfer of photo style (b) onto a painting (a), it is hard so we use the constraints segmentation masks to get better results. As the bottom row shows, increasing achieves more accurate semantic matching. To obtain the best results, we set À to 10 by default.
fix the semantic awareness weight À at 0. On the other
it 10 by default. Based on
Fig. 14 shows the results of photo-realistic style transfer by varying weight wa of photorealism regularization loss L3. Differing from Luan et al. (2017) using a two-stage optimization process based on the outputs of Gatys et al. (2016), we directly solve this problem by optimizing Eq. (TO). This is much simpler, and could produce comparable results due to the synergistic effects of different loss items. As we can see, a too small value of Wy cannot prevent distortions, thus the results in column 2 and 3 have non- pootootooeaa look. On the contrary, too large value of Wy suppresses the style to be transferred and leads to color infidelity (see column and 6). Similar to Luan et al. (017), choose w3 10a by default.
15

---

Under review as a conference paper at ICLR 2020
A.4 QUALITATIVE COMPARISONS WITH OTHER METHODS
Here we demonstrate the results of our methods and twelve other representative single- or multi- domain methods Sheng al. (2018); Li al. (2019); Li & Wand (2016); Champandard (2016); Gu et al. (2018): Liao et al. ( 7 E Luan et al. 2017g Li et al. (2018); Yao et al. (2019); Gatys et al. (2016) Huang & Belongie (2017) Li et al. (211b)) on artistic, semantic and photo-realistic style transfer, respectively. Each nggure shows the representative results obtained by one method. The left, center and right columns show examples of artistic, semantic and photo-realistic style transfer, respectively. For the inputs in each group, the upper one is the content image and the lower one is the style image, Relevant quality and flexibility analyses can found under the caption of each figure. Compared to these methods, our SST schemes perform better on stylistic quality in specific domains, and our PST scheme performs better on stylistic quality and flexibility in all domains.
Figure 15: Results obtained by our SST(a) scheme.
As shown in Fig. 15 this scheme can perform well artistic and semantic style transfer (without segmentation masks). But since its nodes do not support segmentation masks, it could not solve the cases when huge differences exist between the content and style images (e.g.. the case at the top of the center column). This scheme is not suitable for photo-realisti style transfer, as it may produce lot of content distortions and abnormal artifacts (see the right column).
16

---

Under review as a conference paper at ICLR 2020
Figure 16: Results obtained by our SST(b) scheme.
shown in Fig. T6 this scheme can perform well on artistic and semantic style transfer (with and without segmentation masks). However, in artistic style transfer, there are some deficiencies in content preservation because prefers to express more style features (see the two cases at the bottom of the left column). This also limits its capability in photo-realistic style transfer (see the right column).
Figure 17: Results obtained by our SST(c) scheme.
As shown in Fig. 17 this scheme is only suitable for photo-realistic style transfer. As we see, maintains the photorealism of the content photographs and at the same time transfers the global color of the style images. But this also makes difficult to produce the artworks with non-realistic styles. Since the nodes of can incorporate segmentation masks, this scheme is capable to use segmentation masks.
17

---

Under review as a conference paper at ICLR 2020
Figure 18: Results obtained by our PST scheme.
As shown in Fig. 18 this scheme is flexible enough to apply to all these style transfer domains. As we can see, in the left column, both the global color and local patterns of the artistic images can be transferred by this scheme. In the center and right column, can also achieve semantic and photo- realistic style transfer with and without segmentation masks. Moreover, users further improve the stylistic quality by fine-tuning our provided hyperparameters for each input.
Figure 19: Results obtained by the method of Sheng et al. 20188
As shown in Fig. 19 this method can be used in artistic and semantic style transfer (without seg- mentation masks), but the stylistic quality is not so good. As we can see, for artistic style transfer, only transfers the global color of the style images, but lacks local patterns. For semantic style trans- fer, may produce lot hazy blocks, which affect the overall clarity. These problems (including content distortions and abnormal artifacts) also exist in the photo-realistic style transfer.
18

---

Under review as a conference paper at ICLR 2020
Figure 20: Results obtained by the method of Li et al. (0199)
As shown in Fig. 20 this method can be used in artistic and photo-realistic style transfer. But since prefers maintain the structures of the content images, the produced results are often insufficiently stylized. This could help it to perform better in rhoto-realistic style transfer. On the other hand, since the spirit of this method is based on the global statisties, cannot solve the tasks of semantic style transfer. This method is capable to use segmentation masks.
Figure 21: Results obtained by the method Li & Wand (2016)
As shown in Fig. 21 this method can be used in semantic style transfer (without segmentation masks). As we can see, for artistic style transfer, it could transfer adequate local patterns of the style images, but the global effects of the stylized results insufficient global color, see the left column). On the other hand, introduced content distortions also limit its capability for photo-realistic style transfer (see the right column).
unsatisfying (mainly because of the
19

---

Under review as a conference paper at ICLR 2020
Figure 22: Results obtained by the method of Champandard (2016)
As shown in Fig. 22 this method can be used in artistic and semantic style transfer (with and without segmentation masks). For artistic style transfer (see the left column), the most results similar that of Li & Wand (2016), the difference is that this method can perform better on the global color or the local patterns, but it is still unsatisfying. For semantic style transfer, may produce blurred results (e.g.. the two cases at the top of the center column). For photo-realistic style transfer, it cannot adapt to this domain because of the introduced content distortions.
Figure 23: Results obtained by the method Gu et al. 2018
As shown Fig. 23 this method can be used in artistic and semantic style transfer (without seg- mentation masks). The stylized results produced by it obtain sufficient global color and local patterns, but there are also lot of abnormal artifacts which directly affect the final effect. This occurs in almost every case in these three domains.
20

---

Under review as a conference paper at ICLR 2020
Figure 24: Results obtained by the method of Liao al. (2017)
As shown in Fig. 24 this method can be used in semantic and photo-realistic style transfer (without segmentation masks). is more suitable for style transfer of image pairs which have high semantic- level correspondences. Therefore, for the cases of artistic style transfer which are not relevant in semantics, this method yields poor stylized results (see the left column). On the other hand, since this method does not support segmentation masks, mismatching is prone to occur in challenging tasks (c.g., the case at the top of the center column and cases the right column).
Figure 25: Results obtained by the method of Luan et al. (007))
As shown in Fig. 25 similar to our SST(c) scheme (see Fig. this method is only suitable for photo-realistic style transfer. Compared to our SST(c) scheme, may produce some abnormal artifacts, e.g., the hull and eyes in the two cases at the bottom of the right column, respectively. This method is capable segmentation masks.
21

---

Under review as a conference paper at ICLR 2020
Figure 26: Results obtained by the method of Li et al. (2018)
As shown in Fig. 26 similar to our SST(c) scheme (see Fig. 17 and the method Luan et (2017) (see Fig. 25 this method is only suitable for photo-realistic style transfer. See the right column, compared to our SST(c) scheme, it may produce too many undesired effects, c.g., the skies in the case and case 3, the shadows in the case 2 and the eyes in the case (numbers are arranged from top to bottom). This method is capable to use segmentation masks.
Figure 27: Results obtained by the method of Yao et al. (2019)
As shown Fig. 27 this method can be used in artistic and semantic style transfer (without seg- mentation masks). Because of the incorporation with self-attention mechanism, this method can highlight more salient areas (e.g., characters' eyes) of the images. But in other places, the perfor- mance is similar the method Sheng al. (2018) (see Fig. 19
22

---

Under review as a conference paper at ICLR 2020
Figure 28: Results obtained by the method of Gatys al. (016)
As shown in Fig. 28 this method is only suitable for artistic style transfer. As see, could transfer the global color and rough textures of the artistic style images to the content images (see the left column). But since the spirt of its algorithm is based on the global statistics, this method cannot solve the semantic style transfer (see the center column). Of course, without the improvement introduced by Luan et al. (2017), cannot solve the tasks of photo-realistic style transfer (see the right column).
Figure 29: Results obtained by the method of Huang & Belongie (0117)
As shown in Fig. 29 similar to the method of Gatys al. 2016 (see Fig. 28 this method is
only suitable for artistic style transfer. Compared to the method Gatys et al. (011)). some results generated by it not sufficiently stylized (e.g., the cases the bottom of the left column).
23

---

Under review as a conference paper at ICLR 2020
Figure 30: Results obtained by the method of Li et al. (001b))
As shown in Fig. 30 similar the methods of Gatys al. (2016) (see Fig. and Huang & Belongie (017)) (see Fig. 22) this method is only suitable for artistic style transfer. Compared to the methods Gatys al. (2016) and Huang & Belongie (2017), some results generated by excessively stylized, thus introducing a lot of undesired effects (e.g., the case and case in the left column).
A.5
QUANTITATIVE COMPARISONS WITH OTHER METHOSS
A.5.1
USER STUDY ARTISTIC STYLE TRANSFER
#Sheng (2017))
(2029)
Wand (200)) Who
(2014)
Chur (571g)
SST())
wour (s73))
Our PST
Shanp ct (201))
15..34
Uietl (2028)
10.3
Our 99T/0)
9.0%
U and Wond (2005)
Our (RTW)) 14.3%
Getya
SUNN
Figure 31: User preference result of ten style transfer methods for artistic style transfer.
We conduct study to evaluate the proposed schemes against the state-of-the-art style transfer methods Sheng et al. (2018): Li et al. (2019); Li & Wand (2016); Yao et al.] (2019); /aays et al.) Li al. 201 artistic style transfer. We use 10 content images and 10 style im-
2016):
24

---

Under review as a conference paper at ICLR 2020
ages synthesize 100 images in total for each method, and randomly select 50 content and style combinations to each subject. We show stylized images of 10 compared methods (including ours) side-by-side in random order and ask the subjects to select the most visually pleasant one. We col- lect 1500 votes from 30 users and show the percentage of votes for each method in Fig. Overall, proposed PST and (a)(a) are favored among all evaluated methods.
A.5.2 USER STUDY ON SEMANTIC STYLE TRANSFER
We conduct user study to evaluate the proposed schemes against the state-of-the-art style transfer methods Sheng et al. (2018); Li & Wand (2066;; Champandard (2016); ; Gu et al. (2018): Liao et al. (2017); Yao et al. (2019)
semantic style transfer. For each method, we use 30 image groups to synthesize 30 images (all images are produced without using segmentation masks) in total. We show stylized images of 10 compared methods (including ours) side-by-side in random order and ask the subjects to select the most visually pleasant one. We collect 900 votes from 30 users and show the
percentage of votes for each method Fig. 32 (a). Overall, our proposed PST, SST(b) and SST(a) are favored among all evaluated methods.
Rownng aral (231))
Walane (200)
13.9
Wand (2UA)
38:34
Ui Al (2319)
ROur someee 1O00
5570
Rou
Pluen stal (20a.)
PLao p017)
Chanpedded
Las
2.34
#Duu sorg0
(200)
15.5%
MMTW
21.1%
(0)0
SST1))
Rou s5Tid
Luin et (12a))
Mour
Dar SATM
Unsel DSD)
(D00)
14.41
(a)
(b)
Figure 32: (a) User preference result of ten style transfer methods for semantic style transfer. (b)
User preference result of eight style transfer methods for photo-realistic style transfer.
A.5.3 USER STUDY ON
STYLE TRANSFER
We conduct a user study to evaluate the proposed schemes against the state-of-the-art style transfer methods Li al. (019); Liao et al.] (2017): Luan et al. (017): Li (2018) on photo-realistic style transfer. For each method, we use 30 image groups synthesize 30 images in total. We show stylized images of compared methods (including ours) side-by-side
random order and ask the subjects to select the most visually pleasant one. We collect 900 votes from 30 users and show percentage votes for each method Fig. (b). Overall, our proposed SST(c) and PST are favored among all evaluated methods.
25


Reviews:
[{"review_id": "SkxhgEzMtB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nIn this study, the authors propose a new method for performing artistic style transfer for  arbitrary image and styles. The new method employs a cascade/serial architecture for performing the style transfer. The authors test their method using human preference studies.\n\nIn summary, I found the architecture choice to be minimally explored. More importantly, a vast majority of the results to demonstrate the relative merits of this method were qualitative. The minimal quantitative results were unconvincing and left many unanswered questions about how well one could trust these results.\n\nMajor Comments:\n\n1. No experiments to explore the architecture hyperparameters.\nA natural question might be how the quality of the method varies systematically as the number of methods N grows. Presumably, if N=1, this would recover previous methods. \n\n2. Authors are missing an important reference and point of comparison for arbitrary style transfer.\n  Exploring the structure of a real-time, arbitrary neural artistic stylization network\n  Golnaz Ghiasi, Honglak Lee, Manjunath Kudlur, Vincent Dumoulin, Jonathon Shlens\n  https://arxiv.org/abs/1705.06830\n  http://goo.gle/2oiDKaT \n\n3. Minimal quantitative analysis.\nA vast majority of the results (30 of 32 figures) are qualitative comparisons and the paper is sorely lacking an emphasis on quantitative comparisons. This is a large and notable problem in this paper and a quantitative comparison *should* constitute the primary thrust and central result of such a paper to convincingly demonstrate to a reader that the proposed method is indeed to superior to other techniques. I wish the authors dedicated more emphasis in this paper to a detailed quantitative comparison for these methods. As a starter, the analysis presented as the final two appendix figures (31 and 32) should be front and center in the result section of the paper.\n\n4. User study for quantitative comparison is incomplete and unconvincing.\nTable 1 and Appendix Figure 31 and 32 represent the primary result of this paper as these results and comprise the user studies to quantify how much better this method is to previous methods. These studies however are fairly unconvincing as lots of details are omitted and and I am concerned about the rigor of the human studies including but not limited to:\n  4a. How long did each human study each image? What controls were added to the study to ensure that all images were equally studied by humans? For instance, were any golden tests employed to ensure user engagement throughout the study?\n  4b. What was the repeatability of each measurement of preference? If a single human was presented the same image twice, how consistent were there ratings? For that matter, how consistent were the ratings across humans? I presume that some humans preferred some styles over others but how systematic was this?\n  4c. What types of user testing scenarios were explored to ensure minimal bias in the results? Were multi-choice, paired choice or force choice employed? What about minimal or maximal time limit enforcement?\n  4d. How can I have confidence that the authors did not cherry pick images and styles that favored their method? For that matter, I would expect that some methods work better on some styles or images. I would expect to see analysis accordingly to break down which styles/images work better on different slices of the data.\n  4e. The statistical significance of Figure 31 and Figure 32 is not provided. What would an error bar look like with resampling? ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HyeCd0d_cS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors proposed to mix together multiple styles by proposing two frameworks: 1) serial style transfer (SST), which combines style transfer methods in series; and 2) parallel style transfer (PST), which combines style transfer methods in parallel.\n\nThe paper is clearly presented. It is interesting to see work on mixing up different styles, since it is not extensive studied so far. Though not much studied, this topic is not new [ref 1], [ref 2]. The authors didn't provide a thorough literature review on mixing multiple styles in the related work or anywhere else in the submission.\n\nIn terms of the methodology, the novelty is quite limited. The proposed SST and PST are simple frameworks to mix different styles, which, by the way, are fully based on existing style transfer methods. At some point, PST is similar to [ref 2], and the difference is minor. PST linearly combines the losses for different styles to construct the final loss, while [ref 2] linearly combines together the features for different styles.\n\nAbout experiments, it is not clear about how the predefined parameters (e.g. \\alpha, w_1, w_2, etc.) are determined. They were just empirically set and mentioned in the experiment section.\n\nIt is appreciated to see more results in the Appendix, as well as the user study. However, due to lack of novelty, I think this submission may not be qualified for acceptance at this moment.\n\nMinor:\nI think the authors should give their proposed framework another name instead of using \"cascade,\" which has a similar meaning of \"series.\"\n\n\n[ref 1] Google's arty filters one-up Prisma by mixing various styles. https://www.engadget.com/2016/10/27/google-style-transfer-tech/\n[ref 2] Pegios, et al. Style Decomposition for Improved Neural Style Transfer. ArXiv, 2018.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HkgZ0-Ah9H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes two ways to aggregate existing style transfer methods and shows improvements on quality and flexibility. However, the proposed method does not solve the limitations of any previous methods. Instead, it is as simple as an easy combination: the proposed SST is a sequential combination and the proposed PST has no difference with running each single method separately. To me this is more like an engineering effort rather than a research work. \n\n(1) For SST, it just connects N existing methods, using the output of method 1 as the input of method 2. The quality of results might be improved but there is little novelty. I agree combing methods can be a contribution only when there are principle designs and in-depth analysis. \n\n(2) For PST, I do not see its difference with running single method separately. Putting all previous methods altogether cannot be called being more flexible. As said in the paper, when for photorealistic transfer, the proposed PST set the loss weight of other methods except Luan et al. as 0. Then is it the same with running the single method of Luan et al.?\n\nIn general, I do not encourage such a way of exploring research. Authors should focus more on the unsolved issues in style transfer, e.g., how to do geometric style transfer (shape).", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work combines style transfer approaches either in a serial or parallel fashion, and shows that the combination of methods is more powerful than isolated methods.
The novelty in this work is extremely limited and not offset by insightful analysis or very thorough experiments, given that most results are qualitative. Authors have not provided a public response.
Therefore, we recommend rejection.
Decision: Reject