Artistic Style Transfer with Internal-external Learning and Contrastive Learning
Authors: Haibo Chen
Lei Zhao
Zhizhong Wang
Zhang Hui Ming
Zhiwen Zuo
Ailin Li
Wei Xing
Dongming Lu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Artistic style transfer
Internal-external learning
Contrastive learning
Quality
Stability

Abstract: Although existing artistic style transfer methods have achieved significant improvement with deep neural networks, they still suffer from artifacts such as disharmonious colors and repetitive patterns. Motivated by this, we propose an internal-external style transfer method with two contrastive losses. Specifically, we utilize internal statistics of a single style image to determine the colors and texture patterns of the stylized image, and in the meantime, we leverage the external information of the large-scale style dataset to learn the human-aware style information, which makes the color distributions and texture patterns in the stylized image more reasonable and harmonious. In addition, we argue that existing style transfer methods only consider the content-to-stylization and style-to-stylization relations, neglecting the stylization-to-stylization relations. To address this issue, we introduce two contrastive losses, which pull the multiple stylization embeddings closer to each other when they share the same content or style, but push far away otherwise. We conduct extensive experiments, showing that our proposed method can not only produce visually more harmonious and satisfying artistic images, but also promote the stability and consistency of rendered video clips.

Artistic Style Transfer with Internal-external Learning and Contrastive Learning
Haibo Chen
Lei Zhao"
Zhizhong Wang
Huiming Zhang Dongming Lu
Zhiwen Zuo
Ailin Li
Wei Xing
College of Computer Science and Technology. Zhejiang University {cshbchen, cszhl, endywon, qinglanwuji, zzwcs, liailin, wxing, 1dm}@zju. edu. on
Abstract
Although existing artistic style transfer methods have achieved significant im- provement with deep neural networks, they still suffer from artifacts such disharmonious colors and repetitive patterns. Motivated by this, we propose an internal-external style transfer method with two contrastive losses. Specifically, we utilize internal statistics of a single style image to determine the colors and texture patterns of the stylized image, and in the meantime, ternal information of the large-scale style dataset to learn the human-aware style information, which makes the color distributions and texture patterns in the stylized image more reasonable and harmonious. In addition, we argue that existing style transfer methods only consider the content-to-stylizatini and style-to-stylizeee relations, neglecting the we introduce two contrastive losses, which pull the multiple stylization embeddings closer to each other when they share the same content or style, but push far away otherwise. We conduct extensive experiments, showing that our proposed method can not only produce visually more harmonious and satisfying artistic images, but also promote the stability and consistency of rendered video clips.
leverage the ex-
cosattionnto-tylizaaiin relations. To address this issue,
Introduction
Artistic style transfer is long-standing research topic that seeks render a photograph with given artwork style. Ever since Gatys et al. (10) for the first time proposed neural method, which leverages pre-trained Deep Convolutional Neural Network (DCNN) to separate and recombine
contents and styles of arbitrary images, unprecedented booming 20 26 style transfer has been witnessed.
15
30
36
51 48
in
Despite the recent progress, there still exists large gap between real artworks and synthesized stylizations. As shown in Figure [ the stylized images usually contain some disharmonious colors and repetitive patterns, which makes them easily distinguishable from real artworks. We argue that this is because existing style transfer methods often confine themselves to the internal style statistics of a single artistic image. In some other tasks (for example, image-to mnage-to-image translation I7 60 16 25 I8)), the style usually learned from collection of images, which inspires to leverage the external information reserved in the large-scale style dataset to improve the stylization results in style transfer. Why is the external information so importunt for style transfer? Our analyses are as follows:
Although different images in the style dataset vary greatly in fine details, they share key commonality: they are all numan-createe artworks, whose brushstrokes, color distributions, texture patterns, tones,
Corresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Style
Content
Ours
Gatys al.
AdaiN
WCT
Avatar-Net
LST
SANet
Figure 1: Stylization examples. The first and second columns show the style and content images, respectively. The other seven columns show the stylized images produced by our method, Gatys et al. 101, AdaIN [5]. WCT 330, Avatar-Net [41], LST (28). and SANet [36]
efc., are consistent with human perception. Namely, they contain some human-aware style information that is lacked in synthesized stylizations. A natural idea is to utilize such human-aware style information improve stylization results. To this end, employ internal-external learning scheme during training. which takes both internal learning and external learning into consideration.
To be more specific, on the one hand, follow previous methods [10
20
46
54| [58), utilizing
internal statistics of a single artwork to determine the colors and texture patterns of the stylized image. On the other hand, we employ Generative Adversarial Nets (GANs)
M 39 56 3 to
externally learn the human-awar style information from the large-scale style dataset, which is then used make the color distributions and texture patterns in the stylized image more reasonable and harmonious, significantly bridging the gap between human-create artworks and Al-created artworks.
In addition, there is another problem with existing style transfer methods: they usually employ content loss and a style loss to enforce the content-to-stylizaiee relations and style-to-stllizatie relations, respectively, while neglect the dlylization-to-styiiiaiooo relations, which also important for style transfer. What are stylization-ler totuatioontt-sstlizaaton relations? Intuitively, stylized images rendered with the same style image should have closer relations in style than those rendered with different style images. Similarly, stylized images based on the same content image should have closer relations in content than those based on different content images. Inspired by this, in this paper we introduce two contrastive losses: content contrastive loss and style contrastive loss that can pull the multiple stylization embeddings closer to each other when they share the same content or style, but push far away otherwise. To the best of our knowledge, this is the first work that successfully leverages the power of contrastive learning 12
in the style transfer scenario.
Our extensive experiments show that the proposed method can not only produce visually more harmonious and plausible artistic images, but also promote the stability and consistency of rendered video clips.
To summarize, the main contributions of this work are threefold:
We propose novel internal-external style transfer method which takes both internal learning and external learning into consideration, significantly bridging the gap between human- created and Al-created artworks.
stylization results with the learned ayylization-t--styizzaaee liiztttni relations. We for the first time introduce contrastive learning to style transfer, yielding more satisfying
We demonstrate the effectiveness and superiority of our approach by extensive comparisons with several state-of-the toetthe-rt artistic style transfer methods.
2
Related Work
Artistic style transfer. Artistic style transfer an image editing task that aims at transferring artistic styles onto everyday photographs create new artworks. Earlier methods usually resort to traditional

---

techniques such as stroke rendering [13. image analogy [14 9 31, and image filtering [52 to perform artistic style transfer. These methods typically rely on low-level statistics and often fail to capture semantic information. Recently, Garys et al. [10] discovered that the Gram matrix upon deep features extracted from a pre-trained DCNN can notably represent the characteristics of visual styles, which opens up the neural style transfer era. Since then, a suite of neural methods have been proposed, boosting the development of style transfer from different concerns. Specifically, [20] 27 46] utilize feed-forward networks to improve efficiency. 126 54 36
refine various elements in the stylized images (including content preservation, textures, brushstrokes, etc.) to enhance visual quality,
58 35
77 15 30
41 (8 propose universal style transfer methods to achieve generalization. [29 47 51
inject random noise to the generative network to encourage diversity. Despite the rapid progress, these style transfer methods still suffer from spurious artifacts such as disharmonious colors and repetitive patterns.
5 that aims to learn artist's 's style from all his/her artworks. In comparison, instead of learning an artist's 's style, we focus on better leaning an artwork's style (just like the style transfer methods mentioned in the previous paragraph) with the assist of the human-aware style information reserved in the external style dataset. Therefore, our method is orthogonal to these works.
Notice that there is another line of work 40 24 23 45
Image-to-image translation. Image-to-image translation (I21)
aims at learning 25
the mapping between different visual domains, which is closely related to style transfer. [60| [6] have distinguished these two tasks: (i) 12I can only translate between content-simile visual domains (such as horses+>7ebbas and summer summer<-winter), while style transfer does not have such limitation, whose content image and style image can be totally different (e.g., the former is photo of person and the latter is van Gogh's s The Starry Night). (ii) 121 aims to learn the mapping between two image collections, while style transfer aims to learn the mapping between two specific images. However, we argue that we can borrow some insights from 121, and leverage the external information of the large-scale style image collections to improve the stylization quality in style transfer.
Internal-exteraal learning. generation tasks, such as internal-extemaa learning has shown effectiveness in various image image inpainting, and so on. In detail, Soh et al. [44] presented a fast, flexible, and lightweight self-supervised super-resolution method by exploiting both external and internal samples. Park el al. (37 developed an that facilitates super-resolution networks to further enhance the quality of the restored images. Wang et al. [49] proposed a general external-interaa learning inpainting scheme, which learns semantic knowledge externally by training on large datasets while fully utilizes internal statistics of the single test image. However, in the field of style transfer, existing methods only use a single artistic image to learn style, resulting in unsatisfying stylization results. Motivated by this, in this work we propose an internal-external style transfer method that takes both internal learning and external learning into consideration, significantly bridging the gap between
super-resolution method
and Al-created artworks.
Contrastive learning. Generally, there are three key ingredients in a contrastive learning process: query, positive examples, and negative examples. The target of contrastive learning is to associate a "query" with its "positive" example while disassociate the "query" with other examples that are referred to as "negatives". Recently, contrastive learning has demonstrated its effectiveness in the field of conditional image synthesis. To be more specific, ContraGAN 21 introduced conditional contrastive loss (2C loss) to learn both data-to-class and data-to-data relations. Park et al. (38] maximized the mutual information between input and output with contrastive learning to encourage content preservation in unpaired image translation problems. Liu et al. (34] introduced a latent- augmented contrastive loss to encourage images generated from adjacent latent codes to be similar and those generated from distinct latent codes to be dissimilar, achieving diverse image synthesis. Yu et al. (55) proposed a dual contrastive loss in adversarial training that generalizes representation to more effectively distinguish between real and fake, and further incentivizes the image generation quality. Wu et al. (53] improved the image dehazing result by introducing contrastive learning, which ensures that the restored image is pulled closer to the clear image and pushed far away from the hazy image in representation space.
Note that all the above contrastive learning methods cannot be adopted for style transfer. In this work, we make the first attempt to adapt contrastive learning to artistic style transfer, and propose two novel contrastive losses: content contrastive loss and style contrastive loss to learn the stylization-to- stylization relations that are ignored by existing style transfer methods.
3

---

VGG
sample
Style 5
Encoder (V66)
Transformation
Decoder
Disoriminator
sample
1,c
Ladk
VIGG Nertwork
dataset
(a)
conto
Pasliow
Poiliivr
1. (1.)
I. (I.)
S:G
SuC
S:C
S:C.
Negaoiee
SySa
Suc.
Nogotive
Projection
Encoder
Transformation
Encoder (VGG)
Transformation
Decoder
Encoder [VGG]
Decoder
1cc (1..)
Sile
Powiive
Powiiiw
Projection
SC
Negarive
S
Nogathic
(b)
(c)
Figure 2: An overview of the proposed method. (a) illustrates basic framework, which mainly contains a pre-trained encoder, style-attentional transformation module, decoder, and a discrimi- nator. The style loss f. and the content loss Le used learn the style and content information, respectively. The adversarial loss Ladu is used to learn the human-aware style information. (b) and (c) depict the identity loss Lidentity and contrastive losses -on & Le-contra- where Lidentity is used to preserve more content structures and style characteristics in the stylized image, and L.-contra & Co-contra are used learn the stylization-Be
toraatioontotstyliatiinn stylization relations.
3 Proposed Method
Existing style transfer methods usually produce unsatisfying stylization results with disharmonious colors and repetitive patterns, which makes them pretty easy be distinguished from real artworks. As an attempt to bridge the large gap between human-creater and Al-created artworks, we propose novel internal-external style transfer method with contrastive losses. The overview of our method
is shown in Figure 2 It is worth noting that our framework is built on the SANet (36] (one of the state-of-the of-the--rt style transfer methods) backbone, which consists of encoder a transformation module T. and decoder D. In detail, E is pre-trained VGG-19 network 43 used to extract image features, T' a style-attention onto the content features, and D is a generative network used to transform encoded semantic feature maps into stylized images. We extend SANet (36] with our proposed changes, and our full model described below.
network that can flexibly match the semantic nearest style features
3.1 Internal-external Learning
Let C' and be the sets of photographs and artworks, respectively. aim to learn both the internal style characteristics from single artwork Ia S and the external human-aware style information

---

from the dataset S, and then transfer them to an arbitrary content image Le E C' to create new artistic images Ie-
we use a pre-trained VGG-19 network @ capture the internal style characteristics from single artistic image, and the style loss can be generally computed as:
Internal style learning. Following previous style transfer methods 13 36
L= - u(o: 2l2 + (0)() (d)((1)) lla
(I)
where d; denotes the ien layer (Relul_1, Relu___, Relu3 our model) of the VGG-19 network. 1 and or represent the mean and standard deviation of feature maps extracted by 6; respectively.
R, and Relu5_ _alu__1 layers are used in
56 3 to learn the human-awan style information from the style dataset S. GAN is popular generative model consisting of two networks (i.e., generator G and a discriminator D) that compete against each other. Specifically, we input the stylized images produced by the generator and the artworks sampled from to the discriminator as fake data and real data, respectively. In the training process, the generator will try to fool the discriminator by generating a realistic artistic image, while the discriminator will try to distinguish generated fake artworks from real ones. Joint training of these networks leads generator that is able to produce remarkable realistic fake images with the learned human-awan style information. The adversarial training process can be formulated as (note that our generator G contains an encoder E, transformation module T, and a decoder D, as shown in Figure (a)):
External style learning. Here, employ GAN || 39
Lodv
E [log( 1.25"
I,C llo D(D(T(ELL)
Content structure preservation. To preserve the content structure of Ie in the stylized image Ise. we adopt the widely-used perceptual loss:
Le @conu4_2 2 (Tsc) - Pconu4 2aieua222200 (Tc) ||2
(3)
we utilize the identity loss to encourage the generator 9 to be approximate identity mapping when the content image and style image are the same. In this manner, more content structures and style characteristics can be preserved the stylization result. The identity loss is depicted in Figure 2 (b) and defined as:
Identity loss, Similar to [3
32
59
Lidentity =
Aidentity (Il Ien - Ie 112 + Isn |12)+
(4)
Nidentity? Σ (i - (i(Te) ll2 + (b (ba (12)
where Loe is the output image generated when both the content image and style image are Le- Lse is analogous. Aidentityl and Aidentity2 are the weights associated with different loss terms. For di, we choose Relul 1, Relu2_ 1, Relu3 Relu4_1, and Relu5 layers in experiments.
3.2 Contrastive Learning
Intuitively, stylized images rendered with the same style image should have closer relations in style than those rendered with different style images. Similarly, stylized images based on the same content image should have closer relations in content than those based on different content images. We refer such relations as syylization-tosttyiiaaatt to-stylization relations. Generally, existing style transfer methods only consider the content-to- soeenn-tt-styyizttiie stylization and style-t--ttyiizatin style loss (like L. and L. introduced above), while neglect the stylization-to -trtttiiontttttllizaain relations. To tackle this problem, we for the first time introduce contrastive learning to style transfer. The core idea of contrastive learning to associate data points with their "positive" examples while disassociate them from the other points that are regarded "negatives".
relations by applying the content loss and
5

---

Specifically, we propose two contrastive losses: style contrastive loss and a content contrastive loss relations. Note that for clearer expression, hereafter, we use
learn the eyllization-to-ttliiaaiie to represent the feh. style image, Gi to represent the deh content image, and 8;0; to represent the stylized image generated with 8; and G. To perform contrastive learning every training batch, we arrange batch of style and content images in the following manner:
Assume the batch size 6, which is even number. Then we get batch of style images {s), 52, 86/2, 31, $2..... S1/2-1, 1, and a batch of content images {cI, C2. 0b/2. C2 C3 ... 06/2. C1 J. Hence, the corresponding stylized images are {8101, 82C2, ... 6b/2Cb/2- S1C2, 8203, 16/2- 8b/2C1]. In this way, we ensure that for every stylized image s,Cj, we can find a stylized image s,C₂ (x # j) that shares the same style with it, and a stylized image SyC; (y * i) that shares the same content with it in the same batch. Figure 2 (c) depicts this process by taking = as an example.
Style contrastive loss. To associate stylized images that share the same style, for stylized image S,C we select S,C, (x o j) as its positive example (s,C, shares the same style with s,C,), and smCn (m j i and n + j) as its negative examples. Notice that 8m Cn. represents series of stylized images, not just one image. Then we can formulate our style contrastive loss as follows:
L La-contra- == -log( exp(l + "e
(5)
where 1, h (arrrelus_1)) (-)), in which h2 is style projection network. L, is used obtain the style embeddings from stylized images. J is a temperature
to control push and pull force.
Content contrastive loss. Similar to the style contrastive loss, to associate stylized images that share the same content, for a stylized image 8tGj, we select Sy (y A i) as its positive example (8y00 shares the same content with s,Cy), and 5m (m & and n * j) as its negative examples. We express the content contrastive loss as:
Lo-contra == -log( exp(ll(sse))))(((c))))) + (eppl((sscc)(l(ssmmmn))
(6)
where ho(@retu4 (o)ea__(())))))))) (-)), in which he is content projection network. lo used to obtain the content embeddings from stylized images.
3.3 Final Objective
We summarize all aforementioned losses and obtain the final objective of our model,
fiinal == A2 + X2Cade + >ge. + da Lidentiiy + >>gLs-contre + As AG where Åo X2, A3, Ao, As. and As are hyper-parameters for striking proper balance among losses.
(7)
4 Experimental Results
In this section, we first introduce the experimental settings. Then we present qualitative and quantita- tive comparisons between the proposed method and several baseline models. Finally, we discuss the effect of each component in our model by conducting ablation studies.
4.1 Experimental Settings
Implementation details. We build on the recent SANet (36] backbone and extend it with our proposed changes to further push the boundaries in automatic artwork generation. We refer the original paper [36] for the detailed network architecture of the encoder E, transformation module T, and decoder D. As for the discriminator D, we employ the multi-scale discriminator proposed by Wang et al. [50]. The style projection network h2 is a two-layer MLP (Multilayer Perceptron) with 256 units at the first layer and 128 units the second layer. Similarly, the content projection network h, is a two-layer MLP with 1 128 units at each layer. The hyper-parameter in Equation (5) and (6) is set to 0.2. The loss weights in Equation (4) and (7) are set Aidentityl = 50, Aidentity2 = 1, À
1, Ag 5, λ3 1, X4 1, As 0.3, and Ag 0.3. We train network using the Adam optimizer

---

imputs
Ours
Gatys al.
AdaiN
WCT
Avatar Avatar-Net
1ST
SANett
Figure 3: Qualitative comparisons on image style transfer. The first row shows the content and style images. The rest of the rows show the stylization results generated with different style transfer methods.
with learning rate of 0.0001 and batch size of 16 for 160000 iterations. Our code is available at: httss//ghhb..coommlalllCCHooono
Datasets. Like 115 58 36 19,, we take MS-COCO 33 and WikiArt 22 the content dataset and
style dataset, respectively, During the training stage, we first resize the smallest dimension training images to 512 while preserving the aspect ratio, and then randomly crop 256 256 patches from these images as input. Note that the reference stage, our method applicable for content images and style images with any size.
Baselines. We choose several sfte-of-the-art -he--art style transfer methods as our baselines, including Gatys et al. [10], AdaIN (15), WCT [30], Avatar-Net [4]] LST (28), and SANet [36]. All these methods are conducted by using the public codes and default configurations.
4.2 Qualitative Comparisons
In Figure 3 we show the qualitative comparisons between our method and six baselines introduced above. We observe that Gatys et al. [TO] prone fall in bad local minimum (e.g., 1st 2nd and 3rd columns). AdaIN [15] sometimes produces messy stylized images with unseen colors and unwanted halation around the edges (e.g., 1st 3rd and 6th columns). WCT (30] often introduces distorted patterns, yielding less-structured and blunt stylized images (e.g., 2nd 4th and 5th columns). Avatar-Net is hard to produce sharp details and fine brushstrokes (e.g., 1 4th and 5th columns). LST (8 usually produces less stylized images with very limited texture patterns (e.g., 2nd 4th and 6th columns). SANet [36] tends to apply similar repeated texture patterns among different styles (e.g., 1st 3rd and 6th columns). Despite the recent progress, the gap between synthesized artistic
7

---

Olrrs
Getys
AdsiN
WCT
Asstarffet
AST
SANve
Figure 4: Qualitative comparisons on video style transfer. The first row shows several video frames and the style image. The rest of the rows show the stylization results generated with different style transfer methods. The last column shows the heat maps of differences between different frames.
Table 1:
The user study scores for different methods.
The higher the better.
WikiArt
Gatys et al.
AdaIN
WCT
Avatar-Net
LST
SANet
Ours
Preference Score
0.143
0.118
0.099
0.087
0.125
0.161
0.267
Deception Score
0.875
0.438
0.363
0.375
0.275
0.381
0.394
0.624
images and real artworks is still very large. To further narrow this gap, we introduce internal-external learning and contrastive learning to artistic style transfer, leading to visually more harmonious and plausible artistic images, shown in the 2nd row of Figure
We also compare our method with baselines video style transfer, which is conducted between a content video and style image in a frame-wise manner. The stylization results are shown in Figure 4 To visualize the stability and consistency of synthesized video clip, we also show the heat maps of differences between different frames in the last column of Figure 4 As can see, our approach outperforms existing style transfer methods in terms of stability and consistency by significant margin. This can be attributed to two points: (i) external learning smooths the stylization results by eliminating those distorted texture patterns; (ii) the proposed contrastive losses take the stylization-to- tiriiion-ttttttzttiiii relations into consideration, pulling adjacent stylized frames closer to each other since they share the same style and similar content.
4.3 Quantitative Comparisons
As the qualitative assessment presented above could be subjective, in this section, we resort to several evaluation metries to better assess the performance of the proposed method quantitative manner.
User study [54 36 24 23 48 is the most widely adopted evaluation metric in style transfer, which investigates user preference
different stylization results for
objective comparison.
Preference score. We use content images and 15 style images synthesize stylized images for each method. Then 20 content-style pairs are randomly selected for each participant and show

---

Table 2: The average LPIPS distances for different methods, The lower the better.
Inputs
Gatys al.
AdaIN
WCT
Avatar-Net
LST
SANet
Ours
LPIPS Distance
0.231
0.488
0.369
0.460
0.341
0.326
0.372
0.317
Oan denaoo
2.21
Figure 5: Ablation studies of external learning (abbr. EL) and contrastive learning (abbr. CL) on (a)
image style transfer and (b) video style transfer. Please zoom in for better view and details.
them the stylized images generated by our and competing methods side-by-side in a random order. Next, we ask each participant choose his/her favorite stylization result for each content-style pair. We finally collect 1000 votes from 50 participants and present the percentage of votes for each method in the second row of Table [ The results indicate that the stylized images generated by method are more preferred by human participants compared to those generated by the competing methods.
Deception score. To measure the gap between Al-created artistic images and human-created artworks, we conduct another user study: for each participant, we show them 80 artistic images which consist of 10 human-created artworks collected from WikiArt (2 and 70 stylized images generated by our and baseline methods (note that each method provides 10 stylized images). Then for every image, ask these participants to guess if it is a real artwork not. The deception score is calculated as the fraction times that the stylized images generated by this method are identified as "real". For comparison, also report the fraction of times that the numan-created artworks "real". The results are shown in the third row of Table T where we can see that the deception rate of artworks, further demonstrating the effectiveness of
identified as
method is closest to that of our method.
To quantitatively evaluate the stability and consistency of the proposed method on video style transfer, we adopt L.IPS (Learned Perceptual Image Patch Similarity) [57] as the evaluation metric.
LPIPS. LPIPS is a widely used metric in the field of multimodal image-to-image translation (MI2I) measure diversity. In this paper, we employ LPIPS to measure the stability and consistency of rendered clips by computing the average perceptual distances between adjacent frames. Note that contrary to MI2I methods that expect higher LPIPS value achieve better diversity, we expect lower LPIPS value to achieve better stability and consistency. We synthesize 18 stylized video clips for each method and report the average LPIPS distances Table where we observe that our approach obtains the best score among all methods, consistent with the qualitative comparisons Figure
61 [6 25 g
4.4
Ablation Studies
In this section, we conduct several ablation studies to highlight the effect of different components in our model.
We first explore the effect of external learning (abbr. EL) and contrastive learning (abbr: CL) on image style transfer. As for internal learning, since its effect has been fully validated existing style transfer methods, we do not ablate it in this experiment. Figure (a) shows the image stylization results of our method with and without EL/CL. can be observed that, without EL., the stylized images become messier with abrupt colors and obvious distortions. The reason could be that the

---

model without EL only focuses on increasing the style similarity between the stylized image and the style image, without considering whether the color distributions and texture patterns in the stylized image are natural and harmonious. In comparison, the model with EL. can learn the human-aware style information from the large-scale style dataset, leading to more realistic and harmonious stylized images that cannot be distinguished from real artworks by the discriminator. In addition, we also find that our method can better match the target style to the content image with the proposed contrastive losses. This is because our contrastive losses can help the network to learn better style and content representations by taking the stylization-to- toann-ttttttttttiiiiooo relations into consideration, further refining the stylization results. The user preference results reported in the last column of Figure 5 (a) also demonstrate that our full model has the best performance.
Similar ablation studies are also conducted on video style transfer. As shown in Figure 5 (b), the stability degradation can be observed after we remove external learning or contrastive learning from our method (notice the color of hair and skin), which is in line with the reported LPIPS distance. The results indicate that both external learning and contrastive learning can improve the stability of video style transfer. As we analyzed in Section 4.2 external learning obtains stability gains by eliminating distorted texture patterns, and contrastive learning obtains stability gains by pulling adjacent stylized frames closer to each other.
5
Limitations
One limitation of this work is that the proposed internal-external learning scheme and two contrastive losses cannot be applied learning-free style transfer methods, such as WCT [30] Avatar-Net [1), LST (28), etc. This is because the training process is necessary for our method. Therefore, our method can only be incorporated into learning-basch methods, such as Johnson et al. [20), AdaIN [15]. SANet [36] (in this work, we mainly take SANet as our backbone to show the effectiveness and superiority of our method), etc. Another limitation that in the inference stage, the style images that are too different from the training styles may not benefit from the external learning scheme, since they are out of the learned style distributions.
Conclusion
In this paper, we propose internal-external style transfer method with two novel contrastive losses. The internal-externa learning scheme learns simultaneously both the internal statistics from a single artistic image and the human-aware style information from the large-scale style dataset. As for the contrastive losses, they are dedicated to learning the stylization-to-styliiaattt the multiple stylization embeddings closer each other when they share the same content style, but pushing far away otherwise. Extensive experiments show that our method can not only produce visually more harmonious and satisfying artistic images. but also significantly promote the stability and consistency of rendered video clips. The proposed method is simple and effective, and may shed light on more future understandings of artistic style transfer from a new perspective. In the future, we would like to extend our method to other vision tasks, for example, texture synthesis.
relations by pulling
Acknowledgments
This work was supported in part by the projects No. 2020YFC1523202 19ZDA197, LY21F020005 2021009, 2019C03137, MOE Frontier Science Center for Brain Science & Brain-Machine Integration (Zhejiang University). National Natural Science Foundation of China (Research on Key Technologies of art image restoration based decoupling learning), and Key Scientific Research Base for Digital Conservation of Cave Temples (Zhejiang University). State Administration for Cultural Heritage. We would also like to thank the reviewers and AC for their constructive and insightful comments on the early submission.
Funding Transparency Statement
The projects mentioned in our Acknowledgments provided funding and support to this work. There no additional revenues related to this work.
10

---

References
[1] Jie An, Siyu Huang, Yibing Song, Dejing Dou, Wei Liu, and Jicbo Luo. Artflow: Unbiased image style transfer via reversible neural flows. arXiv preprint arXiv:2103. 168727003.16877, 2021.
[2] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. arXiv preprint
2017.
[3] Andrew Brock, Jeff Donahuc, and Karen Simonyan. Large scale gan training for high fidelity natural
image synthesis, arXiv preprint arXiv:1809 /1096, 2018.
[4] Haiho Chen, Lei Zhao, Zhizhong Wang, Huiming Zhang, Zhiwen Zuo, Ailin Li, Wei Xing, and Dongming
Lu. Dualast: Dual style-learning networks for artistic style transfer. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition, pages 872-881, 1, 2021.
[5] Haibo Chen, Lei Zhao, Huiming Zhang, Zhizhong Wang. Zhiwen Zuo, Ailin Li. Wei Xing, and Dongming
Lu. Diverse image style transfer via invertible cross-space mapping. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 14880-14889, October 2021.
[6] Ting Chen, Simon Korblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages
1597-1607. PMLR. 2020.
[7] Tian Qi Chen and Mark Schmidt. Fast patch-based style transfer of arbitrary style. arXiv preprint
1xXiv:161..04337, 2016.
[8] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and lung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages (188-8197, 2020.
[9] Oriel Frigo, Neus Sabater, Julie Delon, and Pierre Hellier. Split and match: Example-based adaptive patch sampling for unsupervised style transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 553-561, 2016.
[10] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2414-2423 2016.
[11] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Farde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.
[12] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-97328 9738, 2020.
[13] Aaron Hertzmann. Painterly rendering with curved brush strokes of multiple sizes. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques, pages 453-460, 1998.
[14] Aaron Hertzmann, Charles Jacobs, Nuria Oliver, Brian Curless, and David Salesin. Image analogies.
In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 327 377-300.2 2001.
[15] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision, pages 1510-1510, 2017. [16] Xun Huang, Ming-Vi Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised maage-to-image translation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 172-189, 2018.
[17] Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexci A Efros. Image-to- taggotto-mmage translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125-1134, 2017.
[18] Somi Jeong, Youngjung Kim, Eungbean Lee, and Kwanghoon Sohn. Memory-guided unsupervised image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putiern Recognition, pages 6558-6567, 2021.
[19] Yongcheng Jing, Xiao Liu, Yukang Ding, Xinchao Wang, Errui Ding, Mingli Song, and Shilci Wen. Dynamic instance normalization for arbitrary style transfer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4369-43767 2020.
[20] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pages 694-711. Springer, 2016.
[21] Minguk Kang and Jaesik Park. ContraGAN: Contrastive Learning for Conditional Image Generation. 2020.
[22] Sergey Karayev, Matthew Trentacoste, Helen Han, Aseem Agarwala, Trevor Darrell, Aaron Hertzmann, and Holger Winnemoeller. Recognizing image style. arXiv preprint arXiv: 131 1.3715, 2013.
[23] Dmytro Kotovenko, Artsiom Sanakoyeu. Sabine Lang, and Bjorn Ommer. Content and style disentangle- ment for artistic style transfer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4422 4431, 2019.
11

---

[24]
Dmytro Kotovenko, Artsiom Sanakoyeu, Pingchuan Ma, Sabine Lang, and Bjom Ommer. A content transformation block for image style transfer. In Proceedings of the IELE Conference on Computer Vision and Pattern Recognition. pages 10032-10041, 2019.
[25] Hsin-Ying Lee, Hung-Y Yu Tseng, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. Diverse image- to-image translation via disentangled representations. In Proceedings of the European conference on computer vision (ECCV). pages 35-51, 2018.
[26] Chuan Li and Michael Wand. Combining markov random fields and convolutional neural networks for image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
pages 2479-2488 2016.
[27] Chuan Li and Michael Wand. Precomputed real-time texture synthesis with markovian generative adver- sarial networks. European Conference on Computer Vision, pages 702-716. Springer, 2016.
[28] Xueting Li, Sifei Liu, Jan Kautz, and Ming-Hsuan Yang. Learning linear transformations for fast image and vidco style transfer. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3809-3817, 2019.
[29] Yijun Li, Chen Fang, Jimei Yang. Zhaowen Wang, Xin Lu, and Ming Hsuan Yang. Diversified texture synthesis with feed-forward networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3920-3928, 2017.
[30] Yijun Li, Chen Fang. Jimei Yang. Zhaowen Wang, Xin La,, and King-Hsuan Yang. Universal style transfer via feature transforms. In Advances in neural information processing systems, pages 386-396, 2017.
[31] Jing Liao, Yuan Yao, Lu Yuan, Gang Hua, and Sing Bing Kang. Visual attribute transfer through deep image analogy. arXiv preprint arXiv: 7000000888, 2017.
[32] Jianxin Lin, Yingxue Pang. Yingee Xia, Zhibo Chen, and Jieho Luo. Tuigan: Learning versatile image-to- image translation with two unpaired images. In European Conference Computer Vision, pages 18-35. Springer, 2020.
[33] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[34] Rui Liu, Yixiao Ge, Ching Lam Choi, Xiaogang Wang, and Hongsheng Li. Diveo:
Diverse conditional image synthesis via contrastive generative adversarial network. arXiv preprint arXiv:2103. 07893, 2021.
[35] Ming Lu, Hao Zhao, Anbang Yao, Yurong Chen, Feng Xu, and Li Zhang. A closed-form solution to universal style transfer. In Proceedings of the IEEE/CVF International Conference on Computer Vision,
pages 5952 122159912222 2019.
[36] Dac Young Park and Kwang Hec Lec. Arbitrary style transfer with style-attentional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5880-5888, 2019.
[37] Scobin Park, Jinsu Yoo, Donghyeon Cho, Jiwon Kim, and Tac Hyun Kim. Fast adaptation super- resolution networks via meta-learning, arXiv preprint arXiv:2001.023 20.0000000000....... 2020.
[38] Taesung Park, Alexei A Efros, Richard Zhang, and Jun-Yan Zhu. Contrastive learning for unpaired image-to-image translation. In European Conference on Computer Vision, pages 319-345. Springer, 2020.
[39] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: orXiv:15/1.06444, 1.06434, 2015.
[40] Artsiom Sanakoyeu, Dmytro Kotovenko, Sabine Lang, and Bjorn Ommer. A style-aware content loss for real-time hd style transfer. In Proceedingy of the European Conference Computer Vision (ECCV), pages 698-714, 2018
[41] Lu Sheng, Ziyi Lin, Jing Shao, and Xiaogang Wang. Avatar-net: Multi-scale zero-shot style transfer by feature decoration. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 8242-8250, 2018
[42] YiChang Shih, Sylvain Paris, Connelly Barnes, William T Freeman, and Frédo Durand. Style transfer for headshot portraits. 2014.
[43] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv:1409.1 1556, 2014.
[44] Jae Woong Soh, Sunwoo Cho, and Nam Ik Cho. Meta-transferr learning for zero-shot super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3516-3525, 2020.
[45] Jan Svoboda, Asha Anoosheh, Christian Osendorfer, and Jonathan Masci. Two-stage peer-regularized feature recombination for arbitrary image style transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13816-13825, 2020.
[46] Dmiiryy Ulyanov, Vadim I. Lehedev, Andrea Vedaldi, and Victor S Lempitsky. Texture networks: Feed- forward synthesis of textures and stylized images. In ICMI., volume 1, page 4, 2016.
[47] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Improved texture networks: Maximizing quality and diversity in Computer Vision and Pattern Recognition, pages 6924 0244-692,220 2017.
stylization and texture synthesis. In Proceedings of the IEEE Conference on
12

---

[48] Huan Wang. Yijun Li, Yuehai Wang. Haoji Hu, and
Yang. Collaborative distillation for ultra-resolution universal style transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1860-1869, 2020.
[49] Tengfei Wang, Hao Ouyang, and Qifeng Chen. Image inpainting with exteraa-intteral learning and monochromic bottleneck. arXiv preprint arXiv:2104.09063 2021.
[50] Ting-Chun Wang, Ming-Yo Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 8798 8807, 2018.
[51] Zhizhong Wang, Lei Zhao, Haibo Chen, Lihong Qiu, Qihang Mo, Sihuan Lin, Wei Xing, and Dongming
Lu. Diversified arbitrary style transfer via deep feature perturbation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7789-7799 2020.
[52] Holger Winnemöller, Sven C Olsen, and Bruce Gooch. Real-time video abstraction. ACM Transactions On Graphics (TOG), 25(3):1221-1222) 2006.
[53] Haiyan Wu, Yanyun Qu, Shaohui Lin. Jian Zhou, Ruizhi Qiao, Zhizhong Zhang, Yuan Xie, and Lizhuang Ma. Contrastive learning for compact single image dehazing. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10551-10560, 2021.
[54] Yuan Yao, Jianqiang Ren, Xuansong Xie, Weidong Liu, Yong-Jin Liu, and Jun Wang. Attention-aware multi-stroke style transfer. In Proceedings of the IEEE Conference on Computer Vision and Patten Recognition, pages 1467-1475, 2019.
[55] Ning Yu, Guilin Liu, Aysegul Dundar, Andrew Tao, Bryan Catanzaro, Larry Davis, and Mario Fritz. Dual contrastive loss and attention for gans. arXiv preprint arXiiv2213,, /6748, 2021.
[56] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. International conference on machine learning, pages 7354-7363, PMLR, 2019.
[57] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition, pages 386-595, 2018.
[58] Yulun Zhang, Chen Fang, Yilin Wang, Zhaowen Wang, Zhe Lin, Yun Fu, and Jimei Yang. Multimodal style transfer via graph cuts. In Proceedings of the IEEE International Conference Computer Vision, pages 5943-599 1943-5951, 2019.
[59] Yihao Zhao, Ruihai Wu, and Hao Dong. Unpaired image-to-imas -iagge translation using adversarial consistency loss. In European Conference Computer Vision, pages 800-81 815, Springer, 2020.
[60] Jun Jun Zhu, Taesung Park, Phillip Isola, and Alexci Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223-2232, 2017.
[61] Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexci Efros, Oliver Wang. and Eli Shechtman. Toward multimodal image-to-image translation. In Advances in neural information processing systems, pages 465-476, 2017.
13


Reviews:
[{"review_id": "uchrUQWRG1m", "reviewer": "Reviewer_vsGU", "summary": "The paper describes a method for feed-forward image stylization that combines several losses. The novel losses are a contrastive loss that optimizes the style embedding of the training image to match the embedding of a target style and content and to be distinct from other style and contents.  Other losses used during training include feature mean/variances and a GAN loss.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the visual quality of the results (Figure 3) looks substantially better than the baselines. The detailed structure and scene contours are preserved better than the baselines, and the regions are filled with sensible textures from the exemplars. I haven't attempted to survey the literature to judge if these are fair representations of the baselines\n\nIt's unclear to me how these improvements results from the contrastive losses. Better representation of image structure doesn't seem to encoded in the contrastive losses.\n\nThe writing of the paper is very difficult to follow and I'm not sure that it's reproducible. Section 3 never lists the inputs or outputs for training. It took a long time to figure out that this section describes training and not test time. How are the parameters of the h_s and h_c networks determined?  What feature space are the first two terms in Eq 4; in pixel space?  Eqs 3 and 4 use ||_||_2 notation; does this mean there are square-roots in these terms?\n\nThe motivation and explanation of the algorithm coudl also use a lot of improvement.  The Englihs is not very good. \n* L26: \"existing style transfer methods often confine themselves to the internal style statistics of a single image\". This is obviously false, e.g., see pix2pix, CycleGAN, CUTG. \n* L31: \"... more consistent with human perception\" This is a bold claim, either remove it provide citations to justify whatever assertion is being made here.  \n* L50: \"... this is the first work successfully leveraging the power of contrastive learning in style transfer\": This is false; see CUT [34].\n* L109: \"Existing style transfer methods usually produce unsatisfying stylization results with disharmonious colors...\"  So the claim here is that all previous methods are terrible?\n\n\nThe video style transfer results are so weak that I think they should be removed if the paper is accepted.  The frame rate of the results is so low that one cannot judge the temporal coherence of the provided videos. These are more like sequences of independent still images.  \n\nThe paper fails to compare to the state-of-the-art in video style transfer, or to cite it:\n\nOndřej Jamriška, Šárka Sochorová, Ondřej Texler, Michal Lukáč, Jakub Fišer, Jingwan Lu, Eli Shechtman, and Daniel Sýkora\nStylizing Video by Example    \nIn ACM Transactions on Graphics 38(4):107, 2019 (SIGGRAPH 2019, Los Angeles, USA, July 2019), ACM.\n\nOndřej Texler, David Futschik, Michal Kučera, Ondřej Jamriška, Šárka Sochorová, Menglei Chai, Sergey Tulyakov, and Daniel Sýkora\nInteractive Video Stylization Using Few-Shot Patch-Based Training    \nIn ACM Transactions on Graphics 39(4):73, 2020 (SIGGRAPH, August 2020)\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper doesn't discuss limitations.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "uVtrykZ1gCA", "reviewer": "Reviewer_bZMr", "summary": "The paper proposes two new additional losses for style transfer:\n1. A contrastive loss to incorporate to bring the style of the stylized images with the same style image closer to each other, while pushing the stylized images from different style images further from each other. The same for content.\n2. An adversarial loss to learn the general aesthetics of stylized images over a larger dataset.\nThe authors demonstrate that a combination of these losses plus the regular style-content loss can generate high quality, and surprisingly consistent stylized images, both qualitative and quantitatively as well as user studies.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "===== Writing\n\nThe paper is very well written but still can be improved.\n- The motivation behind the new additions are clearly explained, although it can use more visual examples to demonstrate the shortcomings of the current approaches. Figure 1 can do this with a longer description as why the previous methods fail and how the proposed method shines. In fact, given the subjective nature of style transfer such descriptions are quite important. \n- The method is clearly described and is visualized in Figure 2 (texts can be slightly bigger for better readability in print). The reasoning behind the design decisions are clearly described and justified.\n- The details of the experiments are noted, however, the training details can be expanded (in supplementary materials). The code is not included and including such details is crucial to attain reproducibility. \n\n\n===== Methodology\n\nThe proposed method is novel, clever and impactful. To the best of my knowledge, and as claimed by the authors, this is the first usage contrastive loss in style transfer and it is applied in a way which is intuitive and makes sense.  The paper tested their proposed method against multiple datasets and the majority of the SOTA models. For qualitative comparison, the authors conducted multiple user studies to capture the deception and preferences score which is pivotal for style transfer papers which lack good reliable metrics. \n\nHowever, I strongly suggest the authors to include a \"non-cherry picked\" section of their results to demonstrate the generalization capability of their model. The current set of results, look impressive but cherry-picked. This can be done as part of the supplementary materials or even better a website :) \n\nAnother  missing section from the paper, is the training and inference complexity. I suspect by adding the proposed losses the training time substantially increases while the generation runtime stays the same. Training time is important mainly because the proposed model cannot be applied to learning-free style transfer methods.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The broader impact and limitation is missing from the paper and instead is in the supplementary materials. The authors enumerated the positive and negative impacts of their work however it is quite minimal and can be expanded.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "ogWl-qNNETi", "reviewer": "Reviewer_JMx5", "summary": "In this paper, the authors introduce an internal-external learning framework for style transfer. In the proposed method, the classical VGG-based style loss, which encourages the stylized result to capture the low-level statistics of the style image, is combined with a GAN loss designed to capture the style priors from the style database. In addition, contrastive losses are designed to encourage the results to better capture the stylization-to-stylization relation. Experiments in the paper demonstrate the proposed method can generate visually more appealing results compared to previous style transfer methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrength:\n + The paper is well written and easy to follow.\n + The idea of incorporating GAN loss to capture the style priors from a style database is interesting. The provided results seem to indicate that such external loss can help produce results that look more natural compared to previous works.\n + The evaluation processes for evaluating the stylization results and comparison with existing works are solid, with both visual results on a range of different content and a thorough user study.\n\nWeaknesses:\n + I found the use of the term \"internal learning\" in this context a bit misleading. Internal learning has been used in the literature to refer to the process of optimizing the model parameters on a single image at **test time**. In this paper, the internal learning part seems to refer to the conventional style-transfer loss (which consists of VGG-based losses for style and content loss), which is used at training time and applied when trained on the whole dataset rather than for test-time optimization. I think it is fine to overload the term if it serves the purpose but clarification needs to be provided to avoid confusion. \n + While I find it interesting to explore applying GAN loss to style transfer problem and I appreciate the improvement it brings to the results, I feel that more discussions and examples should be provided to analyze the effects of it. In particular, one challenge in incorporating GAN loss into this setting could happen when the provided style image has a unique style that does not appear in the style database. How would (and how should) the model behave in that case? I would appreciate more discussions on that regard along with more examples to analyze the benefit of incorporating GAN loss.\n + It's also not clear how much improvement was gained by the incorporation of the contrastive loss. Only three examples were provided (in Fig. 5) and the improvements from using contrastive loss in those examples do not seem obvious to me. More explanations on this would make the contributions of these additional losses better justified.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\nThe limitation and societal impacts have been discussed in the paper (incorporated in the supplementary materials).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "WCoS7Jra2Vs", "reviewer": "Reviewer_zxcr", "summary": "The paper proposes a neural style transfer algorithm, for transferring the artistic style of one image to another.   It is basically SANet [32] with some incremental novelties.\n\nSANet is an encoder-decoder transformer architecture that accepts two images (content/desired style) and outputs a stylized version of the content by optimizing a Gatys-like loss function.\n\nTwo novelties are claimed over SANet.  \n\n1) the use of ‘human external information’ by which the authors mean a corpus of real artworks in the desired style, to train a GAN discriminator added to the SANet.  It is argued that adding knowledge of multiple real artworks of the desired style in this way helps produce more plausible stylizations.\n\n2) the use of contrastive learning to train their architecture.  The idea here is to ensure pair-wise consistency between style descriptors of generated artwork and the aforementioned corpus of artwork sharing that desired style.\n", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contribution of this paper is extending SANet to draw upon external knowledge – i.e. in addition to taking in a single content/style image pair, a supporting corpus of images with similar content and similar style are provided.\nThe extensions are fairly straightforward..  the same exact SANet losses are used (ensuring Gatys-like content and style consistency using pre-trained VGG) – here referred to as ‘internal content/style’ losses – and an identity loss agains, as defined, in SANet.\n\nThe first technical tweak is adding a loss for the discriminator, which conducts the usual real/fake check on the output of the stylization vs. unpaired set of style images.  This is shown via a ‘deception score’ in Sec.4 - a kind of artistic Turing test user study  -  that the output artwork is considered more realistic, and qualitatively this is shown in the figures of the paper.  The idea is quite simple but seems to improve the output aesthetics.  The practical downside is the need to have a large corpus of images in the desired output style.\n\nThe second tweak is adding a contrastive loss and contrastive training methodology to the network – the formation of the batches is explained, but it is not explained how the training curriculum works in tandem with the GAN discriminator.  Is there some alternating training pattern similar to DCGAN?  How is this contrastive training integrated?   A key technical detail omitted from the implementation description is the batch size – it is mentioned that Figure 2 diagram 'uses a batch size b=8 for illustration' but what was used for the experiments reported? Presumably for contrastive learning to make any difference you need batch size in the hundreds or even thousands?  Again, is that practical given the number of style images required – and how would all of this fit into typical GPU VRAM it seems with this architecture it would be challenging.  The actual contrastive loss introduced has limited novelty it is just SimCLR temperature softmax approach with a header network, exactly as in the original SimCLR paper.\n\nOverall I am not convinced by this paper.  The novelty is not significant, and the benefit of the external corpus of style images is not well demonstrated versus the practical disadvantage of requiring it.  The time cost of training such a network to perform a stylization, as well as the undescribed practical issues around large batch size, make me doubt this is a valuable approach.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Societal impact not discussed", "review_text": "", "extra_scores": {"needs_ethics_review": "Yes", "ethics_review_area": ["Research Integrity Issues (e.g., plagiarism)"], "time_spent_reviewing": "1.5"}, "extra_text": {"ethical_concerns": "I am not sure this is an ethics consideration but I notice there are non-anonymized funder acknowledgements to some Korean agencies at the end of the paper, isn’t this a violation of NeurIPS submission policy?  I am unsure so flag it here - it did not have any bearing on my review rating."}}]

Author rebuttal:

Meta-review: The submission proposes an extension of SANet's artistic style transfer loss formulation which adds adversarial and contrastive terms to the loss. The adversarial loss term is computed using a discriminator trained to classify images as either belonging to the style dataset or to the distribution of stylized content images. The contrastive loss terms encourage the embeddings of stylized images which either share the same content or style image to be closer and vice versa.

The proposed approach is compared against SANet and other baselines through qualitative (showing stylizations) and quantitative (user study, LPIPS) means. Stylizations are compared for a few content/style image pairs to qualitatively assess the effect of the adversarial and contrastive loss terms.

Reviewers appreciate the writing quality and agree on the fact that the results presented in the submission represent an improvement over previous approaches, but there is disagreement on whether the analysis of the effect of individual loss components is sufficient, and on whether the limitations of the GAN loss are sufficiently discussed.

On the limitations of the GAN loss, reviewer JMx5 objects that its benefit may not be realized for style images that are very different from the training styles. The authors respond that the style images they evaluate on have not been seen during training. Reviewer zxcr points out in the reviewer discussion that the extent to which the effect of the GAN loss generalizes to held-out style images likely depends on whether they are in-distribution or out-of-distribution (the "distribution" here being the distribution of style images seen during training). This is a good point, and the authors' response here isn't entirely satisfactory: even if the model generalizes to in-distribution held-out style images, there's no guarantee that it would generalize to out-of-distribution style images, and that limitation should be discussed. In the discussion, Reviewer vsGU mentions that discussing the limitations of the proposed approach belongs to the main paper and not the Appendix. I agree with that.

There was also a debate on whether the distinction between style transfer and domain translation is artificial, which I think requires some nuance. Since domain translation approaches work by imitating the statistics of a population of domain-specific images, it does feel weird that the submission dismisses style transfer approaches as "neglecting the external style information reserved in the large-scale style dataset" without acknowledging domain translation approaches. There is a distinction between the two (the former targets a specific style image while the latter targets the common "look and feel" of a collection of style images), but I think the submission would benefit from being more generous in drawing connections between various families of approaches.

On the influence of the added loss terms, I would break the issue down into two questions:
- Are the current ablations convincing enough?
- If they are not, how much does that weaken the submission?

I agree with Reviewers JMx5 and zxcr that a qualitative assessment on three content/style image pairs is a very small number of data points to draw conclusions from. This issue could have been fixed by adding ablated models among the approaches evaluated in Section 4.3's user study.

On whether this is a serious issue, the risk here is that the effect of one of the two loss terms could be marginal, which would nullify one of the submission's two major contributions. The principle of incorporating global information from a corpus of styles into the training signal remains an interesting contribution nevertheless, but obviously the paper's impact would be more substantial if it better characterized the contributions of the two loss terms.

This submission is very much a borderline case. I read the paper attentively, and I agree that the idea of incorporating information from a corpus of styles rather than treating each style as separate is a new and interesting contribution which incorporates insights from domain translation. Even though the submission has its flaws, I recommend accepting it provided the following is addressed in the final manuscript:
- A discussion on the approach's limitations, and in particular those of its GAN loss, is incorporated into the main text.
- The relationship between the proposed approach and domain translation is better highlighted.
- Additional examples are shown in the Appendix to accompany Figure 5.
Decision: Accept (Poster)