Artistic Style Transfer with Internal-external Learning and Contrastive Learning

1 Introduction . Artistic style transfer is a long-standing research topic that seeks to render a photograph with a given artwork style . Ever since Gatys et al . [ 10 ] for the first time proposed a neural method , which leverages a pre-trained Deep Convolutional Neural Network ( DCNN ) to separate and recombine contents and styles of arbitrary images , an unprecedented booming [ 20 , 26 , 15 , 30 , 36 , 51 , 48 ] in style transfer has been witnessed . Despite the recent progress , there still exists a large gap between real artworks and synthesized stylizations . As shown in Figure 1 , the stylized images usually contain some disharmonious colors and repetitive patterns , which makes them easily distinguishable from real artworks . We argue that this is because existing style transfer methods often confine themselves to the internal style statistics of a single artistic image . In some other tasks ( for example , image-to-image translation [ 17 , 60 , 16 , 25 , 8 , 18 ] ) , the style is usually learned from a collection of images , which inspires us to leverage the external information reserved in the large-scale style dataset to improve the stylization results in style transfer . Why is the external information so important for style transfer ? Our analyses are as follows : Although different images in the style dataset vary greatly in fine details , they share a key commonality : they are all human-created artworks , whose brushstrokes , color distributions , texture patterns , tones , ∗Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . AdaIN SANetAvatar-Net LSTGatys et al . WCTOursStyle Content Figure 1 : Stylization examples . The first and second columns show the style and content images , respectively . The other seven columns show the stylized images produced by our method , Gatys et al . [ 10 ] , AdaIN [ 15 ] , WCT [ 30 ] , Avatar-Net [ 41 ] , LST [ 28 ] , and SANet [ 36 ] . etc. , are more consistent with human perception . Namely , they contain some human-aware style information that is lacked in synthesized stylizations . A natural idea is to utilize such human-aware style information to improve stylization results . To this end , we employ an internal-external learning scheme during training , which takes both internal learning and external learning into consideration . To be more specific , on the one hand , we follow previous methods [ 10 , 20 , 46 , 54 , 58 ] , utilizing internal statistics of a single artwork to determine the colors and texture patterns of the stylized image . On the other hand , we employ Generative Adversarial Nets ( GANs ) [ 11 , 39 , 2 , 56 , 3 ] to externally learn the human-aware style information from the large-scale style dataset , which is then used to make the color distributions and texture patterns in the stylized image more reasonable and harmonious , significantly bridging the gap between human-created artworks and AI-created artworks . In addition , there is another problem with existing style transfer methods : they usually employ a content loss and a style loss to enforce the content-to-stylization relations and style-to-stylization relations , respectively , while neglect the stylization-to-stylization relations , which are also important for style transfer . What are stylization-to-stylization relations ? Intuitively , stylized images rendered with the same style image should have closer relations in style than those rendered with different style images . Similarly , stylized images based on the same content image should have closer relations in content than those based on different content images . Inspired by this , in this paper we introduce two contrastive losses : content contrastive loss and style contrastive loss that can pull the multiple stylization embeddings closer to each other when they share the same content or style , but push far away otherwise . To the best of our knowledge , this is the first work that successfully leverages the power of contrastive learning [ 6 , 12 , 21 , 38 ] in the style transfer scenario . Our extensive experiments show that the proposed method can not only produce visually more harmonious and plausible artistic images , but also promote the stability and consistency of rendered video clips . To summarize , the main contributions of this work are threefold : • We propose a novel internal-external style transfer method which takes both internal learning and external learning into consideration , significantly bridging the gap between humancreated and AI-created artworks . • We for the first time introduce contrastive learning to style transfer , yielding more satisfying stylization results with the learned stylization-to-stylization relations . • We demonstrate the effectiveness and superiority of our approach by extensive comparisons with several state-of-the-art artistic style transfer methods . 2 Related Work . Artistic style transfer . Artistic style transfer is an image editing task that aims at transferring artistic styles onto everyday photographs to create new artworks . Earlier methods usually resort to traditional techniques such as stroke rendering [ 13 ] , image analogy [ 14 , 42 , 9 , 31 ] , and image filtering [ 52 ] to perform artistic style transfer . These methods typically rely on low-level statistics and often fail to capture semantic information . Recently , Gatys et al . [ 10 ] discovered that the Gram matrix upon deep features extracted from a pre-trained DCNN can notably represent the characteristics of visual styles , which opens up the neural style transfer era . Since then , a suite of neural methods have been proposed , boosting the development of style transfer from different concerns . Specifically , [ 20 , 27 , 46 ] utilize feed-forward networks to improve efficiency . [ 26 , 54 , 36 , 58 , 35 ] refine various elements in the stylized images ( including content preservation , textures , brushstrokes , etc . ) to enhance visual quality . [ 7 , 15 , 30 , 41 , 28 ] propose universal style transfer methods to achieve generalization . [ 29 , 47 , 51 ] inject random noise to the generative network to encourage diversity . Despite the rapid progress , these style transfer methods still suffer from spurious artifacts such as disharmonious colors and repetitive patterns . Notice that there is another line of work [ 40 , 24 , 23 , 45 , 4 , 5 ] that aims to learn an artist ’ s style from all his/her artworks . In comparison , instead of learning an artist ’ s style , we focus on better leaning an artwork ’ s style ( just like the style transfer methods mentioned in the previous paragraph ) with the assist of the human-aware style information reserved in the external style dataset . Therefore , our method is orthogonal to these works . Image-to-image translation . Image-to-image translation ( I2I ) [ 17 , 60 , 16 , 25 , 8 , 18 ] aims at learning the mapping between different visual domains , which is closely related to style transfer . [ 60 , 16 ] have distinguished these two tasks : ( i ) I2I can only translate between content-similar visual domains ( such as horses↔zebras and summer↔winter ) , while style transfer does not have such limitation , whose content image and style image can be totally different ( e.g. , the former is a photo of a person and the latter is van Gogh ’ s The Starry Night ) . ( ii ) I2I aims to learn the mapping between two image collections , while style transfer aims to learn the mapping between two specific images . However , we argue that we can borrow some insights from I2I , and leverage the external information of the large-scale style image collections to improve the stylization quality in style transfer . Internal-external learning . Internal-external learning has shown effectiveness in various image generation tasks , such as super-resolution , image inpainting , and so on . In detail , Soh et al . [ 44 ] presented a fast , flexible , and lightweight self-supervised super-resolution method by exploiting both external and internal samples . Park et al . [ 37 ] developed an internal-external super-resolution method that facilitates super-resolution networks to further enhance the quality of the restored images . Wang et al . [ 49 ] proposed a general external-internal learning inpainting scheme , which learns semantic knowledge externally by training on large datasets while fully utilizes internal statistics of the single test image . However , in the field of style transfer , existing methods only use a single artistic image to learn style , resulting in unsatisfying stylization results . Motivated by this , in this work we propose an internal-external style transfer method that takes both internal learning and external learning into consideration , significantly bridging the gap between human-created and AI-created artworks . Contrastive learning . Generally , there are three key ingredients in a contrastive learning process : query , positive examples , and negative examples . The target of contrastive learning is to associate a “ query ” with its “ positive ” example while disassociate the “ query ” with other examples that are referred to as “ negatives ” . Recently , contrastive learning has demonstrated its effectiveness in the field of conditional image synthesis . To be more specific , ContraGAN [ 21 ] introduced a conditional contrastive loss ( 2C loss ) to learn both data-to-class and data-to-data relations . Park et al . [ 38 ] maximized the mutual information between input and output with contrastive learning to encourage content preservation in unpaired image translation problems . Liu et al . [ 34 ] introduced a latentaugmented contrastive loss to encourage images generated from adjacent latent codes to be similar and those generated from distinct latent codes to be dissimilar , achieving diverse image synthesis . Yu et al . [ 55 ] proposed a dual contrastive loss in adversarial training that generalizes representation to more effectively distinguish between real and fake , and further incentivizes the image generation quality . Wu et al . [ 53 ] improved the image dehazing result by introducing contrastive learning , which ensures that the restored image is pulled closer to the clear image and pushed far away from the hazy image in representation space . Note that all the above contrastive learning methods can not be adopted for style transfer . In this work , we make the first attempt to adapt contrastive learning to artistic style transfer , and propose two novel contrastive losses : content contrastive loss and style contrastive loss to learn the stylization-tostylization relations that are ignored by existing style transfer methods . 3 Proposed Method . Existing style transfer methods usually produce unsatisfying stylization results with disharmonious colors and repetitive patterns , which makes them pretty easy to be distinguished from real artworks . As an attempt to bridge the large gap between human-created and AI-created artworks , we propose a novel internal-external style transfer method with two contrastive losses . The overview of our method is shown in Figure 2 . It is worth noting that our framework is built on the SANet [ 36 ] ( one of the state-of-the-art style transfer methods ) backbone , which consists of an encoder E , a transformation module T , and a decoder D. In detail , E is a pre-trained VGG-19 network [ 43 ] used to extract image features , T is a style-attentional network that can flexibly match the semantic nearest style features onto the content features , and D is a generative network used to transform encoded semantic feature maps into stylized images . We extend SANet [ 36 ] with our proposed changes , and our full model is described below .

Summary: The paper describes a method for feed-forward image stylization that combines several losses. The novel losses are a contrastive loss that optimizes the style embedding of the training image to match the embedding of a target style and content and to be distinct from other style and contents.  Other losses used during training include feature mean/variances and a GAN loss.