Artistic Style Transfer with Internal-external Learning and Contrastive Learning

1 Introduction . Artistic style transfer is a long-standing research topic that seeks to render a photograph with a given artwork style . Ever since Gatys et al . [ 10 ] for the first time proposed a neural method , which leverages a pre-trained Deep Convolutional Neural Network ( DCNN ) to separate and recombine contents and styles of arbitrary images , an unprecedented booming [ 20 , 26 , 15 , 30 , 36 , 51 , 48 ] in style transfer has been witnessed . Despite the recent progress , there still exists a large gap between real artworks and synthesized stylizations . As shown in Figure 1 , the stylized images usually contain some disharmonious colors and repetitive patterns , which makes them easily distinguishable from real artworks . We argue that this is because existing style transfer methods often confine themselves to the internal style statistics of a single artistic image . In some other tasks ( for example , image-to-image translation [ 17 , 60 , 16 , 25 , 8 , 18 ] ) , the style is usually learned from a collection of images , which inspires us to leverage the external information reserved in the large-scale style dataset to improve the stylization results in style transfer . Why is the external information so important for style transfer ? Our analyses are as follows : Although different images in the style dataset vary greatly in fine details , they share a key commonality : they are all human-created artworks , whose brushstrokes , color distributions , texture patterns , tones , ∗Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . AdaIN SANetAvatar-Net LSTGatys et al . WCTOursStyle Content Figure 1 : Stylization examples . The first and second columns show the style and content images , respectively . The other seven columns show the stylized images produced by our method , Gatys et al . [ 10 ] , AdaIN [ 15 ] , WCT [ 30 ] , Avatar-Net [ 41 ] , LST [ 28 ] , and SANet [ 36 ] . etc. , are more consistent with human perception . Namely , they contain some human-aware style information that is lacked in synthesized stylizations . A natural idea is to utilize such human-aware style information to improve stylization results . To this end , we employ an internal-external learning scheme during training , which takes both internal learning and external learning into consideration . To be more specific , on the one hand , we follow previous methods [ 10 , 20 , 46 , 54 , 58 ] , utilizing internal statistics of a single artwork to determine the colors and texture patterns of the stylized image . On the other hand , we employ Generative Adversarial Nets ( GANs ) [ 11 , 39 , 2 , 56 , 3 ] to externally learn the human-aware style information from the large-scale style dataset , which is then used to make the color distributions and texture patterns in the stylized image more reasonable and harmonious , significantly bridging the gap between human-created artworks and AI-created artworks . In addition , there is another problem with existing style transfer methods : they usually employ a content loss and a style loss to enforce the content-to-stylization relations and style-to-stylization relations , respectively , while neglect the stylization-to-stylization relations , which are also important for style transfer . What are stylization-to-stylization relations ? Intuitively , stylized images rendered with the same style image should have closer relations in style than those rendered with different style images . Similarly , stylized images based on the same content image should have closer relations in content than those based on different content images . Inspired by this , in this paper we introduce two contrastive losses : content contrastive loss and style contrastive loss that can pull the multiple stylization embeddings closer to each other when they share the same content or style , but push far away otherwise . To the best of our knowledge , this is the first work that successfully leverages the power of contrastive learning [ 6 , 12 , 21 , 38 ] in the style transfer scenario . Our extensive experiments show that the proposed method can not only produce visually more harmonious and plausible artistic images , but also promote the stability and consistency of rendered video clips . To summarize , the main contributions of this work are threefold : • We propose a novel internal-external style transfer method which takes both internal learning and external learning into consideration , significantly bridging the gap between humancreated and AI-created artworks . • We for the first time introduce contrastive learning to style transfer , yielding more satisfying stylization results with the learned stylization-to-stylization relations . • We demonstrate the effectiveness and superiority of our approach by extensive comparisons with several state-of-the-art artistic style transfer methods . 2 Related Work . Artistic style transfer . Artistic style transfer is an image editing task that aims at transferring artistic styles onto everyday photographs to create new artworks . Earlier methods usually resort to traditional techniques such as stroke rendering [ 13 ] , image analogy [ 14 , 42 , 9 , 31 ] , and image filtering [ 52 ] to perform artistic style transfer . These methods typically rely on low-level statistics and often fail to capture semantic information . Recently , Gatys et al . [ 10 ] discovered that the Gram matrix upon deep features extracted from a pre-trained DCNN can notably represent the characteristics of visual styles , which opens up the neural style transfer era . Since then , a suite of neural methods have been proposed , boosting the development of style transfer from different concerns . Specifically , [ 20 , 27 , 46 ] utilize feed-forward networks to improve efficiency . [ 26 , 54 , 36 , 58 , 35 ] refine various elements in the stylized images ( including content preservation , textures , brushstrokes , etc . ) to enhance visual quality . [ 7 , 15 , 30 , 41 , 28 ] propose universal style transfer methods to achieve generalization . [ 29 , 47 , 51 ] inject random noise to the generative network to encourage diversity . Despite the rapid progress , these style transfer methods still suffer from spurious artifacts such as disharmonious colors and repetitive patterns . Notice that there is another line of work [ 40 , 24 , 23 , 45 , 4 , 5 ] that aims to learn an artist ’ s style from all his/her artworks . In comparison , instead of learning an artist ’ s style , we focus on better leaning an artwork ’ s style ( just like the style transfer methods mentioned in the previous paragraph ) with the assist of the human-aware style information reserved in the external style dataset . Therefore , our method is orthogonal to these works . Image-to-image translation . Image-to-image translation ( I2I ) [ 17 , 60 , 16 , 25 , 8 , 18 ] aims at learning the mapping between different visual domains , which is closely related to style transfer . [ 60 , 16 ] have distinguished these two tasks : ( i ) I2I can only translate between content-similar visual domains ( such as horses↔zebras and summer↔winter ) , while style transfer does not have such limitation , whose content image and style image can be totally different ( e.g. , the former is a photo of a person and the latter is van Gogh ’ s The Starry Night ) . ( ii ) I2I aims to learn the mapping between two image collections , while style transfer aims to learn the mapping between two specific images . However , we argue that we can borrow some insights from I2I , and leverage the external information of the large-scale style image collections to improve the stylization quality in style transfer . Internal-external learning . Internal-external learning has shown effectiveness in various image generation tasks , such as super-resolution , image inpainting , and so on . In detail , Soh et al . [ 44 ] presented a fast , flexible , and lightweight self-supervised super-resolution method by exploiting both external and internal samples . Park et al . [ 37 ] developed an internal-external super-resolution method that facilitates super-resolution networks to further enhance the quality of the restored images . Wang et al . [ 49 ] proposed a general external-internal learning inpainting scheme , which learns semantic knowledge externally by training on large datasets while fully utilizes internal statistics of the single test image . However , in the field of style transfer , existing methods only use a single artistic image to learn style , resulting in unsatisfying stylization results . Motivated by this , in this work we propose an internal-external style transfer method that takes both internal learning and external learning into consideration , significantly bridging the gap between human-created and AI-created artworks . Contrastive learning . Generally , there are three key ingredients in a contrastive learning process : query , positive examples , and negative examples . The target of contrastive learning is to associate a “ query ” with its “ positive ” example while disassociate the “ query ” with other examples that are referred to as “ negatives ” . Recently , contrastive learning has demonstrated its effectiveness in the field of conditional image synthesis . To be more specific , ContraGAN [ 21 ] introduced a conditional contrastive loss ( 2C loss ) to learn both data-to-class and data-to-data relations . Park et al . [ 38 ] maximized the mutual information between input and output with contrastive learning to encourage content preservation in unpaired image translation problems . Liu et al . [ 34 ] introduced a latentaugmented contrastive loss to encourage images generated from adjacent latent codes to be similar and those generated from distinct latent codes to be dissimilar , achieving diverse image synthesis . Yu et al . [ 55 ] proposed a dual contrastive loss in adversarial training that generalizes representation to more effectively distinguish between real and fake , and further incentivizes the image generation quality . Wu et al . [ 53 ] improved the image dehazing result by introducing contrastive learning , which ensures that the restored image is pulled closer to the clear image and pushed far away from the hazy image in representation space . Note that all the above contrastive learning methods can not be adopted for style transfer . In this work , we make the first attempt to adapt contrastive learning to artistic style transfer , and propose two novel contrastive losses : content contrastive loss and style contrastive loss to learn the stylization-tostylization relations that are ignored by existing style transfer methods . 3 Proposed Method . Existing style transfer methods usually produce unsatisfying stylization results with disharmonious colors and repetitive patterns , which makes them pretty easy to be distinguished from real artworks . As an attempt to bridge the large gap between human-created and AI-created artworks , we propose a novel internal-external style transfer method with two contrastive losses . The overview of our method is shown in Figure 2 . It is worth noting that our framework is built on the SANet [ 36 ] ( one of the state-of-the-art style transfer methods ) backbone , which consists of an encoder E , a transformation module T , and a decoder D. In detail , E is a pre-trained VGG-19 network [ 43 ] used to extract image features , T is a style-attentional network that can flexibly match the semantic nearest style features onto the content features , and D is a generative network used to transform encoded semantic feature maps into stylized images . We extend SANet [ 36 ] with our proposed changes , and our full model is described below .

Summary: The paper proposes two new additional losses for style transfer: 1. A contrastive loss to incorporate to bring the style of the stylized images with the same style image closer to each other, while pushing the stylized images from different style images further from each other. The same for content. 2. An adversarial loss to learn the general aesthetics of stylized images over a larger dataset. The authors demonstrate that a combination of these losses plus the regular style-content loss can generate high quality, and surprisingly consistent stylized images, both qualitative and quantitatively as well as user studies.