Dual-head Genre-instance Transformer Network for Arbitrary Style Transfer

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 16:03:27

Abstract: Arbitrary style transfer aims to render artistic features from a style reference onto an image while retaining its original content. Previous methods either focus on learning the holistic style from a specific artist or extracting instance features from a single artwork. However, they often fail to apply style elements uniformly across the entire image and lack adaptation to the style of different artworks. To solve these issues, our key insight is that the art genre has better generality and adaptability than the overall features of the artist. To this end, we propose a Dual-head Genre-instance Transformer (DGiT) framework to simultaneously capture the genre and instance features for arbitrary style transfer. To the best of our knowledge, this is the first work to integrate the genre features and instance features to generate a high-quality stylized image. Moreover, we design two contrastive losses to enhance the capability of the network to capture two style features. Our approach ensures the uniform distribution of the overall style across the stylized image while enhancing the details of textures and strokes in local regions. Qualitative and quantitative evaluations demonstrate that our approach exhibits its superior performance in terms of visual qualitative and efficiency.

Main paper: 
 introduction: 
 If great artworks tell a complete story, the artistic style acts as its essence, unveiling the thematic core and unique creative perspective of the composition. The objective of Arbitrary Style Transfer (AST) is to transfer the style of a reference image to any given image while preserving its content [9] . Given its great potential in various practical applications, AST has emerged as a prominent research area in computer vision, garnering considerable attention to driving continuous endeavors in both quality and efficiency. The primary challenge of AST lies in effectively representing style and then realistically mapping an image into an artistic rendition. Most existing methods learn style representation using the following two approaches. The first one is to learn the holistic style from a specific artist such as Van Gogh. Prior works [14, 18, 21, 41] have successfully produced high-quality stylized images by treating each artist's style as a domain. However, a common limitation exists: these methods tend to generate only a single type of stylization, making them less adaptable to different artwork styles, as illustrated in Fig. 1 (a) . The alternative approach is to abstract the style representation from a single artwork [3, 6, 25, 26, 37, 38] . Such methods focus on extracting instance features such as specific brushstrokes and textures, enabling controllable generation of images. However, they often struggle to maintain uniform style elements across the entire image due to limited utilization of comprehensive style information present in art collections. As mentioned in [18] , relying on a single artwork may not adequately represent the entirety of an artistic style. This can lead to issues such as local content leaks and inconsistencies in holistic style, as illustrated in Fig. 1 (b) .Recently, some studies [4, 28, 32] attempt to combine the above two approaches to achieve a more comprehensive style representation. Chen et al. propose DualAST [4] to learn the instance features using a pre-trained VGG-19 [19] and artist's style features through GAN-based constraints. Xu et al. introduce DRB-GAN [32] , leveraging Dynamic ResBlocks to integrate both artist's style and instance features. Wu et al. utilize a DSTM [28] to decouple the artist's style and instance style from a single style image and swap them with those from other images to achieve AST. However, these methods are constrained by the scope of their training data, with DualAST [4] trained on just six artists and DRB-GAN [32] on eleven collected from WikiArt, limiting their capability for arbitrary style transfer. As illustrated in Fig. 1 (c ), when presented with an unseen artistic style, the resulting stylized image may exhibit local distortions, compromising the content structure.In this paper, we propose a novel insight that the art genre offers greater generality and adaptability compared to the overall features of individual artists. The genre encompasses distinctive visual elements, techniques, and methods, often corresponding to an art movement or school, such as Baroque, Expressionism, or Impressionism. We recognize that an artist's style in the genre evolves significantly over time. For example, Van Gogh's early artwork "Papa Tanguy" shows Ukiyo-e influences, whereas "Starry Night" embodies Post-Impressionism. These genre differences highlight that even the artworks by the same artist can vary significantly in painting technique and style. To this end, we propose the dual-head genre-instance transformer (DGiT) framework to enable highly effective AST. Specifically, DGiT consists of a content encoder, a dual-head style encoder, and a style transformer decoder. The dual-head style encoder has two heads, a genre-wise head and an instance-wise head, to simultaneously capture the genre features and instance features. The genre-wise head is designed to capture the common features of the art genre such as the overall feeling, while the instance-wise head is utilized to capture the unique features like colors, texture, and brushstrokes of the artwork. We design an effective style transformer decoder to progressively migrate them to obtain the final style representation. Different from [6, 35] based on ViT, our method not only learns two style representations but also accelerates convergence by the designed style transformer decoder. As shown in Fig. 1 (d), our approach can ensure the style elements are uniformly and coherently applied across the entire image while successfully preserving the overall outline and texture details of the content image.Furthermore, we introduced two special contrastive losses, namely genre contrastive loss and instance contrastive loss, to enhance the effectiveness of our model. Specifically, the genre contrastive loss treats artworks from the same genre as positive examples and those from different genres as negative, thereby facilitating the learning of holistic genre styles within the same genre. On the other hand, the instance contrastive loss randomly selects positive-negative patch pairs within a single style image and other distinct style images. Unlike existing methods [17, 26, 38, 39] that process the entire image as an anchor to identify overall differences between various artworks, our instance contrastive loss focuses on the texture details and brushstrokes at any spatial location within the style image. Unlike the patch-methods [17, 23, 33] which considers the relationship between the stylized image and the source image, we consider the relationship between a single artwork and other artworks. This contrastive strategy enables the instance-wise head to identify the unique style variations in each image. Our contributions are summarized as follows:• We propose a dual-head genre-instance transformer (DGiT) framework to simultaneously capture the genre and instance features for achieving arbitrary style transfer. To the best of our knowledge, this is the first work in AST to propose the combination of the genre-wise and instance-wise features to generate high-quality stylized images. • Two contrastive losses are introduced to encourage DGiT to capture two style representations, which can strengthen the model's ability of preserving more texture details and the holistic style of the artwork. • Experimental results show that our approach can achieve the best style performance regarding visual quality. The model allows the style elements to be uniformly and coherently applied across the stylized image, improving the robustness of content-style inputs. 
related work 2.1 arbitrary style transfer: 
 Since the introduction of adaptive instance normalization for achieving AST in [9] , there have been numerous advancements in this area. Some methods [14, 18, 21, 41] focus on learning the holistic style from a specific artist such as Van Gogh. For instance, Svoboda et al. proposed a two-stage model [21] for stylized imaging with enhanced content geometry flexibility, while Zhang et al. created ArtBank [41] to learn artists' style type by guiding pre-trained largescale models. Nevertheless, these methods only learn the style type of a limited number of artists and are unable to achieve AST. To address this issue, most existing methods [1, 3, 6, 13, 26, 31, 34, 37, 38] adopt the other perspective to learn the style representation from a single artwork. For example, StyTr2 [6] , as a transformer-based approach, mitigates biased content representation in style transfer by accounting for the long-range dependencies in input images. A novel dynamic style kernel [31] performs learning spatial adaptation capabilities to achieve per-pixel stylization. Recently, some advanced methods [15, 37] studies on diffusion models to improve the quality of the stylized image. InST [37] learns the artistic style directly based on the inversion method and DiffuseIT [15] employes a pre-trained ViT model to guide the generation process of DDPM models [8] . Despite these progresses, they fail to apply style elements uniformly due to the limited use of comprehensive style information from art collections. Several studies [4, 28, 32] attempt to merge the two perspectives to achieve a more effective style representation. DualAST [4] utilizes a fixed pre-trained VGG-19 [19] to learn instance features and incorporates multiple artist's style features via GAN constraints. DRB-GAN [32] integrates both artist's style and instance features using Dynamic ResBlocks. However, the limited artist style type constrains the flexibility and generalization of arbitrary style transfer. Unlike the above methods, we focus on the art genre features which have better generality and adaptability than the holistic features of the artist. Our method can learn simultaneously the holistic genre features and specific artworks' instance features, to achieve high quality and flexible stylized results in AST.With the rise of contrastive learning, many studies [2, 23, 26, 29, 33, 38, 40] have investigated contrastive learning in style transfer tasks. Inspired by CUT [17] , IEST [17] calculates the feature statistics (mean and standard deviation) as style priors, and confines the calculation of contrastive loss to the generated outputs. Add & Norm 𝐹 ! 𝐹 "#$ Q K V Multi-Head Self-Attention AdaIN 𝐹 "#$ AdaIN Feed Forward Add & Norm Layer 1 Layer M 𝐹 %&# 𝐹 %&# … … ×𝑀 𝐹 !$ ×𝑁 Figure 3:Style transformer decoder includes AdaIN, multihead attention, feed-forward network, and layer normalization. We first fuse the content feature 𝐹 𝑐 with the instance feature 𝐹 𝑖𝑛𝑠 and then fuse it with the genre features 𝐹 𝑔𝑒𝑛 .CLAST [23] employs a similar form to quantify the color and style differences between the generated results and the style reference image. ZeCon [33] effectively preserves the content information in a zero-shot manner by leveraging the patch-wise contrastive loss [17] . Unlike the above patch-based methods, CAST [38] uses extracted holistic style features by the pre-trained VGG-19 layers to compute contrastive loss between images. UCAST [39] further uses an adaptive temperature mechanism to control the proportion of penalties between the positive and negative samples. CCPL [29] introduces contrastive learning for video style transfer by considering the frame-wise patch differences. MicroAST [26] constructs the stylized image with other style images as positive/negative pairs to enhance the capability of the style encoder. CSACT [40] adopts the Gram matrix as a style representation in contrastive learning to capture more style information.In this paper, we adopt supervised contrastive learning [11] to learn the common features of the art genre. An instance contrastive loss is designed to capture the instance features of the style image. Compared with [26, 27, 38] which uses the full image as the sample, our instance contrastive loss focuses on the texture details and brushstrokes at any spatial location within the same artwork. 
method: 
 In this section, we will introduce our Dual-head Genre-instance Transformer (DGiT) framework for arbitrary style transfer. We first present the overall pipeline of our DGiT framework, followed by the dual-head style encoder, style transformer decoder, and contrastive learning strategy in detail.Given a content image 𝐼 𝑐 and a style image 𝐼 𝑠 , we aim to synthesize an image 𝐼 𝑔 that has the style pattern of 𝐼 𝑠 while maintaining the content structure of 𝐼 𝑐 . We propose a dual-head genre-instance transformer (DGiT) which can simultaneously learn the genre and instance features from the style image and then migrate the content and style representations into the stylized image. Compared with Anonymous Authors [4, 28, 32] which capture the artist-style and artwork-style representations, our method ensures that style features are consistently and coherently distributed across the image, effectively maintaining the content image's main outline and texture specifics. The overall framework is illustrated in Fig. 2 . Our DGiT consists of three key components: content encoder, dual-head style encoder, and style transformer decoder. We first extract the content features 𝐹 𝑐 and style feature 𝐹 𝑠 via the transformer encoder, followed by the genre-wise head and instance-wise head to extract the genre features 𝐹 𝑔𝑒𝑛 and instance features 𝐹 𝑖𝑛𝑠 . Then, we employ a designed style transformer decoder to stylize the content feature according to two style representations. Finally, we obtain the final stylized image 𝐼 𝑔 by using an upsampling decoder. To encourage the model to extract the style features effectively, we design two contrastive losses, genre contrastive loss and instance contrastive loss, to guide the dual-head encoder to obtain discriminative genre features and unique instance features. During the training process, the genre labels are provided to the genre-wise head to supervise the learning process. In the test, we can generate the stylized image using the trained model without the genre label.The dual-head style encoder consists of a genre-wise head and an instance-wise head. The genre-wise head is designed to capture the common features of the art genre such as the overall feeling, while the instance-wise head is utilized to capture the unique features like colors, texture, and brushstrokes of the artwork.Specifically, we split a style image 𝐼 𝑠 ∈ R 𝐻 ×𝑊 ×3 into patches and map these input patches by using trainable linear projection to obtain a sequence of patch embeddings in the shape of 𝐿 × 𝐶.Here, 𝐿 = 𝐻 ×𝑊 𝑚×𝑚 represents the sequence length, 𝐶 and 𝑚 denote the embedding dimension and the patch size, respectively. Similarly to Str2 [6] , these feature sequences are fed into the transformer encoder [7] to capture more refined feature sequences 𝐹 𝑠 . Finally, 𝐹 𝑠 is fed into the genre-wise head and instance-wise head to further capture the genre and instance features. Two heads have the same structure but independent parameters. Each head consists of a multi-head self-attention module and a feed-forward network. The multi-head self-attention module can be formulated as:MHA(𝑄, 𝐾, 𝑉 ) = [head 1 , . . . , head ℎ ]𝑊 𝑂 , head 𝑖 = Att(𝑄𝑊 𝑄 𝑖 , 𝐾𝑊 𝐾 𝑖 , 𝑉𝑊 𝑉 𝑖 ), Att(𝑄, 𝐾, 𝑉 ) = softmax 𝑄𝐾 𝑇 √︁ 𝑑 𝑘 𝑉 ,𝑊 𝑄 𝑖 , 𝑊 𝐾 𝑖 , and 𝑊 𝑉 𝑖 are the learnable projection matrices. 𝑊 𝑂 is the output projection matrix. 𝑑 𝑘 is the dimension of the key. The genre-wise head is described as follows:𝐹 ′ 𝑔𝑒𝑛 = MHA(𝐹 𝑠 , 𝐹 𝑠 , 𝐹 𝑠 ) + 𝐹 𝑠 𝐹 𝑔𝑒𝑛 = FFN(𝐹 ′ 𝑔𝑒𝑛 ) + 𝐹 ′ 𝑔𝑒𝑛 ,We apply the layer normalized after the attention block. Similarly, the style sequence 𝐹 𝑠 follows the same calculation after being fed into the instance-wise head. In Section 3.4, we optimize the model by minimizing genre contrastive loss and instance contrastive loss to guide the dual-head encoder to learn genres and instance features.Some existing methods [6, 34, 35] utilize ViT model, which exhibits slow convergence in training procedure due to a large number of parameters [22] . In contrast, we adopt AdaIN [9] combined with multi-head self-attention to alleviate computational cost and accelerate convergence. The more detailed of the style transformer decoder is shown in Fig. 3 .Specifically, we input the content sequence 𝐹 𝑐 and the instancespecific sequence 𝐹 𝑖𝑛𝑠 into AdaIN [9] layer. This process can be formulated as follows:EQUATIONwhere 𝜇 (•) and 𝜎 (•) denote the mean and standard deviation of the input tensor, respectively. The obtained 𝐴 ′ is then fed into a multi-head self-attention module to obtain the intermediate representation as follows:EQUATIONSubsequently, we merge the genre features 𝐹 𝑔𝑒𝑛 with the obtained intermediate representation across 𝑁 layers to generate the final output 𝐹 𝑐𝑠 via the similar processes in (2). Finally, the output feature sequence 𝐹 𝑐𝑠 is fed into three layers CNN decoder to obtain the stylized image 𝐼 𝑔 . Each layer includes two 3 × 3 convolutional layers, a ReLU layer, and an upsample layer.Analysis: As we have two style heads, an immediate challenge is encountered for the decoder design, that is, how to reasonably integrate content and style features to obtain the final stylized image. A reasonable approach is to first merge instance features with content features to preserve texture details in the stylized image. Then, applying genre characteristics ensures uniform and coherent application of style elements across the entire image. We also analyze different fusion strategies in detail in Section 4.3.3.4.1 Genre Contrastive Loss. To encourage the genre-wise head to capture the common features of the art genre, we propose a genre contrastive loss as an implicit measurement to capture the genrewise features effectively. It is well known that artworks in the same art genre share notable similarities in their use of color, line styles, and compositional techniques, while those from different genres exhibit significant differences. Hence, we classify style images from identical genres as positive examples and those from varying genres as negative examples, and design a genre contrastive loss for capturing the holistic discriminative genre style features. Specifically, for a set of 𝑁 randomly sampled sample/lable pairs {𝑥 𝑖 , 𝑦 𝑖 }, where 𝑖 ∈ 𝐼 = {1, . . . 𝑁 } be the index of the sample within the batch. Let 𝐴(𝑖) = 𝐼 \{𝑖} be the index set of all samples except 𝑖. We define 𝑓 𝑔𝑒𝑛 = {𝑧 𝑖 } 𝑁 𝑖=1 ∈ R 𝑁 ×𝑑 ×𝐶 as the output of the genrewise head. We randomly choose 𝑧 𝑖 as the anchor, 𝑃 (𝑖) = {𝑝 ∈ 𝐴( 𝑗) : 𝑦 𝑖 = 𝑦 𝑝 } as the index set of all positive samples, and |𝑃 (𝑖)| is its cardinality. We take 𝐴(𝑖)\𝑃 ( 𝑗) as the index set of all negative samples. The L 𝑔𝑒𝑛 is designed to maximize the similarity between the anchor and positive samples while minimizing the similarity between the anchor and negative samples. The genre contrastive loss is formulated as:EQUATIONwhere ẑ = 𝑓 1 (𝑧) is the nomalized output from the genre-wise head,• symbol denotes the innder product and 𝜏 ∈ R + is a temperature parameter. The function 𝑓 1 (•) is a multi-layer projection that consists of two fully connected layers, to project the features into a 𝑙 2 -normalized space.To assist the instance-wise head in extracting the instance features, we design an instance contrastive loss to focus on the texture details and brushstrokes of the artwork. Some existing methods [26, 38, 39] leverage contrastive learning to enhance the network's proficiency in extracting style representations. As shown in Fig. 4 , CAST [38] employs the style image 𝐼 𝑠 and its augmented version 𝐼 + 𝑠 as positive samples, with other N style images serving as negative samples. MicroAST [26] selects the stylized image 𝐼 𝑐𝑠 1 as the anchor, assigning its associated style image 𝐼 𝑠 1 as a positive sample, and regarding other style images as negative samples. These methods process the entire image as an anchor to identify overall differences between various artworks. In contrastive, our instance contrastive loss focuses on the local textures and unique features at any spatial location within the style image. Different from patch-methods [17, 23] which considers the relationship between the stylized image and the source image, our loss can better assist the instance-wise head to extract the instance features by considering the relationship between a single artwork and other artworks.We randomly select patches from the same style image as positive samples and patches from other style images as negative samples. This contrastive strategy enables the instance-wise head to focus on the distinct style variations in each artwork. In particular, we defined a set of 𝐹 𝑖𝑛𝑠 = {𝑣 𝑖 } 𝑁 𝑖=1 ∈ R 𝑁 ×𝑑 ×𝐶 as the output feature of the instance-specific head. We randomly sample a pair of patches, m𝑖 ∈ R 𝑛×𝑐 and m+ 𝑖 ∈ R 𝑛×𝑐 on 𝑣 𝑖 as positive sample while negative samples {𝑚 - 𝑗 } 𝑗 ∈𝐴( 𝑗 ) are sampled from the remaining 𝑁 -1 images, maintaining the same shape as m𝑖 . Here, 𝑛 denotes the number of the patch pixels.EQUATION)where m = 𝑓 2 (𝑚) is the nomalized output from the genre-wise head and 𝑓 2 (•) is a multi-layer projection layer following the same structure of 𝑓 1 (•).To preserve the original content structures and reference style patterns, we employ content perceptual loss to quantify differences between the generated image 𝐼 𝑔 and the content image 𝐼 𝑐 , and style perceptual loss to measure the difference from the style reference 𝐼 𝑠 to the generated image 𝐼 𝑔 . We use feature maps extracted by a pre-trained VGG-19 [20] to calculate two losses. The content loss can be described as followsEQUATIONwhere 𝜙 𝑖 (•) indicates the features map which is captured from the i-𝑡ℎ layer of the VGG-19. 𝑁 𝑙 denotes the number of layers. The style perceptual loss L 𝑠 can be described as followsEQUATIONWe also adopt our two contrastive objectives to learn genre-wise and instance-specific features from the style image. Hence, the total training objective function is formulated asEQUATIONwhere the hyperparameters 𝜆 𝑐 , 𝜆 𝑠 , 𝜆 𝑔𝑒𝑛 , and 𝜆 𝑖𝑛𝑠 are utilized to fine-tune the loss balance during the training. 
experiments 4.1 experiment settitngs: 
 We utilize MS-COCO [16] and WikiArt [10] datasets as content and style images for the training process, where WikiArt dataset implemented by Pytorch. We utilize Adam [12] as the optimization solver and the learning rate is 5𝑒 -4 using the warm-up adjustment strategy [30] . The network is trained for 160K with a batch size of 6. The loss weights are chosen as 𝜆 𝑠 = 10, 𝜆 𝑐 = 7, 𝜆 𝑔𝑒𝑛 = 5, 𝜆 𝑖𝑛𝑠 = 1.Our model can support any image resolution in the testing phase.All training experiments are developed on two NVIDIA RTX A5000 GPUs, and the testing data are developed on a single NVIDIA RTX A5000 GPU. More details are shown in the supplementary.We conduct comparison experiments with state-of-the-art algorithms on style transfer, including DiffuseIT [5] , Str2 [6] , MicroAST [26] , CAST [38] , Svoboda et al. [21] , ArtBank [41] , DualAST [4] . Among them, DiffuseIT [5] , Str2 [6] , MicroAST [26] , CAST [38] learn the style representation from a single image, while Svoboda et al. [21] , ArtBank [41] learn the style feature of the artist. DualAST [4] considers the both style of the artist and artworks. We evaluate the performance of these methods in terms of both qualitative and quantitative aspects. In the evaluation of all methods, we ensure a fair comparison by employing publicly available code and adhering to the default configurations provided for testing.To evaluate the superiority of our method, we compare our results with the seven methods above in Fig. 5 . DiffuseIT [5] , Str2 [6] , MicroAST [26] , CAST [38] learn the style representation from a single artwork. However, DiffuseIT [5] is unstable and tends to generate unrelated content structures (e.g. 2nd, 4th, and 5th rows). Str2 [6] and CAST [38] exist the problem of content leaks when the content image with complex structure (e.g. 4th, 5th, and 6th rows). MicroAST [26] often suffers from blurred and visual artifacts (e.g. 2nd, 3rd, and 4th rows). Svoboda et al. [21] and ArtBank [41] only capture the overall style of artists, limiting their ability to handle unseen artists and maintain style consistency with reference style images. As shown in (h) and (i), most results generated by the two methods lose the diversity of the style pattern.DualAST [4] considers the style of both artists and artworks, but the generated images often fail to capture the detailed texture and brushstrokes of the style image (e.g. 1st, 3rd, and 4th rows). Benefiting from dual-head style encoder, compared with Diffu-seIT [5] , Str2 [6] , MicroAST [26] , and CAST [38] , our approach can capture both vivid local stroke characteristics and the overall appearance, while maintaining the content's structural integrity. Even when the content image contains complex structures, our method effectively transfers the rich texture of the style image to the stylized image while preserving its structural integrity. For example, as shown in the 1st-3rd row of Fig. 5 , we can achieve the highest score in terms of content and style loss. It indicates our ability to capture a holistic similar style pattern and achieve reasonable texture mapping with the reference style image. In rows 4 to 6, our method transfers the color and texture of the style image to the stylized image while maintaining the structural consistency of the content image. In contrast to Svoboda et al. [21] , ArtBank [41] , and DualAST [4] , we generate more diverse and vivid stylized images across content-style pairs from different artists and artworks.Table 1 shows the quantitative comparison with the above methods. We generated 10,125 stylized images by randomly selecting 75 content images and 135 style images. We calculate the SSIM [24] to assess the stylization quality with respect to content preservation. LPIPS loss [36] is adopted to measure the content fidelity between the content image and stylized images. Additionally, we evaluate the generated image's consistency in style and content with respect to the style and content references using style and content loss, respectively. As we can observe in Table 1 , our method obtains better scores in all metrics. It indicates that our method exhibits superior capability in preserving both finer details and content affinity. We compare our method with recent state-ofthe-art methods to measure which method can generate reasonable images that are most accepted by humans. The users in our study include 25 males and 25 females, spanning the age range from 16 to 60. For each user, we randomly select 50 content-style pairs and present stylized results by our method and other methods in a randomized order. We ask users to evaluate results from the following three aspects: (i) which result exhibits superior style patterns. (ii) which result preserves the content structures more effectively, and (iii) which stylization result appears more natural and reasonable. Finally, we collected 2,500 votes from 50 users. We report the percentage of votes for each method in Table . 1. Our method receives significantly higher preferences in terms of content preservation, natural appearance and style consistency. When 𝛼 is increased to 1, we achieve complete stylization. As illustrated in Fig. 7 , we produce a group of images showcasing gradual changes in style intensity. Our method can generate a wide range of stylization effects by adjusting 𝛼 parameter, allowing users to customize the stylization intensity according to their preferences.Interpolation. The style interpolation allows for interpolating multiple distinct style patterns into one generated image. We show the generated results under our method with different interpolations as exhibited in Fig. 8 . Our method flexibly handles the interpolation between multiple styles. Users can combine various styles based on their preferences, resulting in personalized and diverse outcomes that cater to their individual needs. 
conclusion and future work: 
 In this paper, we propose a dual-head genre-instance transformer (DGiT) framework to simultaneously learn the genre and instance features from the artworks. Moreover, we propose two contrastive losses to enhance the ability of each head to extract the corresponding style. Our method not only generates detailed texture and specific brushstrokes but also ensures that style elements are uniformly and coherently applied across the stylized image. Extensive experiments demonstrate that our method can achieve the best style performance regarding visual quality. In the future, we will explore the application to the video-style transfer task.  
 
 Appendix:  


Decision: This paper introduced genre features for better image style transfer. Convincing qualitative and quantitative evaluations are presented.
All the reviewers are positive about this paper after the authors' rebuttals.

Reviews: review: paper_summary: paper_summary: This work proposes a dual-head genre-instance transformer (DGiT) framework to simultaneously capture genre and instance features for achieving arbitrary style transfer. The paper uses two contrastive losses to encourage DGiT to capture two style representations.
strength_weakness: strength_weakness: 1. As stated in the paper, this work is the first to use art genre in the style transfer task. Numerous visual results demonstrate that it can achieve better performance than other structures.
2. Detailed experiments show the advantages of this method.
3. The results demonstrate the efficiency of the two contrastive losses, with quantitative comparisons supporting this.
4. By fusing two style features, this method can generate an image that combines both styles, which is interesting and has great potential.
limitations: limitations: 1. More analysis of genre labels: The paper firstly introduces genre features to style transfer. However, more analysis of genre labels is needed. Specifically, while the paper mentions that there is no overlap between the test dataset and the training set, it is unclear if this situation is similar with genre information.
2. Generalization of the style loss: The method is intended for Arbitrary Style Transfer (AST), but its generalization to genre and content in the real world.
3. Some confuse of figure: In first row of Figure 6, in intuitions, I think results (e) maybe better preserved the instance information. Especially in structure of the bridge. Maybe more analysis in this situation will be better.
4. Unfair comparison: this paper uses two contrastive losses to train the network. While in quantitative comparison, these losses are used for comparison. This comparison is unfair for other methods that maybe not train with this target. In quantitative comparison, use other SOTA image feature extractor like DINOv2, which also different with the feature extractor of this method can be better.
paper_summary: paper_summary: This paper presents an arbitrary style transfer method by extracting silt features and content features separately. 
Style features are then fed into the style transformer decoder to generate the final results.
The approach is generalizable to arbitrary styles and achieved better results. 
Ablation studies and user studies were also conducted to further demonstrate the effectiveness of the methods
strength_weakness: strength_weakness: The result is better both objectively and subjectively based off the metrics.
limitations: limitations: 1. DiffuseIT performed so bad as reported in the Table 1, can you explain the reasons?
2. Also, feeding the features into the generator is common in the diffusion era, can you compare with some of those methods?
3. Can you share some of the higher resolution results, such as 1K or 2K?
paper_summary: paper_summary: This paper proposes a novel Dual-head Genre-instance Transformer (DGiT) framework for arbitrary style transfer. DGiT simultaneously captures the art genre and instance features. Furthermore, two contrastive losses are proposed to enhance the network's capability to capture style features. The superiority of the proposed method is validated through extensive experiments.
strength_weakness: strength_weakness: 1. The motivation is clear and reasonable. The art genre possesses greater generality than the comprehensive features of a specific artist.
2. The architecture design of the proposed DGiT and two losses make sense to me.
3. Both visual results and metrics demonstrate the excellence of the proposed DGiT.
limitations: limitations: 1. It would be better if the author could include more discussion about the following papers which are related to the proposed method.

(1) Hong K, Jeon S, Yang H, et al. Domain-aware universal style transfer[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 14609-14617.

(2) Kwon G, Ye J C. Clipstyler: Image style transfer with a single text condition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 18062-18071.

(3) Hong K, Jeon S, Lee J, et al. Aespa-net: Aesthetic pattern-aware style transfer networks[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 22758-22767.
paper_summary: paper_summary: The fundamental insight of this paper lies in the recognition that artistic genres possess greater generalizability and adaptability compared to overarching artist-specific features. Consequently, this paper introduces the Dual-head Genre-instance Transformer (DGiT) framework, a novel approach that concurrently captures genre-specific and instance-specific features for arbitrary style transfer. This paper marks the first attempt to integrate genre and instance features in order to generate stylized images of unparalleled quality.

Furthermore, this paper devises two contrastive losses, which significantly enhance the network's capability to capture both style features. the methodology of this paper ensures a uniform distribution of the overarching style across the stylized image, while simultaneously enriching the details of textures and strokes in localized regions. Extensive qualitative and quantitative experiments demonstrate that the proposed method exhibits superior performance in terms of both visual fidelity and computational efficiency.
strength_weakness: strength_weakness: (1) The paper well-written. It is easy for readers to understand the core idea of this paper.

(2) The performance is good. The proposed method can achieve satisfactory qualitative and quantitative results.

(3) The proposed contrastive losses are interesting, which may inspire many other works.
limitations: limitations: (1) I think the concept of "genre" in this paper is similar to the concept of "domain", while the concept of "artwork" i similar to "exemplar". In the field Image-to-Image translation, there exist many works that translates images from one domain to another domain, for example, translating an image from photo domain to Ukiyo-e domain. An exemplar image can be used to control the translated results.

(2) All the visual results shown in this paper is taking the photo as the content image. It is better to show the visual results of transferring the style from photo to painting, that is to say, taking the photo as the style image. 

(3) The designed Dual-head Encoder is somewhat incremental, which is simply a combination of vision transformer and AdaIN.

(4) Lack of experiments on contrastive learning.