AesStyler: Aesthetic Guided Universal Style Transfer

Authors: aesthetic  codebook
usaesa  aesstyler
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-03 10:14:31

Abstract: Recent studies have shown impressive progress in universal style transfer which can integrate arbitrary styles into content images. However, existing approaches struggle with low aesthetics and disharmonious patterns in the final results. To address this problem, we propose AesStyler, a novel Aesthetic Guided Universal Style Transfer method. Specifically, our approach introduces the aesthetic assessment model, trained on a dataset with human-assessed aesthetic scores, into the universal style transfer task to accurately capture aesthetic features that universally resonate with human aesthetic preferences. Unlike previous methods which only consider aesthetics of specific style images, we propose to build a Universal Aesthetic Codebook (UAC) to harness universal aesthetic features that encapsulate the global aspects of aesthetics. Aesthetic features are fed into a novel Universal and Style-specific Aesthetic-Guided Attention (USAesA) module to guide the style transfer process. USAesA empowers our model to integrate the aesthetic attributes of both universal and style-specific aesthetic features with style features and facilitates the fusion of these aesthetically enhanced style features with content features. Extensive experiments and user studies have demonstrated that our approach generates aesthetically more harmonious and pleasing results than the state-of-the-art methods, both aesthetic-free and aesthetic-aware.

Main paper: 
 related work 2.1 aesthetic-free neural style transfer: 
 Gatys et al. [12] found that hierarchical layers within CNNs serve as effective tools for extracting both image content and style texture information and introduced an optimization-driven approach to iteratively generate stylized images. Certain methodologies [17, 22] have embraced an end-to-end model to facilitate real-time style transfer, tailored to a specific style. More generally, arbitrary style transfer has gained more attention in recent years. Huang et al. [15] introduced the concept of adaptive instance normalization (AdaIN), which adaptively applies the mean and standard deviation of each style feature to shift and re-scale the corresponding normalized content feature. Based on the CNN model, works [6, 7, 27, 30, 43] introduced self-attention into the encoder-transfer-decoder framework, enhancing the fusion of features. AdaAttN [27] proposed to take both shallow and deep features into account and properly normalizes content feature. However, it should be noted that existing style transfer methods often overlook the aesthetic aspects of their outputs. In contrast, our proposed model strikes a harmonious balance encompassing style, content, and aesthetic of the results.Similar to our work, Sanakoyeu et al. [32] , Kotovenko et al. [19, 20] , Chen et al. [4, 5] , and Zuo et al. [46] have tried to use aesthetic information to guide the style transfer. However, their aesthetic assessment models are developed to discriminate the aesthetics associated with specific artists, for instance, Claude Monet, whereas our model encapsulates artist-independent and universal aesthetics. Furthermore, their models are restricted to transferring the styles of predefined artists, whereas our approach attains universal style transfer capabilities. Furthermore, AAST [14] proposed aestheticaware style transfer, albeit with aesthetics defined as a fusion of color and texture alone. AesUST [40] tries to integrate aesthetic information into the style transfer process. However, previous works in aesthetic-guided style transfer did not use an aesthetic-annotated dataset to train aesthetic discriminators, potentially limiting their ability to learn true aesthetic patterns. In contrast, we offer a more comprehensive concept of aesthetics, utilizing an aesthetic assessment model pre-trained on the human-annotated aesthetic dataset to more effectively guide the style transfer process.Early Image Aesthetic Assessment (IAA) models mainly focused on extracting handcrafted features from images and mapping the visual features to annotated aesthetics labels either with trained binary classifiers or with regressors [8, 28] . The emergence of large-scale IAA datasets [18, 29] leads to the continued evolution of methods rooted in deep learning. NIMA [36] proposed to use the Earth Mover's Distance (EMD) loss to train the score distribution task. MPada [33] implemented an attention-based mechanism, dynamically fine-tuning the weights of individual image patches during training, thus improving the efficiency of the learning procedure. SAAN [44] proposes a style-specific artistic image assessment. In this work, we employ the state-of-the-art IAA model, TANet [13] to provide aesthetic guidance for the UST task. TANet can accurately predict aesthetic scores leveraging a target-aware network and an RGB-distribution-aware attention network. 
method: 
 In this section, we will first introduce the overall framework of the AesStyler in Section 3.1 and explain the component details in Section 3.2 and loss functions in Section 3.3.Given a style image 𝐼 𝑠 ∈ Φ 𝑠 and a content image 𝐼 𝑐 ∈ Φ 𝑐 , where Φ 𝑠 and Φ 𝑐 are the style dataset and the content dataset respectively, the goal of UST is to learn a transformation which successfully transfers the style of 𝐼 𝑠 to 𝐼 𝑐 while preserving the semantic information of 𝐼 𝑐 .As shown in Fig. 2 , our AesStyler is composed of five principal components: (1) An Encoder-Decoder Module with a pre-trained VGG-19 network [35] 𝐸 𝑣𝑔𝑔 to extract multi-layer feature maps and a decoder network 𝐷 𝑣𝑔𝑔 tasked with reconstructing the stylized Encoder-Decoder Module. Similar to [15] , we use the pre-trained VGG-19 network [35] as our encoder 𝐸 𝑣𝑔𝑔 and freeze it during the whole training stage. The decoder 𝐷 𝑣𝑔𝑔 is trainable and mirrors the encoder, albeit with the substitution of all pooling layers by nearest up-sampling layers. Aesthetic Assessment Module. We employ TANet [13] to assess the aesthetic score of style transfer results and extract aesthetic features that can correctly guide the style transfer. To be specific, TANet is pre-trained with an aesthetic score regression task on TAD66K [13] , a large-scale aesthetic dataset annotated with human-assessed aesthetic scores, ensuring that TANet can accurately capture aesthetic features that resonate with human perceptions of aesthetics. As to the aesthetic features, we extract the feature maps from the InvertedResidual-57, InvertedResidual-93, InvertedResidual-120, InvertedResidual-147, and InvertedResidual-156 layers within the Aesthetic Perceiving Branch of the TANet as the aesthetic features, which are then utilized in the USAesA module to guide the style transfer process. To maintain TANet's capability of detecting aesthetic patterns, we keep it frozen during the entire training process.Universal Aesthetic Codebook (UAC). Besides style-specific aesthetic features, certain aesthetic features should remain universal, unaffected by variations in style. Inspired by [45] , we propose to build a Universal Aesthetic Codebook (UAC). By utilizing images ranked top in terms of aesthetic scores in the whole style dataset and retrieving those with the closest cosine similarity, we can capture universal aesthetic features that embody the overarching elements of aesthetics while minimally disturbing the style information.We first use TANet [13] to assess the aesthetic scores of the whole WikiArt dataset [31] and curate 1000 images with the highest aesthetic scores. Then, as demonstrated in the upper part of Fig. 3 , we compute their style features via the VGG-19 network [35] for later query and extract their aesthetic features using TANet [13] , which are then stored as UAC, as F𝑖 for the 𝑖th selected image.As shown in the lower part of Fig. 3 , during training, we use 𝐸 𝑣𝑔𝑔 to extract the style feature 𝐹 𝑠 of the style image 𝐼 𝑠 . We use 𝐹 𝑠 as the query to retrieve the corresponding universal aesthetic feature 𝐹 𝑢 from the UAC. Specifically, we calculate the cosine similarity between the style feature 𝐹 𝑠 and all style features F𝑖 stored in the UAC as follows:EQUATIONwhere Γ and • denote the feature vectorization and dot product operations respectively, F𝑖 refers to the 𝑖th style feature stored in the UAC. We compute one thousand cosine similarities, subsequently ascertaining the 𝑖𝑛𝑑𝑒𝑥 corresponding to the image with (Thus, we can retrieve the universal aesthetic feature 𝐹 𝑢 = F𝑖𝑛𝑑𝑒𝑥 , which encapsulates the global aspects of aesthetics while minimally disturbing the style information. We are able to employ this feature, along with style-specific aesthetic features, to guide the model towards generating more universally appealing results.Universal and Style-specific Aesthetic-Guided Attention (USAesA) Module. In order to guide the style transfer process with universal and style-specific aesthetic information, we propose a novel attention-based style transfer module, Universal and Stylespecific Aesthetic-Guided Attention (USAesA) module, which can adaptively integrate style patterns into content features, considering both universal and style-specific aesthetic attributes. As shown in Fig. 4 , USAesA works in three phases: (1) Given the universal aesthetic feature 𝐹 𝑢 retrieved from UAC and the style-specific aesthetic feature 𝐹 𝑎 extracted from the style image, we first use 𝐹 𝑢 to enhance global aesthetic attributes of 𝐹 𝑎 . (2) We then use the universally enhanced aesthetic feature 𝐹 𝑢𝑎 to further integrate aesthetic information into the style feature 𝐹 𝑠 . (3) Lastly, we integrate the aesenhanced style feature 𝐹 𝑎𝑠 according to the semantic distributions of the content feature 𝐹 𝑐 .Phase 1: Universal Aesthetic Enhancement. Given the universal aesthetic feature 𝐹 𝑢 retrieved from UAC and the style-specific aesthetic feature 𝐹 𝑎 extracted from the style image, we first use 𝐹 𝑢 to enhance global aesthetic attributes of 𝐹 𝑎 . Inspired by [27] that attention maps and adaptive normalization can fuse style and content information, we propose to employ these two mechanisms to globally enhance the style-specific aesthetic feature by channel distribution.Specifically, given 𝐹 1:𝑥 𝑢 , 𝐹 𝑥 𝑢 , where 𝑥 means 𝑥th layer in VGG features, retrieved from the UAC and 𝐹 1:𝑥 𝑎 , 𝐹 𝑥 𝑎 extracted from the style image by TANet [13] , we first obtain 𝑄 𝑢 (query), 𝐾 𝑢 (key) and 𝑉 𝑎 (value):EQUATIONwhere 𝑓 𝑢 , 𝑓 1 𝑎 and 𝑓 2 𝑎 are 1 × 1 learnable convolutions. Then, we calculate the aesthetic attention between F 1:𝑥 𝑢 and F 1:𝑥 𝑎 according to the following equation:EQUATIONwhere ⊗ denotes matrix multiplication and 𝑇 denotes the matrix transpose operation. We then calculate the attention-weighted mean and standard deviation respectively:EQUATIONwhere • denotes the element-wise product. Finally, for each position and channel of the universal aesthetic feature, the corresponding scale in 𝑆 1 and shift in 𝑀 1 are used to generate the final universally enhanced aesthetic feature:EQUATIONThus, we effectively enhance the style-specific aesthetic feature by endowing it with more universal aesthetic qualities which resonate with human aesthetic preferences Phase 2: Aesthetic Style Guidance. After using the universal aesthetic feature to enhance the style-specific aesthetic feature, we integrate the universally enhanced aesthetic information into the style features. Similarly, given aesthetic features 𝐹 1:𝑥 𝑎 , 𝐹 𝑥 𝑔𝑎 and style features 𝐹 1:𝑥 𝑠 , 𝐹 𝑥 𝑠 extracted by the VGG-19 network [35] , we first obtain 𝑄 𝑎 , 𝐾 𝑠 and 𝑉 𝑠 , the attention map between 𝑄 𝑎 and 𝐾 𝑠 , the attention-weighted mean and standard deviation respectively:EQUATIONwhere 𝑓 3 𝑎 , 𝑓 1 𝑠 and 𝑓 2 𝑠 are 1 × 1 learnable convolutions. Finally, we scale and shift the universally enhanced aesthetic feature with 𝑆 2 and 𝑀 2 respectively to generate the final aesthetically enhanced style feature:EQUATIONwhere 𝑓 𝑢𝑎 is 1 × 1 learnable convolution. Phase 3: Style Content Integration. Upon refining the style feature with aesthetic guidance, our objective is to integrate them into the content feature, thereby accomplishing an aesthetic-aware style transfer. We transfer feature statistics via generating attentionweighted mean and standard variance maps. Given aesthetic-enhanced style features 𝐹 𝑥 𝑎𝑠 , lower-layer style features 𝐹 1:𝑥 𝑠 and content features 𝐹 1:𝑥 𝑐 , 𝐹 𝑥 𝑐 , we first obtain 𝑄 𝑐 , 𝐾 ′ 𝑠 and 𝑉 𝑎𝑠 , the attention map between 𝑄 𝑐 and 𝐾 ′ 𝑠 , the attention-weighted mean and standard deviation respectively:EQUATIONwhere 𝑓 𝑐 , 𝑓 3 𝑠 and 𝑓 𝑎𝑠 are 1 × 1 learnable convolutions. Finally, we scale and shift the normalized content feature with 𝑆 3 and 𝑀 3 respectively to generate the final result:EQUATIONUSAesA enables our model to adaptively and simultaneously integrate the aesthetic attributes of both universal and style-specific aesthetic features with the style feature according to the global aesthetic channel distribution, and subsequently aids in the integration of the aesthetically enhanced style feature into the content feature, in harmony with the semantic spatial distribution of the content feature.Aesthetic Discriminator. As previous works [26, 37] have shown that aesthetic attributes of one image are closely related to its textures, we find that if we directly optimize the aesthetic scores of images, it may lead to strange textures, which we conjecture act as a primitive guise intended to deceive the TANet into judging the results as aesthetically pleasing. To solve this problem, we propose to employ a new aesthetic discriminator D 𝑎𝑒𝑠 which plays the minmax game of discriminating between real artworks from WikiArt dataset [31] and style transfer results along with the generator to avoid the appearance of strange textures. It is important to highlight that the aesthetic discriminator we introduce differs from the one presented in AesUST [40] . Our Aesthetic Discriminator primarily targets and mitigates odd textures, which could mislead TANet into deeming the results aesthetically pleasing. In contrast, the aesthetic discriminator in AesUST aims to preserve aesthetics, yet it encounters an aesthetic assumption bias, as discussed in Section 1.Aesthetic loss. To augment the aesthetic quality of the style transfer results, we employ the negative aesthetic score as the aesthetic loss term:EQUATIONwhere TANet refers to the pre-trained aesthetic assessment model TANet [13] which will produce an aesthetic score for the style transfer result 𝐼 𝑐𝑠 . Adversarial Loss. In order to make the style transfer results look similar to real artistic images and avoid artifacts related to textures, the newly proposed discriminator D 𝑎𝑒𝑠 plays the min-max game with the generator as follows:EQUATIONHere the generator G consists of 𝐸 𝑣𝑔𝑔 , 𝐷 𝑣𝑔𝑔 and USAesA. Style Loss. Following [27] , distances of mean 𝜇 and standard deviation 𝜎 between generated images and style images in VGG feature space are penalized:EQUATIONContent Loss. L 𝑐 constrains the consistency between features of stylized images and transformation results:EQUATIONwhere USAesA * serves as a supervision signal that should be deterministic. Thus, we consider the parameter-free version of USAesA * without learnable 1 × 1 convolution layers and the aesthetic integration branch.To conclude, our final objective is:EQUATION) 
experiments 4.1 implementation details: 
 Dataset. The Aesthetic Discriminator is trained on TAD66K [13] .TAD66K is a large-scale aesthetic dataset that contains 14900 artistic images and 45100 other images annotated with human-assessed aesthetic scores. This pre-training ensures that TANet can accurately capture aesthetic features that resonate with human perceptions of aesthetics. The main network is trained with 82,783 images from MS-COCO [25] as the content dataset Φ 𝑐 and 79,433 images from WikiArt [31] as the style dataset Φ 𝑠 . Hyper-parameter. We set the hyperparameters 𝜆 𝑠 , 𝜆 𝑐 , 𝜆 𝑎𝑒𝑠 , and 𝜆 𝑎𝑑𝑣 to 0.4, 3.5, 1.2, and 5.0, respectively, which are the optimal values obtained from experiments for the best performance.Details. We employ the Adam optimizer with an initial learning rate of 0.0001, and each batch comprises 6 content and 6 style images. The model is trained for 25,000 iterations. During the training stage, all the images are uniformly resized to 512 × 512 and then randomly cropped to 256 × 256. All experiments are carried out with an NVIDIA GeForce RTX 3090 24GB GPU. We compare our proposed AesSTyler against 10 state-of-the-art arbitrary style transfer methods: aesthetic-aware UST methods (Ae-sUST [40] and AAST [14] ), aesthetic-free UST methods (AdaAttN [27] , Avatar [34] , ArtFlow [1] , IECAST [3] , MAST [7] , AdaIN [15] , SANet [30] and StyleFormer [43] ).Style Transfer Comparison. We first provide the qualitative comparison results in Fig. 5 . Due to the rudimentary alignment of mean and variance, AdaIN [15] generates results with crack-like artifacts (2nd, 4th and 5th rows). Avatar [34] produces results with a blurred appearance and noticeable patchiness due to patch matching strategies, evident in all rows. SANet [30] and MAST [7] meticulously transfer style features onto content features only within the deeper layers, leading to compromised content structures and muddled textures (1st, 2nd, 4th, 5th, 8th rows). The constrained feature representation capacity of flow-based models means that the outcomes of ArtFlow [1] typically suffer from a lack of style richness or accuracy (4th and 5th rows). Additionally, the borders of stylized images may exhibit undesirable patterns stemming from numerical overflow (4th, 5th and 7th rows). As to AdaAttN [27] and IECAST [3] , some results show the style degradation problem, rendering the stylized patterns in the generated images inconsistent with those of the input reference in the 1st, 2nd, 4th and 5th rows and 1st, 3rd, 5th and 7th rows respectively. The outputs of StyleFormer [43] lack details. AAST [14] results have apparent disharmonious patterns. AesUST [40] tends to introduce strange artifacts into the results and suffers from the problem of losing content information which is evident in the 1st, 4th, 5th and 8th rows. Besides, the color harmony and appeal of the results from AesUST are inferior to those produced by our method, evident in the 2nd, 3rd, and 7th rows. On the contrary, by accounting for universally appealing aesthetic qualities, AesStyler generates results that are aesthetically more cohesive and pleasing. The style patterns are integrated more harmoniously with fewer artifacts and better consistency in color patches (3rd, 4th, 5th, 6th and 7th rows). The semantic information is better preserved, such as the face in the 1st row and the background and foreground of the 4th row. This shows that AesStyler outperforms other methods in terms of style transfer quality. Aesthetic Comparison. We also show the results of aesthetic comparison in Fig. 6 . Another Universal Style Transfer method that considers aesthetics, AesUST, evidently falls short in generating images with sufficient aesthetic appeal. Artifacts in the 2nd, 3rd, and 4th rows contribute to a decline in aesthetic scores. Furthermore, the overly dark tone in the 2nd and 4th rows also leads to a reduction in aesthetic appeal. On the contrary, it is evident that our AesStyler generates results with markedly superior aesthetic quality. On the one hand, AesStyler avoids the introduction of conspicuous artifacts and overly complex patterns in the final results (1st and 3rd rows). On the other hand, AesStyler ensures color consistency and aesthetic harmony in the final results (2nd, 3rd, 4th and 5th rows). Finally, our method distinctly produces images with more vibrant colors and greater contrast (2nd and 5th rows), which are typically more pleasing to human eyes.Quantitative Comparison Quantitative Metrics. We present quantitative results in Table 1 . Gram loss, which computes the difference in mean and standard deviation between Gram matrices of style and stylized features, indicating the degree of style preservation. Structural Similarity Index Measure (SSIM) considers changes in structural information, [13] indicates the aesthetic level of the final results. Gram loss, SSIM and aesthetic score are all calculated with results produced by 1000 style and content image pairs. As to Deception Rate, following [32] , we train a VGG-16 network [35] on the WikiArt [31] dataset to classify the artist labels. The Deception Rate is calculated as the ratio of generated images that the network misclassifies as the original artworks of the artist who created the style image, which is computed using 18 style images, each emblematic of a distinct artist, paired with 300 content images, culminating in a total of 5,400 style transfer results for evaluation.Gram Loss & SSIM. As delineated in Table 1 , AesStyler has secured the best Gram loss and the second-best SSIM, indicating that AesStyler adeptly strikes a balance between style transformation and content preservation. Although AdaAttN achieved the highest SSIM score, its Gram loss performance is poor. As discussed in Section 4.2, AdaAttN tends to over-preserve content information, including elements related to color and style from the content images, which contributes to the style degradation problem. Furthermore, it is logical for the SSIM metric of AesStyler to be slightly lower than that of AdaAttN, as achieving aesthetic excellence necessitates the compromise of certain content fidelity.Aesthetic Score. As shown in Table 1 , AesStyler achieved the highest aesthetic score, indicating that our method produces results that align more closely with human aesthetic preferences. This is attributed to the strategic incorporation of aesthetic features into the style transfer process and the efficacy of the feature integration module. However, AesUST, another Universal Style Transfer method that takes aesthetics into account, clearly underperforms in aesthetic scoring. This shortfall is attributed to the Aesthetic Assumption Bias and the Style-Constrained Aesthetic Extraction, which we analyze and discuss in Section 1. Moreover, the quantitative results are also consistent with those shown in the qualitative comparison in Fig. 6 , which indicates that AesStyler, along with AdaAttN, renders results that are aesthetically superior to those of other style transfer methods.Deception Rate. AesStyler achieved best in the deception rate, signifying that through the assistance of both universal aesthetic features and style-specific aesthetic features, coupled with a novel feature integration mechanism, our model demonstrates a stronger capacity to produce results that bear a closer resemblance to authentic artist-created paintings. Besides qualitative and quantitative comparison, we conduct 2 user studies to further compare our method with baselines regarding the quality of style transfer and the aesthetic appeal of style transfer results. We choose AesUST [40] , AdaAttN [27] , MAST [7] , IECAST [3] and StyleFormer [43] as baseline methods in user studies due to their commendable qualitative and quantitative performance.In the first user study aiming to compare the style transfer quality, we invited 50 participants to respond to 15 questions. Each question presented a pair of content and style images alongside six images produced by AesStyler and the five aforementioned methods. Users were asked to rank the top-3 images that have the best style transfer quality, striking a good balance between style transformation and content preservation. The 1st, 2nd and 3rd images are given scores of 3, 2 and 1 respectively and others are scored 0. We calculate average scores for 6 methods and report them in Table 2 . From these results, we can conclude that AesStyler, with the support of aesthetic features and a novel feature integration mechanism, achieves almost full score (3), signifying its capacity to realize superior style transfer quality.In the second user study aiming to compare the aesthetic appeal of style transfer results, we invited 50 participants to answer 20 similar ranking questions. This time, however, participants were presented solely with style transfer results produced by six methods, as the intention is for the users to exclusively evaluate aesthetic appeals. The aesthetic scores for six methods are presented in Table 2 . From Table 2 , we can conclude that AesStyler produces results that are aesthetically more harmonious and pleasing.We conduct ablation studies on the UAC, TANet, USAesA, aesthetic discriminator and aesthetic loss. We show the qualitative results in Fig. 7 and report the quantitative results in Table 3 . All experimental settings follow as before.UAC, TANet and USAesA. Without UAC, TANet, USAesA means removing 𝐹 𝑢 , 𝐹 𝑎 and 𝐹 𝑢 &𝐹 𝑎 in style transfer process respectively. Without UAC, the results show some aesthetically unpleasant features originating from the given style image (3rd and 4th rows), which can be ascribed to the fact that, without UAC, AesStyler solely considers style-specific aesthetic features, thus bringing aesthetic deficiency in style images into final results. Without TANet, the results apparently lack aesthetic appeal (2nd and 4th rows). The absence of USAesA leads to the style degradation problem (2nd and 4th rows). Furthermore, the omission of UAC, TANet, USAesA does not improve the Gram loss and SSIM, which, by another measure, corroborates that the incorporation of these modules in guiding the style transfer process confers only benefits upon the results. Aesthetic Discriminator. Without the discriminator module, although the aesthetic scores of the results increase markedly in Table 3 , in Fig. 7 , the images display distinctly odd textures (especially evident upon zooming in on all rows), which we surmise serve as a rudimentary subterfuge to mislead the TANet into deeming the results aesthetically pleasing. Furthermore, as shown in Table 3 , the user study also reveal that, in the absence of D 𝑎𝑒𝑠 , the results demonstrate a reduced aesthetic appeal.Aesthetic Loss. Without the aesthetic loss term, the results generated by our method apparently show lower aesthetic appeal than results of the full model with weird big dark patches, apparent in the 3rd, 4th and 5th rows. Besides, removing the aesthetic loss term also causes the style degradation problem, resulting in failing to capture the true style of style images. 
conclusion: 
 In this paper, we propose AesStyler, a novel Aesthetic Guided Universal Style Transfer method. Our AesStyler, by utilizing TANet [13] as the aesthetic feature extractor, can accurately capture aesthetic features that resonate with human aesthetic preferences. Secondly, we propose to build a Universal Aesthetic Codebook (UAC) to harness universal aesthetic features that encapsulate the global aspects of aesthetics and to employ these features to guide the style transfer process. Thirdly, we propose Universal and Style-specific Aesthetic-Guided Attention (USAesA) module, which empowers our model to adaptively and progressively integrate both universal and style-specific aesthetic features with the style feature and incorporate the aes-enhanced style feature into the content feature. Extensive experiments and user studies have demonstrated the superiority of our method. Compared to state-of-the-art methods of both aesthetic-free and aesthetic-aware, AesStyler yields results of superior aesthetics and better style transfer quality.  
examples of aesthetic assumption bias: 
 AdaIN [15] and ArtFlow [1] ) and local patch-based (e.g. SANet [30] , MAST [7] , AdaAttN [27] , and IECAST [3] ). Although these methods achieve good results they entirely disregard aesthetic aspects of images, making them frequently generate results with aesthetically disharmonious patterns.To mitigate this issue, AesUST [40] proposed to add a discriminator to extract aesthetic features to enhance the aesthetic aspects of style features. However, sometimes AesUST still yields results with evident aesthetic disharmonies and artifacts. This is due to the following problems: (1) Aesthetic Assumption Bias: In the training stage, the aesthetic discriminator of AesUST lacks explicit supervisory signals to define aesthetics, instead it presumes that images from the style training dataset inherently possess aesthetics-a presumption that is not guaranteed to be accurate as shown in Fig. 1 . This can lead to it acting more as a style feature extractor, potentially overlooking true aesthetic elements. (2) Style-Constrained Aesthetic Extraction: AesUST restricts aesthetic feature extraction to style images, leading to a narrow, style-specific aesthetic perspective. However, aesthetics generally have universal qualities and shouldn't be confined as style-specific. (3) Indiscriminate Feature Fusion: attention scores recalibrate high layer feature maps of aesthetic and style features. However, these modified features are then merged with content features without considering the differences in feature distributions and information from lower layers.Inspired by the recent work TANet [13] , which trains an aesthetic assessment model on a large-scale dataset with human-assessed aesthetic scores for precise aesthetic score prediction, we propose to leverage the strong capability of pre-trained aesthetic assessment model in accurately discerning aesthetic features to guide the style transfer process.To this end, we propose AesStyler, a novel Aesthetic Guided Universal Style Transfer method. Firstly, we propose to utilize TANet as the aesthetic feature extractor in AesStyler. Secondly, we propose to build a Universal Aesthetic Codebook (UAC), to harness and utilize universal aesthetic features which encapsulate the global aspects of aesthetics. Thirdly, we propose the Universal and Style-specific Aesthetic-Guided Attention (USAesA) module. USAesA empowers our model to adaptively and progressively integrate both universal and style-specific aesthetic features with the style feature and incorporate the aes-enhanced style feature into the content feature. Extensive experiments and user studies have demonstrated the superiority of our approach. Compared to previous methods, our AesStyler not only yields results of superior aesthetics but also with better style transfer quality.In summary, our contributions are threefold:• We propose AesStyler, a novel Aesthetic Guided Universal Style Transfer method. We introduce the pre-trained aesthetic assessment model into UST task as the aesthetic feature extractor to accurately capture aesthetic features that resonate with human aesthetic preferences.  
 
 Appendix:  


Decision: The paper received three Borderline Accept ratings. While the reviewers acknowledged the novelty and potential impact of the proposed method, they also identified several areas for improvement.

Reviews: review: paper_summary: paper_summary: This paper presents a new Aesthetic Guided Universal Style Transfer method, which includes UAC and USAesA modules to accurately
capture aesthetic features that universally resonate with human aesthetic preferences. Extensive experiments and user studies have demonstrated the superiority of the proposed method.
strength_weakness: strength_weakness: - The idea of this paper is interesting.
- The structure of this paper is well organized.
limitations: limitations: - Recently, diffusion models have achieved very good results in image generation and editing, and the authors need to explain the superiority of the proposed framework compared to diffusion models.
- The authors used the pre-trained aesthetic assessment model in the UST task as the aesthetic feature extractor. However, different methods may have different results. The authors need to explain why TANet is adopted instead of other methods such as NIMA [1], TAVAR [2] and VILA [3].

  [1] NIMA: Neural Image Assessment, TIP.

  [2] Theme-aware Visual Attribute Reasoning for Image Aesthetics Assessment, TCSVT.

  [3] VILA: Learning Image Aesthetics From User Comments With Vision-Language Pretraining, CVPR.

- The proposed method seems similar to AesUST, and the authors need to explain the differences between the two, including motivation and methodology.
- There is no standard format for references; for example, authors' names and conference abbreviations need to remain consistent.
- The experimental results show that the proposed model has a good effect. To improve the contribution, it is recommended that the author open-source the code and related weights.
paper_summary: paper_summary: In this paper, a general image style transfer method based on aesthetic guidance is proposed to improve the beauty and harmony of the result of style transfer. To this end, the author introduces an aesthetic evaluation model in the process of style transfer, which captures the characteristics of human aesthetic preference resonance by establishing a universal aesthetic codebook. In addition, an attention module is constructed to guide the integration of general aesthetic characteristics and specific stylistic aesthetic characteristics. Experiments and user studies have shown that the method produces state-of-the-art results aesthetically.
strength_weakness: strength_weakness: 1.	The paper is easy to follow;
2.	The performance of the proposed method seems great;
3.	The strengths of this paper lie in the originality of its ideas and the network design developed based on these concepts.
limitations: limitations: The author's reference to AesUST's work in section 1, where there is a problem with aesthetic assumption bias, refers to the fact that the images in the style training dataset are inherently aesthetic. In the work of this paper, the design of this network is also designed with this assumption in mind, and I am not sure why UAC is used.
	In the first phase of the USAesA module mentioned in Section 3.2, the author uses general aesthetic features to enhance specific aesthetic features, does the author think that general aesthetic features can better express aesthetics, what is the difference between general aesthetic features and specific aesthetic features, and what is the significance of this? It is recommended to emphasize this point.
	Do the content loss and style loss described in Section 3.3 correspond to L_lf and L_gs in Figure 2 ? If yes, it is better to keep them consistent; if not, please explain L_(lf ) and L_gs.
	Since the aesthetic evaluation model introduced in this work is designed based on TANet, it is better to use different aesthetic evaluation methods in the aesthetic score evaluation of quantitative experiments, which is more persuasive.
	It is recommended to evaluate the effectiveness of each component of USAesA in ablation experiments.
paper_summary: paper_summary: The manuscript introduces "AesStyler," a novel approach for aesthetic-guided universal style transfer. Building on the existing TANet model, AesStyler enhances style transfer outputs by integrating aesthetics assessed by humans. This is achieved through the Universal and Style-specific Aesthetic-Guided Attention (USAesA) module, which merges universal and style-specific aesthetic features with traditional style features.
strength_weakness: strength_weakness: The integration of a pre-trained aesthetic assessment model is a clever solution to the common issue of aesthetic assumption bias in style transfer. This allows for a more nuanced capture of aesthetic features that align with broad human preferences.
limitations: limitations: 1-	The reliance on TANet for extracting aesthetic features could restrict the model’s adaptability to new datasets or evolving definitions of aesthetics.

2- TANet is just used as an aesthetic feature extractor, try something new, for example Q-Align[1].
    [1] Wu et al. Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels, The Forty-first International Conference on 
        Machine Learning(ICML), 2024.

3-	The use of VGG as the backbone architecture raises questions. Exploring or justifying the choice over potentially more suitable architectures like ResNet, ResNext, ViT and Mamba could strengthen the manuscript.

4-	The core methodology primarily extends TANet without substantial modifications tailored to style transfer, potentially weakening the novelty of the approach.

5-	The manuscript details an experimental procedure involving resizing images to 512x512 pixels and randomly cropping them to 256x256. This raises concerns about whether such transformations could alter the aesthetic characteristics of the images. The review does not address how the method ensures the aesthetic properties remain consistent post-transformation, which is crucial for validating the effectiveness of the aesthetic integration. This oversight might impact the perceived reliability of the results, and further clarification on this aspect would strengthen the manuscript.

6-	While AesStyler excels in enhancing aesthetic appeal, it does not perform as well on the Structural Similarity Index Measure (SSIM) compared to AdaAttN, indicating room for improvement in maintaining structural integrity during style transfers.
paper_summary: paper_summary: The authors presented a new aesthetic guided style transfer method, called AesStyler.

(1) Utilizing TANet as the aesthetic feature extractor, AesStyler effectively captures aesthetic features that align with human aesthetic preferences. 

(2) They develop a Universal Aesthetic Codebook (UAC), which aggregates universal aesthetic features representing the global aspects of aesthetics. These features are then used to direct the style transfer process. 

(3) They introduce the USAesA module. USAesA allows AesStyler to adaptively and progressively blend both universal and style-specific aesthetic features with the style feature, integrating this aes-enhanced style feature into the content feature for superior style transfer results.
strength_weakness: strength_weakness: The authors designed several aesthetic-aware modules to generate aesthetically more harmonious and pleasing results. The writing of this paper is logically coherent and the expression is easy to understand.
limitations: limitations: The most recent neural style transfer (NST) work cited in the related work section is from 2022, which is no longer considered state-of-the-art. It is advisable to compare more current NST methods, such as GAN-based and diffusion techniques, in terms of quality.

The primary focus of this paper is $\mathbf{aesthetic}$-guided universal style transfer (UST). To demonstrate the effectiveness of AesStyler, it is essential to compare it with more aesthetic-aware methods. Currently, only two such methods are evaluated, which may not sufficiently establish its advantages.

I suggest that the user study methods should be aesthetic-aware approaches, as these would be more convincing. Additionally, could you clarify the background of the participants? Those with a background in aesthetics would be more suitable. What exactly is the definition of aesthetic features? How participants evaluate aesthetic appeals.

It seems there is no explanation for the pictures in the second row of Figure 1in the main text.

In Phase 2: Aesthetic Style Guidance.  $F_{ga}^x$ is $F_a^x$?