Semantix: An Energy-guided Sampler for Semantic Style Transfer
Authors: Huiang He
Minghui Hu
Chuanxia Zheng
Chaoyue Wang
Tat-Jen Cham
Venue: ICLR 2025 Poster (2025)
Keywords: style transfer
diffusion model
energy guidance
Primary area: generative models
TL;DR: An Energy-Guided Sampler for Semantic Style Transfer across Images and Videos

Abstract: Recent advances in style and appearance transfer are impressive, but most methods isolate global style and local appearance transfer, neglecting semantic correspondence. Additionally, image and video tasks are typically handled in isolation, with little focus on integrating them for video transfer. To address these limitations, we introduce a novel task, *Semantic Style Transfer*, which involves transferring style and appearance features from a reference image to a target visual content based on semantic correspondence. We subsequently propose a training-free method, *Semantix*, an energy-guided sampler designed for Semantic Style Transfer that simultaneously guides both style and appearance transfer based on semantic understanding capacity of pre-trained diffusion models. Additionally, as a sampler, *Semantix* can be seamlessly applied to both image and video models, enabling semantic style transfer to be generic across various visual media. Specifically, once inverting both reference and context images or videos to noise space by SDEs, *Semantix* utilizes a meticulously crafted energy function to guide the sampling process, including three key components: *Style Feature Guidance*, *Spatial Feature Guidance* and *Semantic Distance* as a regularisation term. Experimental results demonstrate that *Semantix* not only effectively accomplishes the task of semantic style transfer across images and videos, but also surpasses existing state-of-the-art solutions in both fields.

Published as a conference paper at ICLR 2025
SEMANTIX: AN ENERGY-GUIDED SAMPLER FOR SE- MANTIC STYLE TRANSFER
Huiang He
Minghui Hu
South China University of Technology mshuianghe@mail scut. edu.cn
SpellBrush & Nanyang Technological University e2000080e ntu. edu.sg ag
Chuanxia Zheng
Chaoyue Wang
VGG, University of Oxford cxzhengerobots. ox ac. uk
The University of Sydney chaoyue. wargeout com
Tat-Jen Cham
College of Computing and Data Science Nanyang Technological University ASTJCham@ utu. www.tu..uuu....... edu.sg
ABSTRACT
Recent advances in style and appearance transfer impressive, but most methods isolate global style and local appearance transfer, neglecting semantic correspon- dence. Additionally, image and video tasks are typically handled in isolation, with little focus on integrating them for video transfer. To address these limitations, we introduce a novel task, Semantie Style Transfer. which involves transferring style and appearance features from a reference image to target visual content based on semantic correspondence. We subsequently propose training- -rree method, Semantix, anergy-guides sampler designed for Semantic Style Transfer that simultaneously guides both style and appearance transfer based on semantic understanding capacity of pre-trained diffusion models. Additionally, pler, Semantix can be seamlessly applied to both image and video models, enabling semantic style transfer to be generic across various visual media. Specifically, once inverting both reference and context images or videos to noise space by SDEs, Semantix utilizes meticulously crafted energy function to guide the sampling process, including three key components: Style Feature Guidance, Spatial Feature Guidance and Semantic Distance as regularisation term. Experimental results demonstrate that Semantix not only effectively accomplishes the task of semantic style transfer across images and videos, but also surpasses existing state-of-the-art solutions in both fields.
sam-
INTRODUCTION
The vision community has rapidly improved the quality of image and video generation over short period. In particular, some powerful bascline systems for Text-to-Image (Podell al., 2023; Saharia et al., 2022; Nichol et al., 2021; Rombach et al., 2022; Ramesh et al., 2022) and Text-to- Video (Blattmann et al., 2023; Guo et al., 2023; Girdhar et al., 2023) have been proposed, which contributes series of applications such controllable generation in ControlNet (Zhang et al., 2023a; Hu et al., 2023), IPAdapter (Ye et al., 2023) and InstantID (Wang et al., 2024c), Among these innovations, significant field is visual transfer, which modifies a context image to fit the style or appearance of the reference image, while preserving the original content or structure.
Prior works have extensively explored visual transfer. However, they typically focus on two distinct scenarios: (1) Style transfer (Gatys et al., 2015; 2016; Huang and Belongie, 2017; Li et al., 2017; Liu et al., 2021; Deng al., 2022; Wang al., 2023b; 2024a; Ye al., 2023) that utilizes the global stylistic features for the entire image style modification, but overly emphasize the overarching style of the reference images; and (2) Appearance transfer (Isola al., 2017: Zhu al., 2017; Park et al., 2020a;b; Zheng et al., 2021; Mou et al., 2023; Alaluf et al., 2023; Wang et al., 2024b) that conveys
"cqual contribution

---

Published as a conference paper at ICLR 2025
Rdoces
Came trugr
teeth
Gonar Vidoo
Reforas Iina
Concen
Figure 1: Examples of our Semantix. Given a visual context and reference image (Top examples), Semantix can perform Semantic Style Transfer based on the semantic correspondence. Besides, our Semantix also can be directly adapted for the videos (Bottom examples) without the need of additional modification. It is important emphasize that, sampler, Semantix directly leverages the knowledge from the pretrained model to guide the sampling process based on our proposed energy function for Semantic Style Transfer, without the need for any additional training or optimization.
the object appearance from the reference to the context image, but exhibits only limited sensitivity to the overall perceptual style. Besides, both of them ignore semantic alignment and video continuity during the transfer process. These factors destroy the video continuity and lead to global style transfer risking content leakage, while local appearance transfer may disrupt structural integrity.
We observe that the tasks of style transfer and appearance transfer share similarities in their underlying objectives: to transfer relevant information from the reference visual content to the context visual content. We assume that a feature transfer task guided by semantic alignment can better integrate these two tasks, mitigating the risks of content leakage and structural disruption. Thus we define task termed Semantic Style Transfer as: given context visual content, e.g., image or video, and reference image with style and (or) appearance features, the objective of Semantic Style Transfer is to analyse and transfer the features from the reference image to the context visual content through precise semantic mapping. Specifically, semantic style transfer considers semantic correspondence between context and reference images. When there is a clear semantic correspondence between the context image and the reference image, the transfer is executed based on the semantic correspondence. For instance, shown in Fig. the body of the giraffe and the zebra exhibit semantic correspondence,
2

---

Published as a conference paper at ICLR 2025
then the visual features of the body of zebra will be applied to the body of giraffe. Conversely, when semantic correspondence is weak, as between village in Van Gogh's 's style and the giraffe, the style is injected based on the other correlations, e.g., color information positional information.
To achieve this goal, we propose Semantix, an energy-guided sampler, which leverages the strong semantic alignment capabilities of pre-trained diffusion models (Epstein et al., 2023) to transfer features from the reference image to the context visual based on the semantic correspondence without Sepierrenn-Siiggggggle et al., 2023;
any training or optimization. Initially, we employ SDE Inversion Nie et al., 2023) to invert given content into the noise manifold, establishing a conducive foundation for semantic style manipulation. We then introduce a specialized energy function for semantic style transfer that guides the sampling process rather than modify the model structure (Alaluf et al., 2023), thus maintaining the original capabilities of the visual models and supporting video continuity. Our proposed energy function comprises three terms: i) Style Feature Guidance, to align the style features with the reference image; ii) Spatial Feature Guidance, to maintain spatial coherence with context; and iii) Semantic Distance, to regularise the whole function. In particular, within the term of style feature guidance, we initially leverage semantic correspondence in the diffusion model (Tang et al., 2023) and position encoding (Vaswani et al., 2017; Dosovitskiy et al., 2020) to align the output and style features. In the spatial feature guidance component, we directly consider the feature distances at corresponding positions to ensure consistency between the generated content and the context. Lastly, we utilize the distance between the cross-attention maps of the generated content and the given context as a regularisation term enhance stability.
Integrating these capabilities, Semantix effectively transfers the features with semantic from the reference image the context with precise spatial alignment, satisfying the requirements of Semantic Style Transfer. Additionally, as Semantix functions as sampler for diffusion models, it can be seamlessly applied to various image or video base models. Since these pre-trained models already encapsulate sufficient semantic information and maintain action coherence, Semantix enables training- free semantic style transfer across image and video simply through guided sampling.
In summary, our contributions include the following key points:
We introduce Semantix, an energy -uidded sampler specifically designed for training free semantic style transfer. We further extend Semantix across images and videos, demonstrating the versatility of energy-guided samplers.
Experimental results demonstrate that Semantix yields superior results training-free semantic style transfer across images and videos, surpassing existing solutions in both style and appearance transfer in terms of accuracy and adaptability.
2 RELATED WORKS
Style and Appearance Transfer Style transfer infuses the style information of a reference image into context image to synthesize stylized context image. Previous convolution-base meth- ods (Huang and Belongie, 2017; Gatys et al., 2016; Johnson et al., 2016; Li et al., 2017; 2018; Park and Lee, 2019; Lai et al., 2017; Gu et al., 2018) and Transformer-based methods (Deng et al., 2022; Wu et al., 2021; Wang et al., 2022; Liu et al., 2021) have successfully facilitated style transfer through the fusion of style and context information. Recently, diffusion-baser style transfer has seen significant attention and progress. Some works (Ye et al., 2023; Wang et al., 2024a; 2023b; Sohn et al., 2023) use additional trained networks to extract style features for guiding image synthesis. These methods inject global style features into the context image, altering its overall color and brush strokes but ignoring the necessary semantic correspondence for meaningful style transfer. Other training-based based methods (Ruiz et al., 2023a;b; Shi et al., 2023; Wei et al., 2023; Li et al., 2024) require fine-tuning the diffusion model to learn specific styles or introducing new text embeddings for style representation through textual inversion (Gal et al., 2022; Zhang et al., 2023b). These methods are time-consuming and often struggle to balance style injection with context preservation. Some studies (Qi et al., 2024; Wang et al., 2023a; Jeong et al., 2023; Frenkel et al., 2024; Gandikota et al., 2023) attempt to achieve style transfer through decoupling, but they either face challenges in acquiring paired style datasets risk losing image context. Contrastive learning (Yang et al., 2023; Chen et al., 2021: Zhang et al., 2022; Park et al., 2020a) has also been employed, requiring carefully designed loss functions for optimization. Howerver, all these methods need additional training. Some
3

---

Published as a conference paper at ICLR 2025
training-free methods (Chung al., 2023; Deng al., 2023; Hertz et al., 2023; Jeong et al., 2024) induce style transfer by manipulating features within the attention blocks of the diffusion model.
Mcanwhile, appearance transfer aims map appearance from image another. Early methods based on GANs (Isola et al., 2017; Zhu et al., 2017; Yi et al., 2017) have faced practical limitations. Other approaches using VAEs (Park al., 2020b; Liu al., 2017; Jha al., 2018; 88; Pidhorskyi et al., 2020) encode images into separate structure and appearance representations to combine features from different images. Recent diffusion-based techniques have significantly advanced appearance transfer (Mou et al., 2023; Epstein et al., 2023; Kwon and Ye, 2022; Alaluf et al., 2023; Wang et al., 2024b). However, these approaches utilize local appearance features for guidance, overlooking precise semantic correspondence between images, leading to inaccurate appearance transfer.
While related dense style transfer work (Ozaydin al.) introduces the concept of semantics, our definitions and tasks differ significantly, particularly in how semantic features are extracted and used. Specifically, unlike some semantic style transfer methods relying on segmentation labels (Shen et al., 2019; Bhattacharjee et al., 2020) or CLIP visual features (Ozaydin et al.), they lose natural language alignment. Furthermore, such existing works only extract dense semantic features and reuse it in given image. Instead, our work defines a more expansive zero-shot semantic style transfer task focused on generation. In detail, rather than merely reusing the low-level features from a given image, e.g., color, leveraged powerful pre-trained diffusion model to generate novel style images that adhere to semantic constraints derived from the features of the given image.
Semantic Correspondence between Images Establishing semantic correspondence between im- ages is crucial. Past methods (Zhang et al., 2021; Zhao et al., 2021) using supervised learning require extensive annotations. To address data limitations, some works (Wang al., 2020; Rocco al., 2018; Lee et al., 2019; Seo al., 2018) use weakly supervised approaches. Recently, supe-supe learning has gained attention, especially with diffusion models. Studies (Tang et al., 2023; Zhang al., 2024; Caron et al., 2021) leverage these models representation abilities calculate feature similarities and establish semantic correspondence during the denoising process.
Energy Functions Previous research interprets diffusion models as energy- models (Liu et al., 2022), where the energy function guides the generation process for precise outputs. The energy function has various applications, such as energy-guided image editing (Mou et al., 2023; 2024; Epstein et al., 2023) and translation (Zhao et al., 2022). The energy function also shows potential in controllable generation, guiding generation through conditions such as sketch (Voynov et al., 2023a), mask (Singh et al., 2023), layout (Chen et al., 2024), concept (Liu et al., 2022) and universal guidance (Bansal et al., 2023; Yu et al., 2023), enabling precise control over the output.
3 PRELIMINARIES
Energy Function Diffusion models can be viewed as score-based generative models (Song et al., 2020b). In classifier guidance (Dhariwal and Nichol, 2021; Ho and Salimans, 2022), the gradient of the classifier V,, log P ()((())) is used to influence generation. From the perspective of score functions, the condition y can be integrated within conditional probability g(xely) via an auxiliary score function and expressed as such:
Va, log g(xely)) Va, log (q(((2))((11) q(y)
Vz, ((zz) + V Nx log (M)l),)
(I)
where the first term is viewed as the unconditional denoiser co(;;; t, 0), and the second term can be interpreted as the gradient of the energy function: E(x;t, y) = log (v)lz.). Alternatively, classifier-free guidance (CFG) (Ho and Salimans, 2022) can also be used, expressed as:
é (1 ( t, y) WEO (x,;
where w is the classifier-free guidance strength. In fact, the diffusion model can also be interpreted as an energy-based model (Liu et al., 2022), guided by any energy function. One can design an energy function beyond blase-based conditioning and use it for guidance (Zhao et al., 2022; Epstein et al., 2023; Bansal et al., 2023; Chen et al., 2024: Yu et al., 2023; Voynov et al., 2023a; Kwon and Ye, 2022). Such guidance provides directional information guide the diffusion process, and if appropriately
4

---

Published as a conference paper at ICLR 2025
Repoat Binss
Poilien Exosding
Festure Afline with DIFT PE
DIDM
Rebweoee Fostures F,
Postice Baodds
M
Stele Guidarcce
Calcolate paired
Reference Inaag
X,9
Spotial Guidlacce
17 7 m 3pp
Contoct Foctunne
Semantic Distance
Farad Rekreeze
Chipus Featarce
Sputial Guidinnce
Strap Icy
Content Imagee Vidco
X/
Ouipat
F.
Cantent Feureer
Ourpue Frature
Desporyy
INtrance
Inapped
Prolicken
X,t=X;
Figure 2: Overview of Semantix. Given a reference image Iref and a context image I or video Vo, we first invert them to the latent 2r through edit- fiit-friendly DDPM inversion. In the denoising process, we then modify the z!" through the designed energy gradient in every sampling step.
designed, shall show in this paper, it can be used to preserve semantic structure while adjusting the style or appearance of the context visual according to the reference image. Following from Eq. 2, the guidance from energy function be expressed follows:
& = (1 + (a) y) weg (x,, 0) + + E.,
(3)
where the classifier-fres guidance strength, and y the newly introduced guidance weight for the energy function E(x;;t, t, y). In this section, 2/, o, 8, 0 represent diffused signal, classifier parameters, noise estimator parameters, null token respectively.
METHODS
To address the task of Semantic Style Transfer, it is essential to establish precise semantic mappings between the content to be transferred and the target. And the style features and visual appearance are required to be transferred with the guidance from semantic correspondence, while preserving the main structure of the original content.
To achieve this, we propose Semantix, a novel energy-guided sampler built upon off-the-shelf diffusion models, illustrated in Fig. Given context image I video Ve and the reference image /ref, we begin with inverting these images or videos to latents through the edit-friendly DDPM inversion (Fig. (left)). In the denoising process, we then modify the target x:!ut which is initialized using the inverse noise of the context visual zG at the final step T, through the designed energy gradient in every sampling sFep (right)). During the sampling process, we integrate the guidance from our sampler with classifier-fre free guidance to generate high-quality samples (Liu et al., 2022; Epstein et al., 2023). Notably, our approach only provides additional guidance during sampling, without altering the generative capabilities of the visual model. As result, the pre-trained video model inherently maintains motion consistency without additional modification. Besides, an AdaIN (Huang and Belongie, 2017) is employed to harmonize color disparities among tout and !nef, which is also widely used in recent works (Alaluf al., 2023; Chung et al., 2023). The algorithm details can be found in Alg. 1.
5

---

Published as a conference paper at ICLR 2025
4.1 DDPM INVERSION
To enhance image or video style editing capabilities, we first employ DDPM inversion (Huberman- Spiegelglas et al., 2023) invert the input the noise space as outlined Appendix A. This method significantly reduces the reconstruction errors associated with CFG. As shown in Fig. 2 (left), given a reference image _ref and context image I or video Vs, we first derive the respective independent inversion noise sequences {x/T") 2 Ie or Ve in the forward process. As shown in Fig. 3, at / = 601, the features we extracted from the diffusion model contain sufficient contextual information and exhibit precise semantic correspondence between context and reference images. Thus, revert the images the T 601 timestep. Once obtained noise sequences, these noise maps are fixed for using in the guided sampling process later. We then initialize TT 21 20ll a 2f and only manipulate the predictions from this sequence.
zref -12....... 21 ref] for r and { 27, 2f-1...... =f} for
Visualizing feature maps. We extracted features from the second block of the diffusion model decoder and visualized the top three PCA components and feature mapping at each timestep.
Figure 3:
4.2 DESIGN OF ENERGY FUNCTION
Inspired by guidance-based image generation and editing methods (Epstein et al., 2023; Mou et al., 2023), we regulate the sampling process through the design of an energy function to achieve semantic style transfer. By leveraging the gradient of the energy function, we can guide the context visual toward the desired style while preserving its structural integrity. Eq. 3 can be re-expressed follows:
é = (I + w)eef (xy; t, C) Wep (x,; t, o) + Vz, F(rritt)
(4)
where is the classifier-freee guidance strength and F is the designed energy function which in- cludes three parts: Fref for Style Feature Guidance, F. for Spatial Feature Guidance, and Freg for regularisation:
Flz = Frej + yeFe + Freg
(5)
where y is the weight of corresponding components. We detail the three components below.
Style Feature Guidance To achieve semantic style transfer, we propose style feature guidance, which accurately captures the semantic correspondence between reference and context features to guide style feature injection. In DIFT (Tang et al., 2023), the authors observed that the internal features of the pre-trained Stable Diffusion model can used to establish accurate semantic correspondence.
Inspired by it, we initially feed the inversion latents xf, _i ref and the infusion latent xt to the diffusion model and acquire the corresponding feature maps FF Ereef & Foout E R (B,'',,'' (,)) from the intermediate layers of the network. For videos, feature maps Pe & Pood are extracted from each individual frame I E Vo, Iref E Vref_ Next, we pre-align the features via DIFT. Specifically, given the features Fr. Frof and pixels Pt, Pj in Te, fref respectively, we calculate the l, distance between the pairwise vectors vs, from F. and ref from Free and then define the pixels with the smallest distance as the corresponding pixels:
Dij IVs
vref
Vy E Fe, Vyref E rref
(6)
p arg min Dig. P,
(7)
Therefore, for any context feature vector of the context image vs, F., the corresponding feature pair can be defined as (v vre For all v in FY, we can identify corresponding feature pairs. Consequently, obtain new set Fres+ according corresponding feature pairs.

---

Published as a conference paper at ICLR 2025
mee- Stylereeneer
CVN-based
Diffusion-bueet
Mechods
AdaiAii
Sky
CAST
OCPL
16
Digge
Choo-lmtge
StyCIl
DecomSifl 2015
Tinndall
Ours
Figure 4: Qualitative comparison with style transfer and appearance transfer methods. The top two rows are comparisons of style transfer, the bottom of appearance transfer.
MCCNet
UNIST
Cross-Image
CCPL.
Ours
MCCNet
UNIST
Cross-Image
CCPL
Ours
Figure 5: Qualitative comparison of video style transfer. Click the images to play the animation clips. (Recommended to use Adobe Reader ensure the GIFs play properly.)
However, such an affinity-based approach overlooks the spatial location of the vectors within the context features and fails to account for their relationships with adjacent vectors. Inspired by Position Encoding (PB) (Vaswani al., 2017; Dosovitskiy et al., 2020), we integrate additional optimizing- free position encoding term to maintain relative position. Thus the feature maps can regarded as:
((n) Fe <- Fr. + Ape pe
(8)
Fref +- Fres + Xpp ps
(9)
Subsequently, we locate the corresponding pairs of features Fres+ using the lo distance, and utilize the obtained new pairs of feature vectors (vg, rre)) to form the new rearranged pref+ Therefore, the optimization objective for Style Feature Guidance is minimize:
Fref x d [reut FFe"t
(10)
where d is a type of distance metric. We use the cosine similarity as implemented in (Mou et al., 2023). Besides, we use self oeffaattention (Tumanyan al., 2023) to define region masks m. and myef- which are generated via h-means clustering and applied to the features, limiting calculations and guidance to the masked regions.
To preserve the spatial structure of the generated content during style guidance, we introduce spatial feature guidance. minimizes the distance between context and output features, ensuring spatial integrity. Unlike previous methods that replace features, calculate feature distances at corresponding positions during sampling and design an energy function to align and maintain spatial structure. Specifically, based the features F. extracted from the context
Spatial Feature Guidance
7

---

Published as a conference paper at ICLR 2025
image video, perform point-to-point feature mapping between F and Ft to obtain feature vector pairs (yp, "p.") where p, E FÇ. Therefore, we can calculate the similarity between the output features Frut and the context features F as Spatial Feature Guidance:
Fe x d (Fout F().
(11)
Semantic Distance To avoid overfitting to style or context and achieve a balance between style and structure, incorporate a commonly used regularisation term in training-based methods. Previous methods have demonstrated that self-attention and cross-attention mechanisms encode context and structural information of images (Hertz al., 2022; Epstein et al., 2023; Tumanyan al., 2023). Additionally, recent methods have achieved style transfer by swapping the keys and values in self- attention (Alaluf et al., 2023; Chung et al., 2023; Deng et al., 2023; Hertz et al., 2023; Wang et al., 2024b; Jeong et al., 2024). Inspired by these works, we design a regularisation term to balance style injection and context preservation. Specifically, the parameters of style and spatial guidance are their respective feature vectors, while the swapped attention features combine the outputs of both. By constraining and penalizing the parameters from the swapped attention features in the regularization terms, it ensures that the style and spatial feature vectors remain within a more compact space, resulting in a smoother solution, reduced overfitting, and enhanced stability during sampling. During the sampling process, we feed zzut into U-net again and replace the original KTnt, Viout in self-attention with KTeS, viro,, which come from the reference image pref, Expressed follows:
(Q K.rSft Vd
Self-Att(Q KTT, v.ree) = Softmax
ViTes
(12)
where is the index of the transformer blocks that need replace K and V. and is the dimension. Consequently, we calculate the L.2 distance between the cross-attention map Cross-Attn.ul out replacing K and V and the cross-attention map Cross-Attn' of the context image:
Stap after
Freg |Cross-Attn sg( (CrosssAAtt"))
(13)
where sg(+) represents the operation of stop gradient.
5 EXPERIMENTS
In this section, we conduct an exhaustive experimen- tal analysis substantiate the efficacy and superiority of our proposed method through qualitative comparison (Sec. 5.1), quantitative comparison (Sec. 5.2) and ablation study (Sec. 5.3). For more experimental details, please refer to the Appendix
5.1 QUALITATIVE COMPARISON
(a) Image Examples w/ Semantix
We compare our proposed method with previous state-of- the-art methods in style and appearance transfer, including Transformer-basee methods AdaAttn (Liu al., 2021). I), StyleFormer (Wu et al., 2021), Sty (Deng et al., 2022), CNN-based method CAST (Zhang et al., 2022), CCPL (Wu et al., 2022) and Diffusion-basee methods 1nST (Zhang al., 2023b). Dragon Diffusion (Mou al., 2023), Cross- Image (Alaluf et al., 2023), StyleID (Chung et al., 2023), DreamStyler (Ahn al., 2024). TI-Guided-Eate (Wang et al., 2024b). As illustrated in Fig. 4, our approach ex- cels at generating visually appealing images in both style and appearance transfer tasks. Specifically, our method not only preserves the structural integrity of the context image, but also integrates style features more effectively based on semantic correspondence. AdaAttn, Styleformer, StyTrr, CAST and CCPL mainly change the color without injecting
SA
(b) Video Examples w/ Semantix
Figure 6: Image and Video Examples for Semantic Style Transfer
8

---

Published as a conference paper at ICLR 2025
Table Quuntitative compurison with style tramsfer and appeurance transfer methods.
Metrics
AdnAtta
StyleFormer
StyTR2
CAST
CCPL
InST
Cross-Imagg
StyleID
DreamSttyler
TI-Guided
Ours
LPIPS
0.581
0,560
0.476
0.465
0.523
0.54K
0.703
0.514
0.580
0.649
0.461
CPSD
0.189
0.156
0.133
01400%
0.232
0.160
0.789
0.183
SSIM
0,403
0.514
0.385
0.454
0.527
0.334
Gram au
7,929
2,822
5,403
6.594
4.861
4,917
5,800
2,878
6.990
4,811
2.524
PickScore
16.87
18,55
16.76
16.72
16.75
16.80
17.45
19.68
16.30
18.39
19.95
HPS
16.81
18.30
16.81
16.77
16.79
16.87
16.59
18.70
16.87
17.56
18,78
compane highiighted bold
second-bes inderlined proservation, style vimilarity image
proarsation capubility caliaalttc CFSD SSIM.
For style similarit Gram Metrics style lea.
We PickSoore HPS acstheiic ovaluation
the whole style information. InST and DreamStyler fail to effectively learn style and disrupt the context image. Dragon Diffusion employs global feature guidance for appearance transfer, yet fails to maintain structural integrity, causing severe distortion of image context. Meanwhile, Cross- Lross-Image leads to significant structural degradation and results in blending of foreground and background. StyleID unable to infuse enough style information, potentially leading to color deviation. Fur- thermore, Tl-Guided Edit struggles with semantic-driven style transfer, causing blurry images and structural damage. For video style transfer, we compare our method with previous video style transfer methods such as MCCNet (Deng et al., 2021), UNIST (Gu et al., 2023) and CCPL (Wu et al., 2022). We also adapt Cross-Image (Alaluf et al., 2023) to a video version for comparison. As shown in Fig. 5, method outperforms others in terms of visual quality, consistency, and continuity. Additional qualitative comparisons of images are provided in the Appendix G. Besides, we provide image and video semantic style transfer results shown in the Fig. 6, as well as additional visual results in the Appendix I.
5.2 QUANTITATIVE COMPARISON
Metrics. For quantitative performance of image semantic style transfer, evaluate stylized images from three aspects: structure preservation ability, style injection capability and image quality. We quantify the structure preservation ability by LPIPS (Zhang al., 2018), CFSD (Chung al., 2023) and SSIM (Wang et al., 2004). For style injection capability, we employ Gram matrices, which are widely used in style and appearance transfer (Gatys et al., 2016; Alaluf al., 2023; Wang al., 2024b). In addition, we utilize the aesthetic score metrics to measure the quality of the generated images such as PickScore (Kirstain al., 2024) and Human Preference Score (HPS) (Wu al., 2023). For video transfer, we evaluate video continuity with Semantic and Object Consistency metrics and Motion Alignment metric for assessing the difference of motions between source and editing videos as described in (Sun et al., 2024). We further evaluate the visual appeal, motion quality and temporal consistency by Visual Quality, Motion Quality and Temporal Consistency metrics respectively as described in EvalCreafter (Liu al., 2024).
Comparison Analysis We quantitatively eval- uate our method on 1000 sampled context-style
Tale: Quantitaive Comparison of Video Style Transfer.
image pairs and compare it with previous state-
Matric
MCCNat UNIST
Cres-Imux
CCPL.
of-the-art methods mentioned above. As demon-
Semantic
0.714
0.936
0,922
strated in Tab. 1, our method not only signif- icantly surpasses previous techniques but also excels over recent diffusion- methods in several context preservation metrics, including LPIPS, CFSD and SSIM. It indicates that our method possesses a significant edge in maintaining structural integrity. For the style similarity metric, we measure the L.2 loss between Gram Matrices (Gatys et al., 2015), which represents the style similarity between the generated image and the reference style image. Our Semantix has the lowest Gram Matrices style loss among all methods. This demonstrates that the images generated by our method most closely resemble the reference style images. Besides, Semantix achieves the highest PickScore and HPS, which indicates that the images generated by Semantix attractive. Therefore, experiments illustrate that Semantix not only solves the problem of context disruption, but also effectively injects style into context images in harmonious way.
Object Consetency
Medion Aligiment
0.723
-:251
-4.178
0.771
0.939
-3378
+1,793
-1.991
0.955
Vicual Qualits
$2.11
45.97
47.35
48.01
55,86
Mation Quality
$1.35
35.07
Tomperal Consisency
59.14
55.35
59.64
The lest mabisan Hightghor holdd free and uwweedaae mall undiutiod
visually

---

Published as a conference paper at ICLR 2025
Ours (default)
Style Feature Guidance
Position Encoding
Spacial Feoture Guidnnce
Semantic Distance
Carcat
Reforose
Figure 7: Ablation study on our proposed components. From the 3rd to the 6th columns, each column has separate component removed compared to our default.
In terms of video semantic style transfer, we conducted evaluations across 100 stylized videos in comparison with MCCNet (Deng et al., 2021), UNIST (Gu et al., 2023), Cross-Image (Alaluf al., 2023) and CCPL (Wu et al., 2022). In contrast to Cross Image, our approach did not modify the model's 's structure, which allowed it to achieve commendable outcomes in generation quality, consistency and continuity aspects, shown in Tab. our approach exhibited superior performance across all metrics among all methods.
Tatl User Study.
Abiition stady proposed compotents.
Coanseage
Owr
Maticc
-Ryle Grbdince
-quatal Cuddanee
-uubisa Eaceding
-eewaati Blazare
Style Stailarity
Caskal Prourvatis
8.2%
31.0
5.2
33,8
1509
38.55
32%
51.3%
31.39
CTSD
A.11
0.116
0.156
E.136
0.131
Appeel
Matrox., an
ek entemnia Be sage lond mile ayeil mnlie
IIS
15,7600
12.945
16.355
16.199
16:358
16.108
16.258
14.500
16,30
14.76
Tris 05 10 wwliiiiaiii 2480 511 ROMMA CWI.GERE -on.e 8: (S086) 01800 drai))
User Study In order to obtain a more convincing comparison, we conducted a user study to compare our method with StyleFormer (Wu et al., 1), Cross-Image (Alaluf et al., 2023), StyleID (Chung et al., 2023) and TI-Guided (Wang et al., 2024b). We enrolled 30 volunteers, and for each volunteer randomly selected 40 generated images for user evaluation in three aspects: structural preservation, style similarity and visual appeal. The results, as detailed in the Tab. 3, demonstrate that our method excels maintaining image structure, enhancing style injection, and achieves superior image quality.
5.3 ABLATION STUDY
To validate the effects of each component, conducted series of ablation studies. In particular, we experiment with style feature guidance, spatial feature guidance, position encoding and semantic distance in both qualitative and quantitative aspects. As illustrated in the Tab. 4 and Fig. 7, style feature guidance injects style features into context visual based on semantic correspondence. Spatial feature guidance effectively maintains the structure, thereby ensuring the coherence of image context. Position Encoding enhances the harmony of stylized images. As regularisation term, semantic distance mitigates overfitting to structure or style and contributes to balancing style injection and structure maintenance. Further analysis is provided in Appendix E. Parameter sensitivity analysis and ablation on semantic distance can be found in Fig. 13 and Fig. 15 respectively.
6 CONCLUSION
We propose Semantix, a carefully crafted energy guided sampler designed to facilitate semantic aligmment feature transfer both image and video diffusion models. By utilizing three proposed components, Semantix effectively transferred both the style and appearance features with seman- tic guidance. Experimental results demonstrate that our approach not only produces high-quality stylized images and videos but also efficiently prevents contextual interference and effectively incor- porates style features. Moreover, video experiments confirm that our method excels in maintaining consistency and continuity across frames.
10

---

Published as a conference paper at ICLR 2025
LIMITATIONS
While our method achieves style and appearance transfer across various visual, there are still certain limitations. Specifically, it tends be less effective when the context visual has strong inherent style features. In such scenarios, our method may lead to failures in semantic style transfer. Moreover, Semantix relies on the semantic information from pre-trained diffusion models, with the UNet architecture (Rombach et al., 2022). Future work will explore semantic alignment in advanced models like UViT (Bao al., 2023) and DiT (Peebles and Xie, 2023).
BROADER IMPACTS
Powerful feature transfer capabilities can facilitate creative content generation. However, there is also a risk that these technologies could be used to generate harmful content with negative societal impacts. A particularly concerning misuse could involve placing an actual people in compromising scene involving illicit activities.
REFERENCES
Aishwarya Agarwal, Srikrishna Karanam, KJ Joseph, Apoorv Saxena, Koustava Goswami, and Balaji Vasan Srinivasan. A-star: Test-time attention segregation and retention for text-to-image synthesis. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2283-2293, 2023a.
Aishwarya Agarwal, Srikrishna Karanam, Tripti Shukla, and Balaji Vasan Srinivasan. An image is worth multiple words: Multi-attribute inversion for constrained text-to-image synthesis. arXiv preprint arXiv:2311.1191 2023b.
Namhyuk Ahn, Junsoo Lee, Chunggi Lee, Kunhee Kim, Daesik Kim, Seung-Hun Nam, and Kibeom Hong. Dreamstyler: Paint by style inversion with text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 674-681, 2024.
Yuval Alaluf, Daniel Garibi, Or Patashnik, Hadar Everbuch-Elor and Daniel Cohen-Or, Cross-image attention for zero-shot appearance transfer. arXiv preprint arXiv:2311.03335. 2023.
Arpit Bansal, Hong-Min -Mn Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 843-852, 2023.
Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 22669-22679, 2023.
Deblina Bhattacharjee, Seungryong Kim, Guillaume Vizier, and Mathieu Salzmann. Dunit: Detection- based unsupervised image-to-image translation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4787-4796, 2020.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
Haibo Chen, Zhizhong Wang, Huiming Zhang, Zhiwen Zuo, Ailin Li, Wei Xing, Dongming Lu, et al. Artistic style transfer with internal-external learning and contrastive learning. Advances in Neural Information Processing Systems, 14:26561-225777 2021.
Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free free layout control with cross- cooss-attention guidance. In Proceedings the pages 5343-5353, 2024.
the IEEE/CVF Winter Conference on Applications of Computer Vision,
11

---

Published as a conference paper at ICLR 2025
Jiwoo Chung, Sangeek Hyun, and Jac-Pil Heo. Style injection in diffusion: A training- free approach for adapting large-scale diffusion models for style transfer. arXiv preprint arXiv:2312.09008, 2023.
Yingying Deng, Fan Tang, Weiming Dong, Haibin Huang. Chongyang Ma, and Changsheng Xu. Arbitrary video style transfer via multi-channel correlation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1210-1217, 2021.
Yingying Deng, Fan Tang, Weiming Dong, Chongyang Ma, Xingjia Pan, Lei Wang, and Changsheng Xu. Stytr2: Image style transfer with transformers. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 1 11326-1 1136,, 2022.
Yingying Deng, Xiangyu He, Fan Tang, and Weiming Dong. Z* Zero-shot style transfer via attention rearrangement. arXiv preprint arXiv:231 16449) 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.119299 2020.
Dave Epstein, Allan Jabri, Ben Poole, Alexei Efros, and Aleksander Holynski. Diffusion self- guidance for controllable image generation. Advances in Neural Information Processing Systems, 36:16222-16239, 2023.
Yarden Frenkel, Yael Vinker, Ariel Shamir, and Daniel Cohen-Or. Implicit style-content separation using b-lora. arXiv preprint 2xXi:24433.1557. 2024.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashmik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint 22t.::2000001111 2022.
Rohit Gandikota, Joanna Materzynska, Tingrui Zhou, Antonio Torralba, and David Bau. Concept sliders: Lora adaptors for precise control in diffusion models. arXiv preprint arXiv:2311.12092 2023.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. A neural algorithm of artistic style. arXiv preprint 1XXiv:5508.06566 2015.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2414-2423, 2016.
Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factorizing text-to-video generation by explicit image conditioning. arXiv preprint arXiv:2311.10700 2023.
Bohai Gu, Heng Fan, and Libo Zhang.
Two birds, one stone: A unified framework for joint learning of image and video style transfers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages -23554,, 2023.
Shuyang Gu, Congliang Chen, Jing Liao, and Lu Yuan. Arbitrary style transfer with deep feature reshuffle. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8222-8222 8231, 2018
Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint vrxii::2007.04255. 2023.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Crhen-Or. Prompt- to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.0166 2022.
12

---

Published as a conference paper at ICLR 2025
Amir Hertz, Andrey Voynov, Shlomi Fruchter, and Daniel Cohen-Ou Or. Style aligned image generation via shared attention. arXiv preprint arXiv:2312.02133, 2023.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12599. 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Minghui Hu, Jianbin Zheng, Daqing Liu, Chuanxia Zheng, Chaoyue Wang, Dacheng Tao, and
Tat-Jen Cham. Cocktail: Mixing aulti-modality control for text-conditional image generation. In seirty-seventh Conference on Neural Information Processing Systems (NeurIPS), 2023.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normaliza- tion. In Proceedings of the IEEE international conference on computer vision, pages 1501-1510, 2017.
Inbar Saitermmnn-Siegglggha Vladimir Kulikov, and Tomer Michaeli. An edit friendly ddpm noise space: Inversion and manipulations. arXiv preprint arXiv:2304.06114) 2023.
Phillip Isola, Lun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125-1134, 2017.
Jaeseok Jeong, Mingi Kwon, and Youngjung Uh. Training-free free style transfer emerges from h-space in diffusion models. arXiv e-prints, pages arXiv-2303, 2023.
Jaeseok Jeong, Junho Kim, Yunjey Choi, Gayoung Lee, and Youngjung Uh. Visual style prompting with swapping self-attention. arXiv preprint arXiv:2402. /2944, 2024.
Ananya Harsh Jha, Saket Anand, Maneesh Singh, and VS Rao Veeravasarapu. Disentangling factors of variation with cycle-consistent variational auto-encoders. In Proceedings of the European Conference on Computer Vision (ECCV), pages 805-820, 2018.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Computer Fsion-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 694-711. Springer, 2016.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pie: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Gihyun Kwon and Jong Chul Ye. celfusiio-bbsed image translation using disentangled style and content representation. arXiv preprint arXiv:2209.15000 2022.
Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming Hsuan Yang. Deep laplacian pyramid networks for fast and accurate super-resolltion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 624-632. 2017.
Junghyup Lee, Dohyung Kim, Jean Ponce, and Bumsub Ham. Sfnet: Learning object-aware semantic correspondence. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2278-22877 2019.
Dongxu Li, Junnan Li, and Steven Hoi. Dlip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing. Advances in Neural Information Processing Systems, 36, 2024.
Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang. Universal style transfer via feature transforms. Advances in neural information processing systems, 30, 2017.
Yijun Li, Ming- Yu Liu, Xueting Li, Ming Hsuan Yang. and Jan Kautz. A closed-form solution to photorealistic image stylization. In Proceedings of the European conference on computer vision (ECCV), pages 453 2018.
13

---

Published as a conference paper at ICLR 2025
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised Advances in neural information processing systems, 30, 2017.
translation networks.
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision, pages 423-439. Springer, 2022.
Songhua Liu, Tianwei Lin, Dongliang He, Fu Li, Meiling Wang, Xin Li, Zhengxing Sun, Qian Li, and Errui Ding. Adaattn: Revisit attention mechanism in arbitrary neural style transfer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6649-6658, 2021.
Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and evaluating large video generation models. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22139-211 22149,222211 2024.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1138-6047, 2023.
Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang. Dragondiffusion: Enabling drag-style manipulation on diffusion models. arXiv preprint arXiv:2307.02421. 2023.
Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang. Diffeditor: Boosting accuracy and flexibility on diffusion-based image editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 8488 8497, 2024.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 12. 2021.
Shen Nie, Hanzhong Allan Guo, Cheng Lu, Yuhao Zhou, Chenyu Zheng, and Chongxuan Li. The blessing of randomness: Sde beats ode in general diffusion-based image editing. arXiv preprint arXiv:2311.01410, 2023.
OpenAl. Sora: Creating video from text. https: : /twp/aippaaaaa///////////....... caciinaaa.coommmmomm coooo/ooraa 2024.
Baran Ozaydin, Tong Zhang, Sabine Susstrunk, and Mathieu Salzmann. Dsi2i: Dense style for unpaired exemplar-based image-to-image translation. Transactions on Machine Learning Research.
Dae Young Park and Kwang Hee Lee. Arbitrary style transfer with style-attentional networks. In proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 5880-5888, 2019.
Taesung Park, Alexei A Efros, Richard Zhang, and Jun- Jun-Yan Zhu. Contrastive learning for unpaired image-to-image -imag translation. In Computer Ession-ECCT 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX 16, pages 319-345. Springer, 2020a.
Taesung Park, Jun-Yen- Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexei Efros, and Richard Zhang. Swapping autoencoder for deep image manipulation. Advances Neural Information Processing Systems, 13:7198-7211 2020b.
the IEFF/CVF International Conference on Computer Vision, pages 4195-4205, 2023. William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of
Stanislav Pidhorskyi, Donald Adjeroh, and Gianfranco Doretto. Adversarial latent autoencoders. In 2020 IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2020.
14

---

Published as a conference paper at ICLR 2025
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint wtXX::2307.01952 2023.
Tianhao Qi, Shancheng Fang, Yanze Wu, Hongtao Xie, Jiawei Liu, Lang Chen, Qian Hc, and Yong- dong Zhang. Deadiff: An efficient stylization diffusion model with disentangled representations. arXiv preprint NXiv:2403.0695/ 2024.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXivv2204.06123, 1(2):3 2022.
Ignacio Rocco, Relja Arandjelović, and Josef Sivic. End-to-end -supery-uperrvieed semantic align- ment. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6917-6925, 2018.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pages 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yacl Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500, 2023a.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yacl Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman. Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models. arXiv preprint (rxXiv2307.069499 2023b.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances neural information processing systems, 35:36479-36494, 2022.
Paul Hongsuck Seo, Jongmin Lee, Deunsol Jung, Bohyung Han, and Minsu Cho. Attentive semantic alignment with offset-aware correlation kernels. In Proceedings of the European Conference Computer Vision (ECCV), pages 349-364, 2018.
Zhiqiang Shen, Mingyang Huang, Jianping Shi, Xiangyang Xue, and Thomas S Huang. Towards instance-level image-to-image translation. In Proceedings of the IEEE/CVF conference on com- puler vision and pattern recognition, pages 3683-3692, 2019.
Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image generation without test-time finetuning, arXiv preprini urXiv:2304.03411 2023.
Jaskirat Singh, Stephen Gould, and Liang Zheng. High-fidelity guided image synthesis with latent diffusion models. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5997-6006. IEEE, 2023.
Jascha Sahl--ickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265 PMLR, 2015.
Kihyuk Sohn, Nataniel Ruiz, Kimin Lee, Daniel Castro Chin, Irina Blok, Huiwen Chang, Jarred Barber, Lu Jiang, Glenn Entis, Yuanzhen Li, al. Styledrop: Text-to-image generation in any style. arXiv preprint 206.023306.00983, 2023.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011 1. 13466, 2020b.
15

---

Published as a conference paper at ICLR 2025
Wenhao Sun, Rong-Chent Tu, Jingyi Liao, and Dacheng Tao. Diffusion model- video editing: A survey. arXiv preprint arXiv:2407.07111. 2024.
Wei Ren Tan, Chee Seng Chan, Hernan E Aguirre, and Kiyoshi Tanaka. Improved artgan for conditional synthesis of natural image and artwork. IEEE Transactions on Image Processing, 28 (1):394-409, 2018.
Luming Tang. Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. Advances in Neural Information Processing Systems, 36: 1363-1389, 2023.
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1921-1930, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Andrey Voynov, Kfir Aberman, and Daniel Cohen-Or. Sketch-guided text-to-image diffusion models. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1-11, 2023a.
Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman, p+: Extended textual condi- tioning in text-to-image generation. arXiv preprint arXiv:2303.09522 2023b.
Haofan Wang, Qixun Wang, Xu Bai, Zekui Qin, and Anthony Chen. Instantstyle: Free lunch towards style-preserving in tex--to-image generation. arXiv preprini arXiv:2404.02733, 2024a.
Jiacheng Wang, Ping Liu, and Wei Xu. Unified diffusion--ased rigid and non-rigid editing with text and image guidance. arXiv preprint arXiv:2401.02 2024b.
Jianbo Wang, Huan Yang, Jianlong Fu, Toshihiko Yamasaki, and Baining Guo. Fine-grained image style transfer with visual transformers. In Proceedings of the Asian Conference on Computer Vision, pages 841-857, 2022.
Qianqian Wang, Xiaowei Zhou, Bharath Hariharan, and Noah Snavely. Learning feature descriptors using camera pose supervision. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16, pages 757-774. Springer, 2020.
Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and Anthony Chen. Instantid: Zero-shot identity- preserving generation in seconds. arXiv preprint arXiv:2401.07519, 2024c.
Zhizhong Wang, Lei Zhao, and Wei Xing. Stylediffusion: Controllable disentangled style transfer via diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7677-7689, 2023a.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from eτror visibility to structural similarity. IEEE transactions on image processing, 13(4):600-612, 2004.
Zhouxia Wang, Xintao Wang, Liangbin Xie, Zhongang Qi, Ying Shan, Wenping Wang, and Ping Luo. Styleadapter: A single-pass lora-free model for stylized image generation. arXiv preprint arXiv:2309.01770. 2023b.
Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15943-15953, 2023.
Chen Henry Wu and Fernando De la Torre. A latent space of stochastic diffusion models for zero- shot image editing and guidance. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 7378-7387, 2023.
Xiaolei Wu, Zhihao Hu, Lu Sheng, and Dong Xu. Styleformer: Real-time arbitrary style transfer via parametric style composition. In Proceedings of the IEEE/CVF International Conference on 2021.
Computer Vision, pages
16

---

Published as a conference paper at ICLR 2025
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306. 09341, 2023.
Zijie Wu, Zhen Zhu, Junping Du, and Xiang Bai. Ccpl: Contrastive coherence preserving loss for versatile style transfer. In European Conference on Computer Vision, pages 189-206. Springer, 2022.
Serin Yang. Hyunmin Hwang, and Jong Chul Ye. Zero-shot contrastive loss for text- text-guided diffusion image style transfer. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 22873-22882, 2023.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for rext-to-image diffusion models. arXiv preprint 200V:22308.0672, 2023.
Zili Yi, Hao Zhang. Ping Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image- to-image translation. In Proceedings of the IEEE international conference on computer vision, pages 2849-2857,, 2017.
Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training- free energy-guided conditional diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2023.
Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. Advances in Neural Information Processing Systems, 36, 2024.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 1836-38477 2023a.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018
Yuxin Zhang, Fan Tang, Weiming Dong, Haibin Huang, Chongyang Ma. Tong-Yee Lee, and Chang- sheng Xu. Domain enhanced arbitrary image style transfer via contrastive learning. In ACM SIGGRAPH 2022 conference proceedings, pages 2022.
Yuxin Zhang, Nisha Huang, Fan Tang, Haibin Huang, Chongyang Ma, Weiming Dong, and Chang- sheng Xu. Inversion- based style transfer with diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10146-10156, 2023b.
Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean- CaanFFrancois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10145-10155, 2021.
Dongyang Zhao, Ziyang Song, Zhenghao Ji, Gangming Zhao, Weifeng Ge, and Yizhou Yu. Multi- scale matching networks for semantic correspondence. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 3354-3364, 2021.
Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. Egsde: Unpaired image-to-image translation via energy-guides stochastic differential equations. Advances in Neural Information Processing Systems, 35:3609-366 -3000993-33--21 2022.
Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. The spatially-correlatvve loss for various image translation tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 16407-16417, 2021.
Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to image-to-image to-imaee translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223-2232, 2017.
17

---

Published as a conference paper at ICLR 2025
Appendix for Semantix: An Energy Guided Sampler for Semantic Style Transfer
A EDIT FRIENDLY INVERSION
This section reproduces, for the reader's convenience, much of the derivation for edit friendly inversion presented in Hluberman--Siieeelee et al. (2023). Diffusion models (Sohl-Dickstein et al., 2015) consist of forward process and denoise process. In the forward process, image xn is transformed into a Gaussian distribution 27 by gradually adding Gaussian noise. This process can be described by stochastic differential equation (SDE) (Song et al., 2020b):
dx f(x, t) dt + g dw.
(14)
here, w represents the standard Wiener process, f(;;; () IR R.d a vector-valued function known as the drift coefficient of =r(f,, and g(.) R - IR. is scalar function referred to as the diffusion coefficient of (t)). In DDPM (Ho et al., 2020), the SDE can be discretized into the following form:
- B + BBEE, t= t=,...,,
(15)
where 6 (0, I) represents standard Gaussian noise, and Be is variance schedule. Formally, the diffusion process can be equivalently expressed as:
Ft v 04 + v1 v
(16)
where at = 1- -1. ā; = II as, and E, ~ (O, In the denoising process, a Gaussian noise #T (0, /) is progressively denoised to sample an image åo. A non-Markovian sampling method was proposed in DDIM (Song et al., 2020a), which can be formulated as follows:
1 (x,) of
T T......
(17)
where 2, ~ (O, 7) represents standard Gaussian noise and independent of xz. The term (x,) can be expressed by the following formula:
(a(xe) va-- vI (ve0((s)
V 1- at-1 ofCo(IE)
(18)
O4g
the fixed schedulers are represented in the general form as: (1((1-64-1) where belongs I-6;
to [0. Within this framework, v = 0 corresponds to the deterministic DDIM scheme, and f/ corresponds to the original DDPM scheme. Owing the deterministic nature of the Ordinary Differential Equations (ODEs), the DDIM does not introduce randomness during the diffusion process. However, classifier-free guidance amplifies the accumulation of errors in text-guided sampling process (Mokady et al., 2023), hence the deterministic DDPMs THuberman-Spicgellgr al., 2023; Wu and De la Torre, 2023) are proposed. In DDPM inversion, a sequence 21,...;; Pr constructed directly from according to the following formula:
St
V /i,50 + vI CrEt, 1 I,
(19)
where é, [), are statistically independent, which are different from Eq. 15. Therefore, compute juy (xx) according to the Eq. 18, which allows us to isolate 2, according to Eq. 17 as shown in the following equation:
f4 (x.)
1- I.....,
(20)
Therefore, the diffusion process evolves into deterministic DDPM. It has been demonstrated that the introduction of randomness in image editing tasks can produce high quality results (Nie et al., 2023).
B IMPLEMENTATION DETAIL.
We use NVIDIA A100 (80G) GPUs for all experiments. For image semantic style transfer, our method is built upon the pre- pre-trained Stable Diffusion v1.5 model. For video task, AnimateDiff (Guo et al., 2023) serves as our base model and we extend the reference image into video sequence. We invert the input images videos into noises through DDPM inversion across 60 timesteps. For
18

---

Published as a conference paper at ICLR 2025
classifier-free guidance, set the scale factor 3.5, aligning it with the sampling procedures. During the sampling process, the features for guidance are extracted from the second and third blocks of the UNet's decoder. In image style transfer tasks, adjust the weights of style feature guidance, spatial feature guidance and semantic distance regularisation to "ref = 3.0, % = 0.9, "reg 1.0, respectively. Additionally, we incorporate a 2D position encoding into the features and assign it weight of Ape = 3.0. For video task, the corresponding hyper-paramelte parameters are set to "/ref = 6.0, Yo- = 3.0, Treg 5.0, Ape 3.0. We further employ a hard clamp in the range of [-1, 1] for all guidance.
To compute the regularisation term, we extract cross-attention maps of context image or video and output image or video the U-net to calculate L2 distance. Besides, we replace the keys and values in the self-attention layer of the decoder at the 32 x 32 and the 64 x 64 resolutions following 10 timesteps, as described in Cross-Image (Alaluf al., 2023). After 20 denoising timesteps, we apply AdaIN (Huang and Belongie, 2017) for the style latents == and output latents æ."d.
C EVALUATION DATASET
We select the COCO (Lin et al., 2014) dataset as the source of context images and obtain style images from WikiArt (Tan et al., 2018) and appearance images from Cross- Masss (Alaluf et al., 2023). We randomly sample 1000 context images in the COCO dataset and pair each with a style image from the WikiArt dataset to form 1000 context-style image pairs. All images are cropped and resized to 512 x 512 resolutions. For context video datas, obtain high quality videos generated by Sora (OpenAI, 2024). Additionally, we also utilize some of our internal data as style images for qualitative result.
D ALGORITHM OF SEMANTIX
In order to make it casier to understand our proposed Semantix, present the detailed algorithm in Algorithm 1.
E FURTHER ANALYSIS.
Sampling speed and memory analysis. Since our sampler is guided by energy gradients, there is a slight increase in sampling time and memory usage. Tab. 5 shows comparison between our method and other diffusion-based approaches, indicating that our approach has a negligible impact on inference time and memory usage.
"Bal Comperace exepeatinat upeoo undraaneer
Droguater
Ti-rebbe
wetccamed Angile
Impact of correspondence accuracy. To further validate the versatility of our approach, we randomly disrupted the feature correspondences between context and reference images. As illustrated in the Fig. 9, this random disruption of feature correspondences led to only slight decline in image quality, without significantly degrading the stylized images. This is attributable to the robustness of our proposed method. Our energy- based approach does not demand high accuracy in semantic correspondences, and the replacement of the K, V in self-attention with semantic correspondences in our semantic distance term also reduces the method's 's reliance on precise semantic correspondence.
exchtd cusciing umeuag
Choices of Position Encoding. In video style transfer, we conducted experiments on 2D and 3D Position Encoding(PE) to explore the impact of different types of PE. The results for 3D PE, as shown in the Fig. 10, illustrate slight decrease in style similarity compared to 2D PE(Fig, 5). Notably, the role of PE is mainly to establish semantic correspondence. Thus, this decline can be attributed the increased difficulty in establishing semantic correspondence across all frames, opposed to within individual frames. Therefore, in video tasks, we employ 2D PE, aligning with image style transfer.
The selection of timestep. In our proposed method, we reverse the input to T=601 timestep by DDPM inversion (Huberman- Spiegelglas et al., 2023). During the experiment, we observed that when T = 601, there was an optimal trade-off between generation quality and sampling speed. Both previous studies (Voynov al., 2023b; Agarwal al., 2023a;b) and Fig. 3 indicate that the image
19

---

Published as a conference paper at ICLR 2025
Algorithm 1: Proposed Semantix
Input:
context image 7 or video Vo; reference image pref or video Vref, prompts P reconstructed and stylized images/videos i'(V"), (ref (Vref), jout (Vrout)
Output:
Require: UNet denoiser ep; UNet feature extractor Fo; hyper-parameters Treef Vo Tregs Apes w; timestep T
Initialization:
(1) Encode image/video:
# = Encoder(I*/V9):
To ref = Encoder(I /V"e));
(2) DDPM inversion to obtain latent æt and noise 2:
pr 2ÿ,, 26 ref ree ref TT T TTT-1
f- DDPM inversion(z.);
ref
DDPM inversion(2;) ( ret
2
-
nout out out
TT 2T TT-1
zpue
Ti 2/ 21:
for t = T,
do
Pt t Concat( vt ref xQuut);
E. tC, CA CC
Eo (zi P);
CA swap - co( P); where Kq = Kret Viout = Viref
me, Mref t Attf-Attn;
è = Et + (ei )
Fú, Fres Fouu =o(
Fref+ = Align (Ff + Axe [m)] (Fred + Ape (F)(mee]))
F Tref (Nout, Frestt + PoFolPutt FF8) + Vreg Freg (CA (CAsuapp CA )
6, = é + V Fi
1 = va--1(
#e -ve + VI ki-1 - -fffe) + 01
--4 = AdaIN(ar x-11);
ref
end
Return i'(Vo), fres (Vros), (Fout))) Decoder(xff zūj a("h);;
structure is formed early in the sampling process. Therefore, when T> 601, the spatial structure of the image has not fully developed, and the semantic correspondence is ambiguous. At this stage, feature guidance can disrupt the image structure, thereby reducing the quality of the generated image. When T < 601, the correspondence tends to be more accurate. However, too few sampling steps weaken the guidance of energy function, reducing style transfer effectiveness. Fewer timesteps with more sampling steps may also lead overfitting due minimal feature variation. Furthermore, conducted an ablation study on the sampling timesteps, with the results shown in Fig. 8.
F QUANTITATIVE COMPARISON IN APPEARANCE TRANSFER.
To quantitatively confirm the effectiveness of our method in the appearance transfer task, we
conduct additional evaluations, similar our
quantitative comparisons of style transfer. We compare our method with some other appear- ance transfer methods (Cross-Iman Image (Alaluf al., 2023), TI-Guided (Wang et al., 2024b)) on the data sourced from Cross-Image Our evaluation metrics include structure preservation ability (LPIPS, CFSD, SSIM), style similarity (Gram matrices) and aesthetic scores (PickScore, HPS) on the images of 11 domains provided by Cross-Image We display the evaluation results of structural preservation capability in Tab. 6,
Figure 8: Ablation of sampling timesteps.
20

---

Published as a conference paper at ICLR 2025
and the results of style similarity and aesthetic scores in Tab. 7. It can be seen that method not only excels in structural preservation but also leads in appearance transfer capability and overall image aesthetics, significantly outperforming competing methods.
Table 6: Structure Preservation Capacity Quantitative Comparisons to Appearance Transfer Methods.
Structure Preservation
LPIPS
CFSD
SSIM
Domain
Choss-maage
Ti-Guided
Ours
Cross-Image
TI-Guided
Ours
Cross-Image
TI-Guided
Animals
0.6389
0.5586
0,4340
1.0949
0.4513
0.3659
0.3996
0.4670
0.5347
Birds
0.0753
Buildings
0.4860
0.4001
0.3004
1.1637
0.5102
0.4950
0.46466
0.4644
0.5788
Cake
0.5737
0.5274
0.3929
0.5588
0.4225
0.2431
0.5099
0.5235
0.6120
Cars
0.3694
0,3505
0.1673
0.1202
0.5012
Fish
0.4850
0.3995
0,,9915
0,5391
0.2234
0.2076
D.4560
0.5133
0.5373
Food
0,4314
0.1112
0.5228
Fruits
0.5417
0,5062
01,8722
0.1657
0.1391
0.0468
D.5498
0.5322
0.6711
House
0.5268
0,4463
0,4135
0,7968
0.4772
0.1886
0.4023
0.3964
0.5433
Landscapes
0.1578
0.0618
0.4186
Vehicles
0.5746
0.5235
0.3990
0,7471
0.3260
0,3477
0,4118
0.3724
0.5564
Average
0,5946
0.2798
0.2057
Table 7: Appearance Similarity and Aesthetics Scores Quantitative Comparisons to Appearance Transfer Methods.
Appeurunce Aesthetics
Metricus 2100
PickScore
Domain
Cross-
T1-Guided
Ours
Cross-Image
Ti TI-Guided
Ours
Cross-Image
TI-Gnided
Ours
Animals
13.4105
10.9154
7.3016
19.13
20.29
20.56
18.99
20.18
20.27
Birds
9,1299
2.4086
1.3395
19.27
19.56
20.20
19.53
19.81
20.31
Duildings
Cake
10.7525
1188198
8.0294
19.43
19.91
20.05
19.19
19.71
19.95
Curs
5,9848
3.6776
2.6616
19.79
20.08
20.29
19.77
19:93
20.01
Fish
9.2579
4.6625
20.87
20.12
Food
13.2022
5.444]
2.8371
20.71
21.67
22.26
19.62
20.27
20.68
Fruits
5.2102
3,6503
1,2086
19.91
House
16.2923
12.8945
3.8237
20.61
20.72
20.70
19.28
19.61
19.44
Landscapes
10.4906
7,1602
2,1185
19.43
19.89
20.35
18.34
18:75
18
Vehicles
20.59
Average
10.9751
8.3295
4.6183
19.82
20.34
20,68
19.22
19.66
19,89
Original
Disrupted
I
Figure 9: Impact of correspon- dence accuracy.
Figure 10: Vidco results with 3D Position Encoding.
G ADDITIONAL QUALITY COMPARISON RESULTS.
We provide additional style and appearance transfer qualitative comparison results with diffusion- based methods (Dragondiff (Mou et al., 2023), Cross- Iosss (Alaluf et al., 2023), StyleID (Chung al., 2023) and TI-Guided (Wang et al., 2024b)) in Fig. 11 and Fig. 12.
21

---

Published as a conference paper at ICLR 2025
DragonDift
Cross-Image
StyleID
TI-Guided
Ours
Context
Reforenes
Content
Refercnce
Context
Referenee
Conkatt
Reference
Coneect
Reforence
Comnett
Reforenes
Figure 11: Additional qualitative comparison with baselines (Dragondiff, Cross-Image, StyleID, T1-Guided) on semantic style transfer (style) task.
H PARAMETER SENSITIVITY
We further demonstrate the parameter effects different levels in Fig. 13. We found that excessively high values of Tre/ can disrupt the structure, while excessively high values of % can diminish the impact of style and appearance. The parameter "reg balances Tree and %e. thereby enhancing image quality. Meanwhile, Ape controls the intensity of the affinity between semantic information and the position information. Besides, also shows that the performance of our proposed sampler is not highly sensitive to these hyper-parameter.
I ADDITIONAL VISUAL RESULTS.
We also provide additional image style transfer results in Fig. 14, Fig. 16, and additional image appearance transfer results in Fig. 17 and Fig. 18. Additional video style and appearance transfer results are shown in Fig. 19. For more video results, please refer to the supplementary materials in the submitted file.
22

---

Published as a conference paper at ICLR 2025
DragonDiff
Cross-Image
StyleID
TI-Guided
Ours
Context
Reforence
Context
Reforenes
Conrect
Reference
Contact
Reference
Context
Reference
Context
Reference
Figure 12: Additional qualitative comparison with baselines (Dragondiff, Cross-Image, StyleID, TI-Guided) on semantic style transfer (appearance) task.
Default
-2.0 2.0
y 0.6
7 .ro 1.5
2 p 1.0
Defaultsctting
7m.440 4.0 -4.0
7,=1.2
7 Y, 22.0
A 5.0
Y =3.0
Y =0.9
reg =1.0
=..0
Figure 13: Parameter sensitivity Treef Ter Preg and Ape-
23

---

Published as a conference paper at ICLR 2025
R Reference
Context
Figure 14: Additional image semantic style transfer (style) results on given context and reference image pairs.
Ours(default)
Semantic Distunce
Context
Reference
Figure 15: Additional ablation study on Semantic Distance.
24

---

Published as a conference paper at ICLR 2025
Reference
Context
Figure 16: Additional image semantic style transfer (style) results given context and reference image pairs.
25

---

Published as a conference paper at ICLR 2025
Reference
Context
Figure 17: Additional image semantic style transfer (appearance) results on given context and reference image pairs.
26

---

Published as a conference paper at ICLR 2025
Reference
Context
Figure 18: Additional image semantic style transfer (appearance) results given context and reference image pairs.
27

---

Published as a conference paper at ICLR 2025
Conteal Viloo
Reforence
Contest Video
Reforms Image
Vilaa
Image
Contect Video
Belanms Image
Contout Vidoo
Image
Figure 19: Additional video semantic style transfer results on given context videos and reference images.
28


Reviews:
[{"review_id": "zmjCYsP9Em", "reviewer": "Reviewer_thdC", "summary": "This paper proposes a training-free semantic style transfer method that innovatively leverages an energy-guided sampler, enabling it to produce both image and video results that surpass existing methods. The task of semantic style transfer is introduced to emphasize the alignment of semantic relationships,  enabling style and appearance transfer while avoiding content degradation and structural disruption.", "questions": "1. I have some confusion regarding spatial feature guidance. Why does calculating the similarity between output features and content features in the feature map preserve spatial structure? Would this impact the effectiveness of style guidance, such as injecting style information from the content reference? \n2. Regarding semantic distance, it seems that you have borrowed from previous methods (\"Visual Style Prompting with Swapping Self-Attention\") in the self-attention part. However, why does directly computing the L2 distance in the cross-attention mechanism help prevent overfitting? It would be helpful to explain the rationale behind this approach.\n3. Acquiring feature maps requires three inference processes in total. Would this significantly increase inference time?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Style Feature Guidance, Spatial Feature Guidance and Semantic Distance are clearly defined and correspond to the theme, successfully achieving semantic style transfer.\n2. The paper is well-written and shows the model’s effectiveness through both quantitative evaluations and user studies.\n3. The authors provide extensive supplementary materials and appendices to validate the effectiveness of the proposed method.", "weaknesses": "1. Some expression in Methods section lacks logical coherence. Some innovations are introduced rather abruptly, with insufficient theoretical foundation and clear explanations, such as the motivation of spatial feature guidance.\n2. The related work section focuses on the field of style transfer but lacks a discussion of energy functions and diffusion-based inference guidance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9oyGjjKsdA", "reviewer": "Reviewer_84Mq", "summary": "The paper introduces Semantix, an energy-guided sampler designed for semantic style transfer that aims to transfer style and appearance features from a reference image to a target visual content based on semantic correspondence. The method leverages pre-trained diffusion models' semantic understanding capabilities to guide both style and appearance transfer. The approach consists of three key components: Style Feature Guidance for aligning style features, Spatial Feature Guidance for maintaining spatial coherence, and Semantic Distance as a regularization term. After inverting reference and context images/videos to noise space using SDEs, Semantix utilizes these components within a carefully crafted energy function to guide the sampling process.", "questions": "Why do you revert the images at step 601 rather than smaller steps since from my point of view, smaller steps seem to induce better correspondence?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method is easy and can be easily applied to both image and video models without requiring additional modifications. \n\n2. The experimental results show that Semantix outperforms existing solutions in both style and appearance transfer tasks while maintaining structural integrity and semantic consistency.\n\n\n3. The writing is mostly clear and the visualizations are decent. For example, I especially like the analysis presented in Fig. 3.", "weaknesses": "1. The motivation for semantic distance is not intuitive. The authors claim that this regularization is meant to prevent overfitting. It is unclear what overfitting the authors indicate, such as to prevent overfitting to the context or reference, or both? Also, it is hard to reason about why the regularization works to handle the problem.\n\n2. Important reference is missing. For example, [1] also deals with universal style transfer which applies to both image and video domains, which is an ideal comparison and discussion.\n\n3. Positional encoding doesn’t seem to have much impact. The qualitative difference between the default version and the deprived version is small. The quantitative difference is also small. \n\n4. Demonstration problems: 1) Fig. 2 is very uninformative. The rightmost part is hard to perceptually understand. The leftmost part does not convey much information. The middle part is hard to observe different arrows and get their exact meanings. 2) Some tables have varied font sizes for titles, such as Tab. 2, 3, 4.\n\n[1] Wu, Zijie, et al. \"Ccpl: Contrastive coherence preserving loss for versatile style transfer.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kfPwqtnr76", "reviewer": "Reviewer_ashT", "summary": "This paper introduces an approach to transferring style between a reference image/video and a context image/video while accounting for the semantics of the elements observed in the images. This is achieved by providing guidance in a diffusion model in the form of an energy function. The proposed energy function leverages semantic correspondences between the reference and context inputs while preserving spatial relationships.", "questions": "Please see Weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Using semantic correspondence to perform style transfer is intuitive\n- The training-free aspect of the method makes it attractive\n- The results are visually convincing\n- The paper is generally clear and reasonably easy to follow (with some exceptions listed in Weaknesses)", "weaknesses": "1) Originality: \n- The idea of accounting for semantics in style transfer has been studied in the past. The most closely related work is Ozaydin et al. TMLR 2024, DSI2I: Dense Style for Unpaired Exemplar-based Image-to-Image Translation, which also uses the notion of semantic correspondence. However, other references therein already looked into leveraging semantics for style transfer (Shen et al., CVPR 2019; Bhattacharjee et al., CVPR 2020; Jeong et al, CVPR 2021; Kim et al. CVPR 2022). I acknowledge that there are methodological differences between these works and the proposed method, but this limits the conceptual originality of this submission, and this line of research should be discussed.\n\n2) Methodology:\n- The choice of T=601 seems a bit arbitrary. Does this work for any scenario/generative model? Is the influence of this studied empirically (this does not seem to be part of the ablation study)?\n- The method has a number of hyper-parameters (\\omega in Eq. 4, the \\gammas in Eq. 5, \\lambda in Eqs. 9-10), which seem to be set to fairly precise values according to Appendix B. Their influence is to some degree studied (in a on-off manner in Table 4, and on one particular example in Fig 12), but this analysis could be made more thorough.\n- Line 313-314: This statement is not clear. How are the object region masks used?\n- Won’t the spatial guidance of Eq. 12 tend to prevent the style transfer, i.e., by encouraging to preserve the appearance from the context image?\n- It seems that the semantic aspect of the transfer is achieved via both the semantic correspondences used in F_{ref}, and the semantic distance employed in F_{reg}. Why is either one of these not sufficient, i.e., why are both needed?\n- In Eq. 14, does sg stand for gradient clipping (as argued below the equation), or stop gradient (as would seem more intuitive based on the name)?\n- As I understand it, in the case of video, motion consistency is assumed to be inherent to the generative model itself. However, even if it is the case, is there any guarantee that the style transfer process won’t destroy, or at least damage, this motion consistency?\n\nExperiments:\n- From Table 1, the proposed method offers a good tradeoff between the different metrics. However, so do StyleID and, to some degree, StyleFormer. Nevertheless, I acknowledge that the qualitative results show that the proposed method yields better quality than these baselines. This raises the question of whether these qualitative examples were cherry-picked to highlight the better performance of the proposed method over StyleID and StyleFormer, or if the metrics are far from being perfect. I acknowledge that the user study of Table 3 helps clarifying things.\n\nClarity:\n- I find the overview of the method, both in the introduction and in Section 4, hard to follow.\n- The purpose of the preliminaries (Section 3) is not clear, with mathematical notation sometimes not explicitly defined (e.g., x_t, \\phi, \\theta), and the link to the proposed method missing.\n\nMinor:\n- The authors should use \\citep instead of \\cite, unless the names of the authors of the corresponding reference should be part of the sentence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PBs2XxtqNk", "reviewer": "Reviewer_uDZ6", "summary": "This paper proposes a semantic style transfer method called Semantix that can handle style transfer and appearance transfer simultaneously. The key idea of this paper is to use the gradients of well-designed energy functions to guide the sampling process of pretrained diffusion models. Specifically, the three terms Style Feature Guidance, Spatial Feature Guidance, and Semantic Distance collectively form the energy function of this paper.", "questions": "please see weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method is capable of performing both style transfer and appearance transfer with a single framework.\n2. The proposed method is training-free and can be extended to video style transfer.\n3. The stylized images generated by the proposed method are plausible.\n4. Extensive experiments are conducted to evaluate the performance of the proposed method.", "weaknesses": "1. My primary concern is the lack of novelty in this paper. The use of energy function gradients to guide the diffusion sampling process has been extensively studied in prior diffusion-based research. Additionally, the energy function itself is not novel: the style feature guidance term resembles the style loss in style transfer, and the spatial feature guidance term aligns with content loss in style transfer.\n\n2. The paper mentions style transfer for videos. What specific adaptations are made in the proposed method for video style transfer? Are motion aspects considered in both the context and synthetic images? Additionally, does applying this method to pre-trained video models impact stability and consistency? If so, why?\n\n3. In the first row of Figure 7, the stylized images generated under different settings appear very similar, without showing significant differences.\n\n4. In Figure 3, why is it claimed that the best results occur at t=601? The results across time steps seem quite similar. It is recommended to compare results across different time steps in the experiments.\n\n5. I’m curious about the runtime of the proposed method. Is its inference speed comparable to or faster than previous methods?\n\n------------------------------------------------------------------------------------\n\nAfter reviewing them, most of my concerns have been addressed. Although the novelty of the proposed method may not fully meet the high expectations for ICLR, this is a strong application with impressive results. As a result, I’ve raised my score from 5 to 6, placing it in the borderline range.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)