PROMOTE: Prior-Guided Diffusion Model with Global-Local Contrastive Learning for Exemplar-Based Image Translation

Authors: conditional  encoder
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-25 08:15:04

Abstract: Exemplar-based image translation has garnered significant interest from researchers due to its broad applications in multimedia/multimodal processing. Existing methods primarily employ Euclidean-based losses to implicitly establish cross-domain correspondences between exemplar and conditional images, aiming to produce high-fidelity images. However, these methods often suffer from two challenges: 1) Insufficient excavation of domain-invariant features leads to low-quality cross-domain correspondences, and 2) Inaccurate correspondences result in errors propagated during the translation process due to a lack of reliable prior guidance. To tackle these issues, we propose a novel prior-guided diffusion model with global-local contrastive learning (PROMOTE), which is trained in a self-supervised manner. Technically, global-local contrastive learning is designed to align two cross-domain images within hyperbolic space and reduce the gap between their semantic correlation distributions using the Fisher-Rao metric, allowing the visual encoders to extract domain-invariant features more effectively. Moreover, a prior-guided diffusion model is developed that propagates the structural prior to all timesteps in the diffusion process. It is optimized by a novel prior denoising loss, mathematically derived from the transitions modified by prior information in a self-supervised manner, successfully alleviating the impact of inaccurate correspondences on image translation. Extensive experiments conducted across seven datasets demonstrate that our proposed PROMOTE significantly exceeds state-of-the-art performance in diverse exemplar-based image translation tasks.

Main paper: 
 introduction: 
 Exemplar-based image translation aims to translate a user-provided conditional image, e.g. pose keypoints, edge maps, or strokes, into a realistic image with styles similar to those of an exemplar image [13, 43] . Compared to traditional image-to-image translation [12, 44, 48] , this task has increasingly attracted the attention of both academic and industrial communities due to its high controllability and flexibility [36] . Meanwhile, it finds applications in various fields including makeup transfer, social media, and the metaverse [24] .Early pioneering works [11, 30] primarily adopt generative adversarial networks (GANs) [6] for exemplar-based image translation, enabling global style control of the generated images. However, these methods ignore spatial correlations between a conditional image and an exemplar, which potentially leads to a lack of faithful details. Consequently, subsequent works [47] attempt to establish cross-domain correspondences between exemplars and conditional images to enhance local style control. They utilize contrastive losses [10, 42] or design adaptive networks [13] , generating a warped exemplar to guide subsequent image synthesis by a generator. More recently, benefiting from the advancements of diffusion models for image generation, Seo et al. [36] first introduced the diffusion model to this task. This approach interleaves cross-domain matching with the diffusion step, iteratively refining a warped image to gradually reduce errors.Overall, existing methods [36, 43, 47] for exemplar-based image translation generally follow a GAN/Diffusion-based matching-thengeneration pipeline: two cross-domain images are first matched via a correspondence network to generate a warped image, and then a realistic image is translated by a generator optimized with several trivial Euclidean-based losses, as illustrated in Fig. 1 (a) . Despite the remarkable success, these methods still suffer from two challenges: 1) Insufficient excavation of domain-invariant features. The Euclidean distance often fails to accurately measure the similarities between exemplars and conditional images due to the inherent domain gap. Relying solely on naive contrastive loss makes it difficult to effectively excavate domain-invariant features, potentially leading to sub-optimal correspondence. 2) Lack of reliable prior guidance in the translation process. Existing methods adopt progressive refinement that ignores reliable prior information from the target structure. Thus low-quality cross-domain correspondences will inevitably produce errors propagated throughout the generation step, significantly reducing the fidelity of the generated image.Motivated by these issues, we propose a novel "Prior-guided Diffusion Model with Global-Local Contrastive Learning" (PROMOTE) for exemplar-based image translation. The simplified framework of PROMOTE is depicted in Fig. 1 (b) . In contrast to previous methods, our PROMOTE mainly designs global-local contrastive learning to effectively learn domain-invariant features within the hyperbolic space [18] and a prior-guided diffusion model that involves a self-supervised training scheme to generate realistic images. Specifically, to tackle the first challenge, we construct a hyperbolic space using exponential mapping [1] to facilitate the global alignment of representations between two cross-domain images. Compared to Euclidean space, hyperbolic geometry accounts for properties like curvature and nonlinearity, allowing it better to capture the consistent and discriminative features of cross-domain images and accurately measure their representation similarity. As an extension, a local contrastive loss defined by Fisher-Rao information [3, 31] is further designed to alleviate the domain gap by aligning their semantic correlation distributions, enabling exemplar and conditional encoders to more effectively capture domain-invariant features for subsequent translation. To address the second challenge, we leverage the structural information from the target, i.e. a conditional image, to modify the diffusion transition to generate prior noisy samples. We then mathematically derive a novel prior denoising loss to emphasize the preservation of the translated image's structure from the prior input. Notably, we employ several data augmentation strategies to construct ground truth and train our prior-guided diffusion model in a self-supervised manner, which results in more realistic images under reliable guidance. Extensive experiments conducted on seven datasets demonstrate that our PROMOTE achieves high-fidelity image translation and significantly outperforms state-of-the-art (SOTA) methods on all benchmarks. Our contributions are listed as follows:• We propose global-local contrastive learning to effectively align two cross-domain images in hyperbolic space and reduce the representation gap between their semantic correlation distributions using the Fisher-Rao metric, thereby better extracting domain-invariant features.  
related work: 
 Exemplar-based Image Translation. Recently, exemplar-based image translation has attracted widespread attention [11, 26, 47] .Early works made use of global styles of exemplars to guide image generation. For instance, Park et al. [30] trained an encoder to transform an exemplar into a global style vector, which is then used to guide the image translation process. While this global control strategy ensures overall style coherence, it falls short in generating intricate details. Recently, numerous approaches [13, 24, 42] have arisen to build dense correspondences to control local detailed styles. Zhang et al. [43] established correspondences on a position-wise basis by using cosine attention to warp an exemplar accordingly, and then the warped image is fed to an image generation process in a manner of SPADE [30] . To explore more accurate correspondences, Zhan et al. [42] employed a marginal contrastive loss to explicitly explore domain-invariant features, and Jiang et al. [13] devised a masked adaptive transformer to suppress unreliable spatial matching and emphasize features of interest. Benefiting from the advance of diffusion models, Seo et al. [36] first introduced a diffusion model in this task, which interleaves the cross-domain matching and the diffusion step to iteratively refine warped images. Compared with these methods, we align cross-domain images in hyperbolic space instead of Euclidean space to learn better domain-invariant features. What's more, we implement image translation in a self-supervised manner by designing a prior-guided diffusion model, emphasizing the modification of diffusion and reverse transitions through reliable prior information from the target structure.Denoising Diffusion Probabilistic Model. Arguing against the training instability and mode collapse of GANs [4, 11, 30] , the denoising diffusion probabilistic model (DDPM) [9] generates highquality samples by reversing the noising process. Notably, Nichol et al. [27] considered quality and speed, additionally predicting variance during the denoising process to improve the sampling process, while Song et al. [39] ensured fast and deterministic sampling by introducing a non-Markovian diffusion process. Rombach et al. [34] adopted an encoder-decoder network [7] to encode the input in a latent space and then trained the diffusion model to reduce computational complexity. Recently, some components containing external knowledge, e.g. classifiers [20, 37] and CLIP [19, 28] , have been incorporated into the diffusion model to control the network for performance boosting. Besides revising network structure, Lee et al. [22] normalized the diffusion variables according to the timestep by leveraging fixed statistical information of conditional prior to improve the performance of speech synthesis. Differently, we exploit the target structure as instance-level prior to adaptively adjust the mean and variance of diffusion distribution at each timestep. Besides, a novel prior denoising loss is mathematically derived to emphasize structure preservation in image translation. 
method: 
 Given a conditional image 𝑥 𝐴 from domain A and an exemplar image 𝑦 𝐵 from domain B, the exemplar-based image translation task aims to generate a target image 𝑥 𝐵 that preserves the semantic structure of 𝑥 𝐴 but adopts the style of 𝑦 𝐵 . The overall architecture of our proposed PROMOTE is illustrated in Fig. 2 , which mainly consists of two components: the visual encoders with global-local contrastive learning for extracting robust domain-invariant features and a prior-guided diffusion model for producing high-fidelity images. Notably, the positive image 𝑦 𝐴 from domain A is derived from 𝑦 𝐵 for global-local contrastive learning. Unlike previous methods where 𝑥 𝐴 and 𝑥 𝐵 are irrelevant with 𝑦 𝐴 and 𝑦 𝐵 , our pipeline applies the same data augmentation techniques to 𝑦 𝐴 and 𝑦 𝐵 , generating credible 𝑥 𝐴 and 𝑥 𝐵 to train our diffusion model in a self-supervised manner.Given 𝑥 𝐴 and 𝑦 𝐵 , the visual domain-invariant representations X 𝐴 ∈ R 𝐶 ×𝐻 ×𝑊 and Y 𝐵 ∈ R 𝐶 ×𝐻 ×𝑊 are extracted by the conditional and exemplar encoders 𝜀 𝐴 and 𝜀 𝐵 , respectively, with 𝐶 dimensions, height 𝐻 and width 𝑊 following [13] . To effectively build the crossdomain correspondence between two images, X 𝐴 and Y 𝐵 are expected to contain accurate and sufficient domain-invariant features [42] . Previous methods mainly apply Euclidean-based contrastive loss [13, 42] to minimize the global similarity between Y 𝐵 and its structurally identical conjugate representation Y 𝐴 to alleviate the domain shift. However, Euclidean space is very sensitive to the overall content and semantics of images, which means that even if two images with different styles have similar structures, their Euclidean distance is still very large, resulting in ineffective representations of images from different domains. Therefore, we propose a novel global-local contrastive learning framework for explicitly excavating domain-invariant features.Global Contrastive Learning in Hyperbolic Geometry. Inspired by the success of hyperbolic embedding in image retrieval [1, 18] , we motivate learning the contrastive and discriminant properties of cross-domain images in hyperbolic geometry. Unlike Euclidean space R 𝑛 , hyperbolic space D 𝑛 , as a Riemannian manifold [41] with constant negative curvature, derives some geometric properties such as curvature, symmetry, and nonlinearity [18] , which can better capture the domain-invariant features and improve the consistency of cross-domain visual representations [14] . Given a Euclidean vector v, the exponential bijective mapping [18] exp 𝑝 :EQUATION)where ⊕ is the differentiable Möbius addition, 𝑝 ∈ D 𝑛 is the reference point empirically set to 0 for simplified computation, and𝜆 𝑝 = 2 1-𝑐 ∥v∥ 2denotes the conformal factor that scales the local distance. Specifically, we follow Eq. ( 1) to obtain the hyperbolic representations Y 𝐴 and Y 𝐵 by applying the exponential mapping to Y 𝐴 and Y 𝐵 before performing the contrastive operation as illustrated in Fig. 2 .The hyperbolic geometry is globally differential to the Euclidean and provides the closed-form distance equation between two points in the hyperbolic space [5] :EQUATIONThen we replace cosine similarity [8, 33] with the distance of Eq.(2) to calculate the global contrastive loss of the cross-domain representations Y 𝐴 , Y 𝐵 in the hyperbolic space:EQUATIONwhere Y -𝑛 (𝑛 = 𝐴, 𝐵) are randomly selected negative samples from the same training batch, half of which come from domain A and half from domain B.Patch-level Contrastive Learning with Semantic Correlation Alignment. Aligning cross-domain images solely at the image level is insufficient as it fails to address the challenges associated with exploring the semantic correlation of appearance attributes, which are crucial for exemplar-based image translation. Thus we design a novel patch-level contrastive learning approach to capture domain-invariant features in a fine-grained manner, as depicted in Fig. 3 . Intuitively, semantic correlations among attributes (e.g. the relative positions of the eyes, nose, and mouth) within an image can be described by the similarity distribution [15] . Given two crossdomain images 𝑦 𝐴 and 𝑦 𝐵 with consistent structures, we propose to bridge their semantic correlation distributions via the shortest path in probability space, which is expected to be sufficiently small.Practically, we first randomly select 𝐾 patches in 𝑦 𝐴 and calculate the semantic correlation distribution D 𝑘 𝐴 for the 𝑘-th patch 𝑦 𝑘 𝐴 in relation to other patches within the hyperbolic space:EQUATIONSimilarly, we extract 𝐾 patches at the same location of 𝑦 𝐵 and obtain its similarity distribution D 𝑘 𝐵 :EQUATIONTo align the semantic correlation distributions between the two images from different domains, we impose the consistency constrain on D 𝑘 𝐴 and D 𝑘 𝐵 for all 𝐾 sampled patches by utilizing the shortest path, which can be given in closed-form 𝑝 FR under the Fisher-Rao information metric and Gaussian assumption, as illustrated in Theorem 3.1:EQUATIONTheorem 3.1. Consider two independent Gaussian distributions: 𝑔 1 with mean 𝜇 1 and standard deviation 𝜎 1 , and 𝑔 2 with mean 𝜇 2 and standard deviation 𝜎 2 . Given the Riemannian metric defined by the Fisher-Rao information [3] , the closed-form expression for the shortest path between these two Gaussian distributions is given by:EQUATION)The proof is provided in Supplementary Material A. Notably, a balancing weight 𝛾 is employed to control the global and local alignment, and the total contrastive loss L cl is:EQUATIONBenefiting from the global-local contrastive learning in the hyperbolic feature space, visual encoders can more accurately excavate domain-invariant features. Then, X 𝐴 is mapped to query Q ∈ R 𝐻𝑊 ×𝐶 , Y 𝐵 is mapped to key K ∈ R 𝐻𝑊 ×𝐶 and value V ∈ R 𝐻𝑊 ×𝐶 . These are fed into the correspondence transformer T corr , which produces the warping estimation A ∈ R 𝐻𝑊 ×𝐻𝑊 and the correspondence feature map X 𝑐𝑜𝑟𝑟 ∈ R 𝐻𝑊 ×𝐶 following [13] . Since our approach effectively bridges the domain gap between the domain 𝐴 and 𝐵 to extract domain-invariant features of X 𝐴 and Y 𝐵 , it facilitates the accurate feature mapping by the correspondence transformer, thereby providing enhanced visual features for the subsequent image reconstruction.In this section, we propose a novel prior-guided diffusion model trained in a self-supervised manner for exemplar-based image translation.Self-supervised Training. Due to the lack of ground truth as a reference in exemplar-based image translation, most previous methods employ the learned correspondences to warp exemplars and implicitly constrain the structure and content of the generated images using several trivial losses. Inspired by [40] , we train our diffusion model in a self-supervised manner by applying several data augmentation techniques (including horizontal flipping, rotation, and elastic deformation) to 𝑦 𝐵 and 𝑦 𝐴 . This generates a ground truth sample 𝑥 𝐵 and its conjugate image 𝑥 𝐴 , which are incorporated into the training scheme. Benefiting from the deterministic guidance of ground truth, the adverse impact of low-quality cross-domain correspondence during image translation is alleviated, which could avoid reliance on several trivial losses typically used by previous methods. Besides, we believe that this self-supervised training compels the network to comprehend the complex nonlinear mapping between cross-domain correspondence X corr and translated image x𝐵 , thereby producing high-fidelity images.Prior-guided Forward Process. Aiming to produce an image x𝐵 that matches the style of 𝑦 𝐵 and maintains the structure of 𝑥 𝐴 , the prior-guided forward process leverages 𝑥 𝐴 to modify the forward prior and generates a new Markov diffusion chain q to corrode 𝑥 𝐵 (denoted as 𝑧 0:𝑇 for brevity) into a non-standard Gaussian distribution N (𝑟, 𝜂 2 I):EQUATIONwhere {𝛽 𝑡 } 𝑇 𝑡 =1 denotes the variance schedule provided in DDPM [9] , 𝛼 𝑡 = 1 -𝛽 𝑡 , 𝑟 = (1 -𝜂)𝑥 𝐴 , and 𝜂 is coefficient schedule that controls the prior intensity. Compared with regular Markov transitions, Eq. ( 9) can adaptively adjust the mean and variance of the distribution of diffusion target 𝑧 𝑡 at each timestep 𝑡 based on the structural prior 𝑟 . We also admit sampling 𝑧 𝑡 given by 𝑧 0 for any timestep 𝑡 in closed-form:EQUATION)where ᾱ𝑡= 𝑡 𝑖=1 𝛼 𝑖 . At timestep 𝑇 , 𝑧 𝑇 = √ ᾱ𝑇 𝑧 0 + √ 1 -ᾱ𝑇 𝜂𝜖 + √ 1 -ᾱ𝑇 𝑟 = 𝜂𝜖 + 𝑟 ∼ N (𝑟, 𝜂 2 I), where 𝜖 ∼ N (0, I). Unlike conventional diffusion trajectories, the diffusion distribution at the end step evolves into a non-standard Gaussian distribution N (𝑟, 𝜂 2 I). Notably, we set 𝜂 = √ ᾱ𝑡 (1-√ ᾱ𝑡 ) as a quadratic schedule to encourage random noise to corrupt the overall style and texture in early phases of the diffusion process, and then emphasize the structural prior in later steps, allowing our diffusion model to effectively preserve target semantic information during the generation process.Denoising Loss for Reverse Process. The reverse process of the diffusion model essentially involves aligning the means of posterior and transition distributions [2] . Therefore, we combine Eq. ( 9) and Eq. ( 10) to derive a tractable posterior (see Supplementary Material B for detailed derivation):q(𝑧 𝑡 -1 |𝑧 𝑡 , 𝑧 0 ) = q(𝑧 𝑡 |𝑧 𝑡 -1 , 𝑧 0 ) • q(𝑧 𝑡 -1 |𝑧 0 ) q(𝑧 𝑡 |𝑧 0 ) , :=N (𝑧 𝑡 -1 ; μ𝑞 (𝑧 𝑡 , 𝑧 0 ), Σ𝑞 (𝑧 𝑡 , 𝑧 0 )),where μ𝑞EQUATIONWe next parameterize p𝜃 (𝑧 𝑡 -1 |𝑧 𝑡 ) := N (𝑧 𝑡 -1 ; μ𝜃 (𝑧 𝑡 ), Σ𝜃 (𝑧 𝑡 )I) by employing a network and leverage the KL-divergence to minimize the difference between p𝜃 (𝑧 𝑡 -1 |𝑧 𝑡 ) and the posterior distribution q(𝑧 𝑡 -1 |𝑧 𝑡 , 𝑧 0 ). Since Σ𝑞 (𝑧 𝑡 , 𝑧 0 ) and Σ𝜃 (𝑧 𝑡 ) are variable-irrelevant constant terms, by aligning μ𝑞 (𝑧 𝑡 , 𝑧 0 ) and μ𝜃 (𝑧 𝑡 ), we can derive the prior denoising loss L diff of our diffusion model (see Supplementary Material C):EQUATIONwhere𝜌 𝑡 = 2𝛿 𝑡 -1 𝛽 𝑡 - √ 𝛼 𝑡 𝛿 𝑡 -1 𝜂 𝛿 𝑡. The first two terms of our denoising loss can be viewed as adaptive modifications to the noises during the diffusion process based on prior, forcing our model to emphasize the target structure and improve the sampling quality under the prior guidance.In addition to the diffusion loss for denoising and the global-local contrastive loss for learning domain-invariant features, two necessary loss functions are incorporated to train our network in an end-to-end way following [24] .Correspondence Loss. Intuitively, we use the reliable correspondence A extracted by the correspondence transformer T corr to warp the exemplar 𝑦 𝐵 , expecting the resulting image should be as close to the ground truth 𝑥 𝐵 as possible. Therefore, the optimization objective for the learned correspondence is denoted as:EQUATIONwhere DS(•) denotes the down-sampling operation used to resize the images 𝑦 𝐵 and 𝑥 𝐵 .Cycle-Consistency Loss. To ensure that visual information is not discarded during the warping process, the warped exemplar is expected to be recovered to the original exemplar 𝑦 𝐵 under the guidance of the inverse correspondence, defined as the cycleconsistency loss [48] :EQUATION)Total Loss. Benefiting from the self-supervised training, our approach constructs ground truth to explicitly guide the image translation, compared to unsupervised schemes, which could produce more faithful images. The overall optimization objective is expressed as:EQUATIONwhere 𝜆 1 , 𝜆 2 , and 𝜆 3 are the balancing weights. 
experiment 4.1 datasets: 
 We mainly conduct experiments on the following seven datasets:(1) CelebA-HQ [21] contains 30, 000 real face images, of which 24, 000 are selected for training and the remaining 6, 000 are used for testing. ( 2) Metfaces [17] and (3) Meticulous consists of 1, 336 artistic facial avatars and 931 Chinese ink paintings, respectively, which are randomly divided at a ratio of 8 : 2 for training and testing. (4) Ukiyo-e [32] contains high-quality Ukiyo-e faces. We randomly select 3, 000 for training and 1, 000 for testing [13] . ( 5) AAHQ [23] consists of facial avatars. We randomly choose 1, 500 samples for training and 1, 000 for testing following [13] . ( 6) DeepFashion [25] consists of 80, 000 fashion images, which are trained and tested according to the official settings. ( 7) ADE-20k [46] contains 20, 210 training images and 2, 000 testing images, which are associated with a 150-class segmentation mask. It is a challenging dataset for most existing methods due to its large diversity.Experimental Settings. All the experiments are deployed on NVIDIA RTX A6000 GPUs and we use 4 GPUs for training and 1 GPU for inference. In all translation tasks, we specify the size of the input and output images as 256 × 256. We build and initialize the visual encoders and correspondence network following [13] . The relevant settings of our diffusion scheduler refer to [9, 45] , with diffusion steps 𝑇 = 1000, a linear noise schedule, and a U-Net noise estimator. For the proposed global-local contrastive learning, we set the curvature 𝑐 = 0.2 to project the data from Euclidian space to hyperbolic space. The weights 𝛾, 𝜆 1 , 𝜆 2 , and 𝜆 3 are set to 0.2, 0.5, 10.0, and 1.0, respectively. We set the batch size to 16 to train our model on DeepFashion for 200 epochs and on the remaining datasets for 100 epochs using the Adam optimizer with the learning rate 1𝑒 -4. Metrics. To evaluate the translation results comprehensively, we adopt several metrics: (1) Fréchet Inception Distance (FID) [35] and Sliced Wasserstein Distance (SWD) [16] to evaluate the perceptual quality of generated images. (2) color, texture, and semantic cosine similarities based on VGG-19 [38] to evaluate style relevance and semantic consistency of generated images [43] .We compare the proposed PROMOTE with several SOTA methods, including CoCosNet [43] , CoCosNet-v2 [47] , DynaST [24] , Midms [36] and MATEBIT [13] across seven datasets. All approaches are replicated by adhering to the settings described in their respective papers and source codes.Quantitative Analysis. The quantitative comparison results are illustrated in Table 1 . Compared to existing methods, our proposed PROMOTE achieves the best FID and SWD scores on almost all datasets, indicating that the perceptual quality of images generated by our method is superior and closest to real images. This superiority stems from our robust feature extraction encoders and the self-supervised training framework integrating prior information. Furthermore, our method also achieves the best performance on semantic and texture metrics tested on CelebA-HQ, with improvements of 0.007 and 0.014, respectively. This demonstrates that the structural prior and self-supervised guidance significantly benefits our model in preserving semantics and achieving more consistent styles. Comparatively, since the structural prior (conditional images) contains little color information, the improvement in color metric is not as pronounced. The comparison results on the ADE-20k dataset are presented in Table 2 , where our method also shows significant gains in challenging scenes with higher diversity and complexity, improving FID and SWD by 0.6 and 0.5, respectively.Qualitative Analysis. Fig. 4 showcases translated images generated by SOTA methods alongside their corresponding input images. The images produced by previous methods exhibit significant flaws, such as geometric distortions, unnatural textures, loss of style, and This experiment validates the effectiveness of the proposed globallocal contrastive learning (Glo. for the global part and Loc. for the local part) and the prior-guided diffusion model (Pri.). between the semantic correlation distributions of the two images. Third, benefiting from the guidance of target structure prior in the diffusion model, the proposed Pri. achieves 0.2 FID improvement on each dataset. This enhancement underscores the value of incorporating structural prior into the diffusion process. Finally, by integrating all three components, our approach reduces the average FID and SWD by 0.75 and 0.7, respectively, which significantly surpasses the baseline model.This experiment analyzes the impact of different balancing weight values in Eq. ( 8) and distance measures on global-local contrastive learning. We vary the values of 𝛾 to observe the performance changes on CelebA-HQ and Metfaces datasets as presented in Table 4 . When 𝛾 is set to 0, i.e. ignoring the optimization by the local 4 and Fig. 6 ) with optimal hyperparameter settings instead of the hyperbolic representation space. Compared with hyperbolic space, using Euclidean space resulted in an average FID and SWD increase of 0.6 and 0.5 across the two datasets. Moreover, images generated in Euclidean spaces exhibit geometric distortion and unnatural texture details (see Fig. 6 ). Both qualitative and quantitative results illustrate that performance in Euclidean space is significantly inferior to that in hyperbolic space. This proves that the inaccurate measurement of two cross-domain visual representations by Euclidean distance makes it challenging to explore domain-invariant features, resulting in sub-optimal cross-domain correspondence.This experiment explores two training manners for the diffusion model including unsupervised ("PROMOTE-un") and self-supervised learning. In the unsupervised deployment, necessary perceptual and contextual losses are incorporated into our PROMOTE and optimized through iterative denoising, akin to Midms [36] . The quantitative and qualitative comparison results are presented in Table 5 and Fig. 6 , respectively. Benefiting from the explicit guidance of ground truth constructed by self-supervised training, our diffusion model effectively addresses geometric distortion and style/semantic inconsistency, bringing significant performance boosting with 0.7 and 2.5 FID improvements as well as 0.9 and 2.8 SWD improvements on the two datasets, respectively. Moreover, images generated through self-supervised training more faithfully preserve details and styles. Meanwhile, the self-supervised PROMOTE avoids iterative refinement training and eliminates several trivial losses commonly employed in unsupervised deployment for performance enhancement, yielding faster training speed. 
conclusion: 
 This work proposes a novel "Prior-guided Diffusion Model with Global-Local Contrastive Learning" (PROMOTE) for exemplarbased image translation. PROMOTE designs global-local contrastive learning to effectively excavate domain-invariant features by aligning cross-domain representations between exemplars and conditional images in hyperbolic space. Furthermore, it emphasizes the target structure as prior information to modify the diffusion transitions and mathematically derives a prior denoising loss. Finally, we tackle this task in a self-supervised manner, constructing deterministic ground truth to train our diffusion model. Our method with theoretical derivations can be generalized to contrastive representation learning and DDPM/DDIM for other generative tasks, facilitating multimedia/multimodal applications. Extensive experiments on seven datasets confirm the effectiveness of PROMOTE, yielding significant performance improvement compared to stateof-the-art methods. 
 
 Appendix:  


Decision: This work has proposed an interesting approach to exemplar-based image restoration with global-local contrastive learning to align two cross-domain images with strong experimental results. The rebuttal has addressed the concerns of the reviewers. In the end, all the reviewers gave "Weak Accept," and the work should be accepted.

Reviews: review: paper_summary: paper_summary: The paper introduces global-local contrastive learning for the alignment of two cross-domain images. With designed prior-guided diffusion model and prior denoising loss, PROMOTE achieves state-of-the-art performance by constructing reliable guidance in the diffusion model.
strength_weakness: strength_weakness: 1. The idea is novel and interesting. In this paper, the authors propose global-local contrastive learning to align images with two domains. And PROMOTE is the first paper that achieves exemplar-based image translation by constructing reliable guidance in the diffusion model.
2. The paper is well-written and easy to follow.
limitations: limitations: For the ablation studies shown in Table. 3, the improvements of FID and SWD are quite small so I am concerned about the effectiveness of the proposed designs.

Although the designs of this paper are novel, the small improvement in the ablation studies cannot show the effectiveness of these designs.
paper_summary: paper_summary: This work proposes a novel "Prior-guided Diffusion Model with Global-Local Contrastive Learning" (PROMOTE) for exemplar-based image translation. PROMOTE designs global-local contrastive learning to effectively excavate domain-invariant features by aligning cross-domain representations between exemplars and conditional images in hyperbolic space. The authors enhance the diffusion model by injecting structural priors and design a novel prior denoising loss to optimize the generation process. Extensive experiments on seven datasets confirm the effectiveness of PROMOTE.
strength_weakness: strength_weakness: 1.	The authors reduce the semantic gap between cross-domain images in hyperbolic space, allowing the visual encoders to extract domain-invariant features more effectively.
2.	The prior-guided diffusion model designed by the authors propagates the structural prior to all timesteps in the diffusion process.
3.	The authors designed a novel prior denoising loss to optimize the generation process.
4.	The authors conducted extensive ablation experiments to demonstrate the effectiveness of the method.
limitations: limitations: 1. When implementing patch-level contrastive learning, how did the authors divide the patches, and what criteria determine if the division is reasonable?
2. By using data augmentation to implement an unsupervised training strategy, the authors leverage the high similarity between the images before and after augmentation. However, could this approach compromise the model's robustness? For instance, can the model generate desirable results when the input consists of condition images with more complex structures?
paper_summary: paper_summary: This paper introduces a novel approach called PROMOTE for exemplar-based image translation, which employs a prior-guided diffusion model and global-local contrastive learning to enhance the extraction of domain-invariant features and improve the accuracy of cross-domain correspondences, resulting in state-of-the-art performance across multiple datasets.
strength_weakness: strength_weakness: 1. The proposed method significantly outperforms SOTA methods on multiple datasets, delivering superior visual results. 
2. The writing is clear and easy to follow, and the topic is both engaging and applicable. 
3. Moreover, the method is grounded in solid mathematical theory, ensuring ease of implementation and excellent outcomes.
limitations: limitations: 1. Given the strong generative capabilities of existing pretrained diffusion models such as Stable Diffusion, why the authors train a new diffusion model from scratch rather than fine-tuning on the pretrained models? I wonder whether the task can be implemented by just combing IP adapter and controlnet.
2. For reproduction and to facilitate community research, would it be possible for the authors to consider release their code and checkpoints?
3. lack of experimental details, for example, the structure of the utilized modules, such as the conditional encoder and denoising network.
4. This work seems unrelated to multimodal and multimedia.
5. Lack of generalization evaluation: the authors can provide the results with mixing conditions, such as combing edge maps and pose keypoints.