Real-World Image Variation by Aligning Diffusion Inversion Chain
Authors: Yuechen ZHANG
Jinbo Xing
Eric Lo
Jiaya Jia
Venue: NeurIPS 2023 spotlight (2023)
Keywords: image variation
diffusion model
image generation
text-driven image editing
TL;DR: RIVAL is a novel approach that uses diffusion models to generate high-quality image variations by aligning the image generation process to the source image's inversion chain.

Abstract: Recent diffusion model advancements have enabled high-fidelity images to be generated using text prompts. However, a domain gap exists between generated images and real-world images, which poses a challenge in generating high-quality variations of real-world images. Our investigation uncovers that this domain gap originates from a latents' distribution gap in different diffusion processes. To address this issue, we propose a novel inference pipeline called Real-world Image Variation by ALignment (RIVAL) that utilizes diffusion models to generate image variations from a single image exemplar. Our pipeline enhances the generation quality of image variations by aligning the image generation process to the source image's inversion chain. 
Specifically, we demonstrate that step-wise latent distribution alignment is essential for generating high-quality variations. 
To attain this, we design a cross-image self-attention injection for feature interaction and a step-wise distribution normalization to align the latent features. Incorporating these alignment processes into a diffusion model allows RIVAL to generate high-quality image variations without further parameter optimization. Our experimental results demonstrate that our proposed approach outperforms existing methods concerning semantic similarity and perceptual quality. This generalized inference pipeline can be easily applied to other diffusion-based generation tasks, such as image-conditioned text-to-image generation and stylization. Project page: https://rival-diff.github.io

Real-World Image Variation by Aligning Diffusion Inversion Chain
Yuechen Zhang'
Jinbo Xing'
Eric Lo' Jiaya Jial
'The Chinese University of Hong Kong SmartMore
(yczhang21, jbxing, ericlo, leojia @cse. cuhk, edu, hk
Image Variations
"Crown"
Generation with style condition
Figure I: Leveraging an image exemplar, our training freen method excels in image generation tasks such generating (a) image variations, (b) images of a similar style (with conditions).
Abstract
Recent diffusion model advancements have enabled high-fidelity images to be generated using text prompts. However, domain gap exists between generated images and real-world images, which poses a challenge in generating high-quality variations of real-world images. Our investigation uncovers that this domain gap originates from a latents' distribution gap in different diffusion processes. To address this issue, we propose a novel inference pipeline called Real-world Image Variation by ALignment (RIVAL) that utilizes diffusion models to generate image variations from single image exemplar. Our pipeline enhances the generation quality of image variations by aligning the image generation process to the source image's inversion chain. Specifically, we demonstrate that step-wise latent dis- tribution alignment is essential for generating high-quality variations. To attain this, we design cross-image self-attention injection for feature interaction and step-wise distribution normalization to align the latent features. Incorporat- ing these alignment processes into a diffusion model allows RIVAL to generate high-quality image variations without further parameter optimization. Our ex- perimental results demonstrate that our proposed approach outperforms existing methods concerning semantic similarity and perceptual quality. This generalized inference pipeline can be easily applied to other diffusion-basse generation tasks, such as image-conditioned text-to-image generation and stylization. Project page: https: http//rialllllleeeeaaaaaaeelle -aiiiiifllii gigh...i.
Introduction
Generating real-world image variation is crucial area of research in computer vision and machine learning, owing to its practical applications in image editing, synthesis, and data augmentation [1, 2]. This task involves the generation of diverse variations of given real-world image while preserving its semantic content and visual quality. Early methods for generating image variations included texture
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

X
Vanillaa demoising
NCO, 2)
Standard
Inverted latent
Gaussian
Distribution Cap
Domain
Real-image inversion chain
Real-image
Inversion space
Deneising step
XX
value
Image exemplar
Figure 2: Conceptual illustration of the challenges in the real-world image variation. Left: Despite using the same text prompt "a lynx sitting in the grass", , a bias exists between the latent distribution in the vanilla generation chain and the image inversion chain, resulting in significant do- main gap between the denoised images. Right: Visualization of the real-image inverted four-channel latent. The distribution of the latent is biased in comparison to standard Gaussian distribution.
synthesis, neural style transfer, and generative models [3, 4, 5, 6, 7, 8]. However, these methods are limited in generating realistic and diverse variations from real-world images and are only suited for generating variations of textures or artistic images.
Denoising Diffusion Probabilistic Models (DDPMs) have resulted in significant progress in text-driven image generation [9, 10, 11]. However, generating images that maintain the style and semantic content of the reference remains a significant challenge. Although advanced training-hased methods [12, 13,
14, 15] can generate images with novel concepts and styles with given images, they require additional training stages and data. Directly incorporating image the input condition [11, 16] results in suboptimal visual quality and content diversity compared to the reference input. Besides, they do not support input with text descriptions. A plug-and and-play method has not been proposed generate high-quality, real-world image variations without extra optimization.
Observing the powerful denoising ability of pre-trained DDPMs in recovering the original image from the inverted latent space [17, 18], we aim to overcome the primary challenge by modifying the vanilla latent denoising chain of the DDPM to fit the real-image inversion chain [17, 18, 19]. Despite generating images with the same text condition, a significant domain gap persists between the generated and source images, as depicted in Fig. 2. We identify distribution misalignment as the primary factor that impedes the diffusion model from capturing certain image features from latents in the inversion chain. As illustrated in the right portion of Fig. 2, an inverted latent may differ significantly from the standard Gaussian distribution. This misalignment accumulates during the denoising process, resulting in domain gap between the generated image and its reference exemplar.
To address this distribution gap problem for generating image variations, we propose an pure inference pipeline called Real-world Image Variation by Alignment (RIVAL). RIVAL is a tuning--rres approach that reduces the domain gap between the generated and real-world images by aligning the denoising chain with the real-image inversion chain. Our method comprises two key components: (i) cross- image self-attention injection that enables cross-image feature interaction in the variation denoising chain, guided by the hidden states from the inversion chain, and (ii) step-wise latent normalization that aligns the latent distribution with the inverted latent in early denoising steps. Notably, this modified inference process requires no training and is suitable for arbitrary image input.
As shown in Fig. 1, our proposed approach produces visually appealing image variations while maintaining semantic and style consistency with a given image exemplar. RIVAL remarkably improves the quality
image variation generation qualitatively and quantitatively compared to existing methods 1, 16, 20]. Furthermore, we have demonstrated that RIVAL's alignment process can be applied to other text-to-image tasks, such as text-driven image generation with real-image condition [21, 22, 23] and example-based inpainting [24, 25).
This paper makes three main contributions.
(1) Using real-world image exemplar, we propose novel tunning-free approach to generate high-quality image variations. (2) We introduce an latent aligmment process enhance the quality of the generated variations. (3) Our proposed method offers a promising denoising pipeline that can be applied across various applications.
2 Related Works
Diffusion models with text control represent advanced techniques for controllable image generation with text prompts. With the increasing popularity of text-to-image diffusion models [11, 16, 26, 10,

---

Real-image inversion
(0,1.)
X
X
X
2-1
X
Noise align
X
injection
Image exemplar
V
V
T2I model
K /
K 7
Q
Alignment
X
X
X
Cross-image self- attennttnn
Image variation
Figure 3: High-level framework of RIVAL. Input exemplar R is inverted to noisy latent X An image variation G is generated from random noise following the same distribution as X For each denoising step we interact X and X by self-attention injection and latent alignment.
27]. high-quality 2D images can be generated through text prompt imput. However, such methods cannot guarantee to generate images with the same low-level textures and tone mapping as the given image reference, which is difficult describe using text prompts. Another line of works aims for diffusion-basse based concept customization, which requires fine-tuning the model and transferring image semantic contents or styles into text space for new concept learning [12, 13, 14, 15, 28]. However, this involves training images and extra tuning, making it unsuitable for plug-ande -ad--aaddpaayy inference.
Real-world image inversion is a commonly employed technique in Generative Adversarial Net- works (GANs) for image editing and attribute manipulation [6, 29, 30, 31, 32). It involves inverting images to the latent space for reconstructions. In diffusion models, the DDIM sampling technique [17] provides a deterministic and approximated invertible diffusion process. Recently developed inversion methods [18, 19] guarantee high-quality reconstruction with step-wise latent alignments. With diffu- sion inversion, real-image text-driven manipulations can be performed in image and video editing methods [33, 34, 23, 22, 35, 36, 37]. These editing methods heavily rely on the original structure of the input, thus cannot generate free-form variations with the same content as the reference image from the perspective of image generation.
Image variation generation involves generating diverse variations of a given image exemplar while preserving its semantic content and visual quality. Several methods have been proposed for this problem, including neural style transfer [5, 38, 39, 40], novel view synthesis [41, 42, 43], and GAN- based methods [7, 44, 32]. However, these methods were limited to generating variations of artistic images or structure- aaructur-preserved ones and were unsuitable for generating realistic and diverse variations of real-world images. Diffusion-baun
methods [16, 11. 20] can generate inconsistent image variants by analyzing the semantic contents from the reference while lacking low- wwwfrequency details. Recent methods using image prompt to generate similar contents or styles [45, 46, 47, 48] requires additional wise tuning or training. Another concurrent approach, MasaCtrl [23], adopts self-attention injection as guidance for the denoising process. Yet, it can only generate variations with generated image and fails real-world image inputs. In contrast, RIVAL leverages the strengths of diffusion chain alignment to generate variations of real-world images.
Real- World Image Variation by Alignment
In this work, we define the image variation as the construction of a diffusion process F that satisfies data distribution function with a given semantic
DRR, C) ((Fc(X, R). C), where (--, C") content condition C. The diffusion process (c(X, R) = G generates the image variation G based on sampled latent feature X, condition C. and the exemplar image R.
The framework of RIVAL is illustrated in Fig. 3. RIVAL generates an inverted latent feature chain (X)..., x by inverting reference exemplar image R using DDIM inversion [17]. Then obtain the initial latent X7, of the inversion chain. Next, random latent feature X is sampled as the initial latent of the image generation (denoising) chain. In this multi-step denoising chain align the latent features to latents from the inversion chain to
{X?...., X.) for generation, obtain perceptually similar generations. The modified step-wise denoising function f at step f. can be represented abstractly as:
X = f((x+1 .x ,C).
(I)
3

---

Generated esceuplar
aligned atrention
Figure 4: Exemplar images and generation results are obtained using the prompt "backpack in the wild". Cross-image self-attention can generate faithful outputs when a vanilla generation chain employed guidance. In the case of real-world images, the generation of faithful variations is dependent on the alignment of the latent.
This function is achieved by performing adaptive cross image attention in self-attention blocks for feature interaction and performing latent distribution alignment the denoising steps. The denoising chain produce similar variations by leveraging latents in the inversion chain as guidance.
3.1 Cross-Image Self-Attention Injection
To generate image variations from an exemplar image R. a feature interaction between the inversion chain and the generation chain is crucial. Previous works [49, 22] have shown that self-attention can efficiently facilitate feature interactions. Similar to its applications [33, 23, 37] in text-driven generation and editing tasks, utilize intermediate hidden states Væ obtained from the inversion chain to modify the self-attention in the generation chain. Specifically, while keeping the inversion chain unchanged, our modified Key-Value Value features for cross-image self-attention for one denoising step t the generation chain are defined as follows:
Q (vG), K WK (va).V WV where
= vc C VR
ift i falign
V
Vn
otherwise
We denote W') the pre-trained, frozen projections in the self-attention block and use @ to represent concatenation in the spatial dimension. Specifically, we adopt an Attention Fusion strategy. In early steps (1 faaign). we replace the KV values with WV (vR). W (vR) using the hidden state væ from the inversion chain. In subsequent steps, we concatenate v and the hidden state from the generation chain itself vG to obtain new Key-Values. We do change Q values and maintain them as Wº (va)- The proposed adaptive cross- cross-image attention mechanism explicitly introduces feature interactions in the denoising process of the generation chain. Moreover, Attention Fusion strategy aligns the content distribution of the latent features in two denoising chains.
Building upon the self-attention mechanism [50], we obtain the updated hidden state output as VG = softmax (QK /dx) V. where dg is the dimensionality of K. Cross-image self-attention and feature injection can facilitate the interaction between hidden features in the generation chain and the inversion chain. It should be noted that the inversion chain is deterministic throughout the inference process, which results in reference feature v remaining independent of the generation chain.
3.2 Inverted Latent Chain Alignment
The cross-image self-attention injection is a potent method for generating image variations from vanilla denoising process originating from standard Gaussian distribution and is demonstrated in [23]. However, as depicted in Fig. 4, direct adaptation to real-world image input is not feasible due domain gap between real-world inverted latent chains and the vanilla generation chains. This leads to the attenuation of attention correlation during the calculation of self-attention. We further visualize this problem in Fig. 12. To facilitate the generation of real-world image variations, the pseudo-geners generation chain (inversion chain) of the reference exemplar can be estimated using the DDIM inversion [17]. Generating an image from latent features X" using a small number of denoising steps is possible with the use of deterministic DDIM sampling:
x Var-1/at X + Var- (8t-1 B+) Ee ttC).
where step-wise coefficient is set 3 1lar - and (x(X,,, C) the pre-trained noise prediction function in timestep. Please refer to Appendix A for the details of a and DDIM

---

sampling. With the assumption that noise predictions are similar in latents with adjacent time steps, DDIM sampling can be reversed small number of steps E [0, T). using the equation:
X = Vat+1/ X + Vac+i B.) so so(('
(x*, ,,,).
(5)
which is known as DDIM inversion [17]. We apply DDIM to acquire the latent representation X of a real image Nevertheless, the deterministic noise predictions in DDIM cause uncertainty in maintaining normal distribution /o I) of the inverted latent X This deviation from the target distribution causes the attention-attenun
problem [51] between generation chains of R and G, leading to the generation of images that diverge from the reference. To tackle this challenge, we find a straightforward distribution alignment on the initial latent X2 useful, that is,
26 E X2 ~ N (M( of (x or X = shufflo(XH))
(6)
In the alignment process, pixel-wise feature elements T6 E X in the spatial dimension can be initialized from one of the two sources: (1) an adaptively normalized Gaussian distribution or (2) permutated samples from the inverted reference latent, shuffle f) Our experiments show that both
types of alignment yield comparable performance. This latent alignment strengthens the association between the variation latents and the exemplar's latents, increasing relevance within the self-attentoon mechanism and thus increasing the efficacy of the proposed cross-image attention.
In the context of initializing the latent variable XT_ further alignment steps within the denoising chain are indispensable. This necessity arises due to the limitations of the Classificr-Free Guidance (CFG) inference proposed in [52]. While effective for improving text prompt guidance in the generation chain, CFG cannot be directly utilized to reconstruct high-quality images through DDIM inversion. On the other hand, the noise scaling of CFG will affect the noise distribution and lead to a shift between latents in two chains, as shown in Fig. 11. To avoid this misalignment while attaining the advantage of the text guidance in CFG, we decouple two inference chains and rescale the noise prediction during denoising inference, formulated as an adaptive normalization [38]:
co (x),, = AdaIN(= ofg (X², t, eo (X t, C)). where
6 (x), (.C) meo (x 1, + (1 - (X
and t fearly-
(8)
5o (X C, is the noise prediction using guidance scale m and unconditional null-text C*. We incorporate this noise alignment approach during the initial stages (1. > learly). After that, a vanilla CFG scaling is applied as the content distribution in G' may not be consistent with R. A Aligning the distributions this early stage proves advantageous for aligning the overall denoising process.
Experiments
We evaluate our proposed pipeline through multiple tasks in Sec. 4.2: image variation, image generation with text and image conditions, and example-based inpainting. Additionally, we provide quantitative assessments and user study results to evaluate generation quality in Sec. 4.3. A series of ablation studies and discussions of interpretability are conducted in Sec. 4.4 to assess the effectiveness of individual modules within RIVAL.
4.1 Implementation Details
Our study obtained a high-quality test set of reference images from the Internet and DreamBooth [12] to ensure diverse image dataset. To generate corresponding text prompts C, we utilized BLIP2 [54].
Our baseline model is Stuble-Diffusion V1.5. During the image inversion and generation, we employed DDIM sample steps T = 50 for each image and set the classifier-free guidance scale
m 7 in Eq. (8). We split two stages at falign tearly 30 for attention alignment in Eq. (3) and latent alignment in Eq. (8). In addition, we employ the shuffle strategy described in Eq. (6) to initialize the starting latent Xτ. Experiments run on a single NVIDIA RTX4090 GPU with 8 seconds to generate image variation with batch size 1.
4.2 Applications and Comparisons
Image variation. As depicted in Fig. 5, real-world image variations exhibit diverse characteristics to evaluate perceptual quality. Text-driven image generation using the basic Stable Diffusion [11] cannot use image exemplars explicitly, thus failing to get faithful image variations. While recent image-conditioned generators such as [20, 53, 16] have achieved significant progress for image input,
5

---

"An black and white drawing of a city."
"A group of people standing around statue."
"A photo of backpack."
"A photo of dog lying on a sofa."
"A gürl sailor uniform posing for a photo."
Source exemplar
Stable Diffusion
SD variation
ELITE
DALLE-2
RIVAL (Ours)
Figure 5: Real-world image variation. We compare RIVAL with recent competitive methods [11, 53, 20] conditioned on the same text prompt (the 2nd column) or exemplar image (3rd-5th columns).
their reliance encoding images into text space limits the representation of certain image features, including texture, color tone, lighting environment, and image style. In contrast, RIVAL generates image variations based text descriptions and reference images, harnessing the real-image inversion chain to facilitate latent distribution alignments. Consequently. RIVAL ensures high visual similarity in both semantic content and low-level features. For instance, approach is the sole technique that generates images with exceedingly bright or dark tones (2nd-row in Fig. 5). More visual comparisons and experimental settings be found in Appendix C.
Text-driven image generation. In addition to the ability to generate images corresponding to exemplar image and text prompts, we have also discovered that RIVAL has strong ability to transfer styles and semantic concepts the exemplar for a casual text-driven image generation. When an inversion with the original text prompt is performed, the alignment denoising processes transfer high-level reference style to the generated images with modified prompts. This process could be directly used in structure-pret structure-preservedd editing. This could be regarded the same task of MasaCtrl and Play [23, 34], which utilizes an initialized inverted latent representation to drive text- -utddg image editing. A comparison is shown in the right part of Fig. 6, here directly use the inverted latent X X for two chains. We adopt the same interaction starting step 45 the MasaCtrl.
still
Furthermore, in the absence a structure- preserving prior, it possible for user-defined text prompt to govern the semantic content of freely generated image G. This novel capability is showcased

---

airinng grase
Mog
MAN AHEALY
"ing"*
(nu dodi
(u) momon: (of))
[C] "odor"
DU) "roor" Augli critor veglume
Bd. "211"
Te SAAR Muse" tog
hoadtiy your
Figure 6: Left: Text-driven image generation results using RIVAL, with the leftmost image as the source exemplar. RIVAL can generate images with diverse text inputs while preserving the reference style. Images noted with indicate peuuture-preserver editing, which starts with the same initialized inverted reference latent X as the exemplar. Right: visual comparison of crucctur-presserved image editing with recent approaches [34, 23].
Canny Edge
Segmentation Map
"portruit"
house"
portrait"
OpenPose
Depth Map
ph
pertni of spiee
Trump
robot"
rabor"
Figure 7: The availability of RIVAL with ControlNet [55]. Two examples are given for each modality of the control condition. Exemplars are shown on the left of each image pair.
Source evemplar
Palut-by-exumplee
RIVAL
SOUICR evemplar
Pulntt mintt-yy-esmpplle
RIVAL
RIVALneriation
Figure 8: RIVAL extended to self-example image inpainting. We use the same image as the example to fill the masked area. Results compared with well-trained inpainting SOTA method [25].
in the left part of Fig. 6, where RIVAL successfully generates images driven by text input while maintaining the original style of the reference image. Further, to extend this ability to other image conditions, we adapt RIVAL with ControlNet Fig. 7 by adding condition residuals of ControlNet blocks on the generation chain, as shown in Fig. 7. With RIVAL, we can casily get style-specific text-to-image generation. For instance, it can produce portrait painting of robot adorned in a sailor uniform while faithfully preserving the stylistic characteristics.
Example-based image inpainting. When abstracting RIVAL as a novel paradigm of image-base based diffusion inference, we can extend this framework enable to encompass other image editing tasks like inpainting. Specifically, by incorporating coarse mask M into the generation chain, we only permute the inverted latent variables within the mask to initialize denoising, we replace latents in the unmasked regions with latents in the inversion chain for each step. shown Fig. 8, we present a visual comparison between our approach and [25]. RIVAL. produces reasonable and visually harmonious inpainted result within the mask.
latent variable X During

---

Metric
SD [11]
Ing Var [53]
ELITE [20]
DALL-E 2 [16]
RIVAL
Text Alignment
0.255 + 0.04
0.223 + 0.04
0.209 = 0.05
0.253 = 0.04
0.275 = 0.03
Image Alignment
0.748 0.08
0.832 0.07
0,736
0.897
0.05
0.840 0.07
Palette Distance 4
3.650 1. 1.30
3.005 1: 0.86
2.885 0.81
2.102
1: 0.83
1.674 1: 0.63
Real-world Authenticity +
2.982 (4.7%)
3.526 (9.6%)
1.961
(28.0%)
1,530 (61.6%)
Condition Adherence
3.146 (7.3%)
3.353 (5.6%)
1.897 (30.6%)
1.603 (56.4%)
Table 1: Quantitative comparisons. We evaluate the quality of image variation regarding feature matching within different levels (color palette, text feature, image feature), highlighted with best and second best results. We also report user preference rankings and the first ranked rate.
Methods
Text Align.
Image Align.
Palette Dist. +
LPIPS
Preparation Time (s)
Inference
Time (s)
PnP [34]
0.249
0.786
1.803
0.245
200
MasaCtrl [23]
0.226
0.827
1.308
0.274
15
RIVAL
0.231
0.831
1.192
0.245
15
Table
Quantitative comparisons with image editing methods. Both in performance and efficiency (time consumption NVIDIA RTX3090 for fair comparison).
Exemplar
Baseline
Attention Inject
Attention fusion
Latent irut.
wio attention operations
wlo noise aligm
w/l atin Fusion
Willlatett inst
w/o noise align
Full model
Figure 9: Left: visual example for module-wise ablation study.
(50,
(40,
(30,30
(0, 0)
(50, 0)
50)
Figure 10: Ablation study for different early-alignment strategies with the exemplar in Fig. 5. We list (fatign, learly) pairs for each image. All images are generated from the same fixed latent.
4.3 Quantitative Evaluation
We compare RIVAL with several state-of-the-art methods, employing the widely used CLIP-score [56] evaluation metric for alignments with text and the image exemplar. The evaluation samples are from two datasets: the DrcamBooth dataset [12] and our collected images. In addition, we cluster 10-color palette using k-means and calculate the minimum bipartite distance to assess the similarity in low- level color tones. In general, the results in Tab. I demonstrate that RIVAL significantly outperforms other methods regarding semantic and low-level feature matching. Notably, our method achieves better text alignment with the real-image condition than the vanilla eoxt-to-image method using the same model. Furthermore, our alignment results are on par with those attained by DALL-E 2 [16], which directly utilizes the image CLIP feature as the condition.
We conducted user study to evaluate further the generated images' perceptual quality. This study utilized results generated by four shuffled methods based on the same source. Besides, the user study

---

CPG-1, origin image
Vanilla Inverted Inference
Inference
CFG-3
CFG-7
CFG-11
CFG-3
CFG+7
CFG-11
Figure 11: High CFG artifacts defense. Left: vanilla generation, starting from the DDIM inverted latent. Right: RIVAL generated images, starting from the permuted latents.
Interaction (R,
Interaction Aigned Latent Chain. 19,
don Pertinn oo txumpplrr
Algned
Dennaing Shop
Dendaing Deg
Figure 12: RIVAL's interpretability. Left: KL divergence between latent among two chains across denoising steps. Right: attention score with the reference feature with different alignment strategies.
includes a test for image authenticity, which lets users identify the "real" image among generated results. We collected responses of ranking results for each case from 41 participants regarding visual quality. The findings indicate clear preference among human evaluators for our proposed approach over existing methods [53, 20, 16]. The detailed ranking results are presented in Tab.
We also compare RIVAL with PnP and MasaCtrl for image editing on a representative test set, with additional metrics in Perceptual Similarity [57] and inference time. Our method presents competitive result Tab. 2.
4.4 Ablation Studies
We perform ablation studies evaluate the efficacy of each module in our design. Additional visual and quantitative results can be found in Appendix D.
Cross-image self-attention injection plays a crucial role in aligning the feature space of the source image and the generated image. We eliminated the cross-image self-attention module and generated image variations verify this. The results in Fig. 9 demonstrate that RIVAL fails align the feature space without attention injection, generating images that do not correspond to the exemplar.
Latent alignment. To investigate the impact of step-wise latent distribution alignment, remove latent alignments and sampling X ~ ( I). An example is shown in Fig. and Fig. 9. Without latent alignment, RIVAL produces inconsistent variations of images, exhibiting biased tone and semantics compared to the source image. Additionally, we perform ablations on attention fusion in Eq. (3) and noise alignment in Eq. (8) during the early steps. Clear color and semantic bias emerge when the early alignment is absent, as shown in Fig. 10. Furthermore, if the values of learly or lalign excessively small, the latent alignment stage becomes protracted, resulting in the problem of over-alignment. Over-alignment gives rise to unwanted artifacts and generates blurry outputs.
Interpretability. We add experiments for the interpretability of RIVAL in the following two aspects. (a) Latent Similarity. To assess RIVAL components' efficacy regarding the distribution gap, we illustrate the KL divergence of noisy latent between chains A and B, X and X in the generation process, as depicted in Fig. 12 left part. Interactions between different distributions (green) widen the gap, while generation chains with the same distribution (orange) can get better alignment by attention interactions. With aligned latent chain alignment and interaction, RIVAL (purple) effectively generates real-world image variations.
(b) Reference Feature Contribution. Attention can be viewed as sampling value features from the key-query attention matrix. RIVAL converts self-attention to image-wise cross-attention. When

---

Vanilla Inference
RIVAL Inference
"photo Tay" source exemplar
"photo bag"
"paintiing" toy girl, In the style of Verimeer"
"photo bag"
"painting toy gtrl"
Figure 13: RIVAL enables seamless customization of optimized novel concepts through text prompt control (<sks>). With various text prompt inputs, we can still generate images preserving the tones, style, and contents of the provided source exemplar (depicted on the left).
latents are sourcing from the same distribution (X), style and content attributes (as in Fig. 12 left orange). This result is beneficial since we do constrain generated images to get similar content and not require complex text conditions style. For more direct explanation, we visualize the bottleneck feature contributions of attention score presented in Fig. 12 right part. Reference contribution of the softmax score is denoted as scoreR Ev,evR vW (W (/ Eve@yR (WQvG (W*(u,)) (. As RIVAL adopts 50% the in the early steps. Latent initialization (orange)
No0 I)) images retain consistent
early fusion in the early steps, and with early alignments (green) play critical roles in ensuring a substantial contribution of the source feature in self-attention. A higher attention contribution helps in resolving the attention attenuation problem (purple) in the generation process.
4.5 Discussions
Integration with concept customization. In addition to its ability to generate image variations from a single source image using a text prompt input for semantic aligmment, RIVAL can be effectively combined with optimization-baseee enable novel concept customization. As illustrated in Fig. 13, an optimized concept can efficiently explore the space of potential image variations by leveraging RIVAL's proficiency real-world image inversion alignment.
concept customization techniques, such as DreamBooth [12], to
Limitations and Future Directions. Despite introducing an innovative technique for crafting high- quality inconsistent image variations, our method contingent upon text prompt input, potentially infusing semantic biases affecting image quality. As evidenced in Fig. 14, prompt like "Pokemon" may lean towards popular choices like "Pikachu" due to training set biases, resulting in Pikachu- dominated generation. Besides, the base model struggles to generate complex scenes and complicated concepts, c.g., "illustration of little boy standing in front of a list of doors with butterflies around them in Fig. 14 (c)". This complexity can degrade the inversion chain and widen the domain gap. leading to less accurate results. Future studies might concentrate on refining diffusion models and exploring novel input avenues besides text prompts to mitigate these constraints.
(e) Semantic bias
(b) Compiex scene
(c Hard concept
Figure 14: Fail cases of RIVAL. The exemplar image is the left for each case.
Conclusion
This paper presents novel pipeline for generating diverse and high-quality variations of real-world images while maintaining their semantic content and style. Our proposed approach addresses previous limitations by modifying the diffusion model's denoising inference chain align with the real-image inversion chain. We introduce a cross-image self-attention injection and step-wise latent alignment technique to facilitate alignment between two chains. Our method exhibits significant improvements in the quality of image variation generation compared to state-of-the-are methods, as demonstrated through qualitative and quantitative evaluations. Moreover, with the novel paradigm of hybrid text-image conditions, our approach can be easily extended to multiple text-to-image tasks.
10

---

Acknowledgements. This work is partially supported by the Research Grants Council under the Areas of Excellence scheme grant AoE/E-601/22-R, Hong Kong General Research Fund (14208023). Hong Kong AoF/P-404/18, and Centre for Perceptual and Interactive Intelligence (CPII) Limited under the Innovation and Technology Fund. We are grateful to Jingtao Zhou for the meaningful discussions.
References
[1] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Liu, and Dilip Krishnan. Supervised contrastive learning. NewrlPS, volume 33, pages 18661-18673, 2020.
[2] Kaiming He, Hanqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pages 9729-9738, 2020.
[3] Alexei A Efros and Thomas K Leung. Texture synthesis by non-parametric sampling. In ICCV, volume 2, pages 1033-1038. IEEE, 1999. 2
[4] Leon Gatys, Alexander S Ecker. and Matthias Bethge. Texture synthesis using convolutional neural networks. In NeurlPS, volume 28, 2015. 2
[5] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In CVPR, pages 2414-2423, 2016. 2,
[6] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139- 144, 2020. 2,
[7] Phillip Isola, lun-Yan Zhu, Tinghui Zhou, and Alexci A Efros, Image-to-image translation with conditional adversarial networks. In CVPR, pages 1125-1134, 2017. 3,
[8] Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelenn decoders. In NewrIPS, volume 29, 2016. 2
[9] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, volume 33, pages 6840-6851, 2020.
[10] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphacl Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, al. Photorealistic text-to- image diffusion models with deep language understanding. NeurlPS, volume 35, pages 36479-36494, 2022. 2
[11] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffussin models. In CVPR, pages 10684-10695, 2022. 2, 3, 5, 6, 8, 21
[12] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yacl Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023. 2, 5, 8, 10, 21
[13] Rinon Gal, Yuval Alaluf. Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image is worth word: Personalizing text-to-image generation using textual inversion. In ICLR, 2023. 2.3
[14] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi-concept customization of text-to-image diffusion. CVPR, 2023. 2,3
[15] Ziyi Dong. Pengxu Wei, and Liang Lin. Dreamartist: Towards controllable one-shot texl-to-image generation via contrastive prompt-tuning. arXiv preprint arXi::2211. 2022. 2,
[16] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditiiii image generation with clip latents. arXiv preprint arXiv:2204.06122, 2022. 2, 3, 5, 8, 9, 14, 15, 16, 21
[17] Jiaming Song, Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021. 2,
3, 4, 5, 14
[18] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen Or. Null-tex text inversion for editing real images using guided diffusion models. arXiv preprint arXi::221..09994, 2022. 2, 14, 20
11

---

[19] Bram Wallace, Akash Gokul, and Nikhil Naik. Edict: Exact diffusion inversion via coupled transformations. arXiv preprint arXiv:2211.1244,, 2022. 2.3
[20] Yuxiang Wei, Yaho Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encod- ing visual concepts into textual embeddings for customized text-to-image generation. arXiv preprint arXiv:2302. 13388, 2023. 2,3,5, 8, 14
[21] Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023. 2
[22] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt- to-prompt image editing with cross attention control. arXiv preprint (rXiv:2208.01626 2022. 2, 3,
(23] Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qic, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. arXiv preprint 1rXiv:2304008465 2023. 2, 3, 4, 6, 7, 14, 15
[24] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu. Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In CVPR, pages 11461-11471, June 2022. 14
[25] Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar-based image editing with diffusion models. arXiv preprint arXiv:221 132277 2022. 2, 7
[26] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In JCML. PMLR, 2021.
[27] Dustin Podell, Zion English. Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXi:2230.001922 2023. 3
[28] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2021. 3
[29]
Weihao Xia, Yulun Zhang, Yujiu Yang. Jing-Hao Hao Xue, Bolei Zhou, and Ming Hsuan Yang. Gan inversion: A survey, PAMI, 45(333112 3138, 2023. 3
[30] Siyu Xing, Chen Gong, Hewei Guo, Xiao Yu Zhang, Xinwen Hou, and Yu Liu. Unsupervised domain adaptation gan inversion for image editing. arXiv preprint arXiv:2211.12122, 2022. 3
[31] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. In ECCV, pages 592-608. Springer, 2020. 3
[32] Tero Karras, Samuli Laine, and Timo Aila. A style- based generator architecture for generative adversarial networks. In CVPR, pages 4401-4410, 2019. 3
[33] Shaoteng Liu, Yucchen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia. Video-p2pp Video editing with cross- attention control. arXiv preprint (oXiv:2303.047//11 2023. 3,4
[34] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven -mage translation. arXiv preprint arXh::2211.12572, 2022. 3, 6, 8
[35] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In ICML. pages 16784-16804. PMLR, 2022. 3
[36] Chenlin Meng. Yutong He. Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2022. 3
[37] Chenyang Qi. Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv preprint arXiv:2303.09533 2023. 3,4
[38] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV, pages 1501-1510, 2017. 5,5
[39] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, pages 711. Springer, 2016. 3
12

---

[40] Nicholas Kolkin, Michal Kucera, Sylvain Paris, Daniel Sykora, Eli Shechtman, and Greg Shakhnarovich. Neural neighbor style transfer. arXiv preprint arXiv:2203. 13222223..111,.... 2022. 3
[41] Kai Zhang, Nick Kolkin, Sai Bi, Fujun Luan, Zexiang Xu, Eli Shechtman, and Noah Snavely, Arf: Artistic radiance fields. In ECCV. pages 717-733. Springer, 2022. 3
[42] Yuechen Zhang, Zexin He, Jinbo Xing, Xufeng Yao, and Jiaya Jia. Ref-NPR: Reference-based non- photorealistic radiance fields for controllable scene stylization. In CVPR, 2023.
[43] Thu Nguyen-Phune, Feng Liu, and Lei Xiao. Snerf: stylized neural implicit representations for 3d scenes. ACM Transactions Graphics (TOG), 41(4):1-11, 2022. 3
[44] Jun Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, pages 2223-2232, 2017. 3
[45] Kihyuk Sohn, Nataniel Ruiz, Kimin Lee, Daniel Castro Chin, Irina Blok, Huiwen Chang, Jarred Barber, Lu Jiang, Glenn Entis, Yuanzhen Li, et al. Styledrop: Text-to-image generation in any style. arXiv preprint arXiv:2306. 00983, 2023.
[46] Yuxin Zhang, Nisha Huang, Fan Tang, Haibin Huang, Chongyang Ma, Weiming Dong, and Changsheng
Xu. Inversion-based style transfer with diffusion models. In CVPR, pages 10146-101530 10156, 2023. 3
[47] Dongxu Li, Jumnan Li, and Steven CH Hoi. Blip-diffusion: Pre-trained subject representation for control- lable lext-to-image generation and editing. arXiv preprint arXiv:2305. 14720, 2023.
[48] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models, arXiv preprint arXiv:2308. 06721, 2023.
[49] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. arXiv preprint arXiv:2212.1 11565, 2022. 4
[50] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrlPS, volume 30, 2017. 4
[51] Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability heyond attention visualization. In CVPR, pages 782-791, 2021. 5
[52] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint ax2iv:2007771725 2022.
[53] Justin Pinkney. Experiments with stable diffusion.
http:://gaiiaaab.cco////uuupippp stable-dee stable-diffus diffusionn on, 2023. 5, 6, 8, 9, 14, 15
[54] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICMI,, 2023. 5
[55] Lvmin Zhang and Mancesh Agrawala. Adding conditional control to text-to-image diffussonn models. arXiv preprint 2302: 05543, 2023. 7, 14
[56] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamola Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In ICML, pages 8748 8763. PMLR, 2021. 8, 20
[57] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018. 9
[58] Jonathan Ho, Ajay Jain, and Pieter Abbecl. Denoising diffusion probabilistic models. NeurlPS, 33:6840- 6851, 2020. 14
[59] Ivar Stangehy. A python library for extracting color palettes from supplied images. https://githi: com/ q htppppyyllllte 2022. 20
[60] Mikotaj Binkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans.
arXiv preprint 10X0:18011.01011, 2018. 21
[61] Ondicj Texler, David Futschik, Michal Kučera, Ondrcj Jamriška, Šárka Sochorová, Menelei Chai, Sergey Tulyakov, and Daniel Sykora. Interactive video stylization using sew-shot patch-based training. ACM Transactions on Graphics (TOG), 39 2020. 21
13

---

Appendix
A
Basic Background of Diffusion Models
This section uses a modified background description provided in [18]. We only consider the condition- free case for the diffusion model here. Diffusion Denoising Probabilistic Models (DDPMs) [58] are generative latent variable models designed to approximate the data distribution ((ra). The diffusion operation starts from the latent To. adding step-wise noise to diffuse data into pure noise IT. It's important to note that this process can be viewed as a Markov chain starting from zo. where noise is gradually added to the data to generate the latent variables IT,-- ....,........................... JT E X. The sequence of latent variables follows the conditional distribution ( 2t rо) II (x, Tt-1). Each step in the forward process is defined by Gaussian transition g(24 2t-1) == N(z;; vI k2 k2 I), which is parameterized by a schedule kg. .o..... ky E (0, 1). As T becomes sufficiently large, the final noise vector 21 approximates an isotropic Gaussian distribution.
The forward process allows us to express the latent variable x; directly as a linear combination of noise and Tp. without the need to sample intermediate latent vectors.
Varg0 + v1 O,,0 w (0,
3t
where O; := II kz). To sample from the distribution g( a reversed denoising process is defined by sampling the posteriors (xx=-1 xz), which connects isotropic Gaussian noise IT to the actual data. However, the reverse process is computationally challenging due its dependence on the unknown data distribution (zz)). To overcome this obstacle, approximation of the reverse process with a parameterized Gaussian transition network denoted as po (2t-1 2x). where po(z;-1 24) follows a normal distribution with mean [0(2,, t) and covariance Eg( t). As an alternative approach, the prediction of the noise eg (xt, () added to xo, which is obtained using equation 9, can replace the use of (1( 1) suggested in [58]. Bayes" theorem could be applied to approximate
ht
no( () = v
VI ai- co (o(
(10)
at
Once we have trained eø (Z t), we using the following sample method Ft-1 ( t) + 0+2, z ~ (N I).
(11)
In DDIM sampling [17], a denoising process could become deterministic when set or 0.
B Details of the Attention Pipeline
We present a comparative analysis of attention injection methods. As depicted in Fig. 15, MasaC- trl [23]. while also adopting a self-attention injection approach, employs more complex control mechanism in its second stage. In the first stage of MasaCtrl, the inverted latent representation X is directly utilized by applying modified prompt. In the second stage, cross-attention mask is introduced to control specific word concepts modified in the prompt, which requires an additional forward pass. In contrast, our proposed method, RIVAL, primarily focuses on generating inconsistent variations. Consequently, we aim to guide feature interaction by replacing KV features with an aligned latent distribution. Unlike MasaCtrl, our approach does not limit content transfer through editing prompts with only few words. Hence, in the second stage, we employ single forward pass without calculating an additional cross-attention mask, allowing fast and flexible text-to-image generation with diverse text prompts.
In recent updates, ControlNet [55] has incorporated attention mechanism resembling the second stage of RIVAL to address image variation. However, a notable distinction lies in using vanilla noised latents guidance, leading to process akin to the attention-only approach employed in RePaint [24] with the Stable Diffusion model. Consequently, this methodology is limited to generating images within the fine-tuned training data domain.
C More About Comparisons
Implementation Details. We compare our work with ELITE [20], Stable Diffusion image varia- tion [53], and DALL-E 2 [16]. We utilize the official demo of ELITE obtain results. To extract
14

---

X
Specific mask control+ replacement
MasaCtrl
V
V
T21 model
T21 model
Q
X
X
X
X
X
K
K
X
Falign
Gllggn
dec enc
Falign
45
(Additional forward get mask)
RIVAL
X
X
Replacement
injection
V
V@V
T2I model
Q
T2I model
Q
K
KOKK
X2
X
X
X
X
Falign
Calign
Faiign
30
(Single forward)
Figure 15: Self-attention control compare with MasaCtrl [23]. The default split of two stages is shown bar for each method.
Experiments
AI
AF
LI
NA
Palette Distance
Text Alignment
Image Alignment
Baseline
3.917
0.266
0.751
AI
3.564
0,277
0.804
AF
3.518
0.274
0.820
LI
3.102
0.268
0.764
NA
3.661
0.251
0.647
w/o AI&AF
2.576
0.276
0.760
w/o AF&NA
2.419
0.279
0.817
w/o AF
1.902
0.274
0.818
w/o LI
3.741
0,242
0.653
w/o NA
2.335
0.267
0.839
Full Model
1.810
0.268
0.846
Table 3: Module-wise ablation (AI: Attention Injection, AF: Attention Fusion, LI: Latent Initialization. NA: Noise Alignment) with quantitative results.
context tokens, we mask the entire image and employ the phrase "A photo/painting of <S>. based on the production method of each test image. Inference for ELITE employs the default setting with denoising steps set to = 300. For Stable Diffusion's 's image variation version, we utilize the default configuration, CFG guidance utilize the official image variation API, specifically requesting using the most advanced API available to generate images of size 1024 X 1024.
and denoising steps T 50. In the case of DALL-E 2,
Comparison with UnCLIP. UnCLIP [16), also known as DALL-E 2, is an image generation framework trained using image CLIP features as direct input. Thanks to its large-scale training and image-direct conditioning design, it generates variations solely based on image conditions when adapted to image variation. However, when faced with hybrid image-text conditions, image-only UnCLIP struggles to produce satisfactory results, particularly when CLIP does not recognize the image content correctly. We provide comparative analysis in Fig. 16. Additionally, we demonstrate in the last two columns of Figure 16 that our approach can enhance the accuracy of low-level details in open-source image variation methods such as SD image variation [53].
Additional Visual Results. We showcase additional results our techniques in variation generation, as illustrated in Fig. 17, and texi-driven image generation with image condition, shown in Fig. 18. The results unequivocally demonstrate the efficacy our approach in generating wide range of image variations that accurately adhere to textual and visual guidance.
15

---

"Pokémon" TM
fail case
"A group of people standing around statue."
"A building with blue and white striped awning."
"Silhouette of woman looking through glass dome filled with fish."
girl in a sailor uniform posing for photo." "
Source exemplar
DALL-E
UnCLIP SDv2 ImgVar SDv1 RIVAL
RIVAL SDv1
Figure 16: Comparision and adaptation with UnCLIP [16]. We highlight texts that enhance the image understanding for each case. Our inference pipeline is adapted the image variation model depicted in the fourth column, in contrast to the variation achieved through vanilla inference in the bottom left corner of each image.
(tolign tearly)
(30, 30)
30)
(30, 0)
(0, 0)
(30, 50)
(50, 30)
(50, 50)
Text Alignment
0.268
0.261
0.269
0.259
0.267
0.274
0.279
Image Alignment
0.846
0.873
0.838
0.865
0.839
0.813
0.817
Palette Distance
1.810
1.421
1.806
1.483
2,359
2.061
2,419
Table 4: Quantitative ablation for different alignment timesteps.
16

---

Exemplars
Variations
Figure 17: Text-driven free-form image generation results. The image reference is in the left column. In the last row, also present variations for one customized concept <sks> bag.
17

---

Figure 18: Text-driven free-form image generation results, with the image reference placed in the top left corner. The prompts used identical those presented in Fig. 6 of the main paper. Every two rows correspond to a shared text prompt.
CFG scale
9
11
Text Alignment
0.260
0.272
0.273
0.273
0.271
Image Alignment
0.859
0.863
0.845
0.845
0.838
Palette Distance
1.749
1.685
1.737
1.829
1.902
Table 5: Quantitative ablation for different Classifier-Free Guidance Scale.
D Additional Ablation Results
Module-wise ablation. We augment visual ablations in Fig. with comprehensive module-wiss experiment. The results in Tab. underscore the role of each component and their combinations. Attention injection facilitates high-level feature interactions for better condition alignment (both text and image). Early fusion, built upon attention injection, aids in early step chain alignment, signifi- cantly enhancing image alignment. Meanwhile, latent noise alignment guarantees the preservation of color. Latent initialization has a pronounced impact, notably enhancing the color palette metric, an effect intensified by noise alignment.
A Ablation on early fusion step. In addition to Fig. 10 of the main paper, we present comprehensive carly-step evaluation results based grid search analysis in Fig. 19. By decreasing the duration of the feature replacement stage (larger falign), we observe an increase in the similarity of textures and contents in the generated images. However, excessively long short early latent alignment durations (fearly) can lead to color misalignment. Users can adjust the size of the early fusion steps hyperparameters achieve the desired outcomes.
We also conduct a quantitative evaluation Tab. It shows that there a balance between two conditions (text-alignment and image-alignment) Besides, low-level texture- texture-pasting pasting artifacts will
18

---

30
40
50
falign image similarity control
30
30
20
10
fearty: color bias control
"A photo of red vase."
Figure 19: Ablation results for alignment steps, with the reference exemplar at the bottom right. We fix each generation's initial latent XT.
present when lalign is small. The choice of bearly primarily influences the generated images' style (color) bias.
Man
Q--mnl (1624)
hitiden euch book
stste befoee selivers
eacly
early seioe
Figure 20: Ablation studies for different feature alignment strategies.
Ablation on different alignment designs. Fig. 20 illustrates ablations conducted on various alignment designs. Two latent initialization methods, as formulated Eq. (6), exhibit comparable performance. Nevertheless, incorporating alignments in additional areas, such as hidden states within each transformer block, may harm performance. Hence, we opt for our RIVAL pipeline's s simplest noise alignment strategy.
19

---

(a) Ablation on different CFG scales
"A PINK building"
(falled)
m=9
(b) Ablation on null-tett Inversion
null
null
w/O
opt. mall
Ablation different prompts
(adjJ: sollor wuforrm"
[nJ: Kilon Musk
[n. original prempt
(n.j: original prompt
original prompt
empry prompe
empty prompt
empty prompt
Figure 21: Ablation studies on different text conditions and guidance scales. Reference exemplars highlighted with golden border.
Ablation on different text conditions. We conduct ablations on text conditions in three aspects. First, we evaluate the impact of different CFG seales m for text prompt guidance, with quantitative results in Tab. 5. As shown in Fig. 21 (a), our latent rescaling technique enables control over the text guidance level while preserving the reference exemplar's low-level features. Second, we employ an optimization-bann based null-text inversion method [18] to obtain an inversion chain with improved recon- struction quality. However, this method is computationally intensive, and the optimized embeddings sensitive the guidance scale. Furthermore, when incorporating this optimized embedding into the unconditional inference branch, there is a variation in generation quality, as depicted in Fig. 21 (b). Third, utilize empty text the source prompt to obtain the latents in the inversion chain while keeping the target prompt unchanged. As depicted in Fig. 21 (c), the empty text leads to weak seman- tie content correspondence between the two chains but sometimes benefits text-driven generation. For example, if users do not want to transfer the "gender" concept to the generated robot.
E Quantitative Evaluations
This section comprehensively evaluates our proposed method with various carefully designed metrics, including CLIP Score, color palette matching, user study, and KID.
CLIP Score. For evaluating the CLIP Score, we employ the official ViT-Large-Patchl CLIP model [56] and compute the cosine similarity between the projected features, yielding the output.
Color palette matching. To perform low-level matching, we utilize the Pylette tool [59] to extract a set of 10 palette colors. Subsequently,
conduct bipartite matching between the color palette of each generated image and the reference palette colors in the RGB color space. Before matching, each color is scaled to [0, 1]. The matching result is obtained by calculating the sum of L,1 distances.
User study. To evaluate the effectiveness of our approach against other methods, we conducted user study using an online form. The user study interface, depicted in Figure 22, was designed to elicit
20

---

01 Specification
Thank you for taking port in this survey. each question in this questiornarr,, we will provide reference image, based on thesa
comsspanding disccrpttion image resutts generated using
each cose,
please
based espects:
Image authenticity (how diftcuut distinguish from the degree matching with the given conditions (both the reference image and the taxt promot).
real world (photo/ (phato/painningg) painting)) and
(a) Specification
oo Imnge authente drawing of
Dng right option dldkk tosent
[Condition metchinn) drawing city
fight
Reference Image
(b) Shuffled results
(c) Ranking questions
Figure 22: User study user interface. In this case, four methods are: (A). SD Image Var [53], (B).
ELITE [20], (C). DALL 2[16), (D). RIVAL (ours).
method
SD [1i]
Img [53]
ELITE [20]
UnCHI [16]
RIVAL.
base model
v1-5
v1-3
V1-4
V2-1
v1-5
KID
|
17.1
18.5
25.7
13.5
13.2
Table 6: Quantitative comparisons for KID (x109). All methods are Stable Diffusion based.
user rankings of image variation results. We collected 41 questionnaire responses, encompassing 16 cases of ranking comparisons.
KID evaluation. To provide a comprehensive assessment of the quality, we utilize Kernel Inception Distance (KID)]60] evaluate the perceptual generation quality of with Stable Diffusion V1-5, our method achieves the best KID score, which is slightly superior to the UnCLIP [16], employing the advanced Stable Diffusion V2-1.
As depicted in Table6,
F Additional Considerations
Data acquisition. To comprehensively evaluate method, collected diverse source exemplars from multiple public datasets, such as DreamBooth [12] and Interactive Video Stylization [61]. Some exemplars were obtained from Google and Behance solely for research purposes. We will not release our self-collected example data due to license restrictions.
Societal impacts. This paper introduces a novel framework for image generation that leverages hybrid image-text condition, facilitating the generation of diverse image variations. Although this application has the potential to be misused by malicious actors for disinformation purposes. significant advancements have been achieved detecting malicious generation. Consequently, we anticipate that our work will contribute to this domain. In forthcoming iterations of our method, we intend introduce NSFW (Not Safe for Work) test for detecting possible malicious generations. Through rigorous experimentation and analysis, our objective is to enhance comprehension of image generation techniques and alleviate their potential misuse.
21


Reviews:
[{"review_id": "rDlSu5KXwe", "reviewer": "Reviewer_4HxC", "summary": "The paper investigates the domain gap between generated images and real-world images, as it is challenging in generating high-quality variations of real-world images. It mainly contains two contributions, the first one is the cross-image self-attention injection for generating images that correspond to the given exemplar, and the second one is the latent alignment, which can help to eliminate the biased tone and semantics. The comparison with stable diffusion, SD variation, and ELITE validates the effectiveness of each component. Lastly, the ablation studies analyze in depth the contribution of each submodule, showing the positive impact of each part.", "questions": "Some deep analyses of the failed cases seem not enough. The provided examples cover very wide ranges, like semantic bias, complex scenes, and hard concepts.\nWhat is the difference between custom image generation and variation generation? It seems that it only has the style difference, as this work needs to keep the style consistent with the given exemplar. In this case, I think the authors should discuss these related works, and compare with them by considering style transfer factors.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper is well-written and easy to follow for most parts, except when introducing the details of the shuffle operation in equation 8. It is not easy to directly understand the operation. \n+ The paper has a nice flow of descriptions and figures to illustrate the qualitative comparison.\n+ The diagrams are easy to understand and are well-made.\n+ The main contribution of the proposed cross-image self-attention injection and latent alignment are intuitive and well-executed in this paper. Each design is reasonable to handle the corresponding challenges, like object custom and style consistency.\n+ The ablation study is sufficient to validate the effectiveness of each component in designing the whole network, the concatenation of cross-image features in different steps, and the alignment of two features.\n+ Experiments show superior performance in comparison with recent works.", "weaknesses": "- Generally speaking, this work is more like the recent customized text-to-image generation, except that this work considered style consistency, like the analyses in Figure 4 (2nd-row). Since these competing methods did not consider this style factor, so they easily have the tones inconsistency problem. Directly comparing with them seems not very fair, as they can generate great results, except for the inconsistent tone distribution. In my opinion, it seems not a crucial issue.\n- The concatenation of v_G and v_R in equation 3 is also like some training-free video generation, which adopts the cross-frame interaction. The second proposed latent alignment is widely used in style transfer tasks. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 4HxC,\n\nThanks for your valuable feedback and constructive comments. We will address your comments below and in the revised paper.\n### Q1 Shuffle Operation\n\nIn Eq. 8, the shuffle operation $X_G^T=\\text{shuffle}(X_R^T)$ is designed to rearrange latent elements within the spatial dimension.\n\nWe present a PyTorch-style code snippet for better understanding:\n```python\n# x_rt: latent variable with the shape (batch size, num. of dimensions, height, width)\ndef shuffle(x_rt):\n    bs, n_dim = x_rt.shape[:2]\n    # Generate a permutation index\n    perm_idx = torch.randperm(x_rt.nelement() // (bs * n_dim))\n    # Reshape, permute, and reshape back to the original\n    x_gt = x_rt.view(bs, n_dim, -1)[:, :, perm_idx].view(x_rt.size())\n    return x_gt\n```\n### Q2 Fairness of Comparison\nWe recognize the importance of ensuring a fair comparison. In the image variation task, RIVAL uses the same objectives and input states as ImgVar and ELITE, ensuring a fair comparison. Our comparisons aim to highlight the benefits of RIVAL, particularly regarding style and content consistency, an overlooked yet critical aspect in image generation.\n#### Q2-1 Comparison and Importance of Image Variation\nImage variation in diffusion-based generation stems from UnCLIP (DALLE2), using image embedding as a condition for tailored generation.\n\nIn Fig. 4 and Tab. 1, the StableDiffusion baseline takes only text as a prompt to illustrate the benefits of introducing a reference image to guide the generation process. Concerning ImgVar and ELITE, these methods also use the original image as a condition and are fine-tuned on large-scale real-world datasets to align the image condition with generated images. However, they often fail to address specific subtle attributes that may not be easily described from the image condition, as shown in Fig 4 (e.g., out-of-focus background in the 3rd row, blinds in the 5th row).\n\nSuch attributes define an image's unique identity. Our approach strives to generate images that closely align with the original image from random noise. This property is vital as it is the fundamental precondition of all the applications presented in the paper and rebuttal, allowing us to generate images more similar to real-world exemplars.\n#### Q2-2 Style-transfer Factor Consideration\nBeyond image variation, RIVAL's capabilities in style transfer are evident. This ability is particularly highlighted in its text-driven image generation (Fig. 5) and customization (Fig. 9). Such demonstrations further accentuate RIVAL's strengths in producing style-consistent images.\n\nFurthermore, we compare methods explicitly crafted for style transfer:  StyleDrop and InST in **rebuttal Fig. C**. This comparison underscores RIVAL's versatility, especially when considering the case-wise fine-tuning (~20min) that StyleDrop and InST demands.\n### Q3 Novelty of Purposed Modules\nHere we will describe our ideas to answer how we can solve problems through the organic combination of modules that might appear \"simple\" at first glance. A comprehensive ablation is presented in **rebuttal Fig. F and mFBx@Q3-1**.\n#### Q3-1 Cross-frame Self-Attention Injection\nAs you correctly pointed out, we have reviewed some video-generation methods (L110-112) that use self-attention injection for cross-frame consistency. Despite sharing a similar objective, our technique is distinct as we are among the first to adapt it to real-world image generation challenges.\n\nA primary challenge we address is the attention-attenuation (L132-135, **rebuttal Fig. D right part**). This problem arises due to the discrepancy between the distributions of latents in image generation. In video generation methods, latents across different frames generally typically maintain the same distribution, whether they originate from inverted latents by DDIM [Video-P2P, Fate-ZeRO] or are derived from the standard Gaussian [Tune-A-Video, Gen-1]. However, uniformity cannot be presumed for generation starting from random noise and referencing with an inverted latent.\n\nAlthough DDIM inversion is deterministic, it does not ensure that the inverted latent adheres to the standard distribution $\\mathcal{N}(0, I)$. This discrepancy in the distribution between the real-inversion and generation chains becomes evident when applying cross-image self-attention (Sec. 3.1). This results in a distribution gap in the Key-Query-Value (KQV) terms. This gap can impact attention computation, leading to varied generation results. Hence, our attention-fusing design combined with latent alignment is proposed to address this challenge effectively.\n#### Q3-2 Latent Noise Alignment\nWhile adaptive normalization is commonly observed in style-transfer and domain adaptation tasks [StyleGAN, AdaIN], our approach to latent alignment tackles a distinct challenge inherent to diffusion generation. Specifically, we aim to solve realism degradation from high classifier-free guidance (CFG).\n\nHigh CFG produces text-aligned images but can cause realism-divergent artifacts **(rebuttal Fig. H)**. To counter this, we introduced a decoupled adaptive normalization (Eq.9-10). It decouples CFG generation from DDIM inversion in one pass, aligning latents to reflect the image exemplar faithfully.\n### Q4 Limitation Analysis\nFor clarity, we will include a more detailed analysis in the revised paper using Fig. 10 examples here.\n> Semantic Bias: A prompt like \"Pokemon\" may lean towards popular choices like \"Pikachu\" due to training set biases, resulting in a Pikachu-dominated generation.\n\n> Complex Scene & Hard Concepts: Stable Diffusion struggles to generate complex scenes and complicated concepts, e.g., \"illustration of a little boy standing in front of a list of doors with butterflies around them in Fig. 10(b)\". This complexity can degrade the inversion chain and widen the domain gap, leading to less accurate results."}, {"review_id": "Yty5i2d5W0", "reviewer": "Reviewer_mFBx", "summary": "Generating real-world image variations is an important research task with practical applications such as image editing, image synthesis, and data augmentation. Past approaches include texture synthesis, neural style transfer, and generative models, among others. This study proposes a method called \"Real-world Image Variation by ALignment\", which generates high-quality image variations from a single image sample through the inference process of an aligned diffusion model. The paper provides experiments on different images.", "questions": "Two general questions:\n1. Ablation study.\n2. Interpretability.\n\nPlease see Weaknesses for full details.", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The overall effect is very good. The proposed method takes full advantage of the properties of the diffusion model.\n\n", "weaknesses": "There are a lot of recent papers based on a very well trained diffusion generative model. The results of these papers are generally very good (after all, it is based on a very mature large model). These papers generally have some micro-designs, which make the academic meaning of the papers more prominent. However, such papers have a common problem, that is, they do not discuss the research issues in a very specific way. Often only stay in the effect. But this hardly accounts for the actual scholarly progress of these papers. Chances are it's just a manifestation of the capabilities of the base model itself.\n\nTaking this paper as an example, the paper proposes the method of Aligning Diffusion Inversion Chain. But the necessity and interpretability of the method is not fully discussed. In fact, there are many \"tricky\" ways to achieve the effect of this paper, and it is not necessary to go through the method described in Section 3 (By the way, the writing of Section 3 is very difficult to understand. In fact, a lot of useless mathematics is unnecessary. It is recommended that authors use more carefully set diagrams to explain their methods). I doubt the necessity of the proposed method. I hope the author can have a full ablation experiment, and compare and discuss it with a wide range of existing tricks. Of course, I also realize that such a requirement is a bit too harsh. Because many methods and tricks have not been written into papers or formally disseminated. The authors may not be able to adequately discuss every possibility.\n\nInterpretability is also another important topic. I want to know the specific physical meaning of each latent and representation in the inversion transfer or the point that it can be exploited in this method. This paper doesn't really explain it (or even try to).\n\n=====================\n\nPost rebuttal:\n\nThanks for the rebuttal. I am willing to rase my score to accept (7). But I recommend authors to add these new discussions in to the final version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer mFBx,\n\nThank you for your valuable feedback and constructive comments. We will address your comments below and in the revised paper.\n### Q1 Utilizing Capability of the Base Model\nWe respect your comments on the trend of recent works leveraging mature models. While we agree these base models are crucial, RIVAL  innovates by addressing the specific challenge of controllable, high-quality, real-world image generation. RIVAL is designed to extend beyond the capabilities of the original model and can be generically applied to broad well-trained diffusion frameworks.\n\nThis generalization ability is demonstrated through extensive experiments with diverse diffusion frameworks and applications, including DreamBooth (Fig.9), SDXL (**rebuttal Fig.A**), ControlNet (**rebuttal Fig.B**), and ImgVar (Appendix Fig.12). We posit that this breadth of applicability constitutes scholarly progress, broadening the base model's utility.\n### Q2 Clarification of Section 3\nFor a detailed and precise workflow of our method, please refer to the Pytorch-style pseudocode for RIVAL in **SLMY@Q3**. A LaTeX algorithm version will also be included in the revision.\n### Q3 Necessity of RIVAL: Wider Discussion and Comparisons\nTo elucidate RIVAL's necessity, we delve into two key aspects:\n#### Q3-1 Module-wise Ablations\nAgreeing with your suggestion for thorough ablations, we augment our visual ablations (**rebuttal Fig. F**, Fig.8 and Appendix Fig.15-17) with a comprehensive module-wise experiment. \n|Module|Attention Inject|Early Fusion(Eq.3)|Latent Init.(Eq.8)|Noise Align(Eq.9-10)|Palette Dist.|Text Align|Image Align|\n|-|-|-|-|-|-|-|-|\n|Baseline (SD)|||||3.917|0.266|0.751|\n|Attn Inject|Y||||3.564|0.277|0.804|\n|Attn Fusion|Y|Y|||3.518|0.274|0.820|\n|Latent Init.|||Y||3.102|0.268|0.764|\n|Noise Align||||Y|3.661|0.251|0.647|\n|w/o Attn Operations|||Y|Y|2.576|0.276|0.760|\n|w/o Fusion&Noise Align |Y||Y||2.419|**0.279**|0.817|\n|w/o Attn Fusion|Y||Y|Y|1.902|0.274|0.818|\n|w/o Latent Init|Y|Y||Y|3.741|0.242|0.653|\n|w/o Noise Align|Y|Y|Y||2.335|0.267|0.839|\n|**Full Model**|Y|Y|Y|Y|**1.810**|0.268|**0.846**|\n\nThe results underscore the role of each component and their combinations. Attention injection facilitates high-level feature interactions for better condition alignment (both text and image). Early fusion, built upon attention injection, aids in early step chain alignment, significantly enhancing image alignment. Meanwhile, latent noise alignment guarantees the preservation of color.\n\nLatent initialization has a pronounced impact, notably enhancing the color palette metric, an effect intensified by noise alignment. A thorough evaluation of hyperparameters $t_{align}$ and $t_{early}$ can be found in **aMYr@Q5**, Appendix Fig.15, and analysis in Appendix L84-89.\n#### Q3-2 Wider Discussions and Comparisons\nTo further ascertain RIVAL's necessity and effectiveness, we compare it with three concurrent related works. **Visual results: rebuttal Fig. C&G. Quantitative comparison: aMYr@Q2**.\n1. **Tunning-based style transfer method [StyleDrop]** Similar to ours, this method yields visually appealing results with a style exemplar image. Despite its high-quality stylization, it relies on case-wise finetuning with intricate prompt design. At the same time, its base model MUSE isn't open-sourced.\n2. **Attention-based image editing [PnP]** PnP employs a plug-and-play approach that uses attention and feature injections for structure-preserving text-based editing. While the results are good, the image style diverges from the exemplar due to a lack of chain alignment consideration.\n3. **Attention-based image synthesis [MasaCtrl]** This work employs a similar attention-injection framework. We've comprehensively analyzed the differences in attention in Appendix Section B and Fig.11. MasaCtrl is generally suited for real-image editing instead of unconstrained generation. We present a visual comparison in **rebuttal Fig. G**. As input latent is fixed, it cannot generate free-form images from a single target prompt, as results in Appendix Fig. 14.\n### Q4. Interpretability of RIVAL\nWe add explanations and experiments for the interpretability of RIVAL in the following two aspects.\n#### Q4-1 Latent Similarity\nTo assess RIVAL components' efficacy regarding the distribution gap, we illustrate the KL divergence of noisy latent between chains A and B, $X_{A}^{t}$ and $X_{B}^{t}$ in the generation process, as depicted in **rebuttal Fig. D left part**. Interactions between different distributions (green line) widen the gap, while two latent chains with the same distribution (yellow line) can get a better alignment by attention interactions. With aligned latent chain alignment and interaction, RIVAL (red line) effectively generates real-world image variations.\n#### Q4-2 Reference Feature Contribution\nAttention can be viewed as sampling value features from the key-query attention matrix. RIVAL converts self-attention to image-wise cross-attention. When latents are sourcing from the same distribution ($X_G^T, X_R^T\\sim\\mathcal{N}(0, I)$) images retain consistent style and content attributes  (Fig. 3, first two images, **rebuttal Fig. D left, yellow line**). This result is beneficial since we do not require complex text conditions $c$ to constrain generated images to get similar content and style.\n\nFor a more direct explanation, we visualize of images' bottleneck feature contributions (as in Fig. 7 right part) of attention score are presented in **rebuttal Fig. D right part.**\nReference contribution of the softmax score is:\n> $$\\text{score}_{R}=\\frac{\\sum^{v_i\\in\\mathbf{v}_R}({W^Q\\mathbf{v}_G\\cdot(W^K (v_i))^{\\top}})}{\\sum^{v_j\\in\\mathbf{v}_G\\oplus\\mathbf{v}_R}({W^Q\\mathbf{v}_G\\cdot(W^K(v_j))^{\\top}})}$$\nAs RIVAL adopts early fusion in the early steps, we use 50% as the score in the early steps. Results indicate that RIVAL leverages 30% of the reference feature in the final step, compared to random initialization (7%) and latent initialization (21%)."}, {"review_id": "Ui6xa2sYYy", "reviewer": "Reviewer_aMYr", "summary": "The authors propose a tunning-free pipeline called RIVAL(Real-world Image Variation by Alignment) for generating diverse and high-quality variations of real-world images.\n\nIn previous works, some models also generate images with novel concepts and styles but require additional training stages and data, and others directly incorporated images as the input condition results in suboptimal visual quality and content diversity.\n\nTo tackle this issue, the authors suggest a pipeline that maintains the style and semantic content of the reference without additional training stages and data.\n\nThe proposed method comprises the inverted latent chain alignment(step-wise latent normalization) and cross-image self-attention injection.\n\nThe authors determine whether aligning the inverted latent chain and injecting the modified self-attention key and value by adopting $t_{align}$ and $t_{early}$.\n\nUnlike the other papers that use the cross-image self-attention injection method with an additional forward to calculate the cross-attention mask, the suggested approach only needs a single forward.\n\nThe authors show that RIVAL enhances performance by applying the alignment process to other text-to-image tasks through experiments and ablation studies.", "questions": "- In Table 1, I wonder if using specific $t_{align}$ and $t_{early}$ values for each metric would lead to better quantitative results.\n- In Fig. 6, the inpainting results show a different masked area. Can you show me the same results by overlaying the masked area at the source image like SD[1] in the paper?\n- Can you provide examples of other tasks to which RIVAL can be applied?\n\n[1] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolutionimage synthesis with latent diffusion models. In *CVPR*, pages 10684–10695, 2022.", "limitations": "The paper needs more extensive experiments and analysis of the limitations of the proposed method.", "rating": 7, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The authors claimed a novel plug-and-play pipeline to generate high-quality, real-world image variations without additional optimization through an inverted latent chain alignment and cross-image self-attention injection.\n- The proposed pipeline enhances the performance of text-to-image models, such as text-driven image generation with real-image conditions and example-based inpainting. The pipeline generates images fast and flexibly with diverse text prompts adopting a new self-attention injection approach with a single forward pass.\n- Compared to CFG, the advantage of the proposed pipeline is its ability to avoid misalignment between latents in two chains and its direct utilization. The authors decouple the two inference chains and rescale noise prediction during denoising inference. Furthermore, they formulate noise prediction as an adaptive normalization.\n- The quantitative results outperform the other pipelines provided in the paper except for one metric, and the qualitative results show that RIVAL generates images with diverse text inputs while preserving the reference style.", "weaknesses": "The experiments need to be more extensive.\n\n- The authors proposed a plug-and-play pipeline, but there are no quantitative results compared with the other plug-and-play pipelines mentioned in the paper. RIVAL used a similar scheme as MasaCtrl[2] and Plug-and-Play[3] except the latents alignment. The authors employed similar schemes to the pipeline, so comparing the quantitative results with the others would be beneficial.\n- In the appendix, the proposed method allows fast and flexible text-to-image generation, but there are no results about how fast it is. It would be reasonable to compare quantitatively with other plug-and-play pipelines.\n\nThe authors need to validate the ability of the pipeline to apply to other diffusion-based generation tasks.\n\n- The authors show the qualitative results of self-example image inpainting in Fig. 6, but the results lack experiments. It would be hard to explain the ability to extend RIVAL’s framework to other image editing tasks.\n\n(Minor) The notation of $t_{align}$ and $t_{early}$ is confusing.\n\n[1] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In *CVPR*, pages 10684–10695, 2022.[2] Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. *arXiv preprint arXiv:2304.08465*, 2023.[3] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. *arXiv preprint arXiv:2211.12572*, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer aMYr,\n\nThank you for your valuable feedback and constructive comments. We will address your comments below and in the revised paper.\n### Q1 Comprehensive Quantitative Experiments\nWe agree that comprehensive quantitative evaluations are vital to support our claims. Please refer to **rebuttal global@Q3** for all quantitative experiments conducted in this rebuttal.\n\n### Q2 Comparison with Other Plug-and-Play Pipelines\nInitially, we excluded quantitative comparisons with training-free pipelines such as PnP and MasaCtrl due to task and input disparities. Specifically, PnP utilizes text to modify image appearance, while MasaCtrl leverages text for structural alterations. We recognize the merit of such a comparison in real-image editing. Please see **rebuttal Fig. G** for qualitative comparisons and the subsequent table for a quantitative evaluation.\n\n|Method|PnP|MasaCtrl|RIVAL|\n|-|-|-|-|\n|Image Alignment $\\uparrow$|0.786|0.827|**0.831**|\n|Text Alignment $\\uparrow$|**0.249**|0.226|0.231|\n|Palatte Distance $\\downarrow$|1.803|1.308|**1.192**|\n|LPIPS $\\downarrow$ |**0.245**|0.274|**0.245**|\n\nIn the experiment, we directly use the inverted latent $X_{G}^{T} = X_{R}^{T}$ for two chains. We adopt the same interaction starting step $t=45$ as the MasaCtrl. Furthermore, as shown in our Appendix Fig. 14, RIVAL can create free-form content that maintains consistency with the image's style, indicating its distinction and effectiveness compared to these methods.\n\n### Q3 Speed of the Pipeline\nAs mentioned in L181-182, RIVAL processes images in roughly 8 seconds each on a single RTX 4090. When tested on the more prevalent RTX 3090, our comparisons with other plug-and-play methods confirm RIVAL's competitive inference speed against recent techniques.\n\n|Comparison on RTX 3090|PnP|MasaCtrl|RIVAL|\n|-|-|-|-|\n|Preparation Time (s)|200|6|6|\n|Inference Time (s)|21|15|15|\n\nPnP has a long preparation time due to the feature storing and long-step inversion. It's worth noting that both MasaCtrl and RIVAL can be sped up using half-precision and xformers, achieving an inference time of 6 seconds on RTX 3090. For our comparison, we used the official implementations and maintained consistent conditions across the board.\n### Q4 Applicability to Other Tasks\nRIVAL's versatility is showcased through diverse experiments spanning multiple diffusion frameworks and applications, such as DreamBooth (Fig.9) and ImgVar[UnCLIP] (Appendix Fig.12).\n\nAdditionally, we have successfully extended RIVAL's application to other tasks and frameworks, which showcases the method's adaptability and broad applicability across diverse diffusion frameworks.\n\n1. Larger diffusion models. We've integrated RIVAL with the recent SDXL in **rebuttal Fig. A** for high-resolution (1024 $\\times$ 1024) image variation and text-driven generation.\n2. Image-conditioned controllable generation. As demonstrated in **rebuttal Fig. B**, RIVAL, in conjunction with ControlNet, produces style-consistent images under varying control conditions.\n3. Image style transfer. Utilizing the aligned inverted latent of the content image, $X_G^T = \\text{AdaIN}(X_G^T, X_R^T)$. The style transfer results are shown in **rebuttal Fig. C, the first row**.\n4. Image editing. We have discussed it in Q2 with visuals in **rebuttal Fig. G**.\n\n### Q5 Use of Specific $t_{align}, t_{early}$\nYour statement is correct. As stated in the grid-wise ablation in Appendix Fig.15, using complete attention feature fusion (replacement) ($t_{align} = 0$) makes the generation closely resemble the original image. However, there is a balance between two conditions (text-alignment and image-alignment). Besides, low-level texture-pasting artifacts will present when $t_{align}$ is small, as shown in Fig. 8, 4th image. The choice of $t_{early}$ primarily influences the generated images' style (color) bias.\n\n|($t_{align}, t_{early}$)|(30, 30)|(0, 30)|(30, 0)|(0, 0)|(30, 50)|(50, 30)|(50, 50)|\n|-|-|-|-|-|-|-|-|\n|Text Alignment  |0.268|0.261|0.269|0.259|0.267|_0.274_|**0.279**|\n|Image Alignment |0.846|**0.873**|0.838|_0.865_|0.839|0.813|0.817|\n|Palette Distance|1.810|**1.421**|1.806|_1.483_|2.359|2.061|2.419|\n### Q6 Inpainting Results\nIn **rebuttal Fig. E**, we've added a mask boundary overlay to highlight the generation area during inpainting. As shown in the last column, we can achieve varied generation results within the masked area by resampling the latent.\n\nRIVAL primarily aims to generate images resembling a reference. Inpainting is just one specific application of RIVAL. Given that self-example inpainting focuses on variation generation within a masked image area, it's challenging to fairly compare it with general inpainting (as RIVAL utilizes the original image as a reference) or example-based inpainting (where RIVAL is only capable when the image itself is the reference). Instead, we've offered a quantitative comparison for the image editing task in Q2.\n\n### Q7 Limitations Analysis\nFor clarity, we will include a more detailed analysis in the revised paper, using Fig. 10 examples here.\n> Semantic Bias: For a prompt like \"Pokemon\", bias in training set towards the popular\"Pikachu\" can skew generated images towards Pikachu-like designs.\n\n> Complex Scene & Hard Concepts: Stable Diffusion struggles to generate complex scenes and complicated concepts, e.g., \"illustration of a little boy standing in front of a list of doors with butterflies around them\". This complexity can lower inversion chain quality and widen the domain gap, leading to less accurate generation results."}, {"review_id": "dYGBRUO73O", "reviewer": "Reviewer_SLMY", "summary": "This paper works on the design of diffusion model to generate image variations given an image examplar as the source image.  The basic idea is to align the image generation process to the image inversion process of source image. This is achieved by designing an cross-image self-attention injection for feature interaction between the source and generated images in the diffusion process. The results demonstrate better performance compared with the existing methods in image-conditioned text-to-image generation and examplar-based image inpainting.", "questions": "I am interested in the problem to tackle and the proposed idea of feature interaction between source and generated image for inverse and generation chain alignment. However, I have many unclear points on the presentation of the proposed method (as mentioned in the weakness). I suggest authors to clarify the method in rebuttal and improve the writings and presentations of this manuscript. ", "limitations": "The paper discussed on the limitation of this work. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The idea to generate image variations given a source image is an interesting task. This paper proposed to align the source image inversion process and image generation process using the cross-image self-attention injection, which is a simple idea and shows good image generation results.", "weaknesses": "1. The presentation and training of the parameters in \"cross-image self-attention injection\" are unclear to me. I understand that, in sect. 3.1, the features of examplar image and the generated image are interacted based on eqns. (2-4), in their inversion and generation chain respectively. However, how to train/determine the parameters in these equations? Are these parameters shared across different diffusion steps?  And how do these parameters affect the final generation results?\n\n2. Does this proposed generation method rely on the off-the-shelf diffusion model without any training? The chain alignment is conducted in the latent feature space or the image space? \n\n3. An algorithmic description of this proposed method for image generation should be given for better understanding the training (possibly) and image generation process?\n\n4. The shuffle operation in sect. 3.2 is described as \"permutated sample from the inverted reference latent\". Please clarify on this description, does this mean permutating the pixels of X_R^T?\n\n5. In the Table I, please clarify on the metric of \"Real-image Rank\" and \"Preference Rank\".  I know that lines 228-233 may describe them, however, it is still unclear to me on the difference between \"Real-image Rank\" and \"Preference Rank\". ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer SLMY,\n\nThank you for your valuable feedback and constructive comments. We will address your comments below and in the revised paper.\n### Q1 Clarification of Cross-Image Self-Attention Injection\nWe employ the pre-trained Stable Diffusion model in RIVAL without adding trainable parameters. All parameters are consistent across steps. Our self-attention injection design modifies the input and interactions of self-attention. Here we consider each chain in the forward process separately:\n\n1. **Inversion Chain**: Because we want to ensure that this chain reconstructs the inverted image, the denoising process is identical to the vanilla diffusion generation forward. No modifications are made in this chain. We use the hidden state feature $\\mathrm{v}_R$ before each self-attention for attention injection to the Generation Chain.\n\n2. **Generation Chain**: For the early generation steps $t > t_\\text{align}$, we replace the KV values with $W^V (\\mathrm{v}_R) $, $W^K (\\mathrm{v}_R) $ using the hidden state $\\mathrm{v}_R$ from the inversion chain for self-attention calculation.  In the later generation steps, we concatenate $\\mathrm{v}_R$ and the hidden state from the generation chain itself $\\mathrm{v}_G$ to obtain new KV values. In all self-attention calculations, we do not change Q values and maintain them as $W^Q(\\mathrm{v}_G)$. It's worth noting that all parameters $W^{(\\cdot)}$ are frozen.\n### Q2 Clarification of Generation Method\n#### Q2-1 Does this proposed generation method rely on the off-the-shelf diffusion model without training? \nAs elaborated in response Q1, our training-free approach is built on the pre-trained Stable Diffusion model. Given its prowess in reconstructing real-world images using DDIM inversion, we infer its potential to generate images within a wide real-world domain. This foundation ensures RIVAL's plug-and-play functionality for arbitrary input images. Furthermore, we've showcased RIVAL's synergy with the recent diffusion model, SDXL, in **rebuttal Fig. A**.\n#### Q2-2 Is the chain alignment conducted in the latent feature or image space?\nIn our approach, we align two chains in the latent feature space. A visual ablation study was conducted and presented in Appendix Fig. 16. One of the reasons is that the chains should be aligned at the initial state $X_G^T$ and $X_R^T$. In the image generation process, we found that aligning in the latent space has a similar effect as aligning the residual part in the self-attention. This result paves the way for RIVAL's adaptability to potential latent-free diffusion models. We align in the latent space for simplicity and computational efficiency by adjusting the predicted noise in both branches, as detailed in Eq. 9-10.\n### Q3 Algorithmic Description\nBelow, we provide a Python-style pseudocode for RIVAL. A LaTeX algorithm version will be included in the revision.\n```py\ndef RIVAL(ref_img, c, cfg, DM, T):\n    '''\n    ref_img: the reference image\n    c: text condition\n    cfg: classifier-free guidance scale\n    DM: a pre-trained diffusion model\n    T: number of total denoising steps\n    return -> the generated image\n    '''\n    t_align = t_early = int(T*0.6)\n    # Eq.(5-6), generate an inversion latent chain of T steps using DDIM\n    inv_chain = DDIM_inv(ref_img, c, T, DM)\n    x_rt = inv_chain[-1] # the last latent X_R^T\n    # Eq.(8)\n    x_gt = shuffle(x_rt) # initialized latent in the generation chain\n\n    # start denoising process, t from T to 0\n    for t in range(T, 0, -1):\n        # Inversion chain is the normal denoising process of the diffusion model\n        # pred_r and pred_g are noise term predictions for each step\n        pred_r, v_rs = DM.unet(x_rt, c, T, cfg=1)\n        pred_g = x_gt\n\n        # use blks to denote the unet blocks\n        for blks in DM.unet.blks:\n            v_g = pred_g\n            # Eq.(3)\n            if t > t_align:\n                # v_r: residual hidden state of generation chain\n                v_r = blk(v_g)\n                v_kv = v_rs[blk]\n            else:\n                v_r = blk(v_g) # [bs, ndim, HW]\n                v_kv = torch.cat([v_rs[blk], v_r]) # [bs, ndim, 2*HW]\n\n            # Eq.(4)\n            pred_g = v_g + blk.self_attn(v_r, v_kv)\n            # Other operations in the unet are omitted for simplification\n            pred_g = blk.others(pred_g, c, T)\n\n        # Eq.(9), cfg rescale\n        pred_g = rescale(pred_g, cfg)\n\n        # Eq.(10), chain latent alignment\n        if t > t_early:\n            pred_g = AdaIN(pred_g, pred_r)\n\n        x_gt =  DDIM_scheduler(x_gt, pred_g, T)\n        x_rt =  DDIM_scheduler(x_rt, pred_r, T)\n\n    return DM.vae.decode(x_gt) # decode X_G^0 to get the variance results\n```\n### Q4 Explanation of the Shuffle Operation\nYou are correct. The shuffle operation $X_G^T = \\text{shuffle}(X_R^T)$ in Eq.8 is designed to perform a permutation in the pixel dimension. We'll ensure this is more explicitly stated in our revised version. Refer to the PyTorch-style code snippet in **4HxC@Q1**.\n### Q5 Clarification of \"Real-Image Rank\" and \"Preference Rank\" Metrics\nThe **Real-Image Rank** originates from the user study, where participants were asked to rank images according to their perceived authenticity - which image they believed was most likely not AI-generated.\n\nThe **Preference Rank**, on the other hand, originates from the user study where participants, given a reference image and text prompt, ranked generated images based on their adherence to the image reference and the text description.\n\nWe acknowledge the potential confusion between these terms and have renamed them \"Real-World Authenticity Preference\" and \"Condition Adherence Preference\" in the revision.\n### Q6 Improving Presentation\nWe appreciate your comments and advice on the presentation of the paper. We will include more details and refine our presentation to make the content more easily understandable in the revision."}, {"review_id": "eoJhAAyaY0", "reviewer": "Reviewer_5k69", "summary": "This paper proposes a framework RIVAL to generate variation of the real image without any tuning. It has two key components, the first one is a cross-image self-attention injection, where it first inverts the real image using DDIM, and then sample another random chain but uses a mix of real inverted chain’s value and denoised value in the denoising process. To mitigate the issue of out-of-distribution issue of the inverted latent of the real image, it further proposes a latent chain alignment process, where the random chain is initialised with the adaptively normalized Gaussian distribution of the inverted real latent. Experiments show improvement over baseline methods in terms of Text alignment and user preference. Overall the paper is well-written, the method is easy to follow.\n", "questions": "In Figure 4, the person's identity seems to change over regeneration, any idea on how to better preserve the identity? \n\nCan the proposed method work with image conditional DM? Would be great to see some visual examples\n", "limitations": "The authors provide limitation analysis, and see my questions above for further clarification", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The proposed method does not require training or finetuning, although the inference time becomes twice as long as the normal inference, it is acceptable in practice.\n\nRIVAL is shown to work on other text-to-image applications such as test-driven image generation with real-image condition and example-based inpainting.\n\nQualitative results show better identity preservation and style matching to the reference image compared to other baselines, plus, the editing results on text conditioning is impressive, considering no finetuning is needed.\n\nThe authors also provide detailed ablation studies on the choice of alignment steps and framework components.\n", "weaknesses": "The author does not provide analysis on the edibility of the proposed method, which I think is an important feature.\n\nHow sensitive is the method to the CFG guidance weight? There is no ablation study on the choice of the CFG weight\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 5k69,\n\nThank you for your valuable feedback and constructive comments. We will address your comments below and in the revised paper.\n### Q1 A Clarification of Inference Time\nWe want to clarify the point on inference time:\n\nAfter the inversion process, the overall inference time is not doubled. This claim is attributed to the concurrent execution of the denoising steps for both the inversion and generation chains when processing batched inputs, as illustrated in Fig. 2. The primary computational overhead emerges during the self-attention phase for $t>t_{\\text{align}}$, due to a doubling in QKV matrix multiplication from the increased size of KV (Eq. 4).\n\nAs mentioned in **aMYr@Q3**, in our experiment, we've cut RIVAL's inference time down to 6 seconds by employing half-precision and x-formers, maintaining 50 DDIM steps.\n\nFurthermore, efficiency could be further improved by caching hidden states of the inversion chain in the inversion process.\n\n### Q2 Sensitivity of the CFG Scale\nCFG scale in our diffusion model is an essential hyperparameter controlling the intensity of guidance. We have conducted a visual ablation study to examine the effect of various CFG scales, and the results are presented in Appendix Fig. 17 (a). The study reveals that the CFG scale influences the variations generated by our model, for instance, the color of the building in Appendix Fig. 17 (a). We performed a quantitative ablation on our dataset with different CFG values. It's worth noting that even with large CFG scales, our generated images did not exhibit color artifacts, as demonstrated in **rebuttal Fig. H**.\n|CFG scale|3|5|7|9|11|\n|-|-|-|-|-|-|\n|Text Alignment  |0.260|0.272|**0.273**|0.273|0.271|\n|Image Alignment |0.859|**0.863**|0.845|0.845|0.838|\n|Palette Distance|1.749|**1.685**|1.737|1.829|1.902|\n\nAs shown in the above table, our RIVAL maintains a balance between the strength of the text guidance and the preservation of features from the reference exemplar, thanks to the proposed latent noise rescaling mechanism in Eq. (9-10).\n### Q3 Identity Preservation in Fig. 4\nIdentity preservation, especially in the context of tuning-free customization methods such as our RIVAL, ELITE, and BLIP Diffusion, is indeed a challenging problem. Unlike customizing some \"easy\" concepts, like specific animal breeds and colors, preserving certain customizable concepts, like personal identity or particular objects, requires intricate control and optimization.\n\nOne feasible solution could be to integrate RIVAL with customized models. We provide an illustrative example in Fig. 9 and the last row of Appendix Fig. 13, where we combined RIVAL with DreamBooth's [sks] doll model. This ability demonstrates how our approach can work synergistically with customized models to achieve better identity preservation, and we will continually explore the direction of tuning-free personalization in our future work.\n\n### Q4 Compatibility with Image Conditional Diffusion Models\nRegarding image conditional diffusion models, we recognize two primary branches. The first is represented by models like UnCLIP (DALLE-2), which accepts an image as a semantic condition. As an example of this type of model's integration with RIVAL, we point to Appendix Fig. 12 (4th column), where we use the open-sourced ImgVar SDv1[29]. RIVAL is able to maintain the tone and low-level style consistency of the reference image, which contrasts with images generated solely using ImgVar (shown in each left-bottom corner of each image).\n\nThe second branch of image conditional diffusion models consists of controllable diffusion models like ControlNet. In this context, we demonstrate the application of the RIVAL integration with ControlNet and present examples in **rebuttal Fig. B**. \nThe implementation is straightforward. We adapt RIVAL with ControlNet by adding condition residuals of ControlNet blocks on the generation chain. Results illustrate that RIVAL, as a plug-and-play module, exhibits strong generalization capabilities when combined with various image-conditioned models.\n\n### Q5 \"Editability\" of RIVAL\nConcerning \"editability\", we understand it in three different ways. If your interpretation is different from ours, we would appreciate your feedback.\n\n1. **Model adaptability**: This refers to the editability of our method to adapt to other models. As discussed in Q4, our approach is highly adaptable across several popular frameworks, such as UnCLIP, ControlNet, and the recent SDXL. The results are shown in **rebuttal Fig. A&B**.\n2. **Text-conditioned image editing**: This refers to the editability for the text-conditioning image editing and generation task. We provide non-cherry-picked visual examples in Appendix Fig. 14 and **rebuttal Fig. C&E**. We also present some ablations concerning editability in Appendix Fig. 17 (c), focusing on selecting source prompts. Using an empty prompt for inversion, we achieve results with more flexibility and consistency with the text prompt. Among those experiments, RIVAL produces results that consistently adhere to the reference image's low-level attributes while maintaining a high-quality match with the text prompt. \n3. **Adjustability of the method itself**: This pertains to the editability of the method in terms of controlling the generation process. Our method provides several editable hyperparameters. For instance, the alignment steps $t_{align}$ and $t_{early}$ can be adjusted (a visual ablation is shown in Appendix Fig. 15, **quantitative results in aMYr@Q5**). Additionally, the target prompt (in image editing) and the CFG value can be adjusted for better text alignment, as mentioned in Q2."}]

Author rebuttal:
Dear all reviewers,

We want to express our gratitude to all reviewers for their thorough reviews and constructive comments. In the global rebuttal, we aim to summarize and address representative questions raised by the reviewers.

**We encourage all reviewers to refer to the global response for Q1 to gain a deeper insight into our motivation and applications.**
### Q1 Motivation and Problem Setting [all reviewers]
The primary goal of our paper is to introduce a tuning-free approach to produce images that resemble real-world samples. With a traditional T2I (Text-to-Image) pipeline, one can generate an image in the "Starry Night" style simply by inputting the prompt "in Starry Night style." However, considering that Van Gogh created 864 oil paintings in his lifetime using various techniques, generating images that closely align with an arbitrary real-world reference remains an essential problem.

While our objective may seem similar to image stylization, a fundamental distinction exists between generation (conditioned on image) and tasks based on the original image (like image stylization and editing). While generation often starts from random noise and produces free-form, unconstrained images, our method can act as a style-condtioned generation technique when references incorporate low-level features. Here, we initialize the latent as a normalized inverted latent of the content image. This process also allows us to harness image editing capabilities directly using the inverted latent.

To provide clarity, we present a table comparing various diffusion-based methods based on four key aspects: (text prompt, image exemplar, initial latent, and requires finetuning)
|Application |Text Prompt|Image Exemplar|Gen. from noise|Gen. from inverted latent|Requires Tuning|Representative Method|
|-|-|-|-|-|-|-|
|T2I generation|Y|-|Y|-|-|Stable Diffusion|
|Image Variation*|C|Y|Y|-|-|DALLE2|
|Style conditioned T2I*|Y|Y|Y|-|Y|StyleDrop|
|Image Stylization**|C|Y|-|Y|Y|InST|
|Image Editing**|Y|Y|-|Y|-|Prompt2Prompt, MasaCtrl|
|Image Customization|C|Y|C|-|C|DreamBooth, Custom Diffusion|
|**Our Method**|Y|Y|C|C|-|**RIVAL**|

>**Y: required, C: required in some methods/application**\
*: Demonstrated ability of RIVAL in the paper submission.\
**: Demonstrated ability of RIVAL in the rebuttal.

### Q2 More Applications [4HxC, aMYr, 5k69]
Given that RIVAL is a broad-based idea for the diffusion model during denoising inference, we received suggestions to expand our method to other diffusion-based applications. Below is a list of tasks RIVAL is adept at:
|Applications of RIVAL |Text Prompt|Image Exemplar|Gen. from noise|Gen. from inverted latent|
|-|-|-|-|-|
|Image Variation|Y|Y|Y|-|
|Style conditioned T2I|Y|Y|Y|-|
|Structure-guided T2I (ControlNet)|Y|Y (+Control condition)|Y|-|
|Image Stylization|Y|Y|-|Y|
|Image Editing|Y|Y|-|Y|

Building on Q1, we have expanded our approach to cover the three new applications listed in the last three columns for adapting RIVAL to different tasks. Generally, RIVAL is suitable for tasks that need consistent style/content generation in line with the exemplar.

### Q3 Table of Content: Experiments [mFBx, aMYr, 5k69]
Outlined below are the quantitative experiments conducted in this rebuttal:

1. **[mFBx@Q3-1] Module-wise ablation study**: We present a detailed module-wise ablation study to evaluate the impact of our method's four components (attention injection, attention feature fusion, latent initialization, and noise alignment).

2. **[aMYr@Q2] Comparison with plug-and-play methods**: We compare our image editing outcomes with PnP Diffusion and MasaCtrl.

3. **[aMYr@Q3] Speed comparison**: We delve into a time comparison for preparation and inference phases against PnP and MasaCtrl, with RIVAL showing competitive inference speed.

4. **[aMYr@Q5, 5k69@Q2] Hyperparameter Ablation**: Quantitative results are shown for selected timestep hyperparameters controlling the denoising process and for different CFG values, proving RIVAL's capability to mitigate artifacts from high CFG values.

### Q4 Table of Content: Visualizations [all reviewers]
We have made available a PDF containing visualization results of all pertinent experiments. We suggest reviewers examine our results, especially Fig. A, B, C, G, and H, within the provided PDF for comprehensive details.
### Q5 Clarifications
#### Q5-1 The algorithm workflow [mFBx, SLMY]
We appreciate the feedback and admit that our paper's presentation could be more precise. We have added an algorithmic description in **SLMY@Q3** to help reviewers better understand RIVAL's workflow.
#### Q5-2 Limitation analysis [4HxC, aMYr]
This is presented in **4HxC@Q7 and aMYr@Q7**.

#### Q5-3 Correction of formula expressions
Despite no direct feedback, we have spotted and rectified a typo in our inversion process in Eq. (5-6). Take Eq. (5) as an example. A coefficient $\sqrt{\alpha_{t-1}}$ should be multiplied before $(\beta_{t-1} - \beta_{t})$,
$$X^{t-1}=\sqrt{{\alpha_{t-1}}/{\alpha_t}} \cdot X^t+\sqrt{\alpha_{t-1}}* (\beta_{t-1} - \beta_{t}) \cdot \varepsilon_\theta(X^t, t, \mathcal{C}).$$

Furthermore, a simpler and corrected version of the adaptive normalization in Eq.(9) is:
$$\epsilon_\theta(X_G^{t}, t, \mathcal{C}) = \text{AdaIN}(\varepsilon_\theta^\mathrm{cfg}(X_G^{t}, t, \mathcal{C}), \varepsilon_\theta(X_R^{t}, t, \mathcal{C}))\text{, where}$$
$$\text{AdaIN}(a, b) = \sigma(b) (\frac{a - \mu(a)}{\sigma(a)}) + \mu(b).$$

In addition to the existing references, we have incorporated a few new ones in this rebuttal, which will appear in the revised version.

> *Inversion-Based Style Transfer With Diffusion Models*; **Yuxin Zhang et al.**; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 10146-10156

> *StyleDrop: Text-To-Image Generation in Any Style*; **Kihyuk Sohn et al.**; arXiv 2023

> *SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis*; **Dustin Podell et al.**; arXiv 2023


Meta-review: All reviewers are positive on the paper, recognizing its technical novelty and potential impact. The AC sees no reason to interfere with reviewer consensus, and is happy to recommend acceptance.
Decision: Accept (spotlight)