Title: Real-World Image Variation by Aligning Diffusion Inversion Chain
Venue: NeurIPS 2023
Accepted: yes
Keywords: image variation

diffusion model

image generation

text-driven image editing
Abstract: Recent diffusion model advancements have enabled high-fidelity images to be generated using text prompts. However, a domain gap exists between generated images and real-world images, which poses a challenge in generating high-quality variations of real-world images. Our investigation uncovers that this domain gap originates from a latents' distribution gap in different diffusion processes. To address this issue, we propose a novel inference pipeline called Real-world Image Variation by ALignment (RIVAL) that utilizes diffusion models to generate image variations from a single image exemplar. Our pipeline enhances the generation quality of image variations by aligning the image generation process to the source image's inversion chain. 
Specifically, we demonstrate that step-wise latent distribution alignment is essential for generating high-quality variations. 
To attain this, we design a cross-image self-attention injection for feature interaction and a step-wise distribution normalization to align the latent features. Incorporating these alignment processes into a diffusion model allows RIVAL to generate high-quality image variations without further parameter optimization. Our experimental results demonstrate that our proposed approach outperforms existing methods concerning semantic similarity and perceptual quality. This generalized inference pipeline can be easily applied to other diffusion-based generation tasks, such as image-conditioned text-to-image generation and stylization. Project page: https://rival-diff.github.io

Real-World Image Variation by Aligning Diffusion Inversion Chain

Abstract

Recent diffusion model advancements have enabled high-fidelity images to be generated using text prompts. However, a domain gap exists between generated images and real-world images, which poses a challenge in generating high-quality variations of real-world images. Our investigation uncovers that this domain gap originates from a latents' distribution gap in different diffusion processes. To address this issue, we propose a novel inference pipeline called R eal-world I mage V ariation by AL ignment (RIVAL) that utilizes diffusion models to generate image variations from a single image exemplar. Our pipeline enhances the generation quality of image variations by aligning the image generation process to the source image's inversion chain. Specifically, we demonstrate that step-wise latent distribution alignment is essential for generating high-quality variations. To attain this, we design a cross-image self-attention injection for feature interaction and a step-wise distribution normalization to align the latent features. Incorporating these alignment processes into a diffusion model allows RIVAL to generate high-quality image variations without further parameter optimization. Our experimental results demonstrate that our proposed approach outperforms existing methods concerning semantic similarity and perceptual quality. This generalized inference pipeline can be easily applied to other diffusion-based generation tasks, such as image-conditioned text-to-image generation and stylization. Project page: https://rival-diff.github.io

1 Introduction

Generating real-world image variation is a crucial area of research in computer vision and machine learning, owing to its practical applications in image editing, synthesis, and data augmentation [1, 2]. This task involves the generation of diverse variations of a given real-world image while preserving its semantic content and visual quality. Early methods for generating image variations included texture

value

distribution

Inverted

latent

!

!

"

"(0, 1)

Image exemplar

(

Real-image

inversion space

Denoising step

Real-image inversion chain

Vanilla denoising chain

Standard

!

!

"

!

!

#

!

$

#

!

$

"

synthesis, neural style transfer, and generative models [3, 4, 5, 6, 7, 8]. However, these methods are limited in generating realistic and diverse variations from real-world images and are only suited for generating variations of textures or artistic images.

Denoising Diffusion Probabilistic Models (DDPMs) have resulted in significant progress in text-driven image generation [9, 10, 11]. However, generating images that maintain the style and semantic content of the reference remains a significant challenge. Although advanced training-based methods [12, 13, 14, 15] can generate images with novel concepts and styles with given images, they require additional training stages and data. Directly incorporating image as the input condition [11, 16] results in suboptimal visual quality and content diversity compared to the reference input. Besides, they do not support input with text descriptions. A plug-and-play method has not been proposed to generate high-quality, real-world image variations without extra optimization.

Observing the powerful denoising ability of pre-trained DDPMs in recovering the original image from the inverted latent space [17, 18], we aim to overcome the primary challenge by modifying the vanilla latent denoising chain of the DDPM to fit the real-image inversion chain [17, 18, 19]. Despite generating images with the same text condition, a significant domain gap persists between the generated and source images, as depicted in Fig. 2. We identify distribution misalignment as the primary factor that impedes the diffusion model from capturing certain image features from latents in the inversion chain. As illustrated in the right portion of Fig. 2, an inverted latent may differ significantly from the standard Gaussian distribution. This misalignment accumulates during the denoising process, resulting in a domain gap between the generated image and its reference exemplar.

To address this distribution gap problem for generating image variations, we propose an pure inference pipeline called Real-world Image Variation by Alignment (RIVAL). RIVAL is a tunning-free approach that reduces the domain gap between the generated and real-world images by aligning the denoising chain with the real-image inversion chain. Our method comprises two key components: (i) a crossimage self-attention injection that enables cross-image feature interaction in the variation denoising chain, guided by the hidden states from the inversion chain, and (ii) a step-wise latent normalization that aligns the latent distribution with the inverted latent in early denoising steps. Notably, this modified inference process requires no training and is suitable for arbitrary image input.

As shown in Fig. 1, our proposed approach produces visually appealing image variations while maintaining semantic and style consistency with a given image exemplar. RIVAL remarkably improves the quality of image variation generation qualitatively and quantitatively compared to existing methods [11, 16, 20]. Furthermore, we have demonstrated that RIVAL's alignment process can be applied to other text-to-image tasks, such as text-driven image generation with real-image condition [21, 22, 23] and example-based inpainting [24, 25].

This paper makes three main contributions. (1) Using a real-world image exemplar, we propose a novel tunning-free approach to generate high-quality image variations. (2) We introduce an latent alignment process to enhance the quality of the generated variations. (3) Our proposed method offers a promising denoising pipeline that can be applied across various applications.

2 Related Works

Diffusion models with text control represent advanced techniques for controllable image generation with text prompts. With the increasing popularity of text-to-image diffusion models [11, 16, 26, 10,

Real-image inversion

T2I model

V

/

K

Q

Cross-image self-attention

…

…

…

…

!

"

!

!

#

!

$

#

!

$

%&'

!

$

%

!

$

%('

!

!

%&'

!

!

%

!

$

"

Image exemplar

!

Image variation

"

Latent

Alignment

" 0,%

Noise

align

V

/

K

injection

!

27], high-quality 2D images can be generated through text prompt input. However, such methods cannot guarantee to generate images with the same low-level textures and tone mapping as the given image reference, which is difficult to describe using text prompts. Another line of works aims for diffusion-based concept customization, which requires fine-tuning the model and transferring image semantic contents or styles into text space for new concept learning [12, 13, 14, 15, 28]. However, this involves training images and extra tuning, making it unsuitable for plug-and-play inference.

Real-world image inversion is a commonly employed technique in Generative Adversarial Networks (GANs) for image editing and attribute manipulation [6, 29, 30, 31, 32]. It involves inverting images to the latent space for reconstructions. In diffusion models, the DDIM sampling technique [17] provides a deterministic and approximated invertible diffusion process. Recently developed inversion methods [18, 19] guarantee high-quality reconstruction with step-wise latent alignments. With diffusion inversion, real-image text-driven manipulations can be performed in image and video editing methods [33, 34, 23, 22, 35, 36, 37]. These editing methods heavily rely on the original structure of the input, thus cannot generate free-form variations with the same content as the reference image from the perspective of image generation.

Image variation generation involves generating diverse variations of a given image exemplar while preserving its semantic content and visual quality. Several methods have been proposed for this problem, including neural style transfer [5, 38, 39, 40], novel view synthesis [41, 42, 43], and GANbased methods [7, 44, 32]. However, these methods were limited to generating variations of artistic images or structure-preserved ones and were unsuitable for generating realistic and diverse variations of real-world images. Diffusion-based methods [16, 11, 20] can generate inconsistent image variants by analyzing the semantic contents from the reference while lacking low-frequency details. Recent methods using image prompt to generate similar contents or styles [45, 46, 47, 48] requires additional case-wise tuning or training. Another concurrent approach, MasaCtrl [23], adopts self-attention injection as guidance for the denoising process. Yet, it can only generate variations with a generated image and fails on real-world image inputs. In contrast, RIV AL leverages the strengths of diffusion chain alignment to generate variations of real-world images.

3 Real-World Image Variation by Alignment

In this work, we define the image variation as the construction of a diffusion process F that satisfies D ( R,C ) ≈ D ( F C ( X,R ) , C ) , where D ( · , C ) is a data distribution function with a given semantic content condition C . The diffusion process F C ( X,R ) = G generates the image variation G based on a sampled latent feature X , condition C , and the exemplar image R .

The framework of RIVAL is illustrated in Fig. 3. RIVAL generates an inverted latent feature chain { X 0 R , ..., X T R } by inverting a reference exemplar image R using DDIM inversion [17]. Then we obtain the initial latent X T R of the inversion chain. Next, a random latent feature X T G is sampled as the initial latent of the image generation (denoising) chain. In this multi-step denoising chain { X T G , ..., X 0 G } for generation, we align the latent features to latents from the inversion chain to obtain perceptually similar generations. The modified step-wise denoising function f at step t can be represented abstractly as:

Real exemplar

+ cross-image attention

+ latent aligned attention

Generated exemplar

+ cross-image attention

This function is achieved by performing adaptive cross-image attention in self-attention blocks for feature interaction and performing latent distribution alignment in the denoising steps. The denoising chain can produce similar variations by leveraging latents in the inversion chain as guidance.

3.1 Cross-Image Self-Attention Injection

To generate image variations from an exemplar image R , a feature interaction between the inversion chain and the generation chain is crucial. Previous works [49, 22] have shown that self-attention can efficiently facilitate feature interactions. Similar to its applications [33, 23, 37] in text-driven generation and editing tasks, we utilize intermediate hidden states v R obtained from the inversion chain to modify the self-attention in the generation chain. Specifically, while keeping the inversion chain unchanged, our modified Key-Value features for cross-image self-attention for one denoising step t in the generation chain are defined as follows:

Wedenote W ( · ) as the pre-trained, frozen projections in the self-attention block and use ⊕ to represent concatenation in the spatial dimension. Specifically, we adopt an Attention Fusion strategy. In early steps ( t > t align ), we replace the KV values with W V (v R ) , W K (v R ) using the hidden state v R from the inversion chain. In subsequent steps, we concatenate v R and the hidden state from the generation chain itself v G to obtain new Key-Values. We do not change Q values and maintain them as W Q (v G ) . The proposed adaptive cross-image attention mechanism explicitly introduces feature interactions in the denoising process of the generation chain. Moreover, Attention Fusion strategy aligns the content distribution of the latent features in two denoising chains.

Building upon the self-attention mechanism [50], we obtain the updated hidden state output as v ∗ G = softmax ( QK ⊤ / √ d k ) V , where d k is the dimensionality of K . Cross-image self-attention and feature injection can facilitate the interaction between hidden features in the generation chain and the inversion chain. It should be noted that the inversion chain is deterministic throughout the inference process, which results in reference feature v R remaining independent of the generation chain.

3.2 Inverted Latent Chain Alignment

The cross-image self-attention injection is a potent method for generating image variations from a vanilla denoising process originating from a standard Gaussian distribution and is demonstrated in [23]. However, as depicted in Fig. 4, direct adaptation to real-world image input is not feasible due to a domain gap between real-world inverted latent chains and the vanilla generation chains. This leads to the attenuation of attention correlation during the calculation of self-attention. We further visualize this problem in Fig. 12. To facilitate the generation of real-world image variations, the pseudo-generation chain (inversion chain) of the reference exemplar can be estimated using the DDIM inversion [17]. Generating an image from latent features X T using a small number of denoising steps is possible with the use of deterministic DDIM sampling:

where step-wise coefficient is set to β t = √ 1 /α t -1 and ε θ ( X t , t, C ) is the pre-trained noise prediction function in one timestep. Please refer to Appendix A for the details of α t and DDIM

sampling. With the assumption that noise predictions are similar in latents with adjacent time steps, DDIM sampling can be reversed In a small number of steps t ∈ [0 , T ] , using the equation:

which is known as DDIM inversion [17]. We apply DDIM to acquire the latent representation X T R of a real image R . Nevertheless, the deterministic noise predictions in DDIM cause uncertainty in maintaining a normal distribution N ( 0 , I ) of the inverted latent X T R . This deviation from the target distribution causes the attention-attenuation problem [51] between generation chains of R and G , leading to the generation of images that diverge from the reference. To tackle this challenge, we find a straightforward distribution alignment on the initial latent X T G useful, that is,

In the alignment process, pixel-wise feature elements x T G ∈ X T G in the spatial dimension can be initialized from one of the two sources: (1) an adaptively normalized Gaussian distribution or (2) permutated samples from the inverted reference latent, shuffle( X T R ) . Our experiments show that both types of alignment yield comparable performance. This latent alignment strengthens the association between the variation latents and the exemplar's latents, increasing relevance within the self-attention mechanism and thus increasing the efficacy of the proposed cross-image attention.

In the context of initializing the latent variable X T G , further alignment steps within the denoising chain are indispensable. This necessity arises due to the limitations of the Classifier-Free Guidance (CFG) inference proposed in [52]. While effective for improving text prompt guidance in the generation chain, CFG cannot be directly utilized to reconstruct high-quality images through DDIM inversion. On the other hand, the noise scaling of CFG will affect the noise distribution and lead to a shift between latents in two chains, as shown in Fig. 11. To avoid this misalignment while attaining the advantage of the text guidance in CFG, we decouple two inference chains and rescale the noise prediction during denoising inference, formulated as an adaptive normalization [38]:

ε cfg θ ( X t G , t, C ) is the noise prediction using guidance scale m and unconditional null-text C ∗ . We incorporate this noise alignment approach during the initial stages ( t > t early ). After that, a vanilla CFG scaling is applied as the content distribution in G may not be consistent with R . Aligning the distributions at this early stage proves advantageous for aligning the overall denoising process.

4 Experiments

We evaluate our proposed pipeline through multiple tasks in Sec. 4.2: image variation, image generation with text and image conditions, and example-based inpainting. Additionally, we provide quantitative assessments and user study results to evaluate generation quality in Sec. 4.3. A series of ablation studies and discussions of interpretability are conducted in Sec. 4.4 to assess the effectiveness of individual modules within RIVAL.

4.1 Implementation Details

Our study obtained a high-quality test set of reference images from the Internet and DreamBooth [12] to ensure a diverse image dataset. To generate corresponding text prompts C , we utilized BLIP2 [54]. Our baseline model is Stable-Diffusion V1.5. During the image inversion and generation, we employed DDIM sample steps T = 50 for each image and set the classifier-free guidance scale m = 7 in Eq. (8). We split two stages at t align = t early = 30 for attention alignment in Eq. (3) and latent alignment in Eq. (8). In addition, we employ the shuffle strategy described in Eq. (6) to initialize the starting latent X T G . Experiments run on a single NVIDIA RTX4090 GPU with 8 seconds to generate image variation with batch size 1.

4.2 Applications and Comparisons

Image variation. As depicted in Fig. 5, real-world image variations exhibit diverse characteristics to evaluate perceptual quality. Text-driven image generation using the basic Stable Diffusion [11] cannot use image exemplars explicitly, thus failing to get faithful image variations. While recent image-conditioned generators such as [20, 53, 16] have achieved significant progress for image input,

'An old black and white drawing of a city.'

SD variation

ELITE

DALLE-2

Source exemplar

Stable Diffusion

'A group of people standing around a statue.'

'A girl in a sailor uniform posing for a photo.'

'A photo of backpack.'

'A photo of a dog lying on a sofa.'

RIVAL (Ours)

their reliance on encoding images into text space limits the representation of certain image features, including texture, color tone, lighting environment, and image style. In contrast, RIV AL generates image variations based on text descriptions and reference images, harnessing the real-image inversion chain to facilitate latent distribution alignments. Consequently, RIV AL ensures high visual similarity in both semantic content and low-level features. For instance, our approach is the sole technique that generates images with exceedingly bright or dark tones (2nd-row in Fig. 5). More visual comparisons and experimental settings can be found in Appendix C.

Text-driven image generation. In addition to the ability to generate images corresponding to the exemplar image and text prompts, we have also discovered that RIVAL has a strong ability to transfer styles and semantic concepts in the exemplar for a casual text-driven image generation. When an inversion with the original text prompt is performed, the alignment on denoising processes can still transfer high-level reference style to the generated images with modified prompts. This process could be directly used in structure-preserved editing. This could be regarded as the same task of MasaCtrl and Plug-and-Play [23, 34], which utilizes an initialized inverted latent representation to drive text-guided image editing. A comparison is shown in the right part of Fig. 6, here we directly use the inverted latent X T G = X T R for two chains. We adopt the same interaction starting step t = 45 as the MasaCtrl.

Furthermore, in the absence of a structure-preserving prior, it is possible for a user-defined text prompt to govern the semantic content of a freely generated image G . This novel capability is showcased

{adj.}: 'in a sailor uniform'

'lynx'*

'dog'*

'dog'

'cat'

w/o RIVAL*

{n.}: 'women',

{adj.}: '' *

{n.}: 'robot'

{n.}: 'girl'

{n.}: 'Elon Musk'

{adj.}: ''

{n.}: 'robot'*

'A {} is sitting in the grass'

'A {n.} {adj.} is posing for a portrait'

'Cat'

'Girl in sailor uniform'

'Cat'

'raising hand'

Plug-and-Play

MasaCtrl

Ours

'city' → 'portrait'

'big house'

'big house'

'a photo of Trump'

'a portrait of spider man'

'illustration of a robot'

'illustration of a robot'

'city' → 'portrait'

'big house'

'big house'

'a photo of Trump'

'a portrait of spider man'

'illustration of a robot'

'illustration of a robot'

'city' → 'portrait'

'big house'

'big house'

'city' → 'portrait'

'big house'

'big house'

'Elon Musk

for a portrait'

'city' → 'portrait'

'Big house'

'Big house'

'A portrait of

spider man'

'A photo of

Trump'

'illustration of a

robot'

'illustration of a

robot'

OpenPose

Segmentation Map

'a photo of Trump'

'a photo of Trump'

'a portrait of spider man'

'illustration of a robot'

'illustration of a robot'

'a portrait of spider man'

'illustration of a robot'

'illustration of a robot'

source exemplar

source exemplar

Paint-by-example

Paint-by-example

RIVAL

RIVAL

RIVAL-variation 2

in the left part of Fig. 6, where RIV AL successfully generates images driven by text input while maintaining the original style of the reference image. Further, to extend this ability to other image conditions, we adapt RIVAL with ControlNet Fig. 7 by adding condition residuals of ControlNet blocks on the generation chain, as shown in Fig. 7. With RIVAL, we can easily get a style-specific text-to-image generation. For instance, it can produce a portrait painting of a robot adorned in a sailor uniform while faithfully preserving the stylistic characteristics.

Example-based image inpainting. When abstracting RIVAL as a novel paradigm of image-based diffusion inference, we can extend this framework to enable it to encompass other image editing tasks like inpainting. Specifically, by incorporating a coarse mask M into the generation chain, we only permute the inverted latent variables within the mask to initialize the latent variable X T G . During denoising, we replace latents in the unmasked regions with latents in the inversion chain for each step. As shown in Fig. 8, we present a visual comparison between our approach and [25]. RIVAL produces a reasonable and visually harmonious inpainted result within the mask.

Baseline

Attention Inject

Attention fusion

Latent init.

Noise align

w/o attention

operations

w/o attn fusion,

noise align

w/o attn fusion

w/o latent init.

w/o noise align

Full model

Exemplar

(50, 50)

(40, 40)

(30, 30)

(0, 0)

(0, 50)

(50, 0)

4.3 Quantitative Evaluation

Wecompare RIVAL with several state-of-the-art methods, employing the widely used CLIP-score [56] evaluation metric for alignments with text and the image exemplar. The evaluation samples are from two datasets: the DreamBooth dataset [12] and our collected images. In addition, we cluster a 10-color palette using k -means and calculate the minimum bipartite distance to assess the similarity in lowlevel color tones. In general, the results in Tab. 1 demonstrate that RIV AL significantly outperforms other methods regarding semantic and low-level feature matching. Notably, our method achieves better text alignment with the real-image condition than the vanilla text-to-image method using the same model. Furthermore, our alignment results are on par with those attained by DALL·E 2 [16], which directly utilizes the image CLIP feature as the condition.

We conducted a user study to evaluate further the generated images' perceptual quality. This study utilized results generated by four shuffled methods based on the same source. Besides, the user study

CFG=3

CFG=7

CFG=11

CFG=1,

origin image

CFG=3

CFG=7

CFG=11

Vanilla Inverted Inference

RIVAL inference

Denoising Step

Denoising Step

includes a test for image authenticity, which lets users identify the "real" image among generated results. We collected responses of ranking results for each case from 41 participants regarding visual quality. The findings indicate a clear preference among human evaluators for our proposed approach over existing methods [53, 20, 16]. The detailed ranking results are presented in Tab. 1.

We also compare RIVAL with PnP and MasaCtrl for image editing on a representative test set, with additional metrics in Perceptual Similarity [57] and inference time. Our method presents a competitive result in Tab. 2.

4.4 Ablation Studies

We perform ablation studies to evaluate the efficacy of each module in our design. Additional visual and quantitative results can be found in Appendix D.

Cross-image self-attention injection plays a crucial role in aligning the feature space of the source image and the generated image. We eliminated the cross-image self-attention module and generated image variations to verify this. The results in Fig. 9 demonstrate that RIV AL fails to align the feature space without attention injection, generating images that do not correspond to the exemplar.

Latent alignment. To investigate the impact of step-wise latent distribution alignment, we remove latent alignments and sampling X T G ∼ N ( 0 , I ) . An example is shown in Fig. 4 and Fig. 9. Without latent alignment, RIVAL produces inconsistent variations of images, exhibiting biased tone and semantics compared to the source image. Additionally, we perform ablations on attention fusion in Eq. (3) and noise alignment in Eq. (8) during the early steps. Clear color and semantic bias emerge when the early alignment is absent, as shown in Fig. 10. Furthermore, if the values of t early or t align are excessively small, the latent alignment stage becomes protracted, resulting in the problem of over-alignment. Over-alignment gives rise to unwanted artifacts and generates blurry outputs.

Interpretability. We add experiments for the interpretability of RIV AL in the following two aspects. (a) Latent Similarity . To assess RIVAL components' efficacy regarding the distribution gap, we illustrate the KL divergence of noisy latent between chains A and B, X t A and X t B in the generation process, as depicted in Fig. 12 left part. Interactions between different distributions (green) widen the gap, while two generation chains with the same distribution (orange) can get a better alignment by attention interactions. With aligned latent chain alignment and interaction, RIV AL (purple) effectively generates real-world image variations.

(b) Reference Feature Contribution . Attention can be viewed as sampling value features from the key-query attention matrix. RIVAL converts self-attention to image-wise cross-attention. When

'A {} of <sks> {}.'

'photo + toy'

source exemplar

'photo + bag'

'photo + bag'

Vanilla Inference

RIVAL Inference

'painting + toy girl, in

'painting + toy girl'

latents are sourcing from the same distribution ( X T G , X T R ∼ N (0 , I ) ) images retain consistent style and content attributes (as in Fig. 12 left orange). This result is beneficial since we do not require complex text conditions c to constrain generated images to get similar content and style. For a more direct explanation, we visualize the bottleneck feature contributions of attention score presented in Fig. 12 right part. Reference contribution of the softmax score is denoted as score R = ∑ v i ∈ v R ( W Q v G · ( W K ( v i )) ⊤ ) / ∑ v j ∈ v G ⊕ v R ( W Q v G · ( W K ( v j )) ⊤ ) . As RIVAL adopts early fusion in the early steps, we use 50% as the score in the early steps. Latent initialization (orange) and with early alignments (green) play critical roles in ensuring a substantial contribution of the source feature in self-attention. A higher attention contribution helps in resolving the attention attenuation problem (purple) in the generation process.

4.5 Discussions

Integration with concept customization. In addition to its ability to generate image variations from a single source image using a text prompt input for semantic alignment, RIVAL can be effectively combined with optimization-based concept customization techniques, such as DreamBooth [12], to enable novel concept customization. As illustrated in Fig. 13, an optimized concept can efficiently explore the space of potential image variations by leveraging RIVAL's proficiency in real-world image inversion alignment.

Limitations and Future Directions. Despite introducing an innovative technique for crafting highquality inconsistent image variations, our method is contingent upon a text prompt input, potentially infusing semantic biases affecting image quality. As evidenced in Fig. 14, a prompt like "Pokemon" may lean towards popular choices like "Pikachu" due to training set biases, resulting in a Pikachudominated generation. Besides, the base model struggles to generate complex scenes and complicated concepts, e.g., "illustration of a little boy standing in front of a list of doors with butterflies around them in Fig. 14 (c)". This complexity can degrade the inversion chain and widen the domain gap, leading to less accurate results. Future studies might concentrate on refining diffusion models and exploring novel input avenues besides text prompts to mitigate these constraints.

(b) Complex scene

(c) Hard concept

(a) Semantic bias

5 Conclusion

This paper presents a novel pipeline for generating diverse and high-quality variations of real-world images while maintaining their semantic content and style. Our proposed approach addresses previous limitations by modifying the diffusion model's denoising inference chain to align with the real-image inversion chain. We introduce a cross-image self-attention injection and a step-wise latent alignment technique to facilitate alignment between two chains. Our method exhibits significant improvements in the quality of image variation generation compared to state-of-the-art methods, as demonstrated through qualitative and quantitative evaluations. Moreover, with the novel paradigm of hybrid text-image conditions, our approach can be easily extended to multiple text-to-image tasks.

Acknowledgements. This work is partially supported by the Research Grants Council under the Areas of Excellence scheme grant AoE/E-601/22-R, Hong Kong General Research Fund (14208023), Hong Kong AoE/P-404/18, and Centre for Perceptual and Interactive Intelligence (CPII) Limited under the Innovation and Technology Fund. We are grateful to Jingtao Zhou for the meaningful discussions.

References

Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NeurIPS , volume 33, pages 18661-18673, 2020. 1

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR , pages 9729-9738, 2020. 1

Alexei A Efros and Thomas K Leung. Texture synthesis by non-parametric sampling. In ICCV , volume 2, pages 1033-1038. IEEE, 1999. 2

Leon Gatys, Alexander S Ecker, and Matthias Bethge. Texture synthesis using convolutional neural networks. In NeurIPS , volume 28, 2015. 2

Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In CVPR , pages 2414-2423, 2016. 2, 3

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM , 63(11):139144, 2020. 2, 3

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In CVPR , pages 1125-1134, 2017. 2, 3

Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelcnn decoders. In NeurIPS , volume 29, 2016. 2

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS , volume 33, pages 6840-6851, 2020. 2

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-toimage diffusion models with deep language understanding. In NeurIPS , volume 35, pages 36479-36494, 2022. 2

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR , pages 10684-10695, 2022. 2, 3, 5, 6, 8, 21

Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR , 2023. 2, 3, 5, 8, 10, 21

Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In ICLR , 2023. 2, 3

Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In CVPR , 2023. 2, 3

Ziyi Dong, Pengxu Wei, and Liang Lin. Dreamartist: Towards controllable one-shot text-to-image generation via contrastive prompt-tuning. arXiv preprint arXiv:2211.11337 , 2022. 2, 3

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022. 2, 3, 5, 8, 9, 14, 15, 16, 21

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR , 2021. 2, 3, 4, 5, 14

Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. arXiv preprint arXiv:2211.09794 , 2022. 2, 3, 14, 20

Bram Wallace, Akash Gokul, and Nikhil Naik. Edict: Exact diffusion inversion via coupled transformations. arXiv preprint arXiv:2211.12446 , 2022. 2, 3

Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. arXiv preprint arXiv:2302.13848 , 2023. 2, 3, 5, 6, 8, 9, 14, 21

Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR , 2023. 2

Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Promptto-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 , 2022. 2, 3, 4

Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. arXiv preprint arXiv:2304.08465 , 2023. 2, 3, 4, 6, 7, 8, 14, 15

Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In CVPR , pages 11461-11471, June 2022. 2, 14

Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar-based image editing with diffusion models. arXiv preprint arXiv:2211.13227 , 2022. 2, 7

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML . PMLR, 2021. 2

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 , 2023. 3

Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR , 2021. 3

Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang. Gan inversion: A survey. PAMI , 45(3):3121-3138, 2023. 3

Siyu Xing, Chen Gong, Hewei Guo, Xiao-Yu Zhang, Xinwen Hou, and Yu Liu. Unsupervised domain adaptation gan inversion for image editing. arXiv preprint arXiv:2211.12123 , 2022. 3

Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. In ECCV , pages 592-608. Springer, 2020. 3

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR , pages 4401-4410, 2019. 3

Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia. Video-p2p: Video editing with crossattention control. arXiv preprint arXiv:2303.04761 , 2023. 3, 4

Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. arXiv preprint arXiv:2211.12572 , 2022. 3, 6, 7, 8

Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In ICML , pages 16784-16804. PMLR, 2022. 3

Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In ICLR , 2022. 3

Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv preprint arXiv:2303.09535 , 2023. 3, 4

Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV , pages 1501-1510, 2017. 3, 5

Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV , pages 694-711. Springer, 2016. 3

Nicholas Kolkin, Michal Kucera, Sylvain Paris, Daniel Sykora, Eli Shechtman, and Greg Shakhnarovich. Neural neighbor style transfer. arXiv preprint arXiv:2203.13215 , 2022. 3

Kai Zhang, Nick Kolkin, Sai Bi, Fujun Luan, Zexiang Xu, Eli Shechtman, and Noah Snavely. Arf: Artistic radiance fields. In ECCV , pages 717-733. Springer, 2022. 3

Yuechen Zhang, Zexin He, Jinbo Xing, Xufeng Yao, and Jiaya Jia. Ref-NPR: Reference-based nonphotorealistic radiance fields for controllable scene stylization. In CVPR , 2023. 3

Thu Nguyen-Phuoc, Feng Liu, and Lei Xiao. Snerf: stylized neural implicit representations for 3d scenes. ACM Transactions on Graphics (TOG) , 41(4):1-11, 2022. 3

Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV , pages 2223-2232, 2017. 3

Kihyuk Sohn, Nataniel Ruiz, Kimin Lee, Daniel Castro Chin, Irina Blok, Huiwen Chang, Jarred Barber, Lu Jiang, Glenn Entis, Yuanzhen Li, et al. Styledrop: Text-to-image generation in any style. arXiv preprint arXiv:2306.00983 , 2023. 3

Yuxin Zhang, Nisha Huang, Fan Tang, Haibin Huang, Chongyang Ma, Weiming Dong, and Changsheng Xu. Inversion-based style transfer with diffusion models. In CVPR , pages 10146-10156, 2023. 3

Dongxu Li, Junnan Li, and Steven CH Hoi. Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing. arXiv preprint arXiv:2305.14720 , 2023. 3

Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721 , 2023. 3

Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. arXiv preprint arXiv:2212.11565 , 2022. 4

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , volume 30, 2017. 4

Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In CVPR , pages 782-791, 2021. 5

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022. 5

Justin Pinkney. Experiments with stable diffusion. https://github.com/justinpinkney/ stable-diffusion , 2023. 5, 6, 8, 9, 14, 15, 21

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML , 2023. 5

Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543 , 2023. 7, 14

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , pages 8748-8763. PMLR, 2021. 8, 20

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR , 2018. 9

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS , 33:68406851, 2020. 14

Ivar Stangeby. A python library for extracting color palettes from supplied images. https://github. com/qTipTip/Pylette , 2022. 20

Mikołaj Bi´ nkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv:1801.01401 , 2018. 21

Ondˇ rej Texler, David Futschik, Michal Kuˇ cera, Ondˇ rej Jamriška, Šárka Sochorová, Menclei Chai, Sergey Tulyakov, and Daniel S` ykora. Interactive video stylization using few-shot patch-based training. ACM Transactions on Graphics (TOG) , 39(4):73-1, 2020. 21

Appendix

A Basic Background of Diffusion Models

This section uses a modified background description provided in [18]. We only consider the conditionfree case for the diffusion model here. Diffusion Denoising Probabilistic Models (DDPMs) [58] are generative latent variable models designed to approximate the data distribution q ( x 0 ) . The diffusion operation starts from the latent x 0 , adding step-wise noise to diffuse data into pure noise x T . It's important to note that this process can be viewed as a Markov chain starting from x 0 , where noise is gradually added to the data to generate the latent variables x 1 , . . . , x T ∈ X . The sequence of latent variables follows the conditional distribution q ( x 1 , . . . , x t | x 0 ) = ∏ t i =1 q ( x t | x t -1 ) . Each step in the forward process is defined by a Gaussian transition q ( x t | x t -1 ) := N ( x t ; √ 1 -k t x t -1 , k t I ) , which is parameterized by a schedule k 0 , . . . , k T ∈ (0 , 1) . As T becomes sufficiently large, the final noise vector x T approximates an isotropic Gaussian distribution.

The forward process allows us to express the latent variable x t directly as a linear combination of noise and x 0 , without the need to sample intermediate latent vectors.

where α t := ∏ t i =1 (1 -k i ) . To sample from the distribution q ( x 0 ) , a reversed denoising process is defined by sampling the posteriors q ( x t -1 | x t ) , which connects isotropic Gaussian noise x T to the actual data. However, the reverse process is computationally challenging due to its dependence on the unknown data distribution q ( x 0 ) . To overcome this obstacle, an approximation of the reverse process with a parameterized Gaussian transition network denoted as p θ ( x t -1 | x t ) , where p θ ( x t -1 | x t ) follows a normal distribution with mean µ θ ( x t , t ) and covariance Σ θ ( x t , t ) . As an alternative approach, the prediction of the noise ϵ θ ( x t , t ) added to x 0 , which is obtained using equation 9, can replace the use of µ θ ( x t , t ) as suggested in [58]. Bayes' theorem could be applied to approximate

Once we have a trained ϵ θ ( x t , t ) , we can using the following sample method

In DDIM sampling [17], a denoising process could become deterministic when set σ t = 0 .

B Details of the Attention Pipeline

We present a comparative analysis of attention injection methods. As depicted in Fig. 15, MasaCtrl [23], while also adopting a self-attention injection approach, employs a more complex control mechanism in its second stage. In the first stage of MasaCtrl, the inverted latent representation X R T is directly utilized by applying a modified prompt. In the second stage, a cross-attention mask is introduced to control specific word concepts modified in the prompt, which requires an additional forward pass. In contrast, our proposed method, RIV AL, primarily focuses on generating inconsistent variations. Consequently, we aim to guide feature interaction by replacing KV features with an aligned latent distribution. Unlike MasaCtrl, our approach does not limit content transfer through editing prompts with only a few words. Hence, in the second stage, we employ a single forward pass without calculating an additional cross-attention mask, allowing fast and flexible text-to-image generation with diverse text prompts.

In recent updates, ControlNet [55] has incorporated an attention mechanism resembling the second stage of RIVAL to address image variation. However, a notable distinction lies in using vanilla noised latents as guidance, leading to a process akin to the attention-only approach employed in RePaint [24] with the Stable Diffusion model. Consequently, this methodology is limited to generating images within the fine-tuned training data domain.

C More About Comparisons

Implementation Details. We compare our work with ELITE [20], Stable Diffusion image variation [53], and DALL·E 2 [16]. We utilize the official demo of ELITE to obtain results. To extract

T2I model

Q

!

!

"

!

#

"

!

#

"$%

T2I model

V

Q

!

#

"

!

#

"$%

K

V

K

T2I model

Q

!

!

"

!

#

"

!

#

"$%

Replacement

T2I model

V

Q

!

!

"

!

#

"

V

!

#

"$%

injection

K

K

V

K

!

#

&

= !

!

&

V

K

enc

dec

!

#

&

…

…

# ≥ #

'()*+

# < #

'()*+

# ≥ #

'()*+

# < #

'()*+

MasaCtrl

RIVAL

Specific mask

control+ replacement

(Additional forward to get mask)

#

'()*+

= 45

#

'()*+

= 30

(Single forward)

context tokens, we mask the entire image and employ the phrase "A photo/painting of <S>." based on the production method of each test image. Inference for ELITE employs the default setting with denoising steps set to T = 300 . For Stable Diffusion's image variation version, we utilize the default configuration, CFG guidance m = 3 , and denoising steps T = 50 . In the case of DALL·E 2, we utilize the official image variation API, specifically requesting using the most advanced API available to generate images of size 1024 × 1024 .

Comparison with UnCLIP. UnCLIP [16], also known as DALL·E 2, is an image generation framework trained using image CLIP features as direct input. Thanks to its large-scale training and image-direct conditioning design, it generates variations solely based on image conditions when adapted to image variation. However, when faced with hybrid image-text conditions, image-only UnCLIP struggles to produce satisfactory results, particularly when CLIP does not recognize the image content correctly. We provide comparative analysis in Fig. 16. Additionally, we demonstrate in the last two columns of Figure 16 that our approach can enhance the accuracy of low-level details in open-source image variation methods such as SD image variation [53].

Additional Visual Results. We showcase additional results of our techniques in variation generation, as illustrated in Fig. 17, and text-driven image generation with image condition, as shown in Fig. 18. The results unequivocally demonstrate the efficacy of our approach in generating a wide range of image variations that accurately adhere to textual and visual guidance.

'A girl in a sailor uniform posing for a photo.'

'Silhouette of a woman looking through a glass dome filled with fish.'

.'

'A building with a blue and white striped awning.'

'A group of people standing around a statue.'

RIVAL SDv1

Source exemplar

DALL·E 2

fail case

Exemplars

Variations

RO

STTEAATTE

SILE

D Additional Ablation Results

Module-wise ablation. We augment our visual ablations in Fig. 9 with a comprehensive module-wise experiment. The results in Tab. 3 underscore the role of each component and their combinations. Attention injection facilitates high-level feature interactions for better condition alignment (both text and image). Early fusion, built upon attention injection, aids in early step chain alignment, significantly enhancing image alignment. Meanwhile, latent noise alignment guarantees the preservation of color. Latent initialization has a pronounced impact, notably enhancing the color palette metric, an effect intensified by noise alignment.

Ablation on early fusion step. In addition to Fig. 10 of the main paper, we present comprehensive early-step evaluation results based on a grid search analysis in Fig. 19. By decreasing the duration of the feature replacement stage (larger t align ), we observe an increase in the similarity of textures and contents in the generated images. However, excessively long or short early latent alignment durations ( t early ) can lead to color misalignment. Users can adjust the size of the early fusion steps as hyperparameters to achieve the desired outcomes.

We also conduct a quantitative evaluation in Tab. 4. It shows that there is a balance between two conditions (text-alignment and image-alignment). Besides, low-level texture-pasting artifacts will

0

10

20

30

40

50

0

10

20

30

40

50

!

!"#$%

: color bias control

!

&'()*

: image similarity control

'A photo of a red vase.'

present when t align is small. The choice of t early primarily influences the generated images' style (color) bias.

w/o initialize

!

!

"

AdaIN

!

!

"

early noise

early latent

hidden state before self-attn

!

!

"

~#ℎ%&&'((!

#

"

)

!

!

"

~+(,,.)

hidden state each block

Ablation on different alignment designs. Fig. 20 illustrates ablations conducted on various alignment designs. Two latent initialization methods, as formulated in Eq. (6), exhibit comparable performance. Nevertheless, incorporating alignments in additional areas, such as hidden states within each transformer block, may harm performance. Hence, we opt for our RIVAL pipeline's simplest noise alignment strategy.

(a) Ablation on different CFG scales

'A

PINK

building'

m=1 (failed)

m=3

m=5

m=7

m=9

empty prompt

empty prompt

empty prompt

original prompt

original prompt

original prompt

{adj.}: 'in a sailor uniform'

{n.}: 'girl'

{n.}: 'Elon Musk'

{n.}: 'robot'

(b) Ablation on null-text inversion

(c) Ablation on different source prompts

Ablation on different text conditions. We conduct ablations on text conditions in three aspects. First, we evaluate the impact of different CFG scales m for text prompt guidance, with quantitative results in Tab. 5. As shown in Fig. 21 (a), our latent rescaling technique enables control over the text guidance level while preserving the reference exemplar's low-level features. Second, we employ an optimization-based null-text inversion method [18] to obtain an inversion chain with improved reconstruction quality. However, this method is computationally intensive, and the optimized embeddings are sensitive to the guidance scale. Furthermore, when incorporating this optimized embedding into the unconditional inference branch, there is a variation in generation quality, as depicted in Fig. 21 (b). Third, we utilize empty text as the source prompt to obtain the latents in the inversion chain while keeping the target prompt unchanged. As depicted in Fig. 21 (c), the empty text leads to weak semantic content correspondence between the two chains but sometimes benefits text-driven generation. For example, if users do not want to transfer the "gender" concept to the generated robot.

E Quantitative Evaluations

This section comprehensively evaluates our proposed method with various carefully designed metrics, including CLIP Score, color palette matching, user study, and KID.

CLIP Score. For evaluating the CLIP Score, we employ the official ViT-Large-Patch14 CLIP model [56] and compute the cosine similarity between the projected features, yielding the output.

Color palette matching. To perform low-level matching, we utilize the Pylette tool [59] to extract a set of 10 palette colors. Subsequently, we conduct a bipartite matching between the color palette of each generated image and the reference palette colors in the RGB color space. Before matching, each color is scaled to [0 , 1] . The matching result is obtained by calculating the sum of L1 distances.

User study. To evaluate the effectiveness of our approach against other methods, we conducted a user study using an online form. The user study interface, depicted in Figure 22, was designed to elicit

* 01 Specification

pleaserankthefourmethodsfromhightolowbasedontwoaspects:

image authenticity (how diffcult it is to distinguish from images in the real world (photo/painting) and

OI understood.

(a) Specification

(b) Shuffled results

(c) Ranking questions

* 02 [image authenticity] An old black and white drawing of a city

Drag the right option or click to the left to sort

A

B

C

D

A

B

*03[Conditionmatching]Anoldblackandwhitedrawingofacity

Drag the right option or click to the left to sort

A

Referencelmage

B

C

D

C

D

user rankings of image variation results. We collected 41 questionnaire responses, encompassing 16 cases of ranking comparisons.

KID evaluation. To provide a comprehensive assessment of the quality, we utilize Kernel Inception Distance (KID)[60] to evaluate the perceptual generation quality of our test set. As depicted in Table6, with Stable Diffusion V1-5, our method achieves the best KID score, which is slightly superior to the UnCLIP [16], employing the advanced Stable Diffusion V2-1.

F Additional Considerations

Data acquisition. To comprehensively evaluate our method, we collected diverse source exemplars from multiple public datasets, such as DreamBooth [12] and Interactive Video Stylization [61]. Some exemplars were obtained from Google and Behance solely for research purposes. We will not release our self-collected example data due to license restrictions.

Societal impacts. This paper introduces a novel framework for image generation that leverages a hybrid image-text condition, facilitating the generation of diverse image variations. Although this application has the potential to be misused by malicious actors for disinformation purposes, significant advancements have been achieved in detecting malicious generation. Consequently, we anticipate that our work will contribute to this domain. In forthcoming iterations of our method, we intend to introduce the NSFW (Not Safe for Work) test for detecting possible malicious generations. Through rigorous experimentation and analysis, our objective is to enhance comprehension of image generation techniques and alleviate their potential misuse.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a framework RIVAL to generate variation of the real image without any tuning. It has two key components, the first one is a cross-image self-attention injection, where it first inverts the real image using DDIM, and then sample another random chain but uses a mix of real inverted chain’s value and denoised value in the denoising process. To mitigate the issue of out-of-distribution issue of the inverted latent of the real image, it further proposes a latent chain alignment process, where the random chain is initialised with the adaptively normalized Gaussian distribution of the inverted real latent. Experiments show improvement over baseline methods in terms of Text alignment and user preference. Overall the paper is well-written, the method is easy to follow.\n\nSTRENGTHS:\nThe proposed method does not require training or finetuning, although the inference time becomes twice as long as the normal inference, it is acceptable in practice.\n\nRIVAL is shown to work on other text-to-image applications such as test-driven image generation with real-image condition and example-based inpainting.\n\nQualitative results show better identity preservation and style matching to the reference image compared to other baselines, plus, the editing results on text conditioning is impressive, considering no finetuning is needed.\n\nThe authors also provide detailed ablation studies on the choice of alignment steps and framework components.\n\nWEAKNESSES:\nThe author does not provide analysis on the edibility of the proposed method, which I think is an important feature.\n\nHow sensitive is the method to the CFG guidance weight? There is no ablation study on the choice of the CFG weight\n\nQUESTIONS:\nIn Figure 4, the person's identity seems to change over regeneration, any idea on how to better preserve the identity? \n\nCan the proposed method work with image conditional DM? Would be great to see some visual examples\n\nLIMITATIONS:\nThe authors provide limitation analysis, and see my questions above for further clarification", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper works on the design of diffusion model to generate image variations given an image examplar as the source image.  The basic idea is to align the image generation process to the image inversion process of source image. This is achieved by designing an cross-image self-attention injection for feature interaction between the source and generated images in the diffusion process. The results demonstrate better performance compared with the existing methods in image-conditioned text-to-image generation and examplar-based image inpainting.\n\nSTRENGTHS:\nThe idea to generate image variations given a source image is an interesting task. This paper proposed to align the source image inversion process and image generation process using the cross-image self-attention injection, which is a simple idea and shows good image generation results.\n\nWEAKNESSES:\n1. The presentation and training of the parameters in \"cross-image self-attention injection\" are unclear to me. I understand that, in sect. 3.1, the features of examplar image and the generated image are interacted based on eqns. (2-4), in their inversion and generation chain respectively. However, how to train/determine the parameters in these equations? Are these parameters shared across different diffusion steps?  And how do these parameters affect the final generation results?\n\n2. Does this proposed generation method rely on the off-the-shelf diffusion model without any training? The chain alignment is conducted in the latent feature space or the image space? \n\n3. An algorithmic description of this proposed method for image generation should be given for better understanding the training (possibly) and image generation process?\n\n4. The shuffle operation in sect. 3.2 is described as \"permutated sample from the inverted reference latent\". Please clarify on this description, does this mean permutating the pixels of X_R^T?\n\n5. In the Table I, please clarify on the metric of \"Real-image Rank\" and \"Preference Rank\".  I know that lines 228-233 may describe them, however, it is still unclear to me on the difference between \"Real-image Rank\" and \"Preference Rank\".\n\nQUESTIONS:\nI am interested in the problem to tackle and the proposed idea of feature interaction between source and generated image for inverse and generation chain alignment. However, I have many unclear points on the presentation of the proposed method (as mentioned in the weakness). I suggest authors to clarify the method in rebuttal and improve the writings and presentations of this manuscript.\n\nLIMITATIONS:\nThe paper discussed on the limitation of this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a tunning-free pipeline called RIVAL(Real-world Image Variation by Alignment) for generating diverse and high-quality variations of real-world images.\n\nIn previous works, some models also generate images with novel concepts and styles but require additional training stages and data, and others directly incorporated images as the input condition results in suboptimal visual quality and content diversity.\n\nTo tackle this issue, the authors suggest a pipeline that maintains the style and semantic content of the reference without additional training stages and data.\n\nThe proposed method comprises the inverted latent chain alignment(step-wise latent normalization) and cross-image self-attention injection.\n\nThe authors determine whether aligning the inverted latent chain and injecting the modified self-attention key and value by adopting $t_{align}$ and $t_{early}$.\n\nUnlike the other papers that use the cross-image self-attention injection method with an additional forward to calculate the cross-attention mask, the suggested approach only needs a single forward.\n\nThe authors show that RIVAL enhances performance by applying the alignment process to other text-to-image tasks through experiments and ablation studies.\n\nSTRENGTHS:\n- The authors claimed a novel plug-and-play pipeline to generate high-quality, real-world image variations without additional optimization through an inverted latent chain alignment and cross-image self-attention injection.\n- The proposed pipeline enhances the performance of text-to-image models, such as text-driven image generation with real-image conditions and example-based inpainting. The pipeline generates images fast and flexibly with diverse text prompts adopting a new self-attention injection approach with a single forward pass.\n- Compared to CFG, the advantage of the proposed pipeline is its ability to avoid misalignment between latents in two chains and its direct utilization. The authors decouple the two inference chains and rescale noise prediction during denoising inference. Furthermore, they formulate noise prediction as an adaptive normalization.\n- The quantitative results outperform the other pipelines provided in the paper except for one metric, and the qualitative results show that RIVAL generates images with diverse text inputs while preserving the reference style.\n\nWEAKNESSES:\nThe experiments need to be more extensive.\n\n- The authors proposed a plug-and-play pipeline, but there are no quantitative results compared with the other plug-and-play pipelines mentioned in the paper. RIVAL used a similar scheme as MasaCtrl[2] and Plug-and-Play[3] except the latents alignment. The authors employed similar schemes to the pipeline, so comparing the quantitative results with the others would be beneficial.\n- In the appendix, the proposed method allows fast and flexible text-to-image generation, but there are no results about how fast it is. It would be reasonable to compare quantitatively with other plug-and-play pipelines.\n\nThe authors need to validate the ability of the pipeline to apply to other diffusion-based generation tasks.\n\n- The authors show the qualitative results of self-example image inpainting in Fig. 6, but the results lack experiments. It would be hard to explain the ability to extend RIVAL’s framework to other image editing tasks.\n\n(Minor) The notation of $t_{align}$ and $t_{early}$ is confusing.\n\n[1] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In *CVPR*, pages 10684–10695, 2022.[2] Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. *arXiv preprint arXiv:2304.08465*, 2023.[3] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. *arXiv preprint arXiv:2211.12572*, 2022.\n\nQUESTIONS:\n- In Table 1, I wonder if using specific $t_{align}$ and $t_{early}$ values for each metric would lead to better quantitative results.\n- In Fig. 6, the inpainting results show a different masked area. Can you show me the same results by overlaying the masked area at the source image like SD[1] in the paper?\n- Can you provide examples of other tasks to which RIVAL can be applied?\n\n[1] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolutionimage synthesis with latent diffusion models. In *CVPR*, pages 10684–10695, 2022.\n\nLIMITATIONS:\nThe paper needs more extensive experiments and analysis of the limitations of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nGenerating real-world image variations is an important research task with practical applications such as image editing, image synthesis, and data augmentation. Past approaches include texture synthesis, neural style transfer, and generative models, among others. This study proposes a method called \"Real-world Image Variation by ALignment\", which generates high-quality image variations from a single image sample through the inference process of an aligned diffusion model. The paper provides experiments on different images.\n\nSTRENGTHS:\nThe overall effect is very good. The proposed method takes full advantage of the properties of the diffusion model.\n\nWEAKNESSES:\nThere are a lot of recent papers based on a very well trained diffusion generative model. The results of these papers are generally very good (after all, it is based on a very mature large model). These papers generally have some micro-designs, which make the academic meaning of the papers more prominent. However, such papers have a common problem, that is, they do not discuss the research issues in a very specific way. Often only stay in the effect. But this hardly accounts for the actual scholarly progress of these papers. Chances are it's just a manifestation of the capabilities of the base model itself.\n\nTaking this paper as an example, the paper proposes the method of Aligning Diffusion Inversion Chain. But the necessity and interpretability of the method is not fully discussed. In fact, there are many \"tricky\" ways to achieve the effect of this paper, and it is not necessary to go through the method described in Section 3 (By the way, the writing of Section 3 is very difficult to understand. In fact, a lot of useless mathematics is unnecessary. It is recommended that authors use more carefully set diagrams to explain their methods). I doubt the necessity of the proposed method. I hope the author can have a full ablation experiment, and compare and discuss it with a wide range of existing tricks. Of course, I also realize that such a requirement is a bit too harsh. Because many methods and tricks have not been written into papers or formally disseminated. The authors may not be able to adequately discuss every possibility.\n\nInterpretability is also another important topic. I want to know the specific physical meaning of each latent and representation in the inversion transfer or the point that it can be exploited in this method. This paper doesn't really explain it (or even try to).\n\n=====================\n\nPost rebuttal:\n\nThanks for the rebuttal. I am willing to rase my score to accept (7). But I recommend authors to add these new discussions in to the final version.\n\nQUESTIONS:\nTwo general questions:\n1. Ablation study.\n2. Interpretability.\n\nPlease see Weaknesses for full details.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the domain gap between generated images and real-world images, as it is challenging in generating high-quality variations of real-world images. It mainly contains two contributions, the first one is the cross-image self-attention injection for generating images that correspond to the given exemplar, and the second one is the latent alignment, which can help to eliminate the biased tone and semantics. The comparison with stable diffusion, SD variation, and ELITE validates the effectiveness of each component. Lastly, the ablation studies analyze in depth the contribution of each submodule, showing the positive impact of each part.\n\nSTRENGTHS:\n+ The paper is well-written and easy to follow for most parts, except when introducing the details of the shuffle operation in equation 8. It is not easy to directly understand the operation. \n+ The paper has a nice flow of descriptions and figures to illustrate the qualitative comparison.\n+ The diagrams are easy to understand and are well-made.\n+ The main contribution of the proposed cross-image self-attention injection and latent alignment are intuitive and well-executed in this paper. Each design is reasonable to handle the corresponding challenges, like object custom and style consistency.\n+ The ablation study is sufficient to validate the effectiveness of each component in designing the whole network, the concatenation of cross-image features in different steps, and the alignment of two features.\n+ Experiments show superior performance in comparison with recent works.\n\nWEAKNESSES:\n- Generally speaking, this work is more like the recent customized text-to-image generation, except that this work considered style consistency, like the analyses in Figure 4 (2nd-row). Since these competing methods did not consider this style factor, so they easily have the tones inconsistency problem. Directly comparing with them seems not very fair, as they can generate great results, except for the inconsistent tone distribution. In my opinion, it seems not a crucial issue.\n- The concatenation of v_G and v_R in equation 3 is also like some training-free video generation, which adopts the cross-frame interaction. The second proposed latent alignment is widely used in style transfer tasks.\n\nQUESTIONS:\nSome deep analyses of the failed cases seem not enough. The provided examples cover very wide ranges, like semantic bias, complex scenes, and hard concepts.\nWhat is the difference between custom image generation and variation generation? It seems that it only has the style difference, as this work needs to keep the style consistent with the given exemplar. In this case, I think the authors should discuss these related works, and compare with them by considering style transfer factors.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}