SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuechen Zhang $^{1}$  Jinbo Xing $^{1}$  Eric Lo $^{1}$  Jiaya Jia $^{1,2}$
<sup>1</sup>The Chinese University of Hong Kong <sup>2</sup>SmartMore
{yczhang21, jbxing, ericlo, leojia}@cse.cuhk.edu.hk
Image Variations
Generation with style condition
Figure 1: Leveraging an image exemplar, our training-free method excels in image generation tasks such as generating (a) image variations, (b) images of a similar style (with conditions).
Recent diffusion model advancements have enabled high-fidelity images to be generated using text prompts. However, a domain gap exists between generated images and real-world images, which poses a challenge in generating high-quality variations of real-world images. Our investigation uncovers that this domain gap originates from a latents' distribution gap in different diffusion processes. To address this issue, we propose a novel inference pipeline called Real-world Image Variation by ALignment (RIVAL) that utilizes diffusion models to generate image variations from a single image exemplar. Our pipeline enhances the generation quality of image variations by aligning the image generation process to the source image's inversion chain. Specifically, we demonstrate that step-wise latent distribution alignment is essential for generating high-quality variations. To attain this, we design a cross-image self-attention injection for feature interaction and a step-wise distribution normalization to align the latent features. Incorporating these alignment processes into a diffusion model allows RIVAL to generate high-quality image variations without further parameter optimization. Our experimental results demonstrate that our proposed approach outperforms existing methods concerning semantic similarity and perceptual quality. This generalized inference pipeline can be easily applied to other diffusion-based generation tasks, such as image-conditioned text-to-image generation and stylization. Project page:
https://rival-diff.github.io
Generating real-world image variation is a crucial area of research in computer vision and machine learning, owing to its practical applications in image editing, synthesis, and data augmentation [1, 2]. This task involves the generation of diverse variations of a given real-world image while preserving its semantic content and visual quality. Early methods for generating image variations included texture
Figure 2: Conceptual illustration of the challenges in the real-world image variation. Left: Despite using the same text prompt "a lynx sitting in the grass", a bias exists between the latent distribution in the vanilla generation chain and the image inversion chain, resulting in a significant domain gap between the denoised images. Right: Visualization of the real-image inverted four-channel latent. The distribution of the latent is biased in comparison to a standard Gaussian distribution.
synthesis, neural style transfer, and generative models [3, 4, 5, 6, 7, 8]. However, these methods are limited in generating realistic and diverse variations from real-world images and are only suited for generating variations of textures or artistic images.
Denoising Diffusion Probabilistic Models (DDPMs) have resulted in significant progress in text-driven image generation [9, 10, 11]. However, generating images that maintain the style and semantic content of the reference remains a significant challenge. Although advanced training-based methods [12, 13, 14, 15] can generate images with novel concepts and styles with given images, they require additional training stages and data. Directly incorporating image as the input condition [11, 16] results in suboptimal visual quality and content diversity compared to the reference input. Besides, they do not support input with text descriptions. A plug-and-play method has not been proposed to generate high-quality, real-world image variations without extra optimization.
Observing the powerful denoising ability of pre-trained DDPMs in recovering the original image from the inverted latent space [17, 18], we aim to overcome the primary challenge by modifying the vanilla latent denoising chain of the DDPM to fit the real-image inversion chain [17, 18, 19]. Despite generating images with the same text condition, a significant domain gap persists between the generated and source images, as depicted in Fig. 2. We identify distribution misalignment as the primary factor that impedes the diffusion model from capturing certain image features from latents in the inversion chain. As illustrated in the right portion of Fig. 2, an inverted latent may differ significantly from the standard Gaussian distribution. This misalignment accumulates during the denoising process, resulting in a domain gap between the generated image and its reference exemplar.
To address this distribution gap problem for generating image variations, we propose an pure inference pipeline called Real-world Image Variation by Alignment (RIVAL). RIVAL is a tunning-free approach that reduces the domain gap between the generated and real-world images by aligning the denoising chain with the real-image inversion chain. Our method comprises two key components: (i) a cross-image self-attention injection that enables cross-image feature interaction in the variation denoising chain, guided by the hidden states from the inversion chain, and (ii) a step-wise latent normalization that aligns the latent distribution with the inverted latent in early denoising steps. Notably, this modified inference process requires no training and is suitable for arbitrary image input.
As shown in Fig. 1, our proposed approach produces visually appealing image variations while maintaining semantic and style consistency with a given image exemplar. RIVAL remarkably improves the quality of image variation generation qualitatively and quantitatively compared to existing methods [11, 16, 20]. Furthermore, we have demonstrated that RIVAL's alignment process can be applied to other text-to-image tasks, such as text-driven image generation with real-image condition [21, 22, 23] and example-based inpainting [24, 25].
This paper makes three main contributions. (1) Using a real-world image exemplar, we propose a novel tunneling-free approach to generate high-quality image variations. (2) We introduce an latent alignment process to enhance the quality of the generated variations. (3) Our proposed method offers a promising denoising pipeline that can be applied across various applications.
Diffusion models with text control represent advanced techniques for controllable image generation with text prompts. With the increasing popularity of text-to-image diffusion models [11, 16, 26, 10,
Figure 3: High-level framework of RIVAL. Input exemplar  $R$  is inverted to a noisy latent  $X_{R}^{T}$ . An image variation  $G$  is generated from random noise following the same distribution as  $X_{R}^{T}$ . For each denoising step  $t$ , we interact  $X_{R}^{t}$  and  $X_{G}^{t}$  by self-attention injection and latent alignment.
27], high-quality 2D images can be generated through text prompt input. However, such methods cannot guarantee to generate images with the same low-level textures and tone mapping as the given image reference, which is difficult to describe using text prompts. Another line of works aims for diffusion-based concept customization, which requires fine-tuning the model and transferring image semantic contents or styles into text space for new concept learning [12, 13, 14, 15, 28]. However, this involves training images and extra tuning, making it unsuitable for plug-and-play inference.
Real-world image inversion is a commonly employed technique in Generative Adversarial Networks (GANs) for image editing and attribute manipulation [6, 29, 30, 31, 32]. It involves inverting images to the latent space for reconstructions. In diffusion models, the DDIM sampling technique [17] provides a deterministic and approximated invertible diffusion process. Recently developed inversion methods [18, 19] guarantee high-quality reconstruction with step-wise latent alignments. With diffusion inversion, real-image text-driven manipulations can be performed in image and video editing methods [33, 34, 23, 22, 35, 36, 37]. These editing methods heavily rely on the original structure of the input, thus cannot generate free-form variations with the same content as the reference image from the perspective of image generation.
Image variation generation involves generating diverse variations of a given image exemplar while preserving its semantic content and visual quality. Several methods have been proposed for this problem, including neural style transfer [5, 38, 39, 40], novel view synthesis [41, 42, 43], and GAN-based methods [7, 44, 32]. However, these methods were limited to generating variations of artistic images or structure-preserved ones and were unsuitable for generating realistic and diverse variations of real-world images. Diffusion-based methods [16, 11, 20] can generate inconsistent image variants by analyzing the semantic contents from the reference while lacking low-frequency details. Recent methods using image prompt to generate similar contents or styles [45, 46, 47, 48] requires additional case-wise tuning or training. Another concurrent approach, MasaCtrl [23], adopts self-attention injection as guidance for the denoising process. Yet, it can only generate variations with a generated image and fails on real-world image inputs. In contrast, RIVAL leverages the strengths of diffusion chain alignment to generate variations of real-world images.
3 Real-World Image Variation by Alignment
In this work, we define the image variation as the construction of a diffusion process  $F$  that satisfies  $D(R,C) \approx D(F_C(X,R),C)$ , where  $D(\cdot ,C)$  is a data distribution function with a given semantic content condition  $C$ . The diffusion process  $F_{C}(X,R) = G$  generates the image variation  $G$  based on a sampled latent feature  $X$ , condition  $C$ , and the exemplar image  $R$ .
The framework of RIVAL is illustrated in Fig. 3. RIVAL generates an inverted latent feature chain  $\{X_R^0, \dots, X_R^T\}$  by inverting a reference exemplar image  $R$  using DDIM inversion [17]. Then we obtain the initial latent  $X_R^T$  of the inversion chain. Next, a random latent feature  $X_G^T$  is sampled as the initial latent of the image generation (denoising) chain. In this multi-step denoising chain  $\{X_G^T, \dots, X_G^0\}$  for generation, we align the latent features to latents from the inversion chain to obtain perceptually similar generations. The modified step-wise denoising function  $f$  at step  $t$  can be represented abstractly as:
$$
X _ {G} ^ {t} = f _ {t} \left(X _ {G} ^ {t + 1}, X _ {R} ^ {t + 1}, C\right). \tag {1}
$$
Generated exemplar
+ cross-image attention
Figure 4: Exemplar images and generation results are obtained using the prompt "backpack in the wild". Cross-image self-attention can generate faithful outputs when a vanilla generation chain is employed as guidance. In the case of real-world images, the generation of faithful variations is dependent on the alignment of the latent.
Real exemplar
+ cross-image attention
+ latent aligned attention
This function is achieved by performing adaptive cross-image attention in self-attention blocks for feature interaction and performing latent distribution alignment in the denoising steps. The denoising chain can produce similar variations by leveraging latents in the inversion chain as guidance.
3.1 Cross-Image Self-Attention Injection
To generate image variations from an exemplar image  $R$ , a feature interaction between the inversion chain and the generation chain is crucial. Previous works [49, 22] have shown that self-attention can efficiently facilitate feature interactions. Similar to its applications [33, 23, 37] in text-driven generation and editing tasks, we utilize intermediate hidden states  $\mathbf{v}_R$  obtained from the inversion chain to modify the self-attention in the generation chain. Specifically, while keeping the inversion chain unchanged, our modified Key-Value features for cross-image self-attention for one denoising step  $t$  in the generation chain are defined as follows:
$$
Q = W ^ {Q} \left(\mathbf {v} _ {G}\right), K = W ^ {K} \left(\mathbf {v} _ {G} ^ {\prime}\right), V = W ^ {V} \left(\mathbf {v} _ {G} ^ {\prime}\right), \text {w h e r e} \tag {2}
$$
$$
\mathbf {v} _ {G} ^ {\prime} = \left\{ \begin{array}{l l} \mathbf {v} _ {G} \oplus \mathbf {v} _ {R} & \text {i f} t \leq t _ {\text {a l i g n}} \\ \mathbf {v} _ {R} & \text {o t h e r w i s e} \end{array} . \right. \tag {3}
$$
We denote  $W^{(\cdot)}$  as the pre-trained, frozen projections in the self-attention block and use  $\oplus$  to represent concatenation in the spatial dimension. Specifically, we adopt an Attention Fusion strategy. In early steps ( $t > t_{\mathrm{align}}$ ), we replace the KV values with  $W^{V}(\mathrm{v}_{R})$ ,  $W^{K}(\mathrm{v}_{R})$  using the hidden state  $\mathrm{v}_R$  from the inversion chain. In subsequent steps, we concatenate  $\mathrm{v}_R$  and the hidden state from the generation chain itself  $\mathrm{v}_G$  to obtain new Key-Values. We do not change Q values and maintain them as  $W^{Q}(\mathrm{v}_{G})$ . The proposed adaptive cross-image attention mechanism explicitly introduces feature interactions in the denoising process of the generation chain. Moreover, Attention Fusion strategy aligns the content distribution of the latent features in two denoising chains.
Building upon the self-attention mechanism [50], we obtain the updated hidden state output as  $\mathbf{v}_G^* = \mathrm{softmax}\big(QK^\top /\sqrt{d_k}\big)V$ , where  $d_{k}$  is the dimensionality of  $K$ . Cross-image self-attention and feature injection can facilitate the interaction between hidden features in the generation chain and the inversion chain. It should be noted that the inversion chain is deterministic throughout the inference process, which results in reference feature  $\mathbf{v}_R$  remaining independent of the generation chain.
3.2 Inverted Latent Chain Alignment
The cross-image self-attention injection is a potent method for generating image variations from a vanilla denoising process originating from a standard Gaussian distribution and is demonstrated in [23]. However, as depicted in Fig. 4, direct adaptation to real-world image input is not feasible due to a domain gap between real-world inverted latent chains and the vanilla generation chains. This leads to the attenuation of attention correlation during the calculation of self-attention. We further visualize this problem in Fig. 12. To facilitate the generation of real-world image variations, the pseudo-generation chain (inversion chain) of the reference exemplar can be estimated using the DDIM inversion [17]. Generating an image from latent features  $X^T$  using a small number of denoising steps is possible with the use of deterministic DDIM sampling:
$$
X ^ {t - 1} = \sqrt {\alpha_ {t - 1} / \alpha_ {t}} \cdot X ^ {t} + \sqrt {\alpha_ {t - 1}} \left(\beta_ {t - 1} - \beta_ {t}\right) \cdot \varepsilon_ {\theta} \left(X ^ {t}, t, \mathcal {C}\right), \tag {4}
$$
where step-wise coefficient is set to  $\beta_{t} = \sqrt{1 / \alpha_{t} - 1}$  and  $\varepsilon_{\theta}(X^{t},t,\mathcal{C})$  is the pre-trained noise prediction function in one timestep. Please refer to Appendix A for the details of  $\alpha_{t}$  and DDIM
sampling. With the assumption that noise predictions are similar in latents with adjacent time steps, DDIM sampling can be reversed In a small number of steps  $t\in [0,T]$  , using the equation:
$$
X ^ {t + 1} = \sqrt {\alpha_ {t + 1} / \alpha_ {t}} \cdot X ^ {t} + \sqrt {\alpha_ {t + 1}} \left(\beta_ {t + 1} - \beta_ {t}\right) \cdot \varepsilon_ {\theta} \left(X ^ {t}, t, \mathcal {C}\right), \tag {5}
$$
which is known as DDIM inversion [17]. We apply DDIM to acquire the latent representation  $X_{R}^{T}$  of a real image  $R$ . Nevertheless, the deterministic noise predictions in DDIM cause uncertainty in maintaining a normal distribution  $\mathcal{N}(\mathbf{0},\mathbf{I})$  of the inverted latent  $X_{R}^{T}$ . This deviation from the target distribution causes the attention-attenuation problem [51] between generation chains of  $R$  and  $G$ , leading to the generation of images that diverge from the reference. To tackle this challenge, we find a straightforward distribution alignment on the initial latent  $X_{G}^{T}$  useful, that is,
$$
x _ {G} ^ {T} \in X _ {G} ^ {T} \sim \mathcal {N} \left(\mu \left(X _ {R} ^ {T}\right), \sigma^ {2} \left(X _ {R} ^ {T}\right)\right), \text {o r} X _ {G} ^ {T} = \operatorname {s h u f f l e} \left(X _ {R} ^ {T}\right). \tag {6}
$$
In the alignment process, pixel-wise feature elements  $x_{G}^{T} \in X_{G}^{T}$  in the spatial dimension can be initialized from one of the two sources: (1) an adaptively normalized Gaussian distribution or (2) permuted samples from the inverted reference latent, shuffle  $(X_{R}^{T})$ . Our experiments show that both types of alignment yield comparable performance. This latent alignment strengthens the association between the variation latents and the exemplar's latents, increasing relevance within the self-attention mechanism and thus increasing the efficacy of the proposed cross-image attention.
In the context of initializing the latent variable  $X_G^T$ , further alignment steps within the denoising chain are indispensable. This necessity arises due to the limitations of the Classifier-Free Guidance (CFG) inference proposed in [52]. While effective for improving text prompt guidance in the generation chain, CFG cannot be directly utilized to reconstruct high-quality images through DDIM inversion. On the other hand, the noise scaling of CFG will affect the noise distribution and lead to a shift between latents in two chains, as shown in Fig. 11. To avoid this misalignment while attaining the advantage of the text guidance in CFG, we decouple two inference chains and rescale the noise prediction during denoising inference, formulated as an adaptive normalization [38]:
$$
\epsilon_ {\theta} \left(X _ {G} ^ {t}, t, \mathcal {C}\right) = \operatorname {A d a I N} \left(\varepsilon_ {\theta} ^ {\text {c f g}} \left(X _ {G} ^ {t}, t, \mathcal {C}\right), \varepsilon_ {\theta} \left(X _ {R} ^ {t}, t, \mathcal {C}\right)\right), \text {w h e r e} \tag {7}
$$
$$
\varepsilon_ {\theta} ^ {\mathrm {c f g}} \left(X _ {G} ^ {t}, t, \mathcal {C}\right) = m \varepsilon_ {\theta} \left(X _ {G} ^ {t}, t, \mathcal {C}\right) + (1 - m) \varepsilon_ {\theta} \left(X _ {G} ^ {t}, t, \mathcal {C} ^ {*}\right), \text {a n d} t > t _ {\text {e a r l y}}. \tag {8}
$$
$\varepsilon_{\theta}^{\mathrm{cfg}}(X_G^t,t,\mathcal{C})$  is the noise prediction using guidance scale  $m$  and unconditional null-text  $C^*$ . We incorporate this noise alignment approach during the initial stages  $(t > t_{\mathrm{early}})$ . After that, a vanilla CFG scaling is applied as the content distribution in  $G$  may not be consistent with  $R$ . Aligning the distributions at this early stage proves advantageous for aligning the overall denoising process.
We evaluate our proposed pipeline through multiple tasks in Sec. 4.2: image variation, image generation with text and image conditions, and example-based inpainting. Additionally, we provide quantitative assessments and user study results to evaluate generation quality in Sec. 4.3. A series of ablation studies and discussions of interpretability are conducted in Sec. 4.4 to assess the effectiveness of individual modules within RIVAL.
Our study obtained a high-quality test set of reference images from the Internet and DreamBooth [12] to ensure a diverse image dataset. To generate corresponding text prompts  $C$ , we utilized BLIP2 [54]. Our baseline model is Stable-Diffusion V1.5. During the image inversion and generation, we employed DDIM sample steps  $T = 50$  for each image and set the classifier-free guidance scale  $m = 7$  in Eq. (8). We split two stages at  $t_{\mathrm{align}} = t_{\mathrm{early}} = 30$  for attention alignment in Eq. (3) and latent alignment in Eq. (8). In addition, we employ the shuffle strategy described in Eq. (6) to initialize the starting latent  $X_G^T$ . Experiments run on a single NVIDIA RTX4090 GPU with 8 seconds to generate image variation with batch size 1.
4.2 Applications and Comparisons
Image variation. As depicted in Fig. 5, real-world image variations exhibit diverse characteristics to evaluate perceptual quality. Text-driven image generation using the basic Stable Diffusion [11] cannot use image exemplars explicitly, thus failing to get faithful image variations. While recent image-conditioned generators such as [20, 53, 16] have achieved significant progress for image input,
"An old black and white drawing of a city."
"A group of people standing around a statue."
"A photo of backpack."
"A photo of a dog lying on a sofa."
"A girl in a sailor uniform posing for a photo."
Source exemplar
SD variation
ELITE
DALLE-2
Figure 5: Real-world image variation. We compare RIVAL with recent competitive methods [11, 53, 20] conditioned on the same text prompt (the 2nd column) or exemplar image (3rd-5th columns).
RIVAL (Ours)
their reliance on encoding images into text space limits the representation of certain image features, including texture, color tone, lighting environment, and image style. In contrast, RIVAL generates image variations based on text descriptions and reference images, harnessing the real-image inversion chain to facilitate latent distribution alignments. Consequently, RIVAL ensures high visual similarity in both semantic content and low-level features. For instance, our approach is the sole technique that generates images with exceedingly bright or dark tones (2nd-row in Fig. 5). More visual comparisons and experimental settings can be found in Appendix C.
Text-driven image generation. In addition to the ability to generate images corresponding to the exemplar image and text prompts, we have also discovered that RIVAL has a strong ability to transfer styles and semantic concepts in the exemplar for a casual text-driven image generation. When an inversion with the original text prompt is performed, the alignment on denoising processes can still transfer high-level reference style to the generated images with modified prompts. This process could be directly used in structure-preserved editing. This could be regarded as the same task of MasaCtrl and Plug-and-Play [23, 34], which utilizes an initialized inverted latent representation to drive text-guided image editing. A comparison is shown in the right part of Fig. 6, here we directly use the inverted latent  $X_{G}^{T} = X_{R}^{T}$  for two chains. We adopt the same interaction starting step  $t = 45$  as the MasaCtrl.
Furthermore, in the absence of a structure-preserving prior, it is possible for a user-defined text prompt to govern the semantic content of a freely generated image  $G$ . This novel capability is showcased
"A  $\{\}$  is sitting in the grass"
"lynx"
w/o RIVAL\*
"dog"
"dog"
"cat"
"dog"
"A  $\{n\} \{\mathrm{adj.}\}$  is posing for a portrait"
{n.]:“women”
{adj.}:"\*"
fn.]: "robot"*
{adj.}: "in a sailor uniform"
{n]:“robot”
{n.]:“girl”
fn.1: "Elon Musk"
{adj.}
"Cat"
"Cat"
"raising hand"
Figure 6: Left: Text-driven image generation results using RIVAL, with the leftmost image as the source exemplar. RIVAL can generate images with diverse text inputs while preserving the reference style. Images noted with * indicate structure-preserved editing, which starts with the same initialized inverted reference latent  $X_{R}^{T}$  as the exemplar. Right: a visual comparison of structure-preserved image editing with recent approaches [34, 23].
"A photo of
Trump"
"Elon Musk
for a portrait"
Canny Edge
"city" → "portrait"
OpenPose
"A portrait of
Segmentation Map
"Big house"
"Big house"
"illustration of a
robot"
Figure 7: The availability of RIVAL with ControlNet [55]. Two examples are given for each modality of the control condition. Exemplars are shown on the left of each image pair.
spider man"
"illustration of a
illustrations robot
source exemplar
Paint-by-example
RIVAL