SlowGuess/NeurIPS_2023_OCR huggingface dataset:
source exemplar
Figure 8: RIVAL extended to self-example image inpainting. We use the same image as the example to fill the masked area. Results are compared with a well-trained inpainting SOTA method [25].
RIVAL-variation 2
in the left part of Fig. 6, where RIVAL successfully generates images driven by text input while maintaining the original style of the reference image. Further, to extend this ability to other image conditions, we adapt RIVAL with ControlNet Fig. 7 by adding condition residuals of ControlNet blocks on the generation chain, as shown in Fig. 7. With RIVAL, we can easily get a style-specific text-to-image generation. For instance, it can produce a portrait painting of a robot adorned in a sailor uniform while faithfully preserving the stylistic characteristics.
Example-based image inpainting. When abstracting RIVAL as a novel paradigm of image-based diffusion inference, we can extend this framework to enable it to encompass other image editing tasks like inpainting. Specifically, by incorporating a coarse mask  $M$  into the generation chain, we only permute the inverted latent variables within the mask to initialize the latent variable  $X_{G}^{T}$ . During denoising, we replace latents in the unmasked regions with latents in the inversion chain for each step. As shown in Fig. 8, we present a visual comparison between our approach and [25]. RIVAL produces a reasonable and visually harmonious inpainted result within the mask.
Metric	SD [11]	ImgVar [53]	ELITE [20]	DALL·E 2 [16]	RIVAL
Text Alignment ↑	0.255 ± 0.04	0.223 ± 0.04	0.209 ± 0.05	0.253 ± 0.04	0.275 ± 0.03
Image Alignment ↑	0.748 ± 0.08	0.832 ± 0.07	0.736 ± 0.09	0.897 ± 0.05	0.840 ± 0.07
Palette Distance ↓	3.650 ± 1.30	3.005 ± 0.86	2.885 ± 0.81	2.102 ± 0.83	1.674 ± 0.63
Real-world Authenticity ↓	-	2.982 (4.7%)	3.526 (9.6%)	1.961 (28.0%)	1.530 (61.6%)
Condition Adherence ↓	-	3.146 (7.3%)	3.353 (5.6%)	1.897 (30.6%)	1.603 (56.4%)
Table 1: Quantitative comparisons. We evaluate the quality of image variation regarding feature matching within different levels (color palette, text feature, image feature), highlighted with best and second best results. We also report user preference rankings and the first ranked rate.
Methods	Text Align. ↑	Image Align. ↑	Palette Dist. ↓	LPIPS ↓	Preparation Time (s) ↓	Inference Time (s) ↓
PnP [34]	0.249	0.786	1.803	0.245	200	21
MasaCtrl [23]	0.226	0.827	1.308	0.274	6	15
RIVAL	0.231	0.831	1.192	0.245	6	15
Table 2: Quantitative comparisons with image editing methods. Both in performance and efficiency (time consumption on one NVIDIA RTX3090 for fair comparison).
Figure 9: Left: a visual example for a module-wise ablation study.
Figure 10: Ablation study for different early-alignment strategies with the exemplar in Fig. 5. We list  $(t_{\mathrm{align}}, t_{\mathrm{early}})$  pairs for each image. All images are generated from the same fixed latent.
We compare RIVAL with several state-of-the-art methods, employing the widely used CLIP-score [56] evaluation metric for alignments with text and the image exemplar. The evaluation samples are from two datasets: the DreamBooth dataset [12] and our collected images. In addition, we cluster a 10-color palette using  $k$ -means and calculate the minimum bipartite distance to assess the similarity in low-level color tones. In general, the results in Tab. 1 demonstrate that RIVAL significantly outperforms other methods regarding semantic and low-level feature matching. Notably, our method achieves better text alignment with the real-image condition than the vanilla text-to-image method using the same model. Furthermore, our alignment results are on par with those attained by DALL-E 2 [16], which directly utilizes the image CLIP feature as the condition.
We conducted a user study to evaluate further the generated images' perceptual quality. This study utilized results generated by four shuffled methods based on the same source. Besides, the user study
CFG=1, origin image
Va CFG=3
Vanilla Inverted Inference CFG=7 C
ce CFG=11
CFG=3
RIVAL inference  
CFG=7
CFG=11
Figure 11: High CFG artifacts defense. Left: vanilla generation, starting from the DDIM inverted latent. Right: RIVAL generated images, starting from the permuted latents.
Figure 12: RIVAL's interpretability. Left: KL divergence between latent among two chains across denoising steps. Right: attention score with the reference feature with different alignment strategies.
includes a test for image authenticity, which lets users identify the "real" image among generated results. We collected responses of ranking results for each case from 41 participants regarding visual quality. The findings indicate a clear preference among human evaluators for our proposed approach over existing methods [53, 20, 16]. The detailed ranking results are presented in Tab. 1.
We also compare RIVAL with PnP and MasaCtrl for image editing on a representative test set, with additional metrics in Perceptual Similarity [57] and inference time. Our method presents a competitive result in Tab. 2.
We perform ablation studies to evaluate the efficacy of each module in our design. Additional visual and quantitative results can be found in Appendix D.
Cross-image self-attention injection plays a crucial role in aligning the feature space of the source image and the generated image. We eliminated the cross-image self-attention module and generated image variations to verify this. The results in Fig. 9 demonstrate that RIVAL fails to align the feature space without attention injection, generating images that do not correspond to the exemplar.
Latent alignment. To investigate the impact of step-wise latent distribution alignment, we remove latent alignments and sampling  $X_G^T \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ . An example is shown in Fig. 4 and Fig. 9. Without latent alignment, RIVAL produces inconsistent variations of images, exhibiting biased tone and semantics compared to the source image. Additionally, we perform ablations on attention fusion in Eq. (3) and noise alignment in Eq. (8) during the early steps. Clear color and semantic bias emerge when the early alignment is absent, as shown in Fig. 10. Furthermore, if the values of  $t_{\text{early}}$  or  $t_{\text{align}}$  are excessively small, the latent alignment stage becomes protracted, resulting in the problem of over-alignment. Over-alignment gives rise to unwanted artifacts and generates blurry outputs.
Interpretability. We add experiments for the interpretability of RIVAL in the following two aspects. (a) Latent Similarity. To assess RIVAL components' efficacy regarding the distribution gap, we illustrate the KL divergence of noisy latent between chains A and B,  $X_A^t$  and  $X_B^t$  in the generation process, as depicted in Fig. 12 left part. Interactions between different distributions (green) widen the gap, while two generation chains with the same distribution (orange) can get a better alignment by attention interactions. With aligned latent chain alignment and interaction, RIVAL (purple) effectively generates real-world image variations.
(b) Reference Feature Contribution. Attention can be viewed as sampling value features from the key-query attention matrix. RIVAL converts self-attention to image-wise cross-attention. When
"A  $\{\}$  of  $<  s k s >$  {}.
"photo + toy" source exemplar
"photo + bag"
Figure 13: RIVAL enables seamless customization of optimized novel concepts through text prompt control (<sks>). With various text prompt inputs, we can still generate images preserving the tones, style, and contents of the provided source exemplar (depicted on the left).
"painting + toy girl, in the style of Vermeer."
"photo + bag"
RIVAL Inference
"painting + toy girl"
latents are sourcing from the same distribution  $(X_{G}^{T}, X_{R}^{T} \sim \mathcal{N}(0, I))$  images retain consistent style and content attributes (as in Fig. 12 left orange). This result is beneficial since we do not require complex text conditions  $c$  to constrain generated images to get similar content and style. For a more direct explanation, we visualize the bottleneck feature contributions of attention score presented in Fig. 12 right part. Reference contribution of the softmax score is denoted as  $\mathrm{score}_{R} = \sum_{v_{i} \in \mathbf{v}_{R}} (W^{Q} \mathbf{v}_{G} \cdot (W^{K}(v_{i}))^{\top}) / \sum_{v_{j} \in \mathbf{v}_{G} \oplus \mathbf{v}_{R}} (W^{Q} \mathbf{v}_{G} \cdot (W^{K}(v_{j}))^{\top})$ . As RIVAL adopts early fusion in the early steps, we use  $50\%$  as the score in the early steps. Latent initialization (orange) and with early alignments (green) play critical roles in ensuring a substantial contribution of the source feature in self-attention. A higher attention contribution helps in resolving the attention attenuation problem (purple) in the generation process.
4.5 Discussions
Integration with concept customization. In addition to its ability to generate image variations from a single source image using a text prompt input for semantic alignment, RIVAL can be effectively combined with optimization-based concept customization techniques, such as DreamBooth [12], to enable novel concept customization. As illustrated in Fig. 13, an optimized concept can efficiently explore the space of potential image variations by leveraging RIVAL's proficiency in real-world image inversion alignment.
Limitations and Future Directions. Despite introducing an innovative technique for crafting high-quality inconsistent image variations, our method is contingent upon a text prompt input, potentially infusing semantic biases affecting image quality. As evidenced in Fig. 14, a prompt like "Pokemon" may lean towards popular choices like "Pikachu" due to training set biases, resulting in a Pikachu-dominated generation. Besides, the base model struggles to generate complex scenes and complicated concepts, e.g., "illustration of a little boy standing in front of a list of doors with butterflies around them in Fig. 14 (c)". This complexity can degrade the inversion chain and widen the domain gap, leading to less accurate results. Future studies might concentrate on refining diffusion models and exploring novel input avenues besides text prompts to mitigate these constraints.
(a) Semantic bias
Figure 14: Fail cases of RIVAL. The exemplar image is on the left for each case.
(b) Complex scene
(c) Hard concept
This paper presents a novel pipeline for generating diverse and high-quality variations of real-world images while maintaining their semantic content and style. Our proposed approach addresses previous limitations by modifying the diffusion model's denoising inference chain to align with the real-image inversion chain. We introduce a cross-image self-attention injection and a step-wise latent alignment technique to facilitate alignment between two chains. Our method exhibits significant improvements in the quality of image variation generation compared to state-of-the-art methods, as demonstrated through qualitative and quantitative evaluations. Moreover, with the novel paradigm of hybrid text-image conditions, our approach can be easily extended to multiple text-to-image tasks.
Acknowledgements. This work is partially supported by the Research Grants Council under the Areas of Excellence scheme grant AoE/E-601/22-R, Hong Kong General Research Fund (14208023), Hong Kong AoE/P-404/18, and Centre for Perceptual and Interactive Intelligence (CPII) Limited under the Innovation and Technology Fund. We are grateful to Jingtao Zhou for the meaningful discussions.
A Basic Background of Diffusion Models
This section uses a modified background description provided in [18]. We only consider the condition-free case for the diffusion model here. Diffusion Denoising Probabilistic Models (DDPMs) [58] are generative latent variable models designed to approximate the data distribution  $q(x_0)$ . The diffusion operation starts from the latent  $x_0$ , adding step-wise noise to diffuse data into pure noise  $x_{T}$ . It's important to note that this process can be viewed as a Markov chain starting from  $x_0$ , where noise is gradually added to the data to generate the latent variables  $x_{1},\ldots ,x_{T}\in X$ . The sequence of latent variables follows the conditional distribution  $q(x_{1},\dots,x_{t}\mid x_{0}) = \prod_{i = 1}^{t}q(x_{t}\mid x_{t - 1})$ . Each step in the forward process is defined by a Gaussian transition  $q(x_{t}\mid x_{t - 1})\coloneqq \mathcal{N}(x_{t};\sqrt{1 - k_{t}} x_{t - 1},k_{t}I)$  which is parameterized by a schedule  $k_{0},\ldots ,k_{T}\in (0,1)$ . As  $T$  becomes sufficiently large, the final noise vector  $x_{T}$  approximates an isotropic Gaussian distribution.
The forward process allows us to express the latent variable  $x_{t}$  directly as a linear combination of noise and  $x_0$ , without the need to sample intermediate latent vectors.
$$
x _ {t} = \sqrt {\alpha_ {t}} x _ {0} + \sqrt {1 - \alpha_ {t}} w, w \sim \mathcal {N} (\mathbf {0}, I), \tag {9}
$$
where  $\alpha_{t} \coloneqq \prod_{i=1}^{t}(1 - k_{i})$ . To sample from the distribution  $q(x_0)$ , a reversed denoising process is defined by sampling the posteriors  $q(x_{t-1} \mid x_t)$ , which connects isotropic Gaussian noise  $x_T$  to the actual data. However, the reverse process is computationally challenging due to its dependence on the unknown data distribution  $q(x_0)$ . To overcome this obstacle, an approximation of the reverse process with a parameterized Gaussian transition network denoted as  $p_\theta(x_{t-1} \mid x_t)$ , where  $p_\theta(x_{t-1} \mid x_t)$  follows a normal distribution with mean  $\mu_\theta(x_t, t)$  and covariance  $\Sigma_\theta(x_t, t)$ . As an alternative approach, the prediction of the noise  $\epsilon_\theta(x_t, t)$  added to  $x_0$ , which is obtained using equation 9, can replace the use of  $\mu_\theta(x_t, t)$  as suggested in [58]. Bayes' theorem could be applied to approximate
$$
\mu_ {\theta} \left(x _ {t}, t\right) = \frac {1}{\sqrt {\alpha_ {t}}} \left(x _ {t} - \frac {k _ {t}}{\sqrt {1 - \alpha_ {t}}} \epsilon_ {\theta} \left(x _ {t}, t\right)\right). \tag {10}
$$
Once we have a trained  $\epsilon_{\theta}(x_t,t)$ , we can using the following sample method
$$
x _ {t - 1} = \mu_ {\theta} \left(x _ {t}, t\right) + \sigma_ {t} z, \quad z \sim \mathcal {N} (\mathbf {0}, I). \tag {11}
$$
In DDIM sampling [17], a denoising process could become deterministic when set  $\sigma_t = 0$ .
B Details of the Attention Pipeline
We present a comparative analysis of attention injection methods. As depicted in Fig. 15, MasaCtrl [23], while also adopting a self-attention injection approach, employs a more complex control mechanism in its second stage. In the first stage of MasaCtrl, the inverted latent representation  $X_{T}^{R}$  is directly utilized by applying a modified prompt. In the second stage, a cross-attention mask is introduced to control specific word concepts modified in the prompt, which requires an additional forward pass. In contrast, our proposed method, RIVAL, primarily focuses on generating inconsistent variations. Consequently, we aim to guide feature interaction by replacing  $KV$  features with an aligned latent distribution. Unlike MasaCtrl, our approach does not limit content transfer through editing prompts with only a few words. Hence, in the second stage, we employ a single forward pass without calculating an additional cross-attention mask, allowing fast and flexible text-to-image generation with diverse text prompts.
In recent updates, ControlNet [55] has incorporated an attention mechanism resembling the second stage of RIVAL to address image variation. However, a notable distinction lies in using vanilla noised latents as guidance, leading to a process akin to the attention-only approach employed in RePaint [24] with the Stable Diffusion model. Consequently, this methodology is limited to generating images within the fine-tuned training data domain.
C More About Comparisons
Implementation Details. We compare our work with ELITE [20], Stable Diffusion image variation [53], and DALL-E 2 [16]. We utilize the official demo of ELITE to obtain results. To extract
Figure 15: Self-attention control compare with MasaCtrl [23]. The default split of two stages is shown as a bar for each method.
Experiments	AI	AF	LI	NA	Palette Distance	Text Alignment	Image Alignment
Baseline	3.917	0.266	0.751
AI	✓	3.564	0.277	0.804
AF	✓	✓	3.518	0.274	0.820
LI.	✓	3.102	0.268	0.764
NA	✓	3.661	0.251	0.647
w/o AI&AF	✓	✓	2.576	0.276	0.760
w/o AF&NA	✓	✓	2.419	0.279	0.817
w/o AF	✓	✓	✓	1.902	0.274	0.818
w/o LI	✓	✓	✓	3.741	0.242	0.653
w/o NA	✓	✓	✓	2.335	0.267	0.839
Full Model	✓	✓	✓	✓	1.810	0.268	0.846
Table 3: Module-wise ablation (AI: Attention Injection, AF: Attention Fusion, LI: Latent Initialization, NA: Noise Alignment) with quantitative results.
context tokens, we mask the entire image and employ the phrase "A photo/painting of  $< S>$ ." based on the production method of each test image. Inference for ELITE employs the default setting with denoising steps set to  $T = 300$ . For Stable Diffusion's image variation version, we utilize the default configuration, CFG guidance  $m = 3$ , and denoising steps  $T = 50$ . In the case of DALL-E 2, we utilize the official image variation API, specifically requesting using the most advanced API available to generate images of size  $1024 \times 1024$ .
Comparison with UnCLIP. UnCLIP [16], also known as DALL-E 2, is an image generation framework trained using image CLIP features as direct input. Thanks to its large-scale training and image-direct conditioning design, it generates variations solely based on image conditions when adapted to image variation. However, when faced with hybrid image-text conditions, image-only UnCLIP struggles to produce satisfactory results, particularly when CLIP does not recognize the image content correctly. We provide comparative analysis in Fig. 16. Additionally, we demonstrate in the last two columns of Figure 16 that our approach can enhance the accuracy of low-level details in open-source image variation methods such as SD image variation [53].
Additional Visual Results. We showcase additional results of our techniques in variation generation, as illustrated in Fig. 17, and text-driven image generation with image condition, as shown in Fig. 18. The results unequivocally demonstrate the efficacy of our approach in generating a wide range of image variations that accurately adhere to textual and visual guidance.
"Pokémon"
"A building with a blue and white striped awning."
"Silhouette of a woman looking through a glass dome filled with fish."
DALL·E 2
UnCLIP SDv2
ImgVar SDv1 + RIVAL
Figure 16: Comparison and adaptation with UnCLIP [16]. We highlight texts that enhance the image understanding for each case. Our inference pipeline is adapted to the image variation model depicted in the fourth column, in contrast to the variation achieved through vanilla inference in the bottom left corner of each image.
RIVAL SDv1
(talign, tearly)	(30, 30)	(0, 30)	(30, 0)	(0, 0)	(30, 50)	(50, 30)	(50, 50)
Text Alignment	0.268	0.261	0.269	0.259	0.267	0.274	0.279
Image Alignment	0.846	0.873	0.838	0.865	0.839	0.813	0.817
Palette Distance	1.810	1.421	1.806	1.483	2.359	2.061	2.419
Table 4: Quantitative ablation for different alignment timesteps.
Figure 17: Text-driven free-form image generation results. The image reference is in the left column. In the last row, we also present variations for one customized concept <sks> bag.
Figure 18: Text-driven free-form image generation results, with the image reference placed in the top left corner. The text prompts used are identical to those presented in Fig. 6 of the main paper. Every two rows correspond to a shared text prompt.
CFG scale	3	5	7	9	11
Text Alignment	0.260	0.272	0.273	0.273	0.271
Image Alignment	0.859	0.863	0.845	0.845	0.838
Palette Distance	1.749	1.685	1.737	1.829	1.902
Table 5: Quantitative ablation for different Classifier-Free Guidance Scale.
D Additional Ablation Results
Module-wise ablation. We augment our visual ablations in Fig. 9 with a comprehensive module-wise experiment. The results in Tab. 3 underscore the role of each component and their combinations. Attention injection facilitates high-level feature interactions for better condition alignment (both text and image). Early fusion, built upon attention injection, aids in early step chain alignment, significantly enhancing image alignment. Meanwhile, latent noise alignment guarantees the preservation of color. Latent initialization has a pronounced impact, notably enhancing the color palette metric, an effect intensified by noise alignment.
Ablation on early fusion step. In addition to Fig. 10 of the main paper, we present comprehensive early-step evaluation results based on a grid search analysis in Fig. 19. By decreasing the duration of the feature replacement stage (larger  $t_{\mathrm{align}}$ ), we observe an increase in the similarity of textures and contents in the generated images. However, excessively long or short early latent alignment durations ( $t_{\mathrm{early}}$ ) can lead to color misalignment. Users can adjust the size of the early fusion steps as hyperparameters to achieve the desired outcomes.
We also conduct a quantitative evaluation in Tab. 4. It shows that there is a balance between two conditions (text-alignment and image-alignment). Besides, low-level texture-pasting artifacts will
Figure 19: Ablation results for alignment steps, with the reference exemplar at the bottom right. We fix each generation's initial latent  $X_G^T$ .
present when  $t_{\text{align}}$  is small. The choice of  $t_{\text{early}}$  primarily influences the generated images' style (color) bias.
Figure 20: Ablation studies for different feature alignment strategies.
Ablation on different alignment designs. Fig. 20 illustrates ablations conducted on various alignment designs. Two latent initialization methods, as formulated in Eq. (6), exhibit comparable performance. Nevertheless, incorporating alignments in additional areas, such as hidden states within each transformer block, may harm performance. Hence, we opt for our RIVAL pipeline's simplest noise alignment strategy.
(a) Ablation on different CFG scales
$\mathrm{m = 1}$  (failed)
m=3
m=5
m=7
m=9
(b) Ablation on null-text inversion
w/o opt. null
w/ opt. null
(c) Ablation on different source prompts
original prompt
Figure 21: Ablation studies on different text conditions and guidance scales. Reference exemplars are highlighted with a golden border.
original prompt
Ablation on different text conditions. We conduct ablations on text conditions in three aspects. First, we evaluate the impact of different CFG scales  $m$  for text prompt guidance, with quantitative results in Tab. 5. As shown in Fig. 21 (a), our latent rescaling technique enables control over the text guidance level while preserving the reference exemplar's low-level features. Second, we employ an optimization-based null-text inversion method [18] to obtain an inversion chain with improved reconstruction quality. However, this method is computationally intensive, and the optimized embeddings are sensitive to the guidance scale. Furthermore, when incorporating this optimized embedding into the unconditional inference branch, there is a variation in generation quality, as depicted in Fig. 21 (b). Third, we utilize empty text as the source prompt to obtain the latents in the inversion chain while keeping the target prompt unchanged. As depicted in Fig. 21 (c), the empty text leads to weak semantic content correspondence between the two chains but sometimes benefits text-driven generation. For example, if users do not want to transfer the "gender" concept to the generated robot.
E Quantitative Evaluations
This section comprehensively evaluates our proposed method with various carefully designed metrics, including CLIP Score, color palette matching, user study, and KID.
CLIP Score. For evaluating the CLIP Score, we employ the official ViT-Large-Patch14 CLIP model [56] and compute the cosine similarity between the projected features, yielding the output.