SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sihan Xu $^{1*}$  Ziqiao Ma $^{1*}$  Yidong Huang $^{1}$  Honglak Lee $^{1,2}$  Joyce Chai $^{1}$
<sup>1</sup>University of Michigan, <sup>2</sup>LG AI Research
{sihanxu,marstin,owenhji,honglak,chaijy}@umich.edu
Diffusion models (DMs) have enabled breakthroughs in image synthesis tasks but lack an intuitive interface for consistent image-to-image (I2I) translation. Various methods have been explored to address this issue, including mask-based methods, attention-based methods, and image-conditioning. However, it remains a critical challenge to enable unpaired I2I translation with pre-trained DMs while maintaining satisfying consistency. This paper introduces CycleNet, a novel but simple method that incorporates cycle consistency into DMs to regularize image manipulation. We validate CycleNet on unpaired I2I tasks of different granularities. Besides the scene and object level translation, we additionally contribute a multi-domain I2I translation dataset to study the physical state changes of objects. Our empirical studies show that CycleNet is superior in translation consistency and quality, and can generate high-quality images for out-of-domain distributions with a simple change of the textual prompt. CycleNet is a practical framework, which is robust even with very limited training data (around 2k) and requires minimal computational resources (1 GPU) to train.
Figure 1: A high-resolution example of CycleNet for diffusion-based image-to-image translation compared to other diffusion-based methods. CycleNet produces high-quality translations with satisfactory consistency. The areas in the boxes are enlarged for detailed comparisons.
Recently, pre-trained diffusion models (DMs) [38, 37, 39] have enabled an unprecedented breakthrough in image synthesis tasks. Compared to GANs [9] and VAEs [22], DMs exhibit superior stability and quality in image generation, as well as the capability to scale up to open-world multimodal data. As such, pre-trained DMs have been applied to image-to-image (I2I) translation, which is to acquire a mapping between images from two distinct domains, e.g., different scenes, different
objects, and different object states. For such translations, text-guided diffusion models typically require mask layers [32, 2, 7, 1] or attention control [10, 30, 25, 35]. However, the quality of masks and attention maps can be unpredictable in complex scenes, leading to semantic and structural changes that are undesirable. Recently, researchers have explored using additional image-conditioning to perform paired I2I translations with the help of a side network [51] or an adapter [31]. Still, it remains an open challenge to adapt pre-trained DMs in unpaired I2I translation with a consistency guarantee.
We emphasize that consistency, a desirable property in image manipulation, is particularly important in unpaired I2I scenarios where there is no guaranteed correspondence between images in the source and target domains. Various applications of DMs, including video prediction and infilling [14], imagination-augmented language understanding [49], robotic manipulation [18, 8] and world models [46], would rely on strong consistency across the source and generated images.
To enable unpaired I2I translation using pre-trained DMs with satisfactory consistency, this paper introduces CycleNet, which allows DMs to translate a source image by conditioning on the input image and text prompts. More specifically, we adopt ControlNet [51] with pre-trained Stable Diffusion (SD) [38] as the latent DM backbone. Motivated by cycle consistency in GAN-based methods [55], CycleNet leverages consistency regularization over the image translation cycle. As illustrated in Figure 2, the image translation cycle includes a forward translation from  $x_0$  to  $\bar{y}_0$  and a backward translation to  $\bar{x}_0$ . The key idea of our method is to ensure that when conditioned on an image  $c_{\mathrm{img}}$  that falls into the target domain specified by  $c_{\mathrm{text}}$ , the DM should be able to reproduce this image condition through the reverse process.
We validate CycleNet on I2I translation tasks of different granularities. Besides the scene and object level tasks introduced by Zhu et al. [55], we additionally contribute ManiCups, a multi-domain I2I translation dataset for manipulating physical state changes of objects. ManiCups contains 6k images of empty cups and cups of coffee, juice, milk, and water, collected from human-annotated bounding boxes. The empirical results demonstrate that compared to previous approaches, CycleNet is superior in translation faithfulness, cycle consistency, and image quality. Our approach is also computationally friendly, which is robust even with very limited training data (around 2k) and requires minimal computational resources (1 GPU) to train. Further analysis shows that CycleNet is a robust zero-shot I2I translator, which can generate faithful and high-quality images for out-of-domain distributions with a simple change of the textual prompt. This opens up possibilities to develop consistent diffusion-based image manipulation models with image conditioning and free-form language instructions.
We start by introducing a set of notations to characterize image-to-image translation with DMs.
Diffusion Models Diffusion models progressively add Gaussian noise to a source image  $z_0 \sim q(z_0)$  through a forward diffusion process and subsequently reverse the process to restore the original image. Given a variance schedule  $\beta_1, \ldots, \beta_T$ , the forward process is constrained to a Markov chain  $q(z_t | z_{t-1}) \coloneqq \mathcal{N}(z_t; \sqrt{1 - \beta_t} z_{t-1}, \beta_t \mathbf{I})$ , in which  $z_{1:T}$  are latent variables with dimensions matching  $z_0$ . The reverse process  $p_\theta(z_{0:T})$  is as well Markovian, with learned Gaussian transitions that begin at  $z_T \sim \mathcal{N}(0, \mathbf{I})$ . Ho et al. [13] noted that the forward process allows the sampling of  $z_t$  at any time step  $t$  using a closed-form sampling function (Eq. 1).
$$
z _ {t} = S \left(z _ {0}, \varepsilon , t\right) := \sqrt {\bar {\alpha} _ {t}} z _ {0} + \sqrt {1 - \bar {\alpha} _ {t}} \varepsilon , \varepsilon \sim \mathcal {N} (0, \mathbf {I}) \text {a n d} t \sim [ 1, T ] \tag {1}
$$
in which  $\alpha_{t} := 1 - \beta_{t}$  and  $\bar{\alpha}_{t} := \prod_{s=1}^{t} \alpha_{s}$ . Thus, the reverse process can be carried out with a UNet-based network  $\varepsilon_{\theta}$  that predicts the noise  $\varepsilon$ . By dropping time-dependent variances, the model can be trained according to the objective in Eq. 2.
$$
\min  _ {\theta} \mathbb {E} _ {z _ {0}, \varepsilon , t} \| \varepsilon - \varepsilon_ {\theta} (z _ {t}, t) \| _ {2} ^ {2} \tag {2}
$$
Eq. 2 implies that in principle, one could estimate the original source image  $z_0$  given a noised latent  $z_t$  at any time  $t$ . The reconstructed  $\bar{z}_0$  can be calculated with the generation function:
$$
\bar {z} _ {0} = G \left(z _ {t}, t\right) := \left[ z _ {t} - \sqrt {1 - \bar {\alpha} _ {t}} \varepsilon_ {\theta} \left(z _ {t}, t\right) \right] / \sqrt {\bar {\alpha} _ {t}} \tag {3}
$$
For simplicity, we drop the temporal conditioning  $t$  in the following paragraphs.
(a) The translation cycle in diffusion-based I2I translation.
(b) CycleNet adopts Stable Diffusion (SD) as the backbone and ControlNet for conditioning.
Figure 2: The image translation cycle includes a forward translation from  $x_0$  to  $\bar{y}_0$  and a backward translation to  $\bar{x}_0$ . The key idea of our method is to ensure that when conditioned on an image  $c_{\mathrm{img}}$  that falls into the target domain specified by  $c_{\mathrm{text}}$ , the LDM should reproduce this image condition through the reverse process. The dashed lines indicate the regularization in the loss functions.
Conditioning in Latent Diffusion Models Latent diffusion models (LDMs) like Stable Diffusion [38] can model conditional distributions  $p_{\theta}(z_0|c)$  over condition  $c$ , e.g., by augmenting the UNet backbone with a condition-specific encoder using cross-attention mechanism [45]. Using textual prompts is the most common approach for enabling conditional image manipulation with LDMs. With a textual prompt  $c_{z}$  as conditioning, LDMs strive to learn a mapping from a latent noised sample  $z_{t}$  to an output image  $z_{0}$ , which falls into a domain  $\mathcal{Z}$  that is specified by the conditioning prompt. To enable more flexible and robust conditioning in diffusion-based image manipulation, especially a mixture of text and image conditioning, recent work obtained further control over the reverse process with a side network [51] or an adapter [31]. We denote such conditional denoising autoencoder as  $\varepsilon_{\theta}(z_t,c_{\mathrm{text}},c_{\mathrm{img}})$ , where  $c_{\mathrm{img}}$  is the image condition and the text condition  $c_{\mathrm{text}}$ . Eq. 3 can thus be rewritten as:
$$
\bar {z} _ {0} = G \left(z _ {t}, c _ {\text {t e x t}}, c _ {\text {i m g}}\right) := \left[ z _ {t} - \sqrt {1 - \bar {\alpha} _ {t}} \varepsilon_ {\theta} \left(z _ {t}, c _ {\text {t e x t}}, c _ {\text {i m g}}\right) \right] / \sqrt {\bar {\alpha} _ {t}} \tag {4}
$$
The text condition  $c_{\mathrm{text}}$  contains a pair of conditional and unconditional prompts  $\{c^{+}, c^{-}\}$ . A conditional prompt  $c^{+}$  guides the diffusion process towards the images that are associated with it, whereas a negative prompt  $c^{-}$  drives the diffusion process away from those images.
Consistency Regularization for Unpaired Image-to-Image Translation The goal of unpaired image-to-image (I2I) translation is to learn a mapping between two domains  $\mathcal{X} \subset \mathbb{R}^d$  and  $\mathcal{Y} \subset \mathbb{R}^d$  with unpaired training samples  $\{x_i\}$  for  $i = 1, \dots, N$ , where  $x_i \in \mathcal{X}$  belongs to  $X$ , and  $\{y_j\}$  for  $j = 1, \dots, M$ , where  $y_j \in \mathcal{Y}$ . In traditional GAN-based translation frameworks, the task typically requires two mappings  $G: \mathcal{X} \to \mathcal{Y}$  and  $F: \mathcal{Y} \to \mathcal{X}$ . Cycle consistency enforces transitivity between forward and backward translation functions by regularizing pairs of samples, which is crucial in I2I translation, particularly in unpaired settings where no explicit correspondence between images in source and target domains is guaranteed [55, 27, 50, 44]. To ensure cycle consistency, CycleGAN [55] explicitly regularizes the translation cycle, bringing  $F(G(x))$  back to the original image  $x$ , and vice versa for  $y$ . Motivated by consistency regularization, we seek to enable consistent unpaired I2I translation with LDMs. Without introducing domain-specific generative models, we use one single denoising network  $\varepsilon_{\theta}$  for translation by conditioning it on text and image prompts.
In the following, we discuss only the translation from domain  $\mathcal{X}$  to  $\mathcal{Y}$  due to the symmetry of the backward translation. Our goal, at inference time, is to enable LDMs to translate a source image  $x_0$  by using it as the image condition  $c_{\mathrm{img}} = x_0$ , and then denoise the noised latent  $y_t$  to  $y_{t-1}$  with text prompts  $c_{\mathrm{text}} = c_y$ . To learn such a translation model  $\varepsilon_\theta(y_t, c_y, x_0)$ , we consider two types of training objectives. In the following sections, we describe the cycle consistency regularization to ensure cycle consistency so that the structures and unrelated semantics are preserved in the generated images, and the self regularization to match the distribution of generated images with the target
domain, As illustrated in Figure 2, the image translation cycle includes a forward translation from a source image  $x_0$  to  $\bar{y}_0$ , followed by a backward translation to the reconstructed source image  $\bar{x}_0$ .
3.1 Cycle Consistency Regularization
We assume a likelihood function  $P(z_0, c_{\mathrm{text}})$  that the image  $z_0$  falls into the data distribution specified by the text condition  $c_{\mathrm{text}}$ . We consider a generalized case of cycle consistency given the conditioning mechanism in LDMs. If  $P(c_{\mathrm{img}}, c_{\mathrm{text}})$  is close to 1, i.e., the image condition  $c_{\mathrm{img}}$  falls exactly into the data distribution described by the text condition  $c_{\mathrm{text}}$ , we should expect that  $G(z_t, c_{\mathrm{text}}, c_{\mathrm{img}}) = c_{\mathrm{img}}$  for any noised latent  $z_t$ . With the translation cycle in Figure 2, the goal is to optimize (1)  $\mathcal{L}_{x \to x} = \mathbb{E}_{x_0, \varepsilon_x} ||x_0 - G(x_t, c_x, x_0)||_2^2$ ; (2)  $\mathcal{L}_{y \to y} = \mathbb{E}_{x_0, \varepsilon_x, \varepsilon_y} ||\bar{y}_0 - G(y_t, c_y, \bar{y}_0)||_2^2$ ; (3)  $\mathcal{L}_{x \to y \to x} = \mathbb{E}_{x_0, \varepsilon_x, \varepsilon_y} ||x_0 - G(y_t, c_x, x_0)||_2^2$ ; and (4)  $\mathcal{L}_{x \to y \to y} = \mathbb{E}_{x_0, \varepsilon_x} ||\bar{y}_0 - G(x_t, c_y, \bar{y}_0)||_2^2$ .
Proposition 1 (Cycle Consistency Regularization). With the translation cycle in Figure 2, a set of consistency losses is given by dropping time-dependent variances:
$$
\mathcal {L} _ {x \rightarrow x} = \mathbb {E} _ {x _ {0}, \varepsilon_ {x}} | | \varepsilon_ {\theta} (x _ {t}, c _ {x}, x _ {0}) - \varepsilon_ {x} | | _ {2} ^ {2} \tag {5}
$$
$$
\mathcal {L} _ {y \rightarrow y} = \mathbb {E} _ {x _ {0}, \varepsilon_ {x}, \varepsilon_ {y}} \left|\left| \varepsilon_ {\theta} \left(y _ {t}, c _ {y}, \bar {y} _ {0}\right) - \varepsilon_ {y} \right|\right| _ {2} ^ {2} \tag {6}
$$
$$
\mathcal {L} _ {x \rightarrow y \rightarrow x} = \mathbb {E} _ {x _ {0}, \varepsilon_ {x}, \varepsilon_ {y}} | | \varepsilon_ {\theta} (y _ {t}, c _ {x}, x _ {0}) + \varepsilon_ {\theta} (x _ {t}, c _ {y}, x _ {0}) - \varepsilon_ {x} - \varepsilon_ {y} | | _ {2} ^ {2} \tag {7}
$$
$$
\mathcal {L} _ {x \rightarrow y \rightarrow y} = \mathbb {E} _ {x _ {0}, \varepsilon_ {x}} | | \varepsilon_ {\theta} (x _ {t}, c _ {y}, x _ {0}) - \varepsilon_ {\theta} (x _ {t}, c _ {y}, \bar {y} _ {0}) | | _ {2} ^ {2} \tag {8}
$$
We leave the proof in Section A.2. Proposition 1 states that pixel-level consistency can be acquired by regularizing the conditional denoising autoencoder  $\varepsilon_{\theta}$ . Specifically, the reconstruction loss  $\mathcal{L}_{x\rightarrow x}$  and  $\mathcal{L}_{y\rightarrow y}$  ensures that CycleNet can function as a LDM to reverse an image similar to Eq. 2. The cycle consistency loss  $\mathcal{L}_{x\rightarrow y\rightarrow x}$  serves as the transitivity regularization, which ensures that the forward and backward translations can reconstruct the original image  $x_0$ . The invariance loss  $\mathcal{L}_{x\rightarrow y\rightarrow y}$  requires that the target image domain stays invariant under forward translation, i.e., given a forward translation from  $x_{t}$  to  $\bar{y}_0$  conditioned on  $x_0$ , repeating the translation conditioned on  $\bar{y}_0$  would reproduce  $\bar{y}_0$ .
3.2 Self Regularization
In the previous section, while  $x_0$  is naturally sampled from domain  $\mathcal{X}$ , we need to ensure that the generated images fall in the target domain  $\mathcal{V}$ , i.e., the translation leads to  $G(x_{t},c_{y},x_{0})\in \mathcal{V}$ . Our goal is therefore to maximize  $P(\bar{y}_0,c_y)$ , or equivalently to minimize
$$
\mathcal {L} _ {\mathrm {L D M}} = - \mathbb {E} _ {x _ {0}, \varepsilon_ {x}} P [ G (S (x _ {0}, \varepsilon), c _ {y}, x _ {0}), c _ {y} ] \tag {9}
$$
Assumption 1 (Domain Smoothness). For any text condition,  $P(\cdot ,c_{\mathrm{text}})$  is  $L$ -Lipschitz.
$$
\exists L <   \infty , | P \left(z _ {0} ^ {1}, c _ {\text {t e x t}}\right) - P \left(z _ {0} ^ {2}, c _ {\text {t e x t}}\right) | \leq L \| z _ {0} ^ {1} - z _ {0} ^ {2} \| _ {2} \tag {10}
$$
Proposition 2 (Self Regularization). Let  $\varepsilon_{\theta}^{*}$  denote the denoising autoencoder of the pre-trained text-guided LDM backbone. Let  $x_{t} = S(x_{0},\varepsilon_{x})$  be a noised latent. A self-supervised upper bound of  $\mathcal{L}_{\mathrm{LDM}}$  is given by:
$$
\mathcal {L} _ {\text {s e l f}} = \mathbb {E} _ {x _ {0}, \varepsilon_ {x}} \left[ L \sqrt {\frac {1 - \bar {\alpha} _ {t}}{\bar {\alpha} _ {t}}} | | \varepsilon_ {\theta} \left(x _ {t}, c _ {y}, x _ {0}\right) - \varepsilon_ {\theta} ^ {*} \left(x _ {t}, c _ {y}\right) | | _ {2} \right] + \text {c o n s t} \tag {11}
$$
Lipschitz assumptions have been widely adopted in diffusion methods [53, 48]. Assumption 1 hypothesizes that similar images share similar domain distributions. A self-supervised upper bound  $\mathcal{L}_{\mathrm{self}}$  can be obtained in Proposition 2, which intuitively states that if the output of the conditional translation model does not deviate far from the pre-trained LDM backbone, the outcome image should still fall in the same domain specified by the textual prompt. We leave the proof in Section A.3.
3.3 CycleNet
In practice,  $\mathcal{L}_{\mathrm{self}}$  can be minimized from the beginning of training by using a ControlNet [51] with pre-trained Stable Diffusion (SD) [38] as the LDM backbone, which is confirmed through preliminary
experiments. As shown in Figure 2, the model keeps the SD encoder frozen and makes a trainable copy in the side network. Additional zero convolution layers are introduced to encode the image condition and control the SD decoder. These zero convolution layers are 1D convolutions whose initial weights and biases vanish and can gradually acquire the optimized parameters from zero. Since the zero convolution layers keep the SD encoder features untouched,  $\mathcal{L}_{\mathrm{self}}$  is minimal at the beginning of the training, and the training process is essentially fine-tuning a pre-trained LDM with a side network.
The text condition  $c_{\mathrm{text}} = \{c^{+}, c^{-}\}$  contains a pair of conditional and unconditional prompts. We keep the conditional prompt in the frozen SD encoder and the unconditional prompt in the ControlNet, so that the LDM backbone focuses on the translation and the side network looks for the semantics that needs modification. For example, to translate an image of summer to winter, we rely on a conditional prompt  $l_{x} =$  "summer" and unconditional prompt  $l_{y} =$  "winter". Specifically, we use CLIP [36] encoder to encode the language prompts  $l_{x}$  and  $l_{y}$  such that  $c_{x} = \{\mathrm{CLIP}(l_{x}), \mathrm{CLIP}(l_{y})\}$  and  $c_{y} = \{\mathrm{CLIP}(l_{y}), \mathrm{CLIP}(l_{x})\}$ .
We also note that  $\mathcal{L}_{y\to y}$  can be omitted, as  $\mathcal{L}_{x\rightarrow x}$  can serve the same purpose in the symmetry of the translation cycle from  $\mathcal{V}$  to  $\mathcal{X}$ , and early experiments confirmed that dropping this term lead to significantly faster convergence. The simplified objective is thus given by:
$$
\mathcal {L} _ {x} = \lambda_ {1} \mathcal {L} _ {x \rightarrow x} + \lambda_ {2} \mathcal {L} _ {x \rightarrow y \rightarrow y} + \lambda_ {3} \mathcal {L} _ {x \rightarrow y \rightarrow x} \tag {12}
$$
Consider both translation cycle from  $\mathcal{X}\leftrightarrow \mathcal{Y}$ , the complete training objective of CycleNet is:
$$
\mathcal {L} _ {\text {C y c l e N e t}} = \mathcal {L} _ {x} + \mathcal {L} _ {y} \tag {13}
$$
The pseudocode for training is given in Algo. 1.
3.4 FastCycleNet
Similar to previous cycle-consistent GAN-based models for unpaired I2I translation, there is a trade-off between the image translation quality and cycle consistency. Also, the cycle consistency loss  $\mathcal{L}_{x\to y\to x}$  requires deeper gradient descent, and therefore more computation expenses during training (Table 6). In order to speed up the training process in this situation, one may consider further removing  $\mathcal{L}_{x\to y\to x}$  from the training objective, and name this variation FastCycleNet. Through experiments, FastCycleNet can achieve satisfying consistency and competitive translation quality, as shown in Table 1. Different variations of models can be chosen depending on the practical needs.
4.1 Benchmarks
Scene/Object-Level Manipulation We validate CycleNet on I2I translation tasks of different granularities. We first consider the benchmarks used in CycleGAN by Zhu et al. [55], which contains:
- (Scene Level) Yosemite summer  $\leftrightarrow$  winter: We use around 2k images of summer and winter Yosemite, with default prompts "summer" and "winter";
- (Object Level) horse  $\leftrightarrow$  zebra: We use around 2.5k images of horses and zebras from the dataset with default prompts "horse" and "zebra";
- (Object Level) apple  $\leftrightarrow$  orange: We use around 2k apple and orange images with default prompts of "apple" and "orange".
State Level Manipulation Additionally, we introduce ManiCups $^1$ , a dataset of state-level image manipulation that tasks models to manipulate cups by filling or emptying liquid to/from containers, formulated as a multi-domain I2I translation dataset for object state changes:
- (State Level) ManiCups: We use around 5.7k images of empty cups and cups of coffee, juice, milk, and water for training. The default prompts are set as "empty cup" and "cup of <liquid>". The task is to either empty a full cup or fill an empty cup with liquid as prompted.
<sup>1</sup>Our data is available at https://huggingface.co/datasets/sled-umich/ManiCups.
ManiCups is curated from human-annotated bounding boxes in publicly available datasets and Bing Image Search (under Share license for training and Modify for test set). We describe our three-stage data collection pipeline. In the image collection stage, we gather raw images of interest from MSCOCO [26], Open Images [23], as well as Bing Image Search API. In the image extraction stage, we extract regions of interest from the candidate images and resize them to a standardized size. Specifically, for subsets obtained from MSCOCO and Open Images, we extract the bounding boxes with labels of interest. All bounding boxes with an initial size less than  $128 \times 128$  are discarded, and the remaining boxes are extended to squares and resized to a standardized size of  $512 \times 512$  pixels. After this step, we obtained approximately  $20\mathrm{k}$  extracted and resized candidate images. We then control the data quality through a filtering and labeling stage. Our filtering process first discards replicated images using the L2 distance metric and remove images containing human faces, as well as cups with a front-facing perspective with a CLIP processor. Our labeling process starts with an automatic annotation with a CLIP classifier. To ensure the accuracy of the dataset, three human annotators thoroughly review the collected images, verifying that the images portray a top-down view of a container and assigning the appropriate labels to the respective domains. The resulting ManiCups dataset contains 5 domains, including 3 abundant domains (empty, coffee, juice) with more than 1K images in each category and 2 low-resource domains (water, milk) with less than 1K images to facilitate research and analysis in data-efficient learning.
To our knowledge, ManiCups is one of the first datasets targeted to the physical state changes of objects, other than stylistic transfers or type changes of objects. The ability to generate consistent state changes based on manipulation is fundamental for future coherent video prediction [14] as well as understanding and planning for physical agents [49, 18, 8, 46]. For additional details on data collection, processing, and statistics, please refer to Appendix B.
Baselines We compare our proposed models FastCycleNet and CycleNet to state-of-the-art methods for unpaired or zero-shot image-to-image translation.
- GAN-based methods: CycleGAN [55] and CUT [34];
- Mask-based diffusion methods: Direct inpainting with CLIPSeg [28] and Text2LIVE [2];
- Mask-free diffusion methods: ControlNet with Canny Edge [51], ILVR [5], EGSDE [53], SDEdit [29], Pix2Pix-Zero [35], MasaCtrl [3], CycleDiffusion [48], and Prompt2Prompt [10] with null-text inversion [30].
Training We train our model with a batch size of 4 on only one single A40 GPU. Additional details on the implementations are available in Appendix C.