SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sampling As shown in Figure 3, CycleNet has a good efficiency at inference time and more sampling steps lead to better translation quality. We initialize the sampling process with the latent noised input image  $z_{t}$ , collected using Equation 1. Following [29], a standard 50-step sampling is applied at inference time with  $t = 100$  for fair comparison.
Figure 3: Step skipping during sampling. The source image is from MSCOCO [26].
4.3 Qualitative Evaluation
We present qualitative results comparing various image translation models. Due to the space limit, additional examples will be available in Appendix E. In Figure 4, we present the two unpaired translation tasks: summer→winter, and horse→zebra. To demonstrate the image quality, translation quality, and consistency compared to the original images, we provide a full image for each test case
2Our code is available at https://github.com/sled-group/CycleNet.
Figure 4: Qualitative comparison in scene/object-level tasks. CycleNet produces high-quality translations with satisfactory consistency. The areas in the boxes are enlarged for detailed comparisons.
Table 1: A quantitative comparison of various image translation models for the summer→winter, horse→zebra, empty→coffee and juice→empty. Performances are quantified in terms of FID, CLIP Score, LPIPS, PSNR, SSIM, and L2.
and enlarge the boxed areas for detailed comparisons. As presented with the qualitative examples, our methods are able to perform image manipulation with high quality like the other diffusion-based methods, while preserving the structures and unrelated semantics.
In Figure 5, we present qualitative results for filling and emptying a cup: coffee  $\leftrightarrow$  empty and empty  $\leftrightarrow$  juice. As demonstrated, image editing tasks that require physical state changes pose a significant challenge to baselines, which struggle with the translation itself and/or maintaining strong consistency. CycleNet, again, is able to generate faithful and realistic images that reflect the physical state changes.
4.4 Quantitative Evaluation
We further use three types of evaluation metrics respectively to assess the quality of the generated image, the quality of translation, and the consistency of the images. For a detailed explanation of these evaluation metrics, we refer to Appendix C.4.
- Image Quality. To evaluate the quality of images, we employ two metrics: The naïve Fréchet Inception Distance (FID) [12] and  $\mathrm{FID}_{\mathrm{clip}}$  [24] with CLIP [36];
Figure 5: Qualitative comparison in coffee  $\leftrightarrow$  empty and empty  $\leftrightarrow$  juice tasks.
- Translation Quality. We use CLIPScore [11] to quantify the semantic similarity of the generated image and conditional prompt;
- Translation Consistency. We measure translation consistency using four different metrics: L2 Distance, Peak Signal-to-Noise Ratio (PSNR) [4], Structural Similarity Index Measure (SSIM) [47], and Learned Perceptual Image Patch Similarity (LPIPS) [52].
The evaluations are performed on the reserved test set. As demonstrated in the quantitative results from Table 1, we observe that some baselines display notably improved consistency in the ManiCups tasks. This could possibly be attributed to the fact that a considerable number of the test images were not successfully translated to the target domain, as can be seen in the qualitative results presented in Figure 5. Overall, CycleNet exhibits competitive and comprehensive performance in generating high-quality images in both global and local translation tasks, especially compared to the mask-based diffusion methods. Meanwhile, our methods ensure successful translations that fulfill the domain specified by text prompts while maintaining an impressive consistency from the original images.
5 Analysis and Discussions
5.1 Ablation study
Recall that FastCycleNet removes the cycle consistency loss  $\mathcal{L}_{x\to y\to x}$  from CycleNet. We further remove the invariance loss  $\mathcal{L}_{x\to y\to y}$  to understand the role of each loss term. For better control, we initialize the sampling process with the same random noise  $\varepsilon$  rather than the latent noised image  $z_{t}$ . Table 2 shows our ablation study on winter  $\rightarrow$  summer task with FID, CLIP score, and LPIPS. When both losses are removed, the model can be considered as a fine-tuned LDM backbone (row 4), which produces a high CLIP similarity score of 24.35. This confirms that the pre-trained LDM backbone can already make qualitative translations, while the LPIPS score of 0.61 implies a poor consistency from the original images. When we introduced the consistency constraint (row 3), the model's LPIPS score improved significantly with a drop of the CLIP score to 19.89. This suggests a trade-off between cycle consistency and translation quality. When we introduced the invariance constraint (row 2), the model achieved the best translation quality with fair consistency. By introducing both constraints (row 1), CycleNet strikes the best balance between consistency at a slight cost of translation quality.
Table 2: Ablation study over the cycle consistency loss and invariance loss.
5.2 Zero-shot Generalization to Out-of-Distribution Domains
CycleNet performs image manipulation with text and image conditioning, making it potentially generalizable to out-of-distribution (OOD) domains with a simple change of the textual prompt. As illustrated in Figure 6, we demonstrate that CycleNet has a remarkable capability to generate faithful and high-quality images for unseen domains. These results highlight the robustness and adaptability of CycleNet to make the most out of the pre-trained LDM backbone to handle unseen scenarios. This underscores the potential to apply CycleNet for various real-world applications and paves the way for future research in zero-shot learning and OOD generalization.
5.3 Translation Diversity
Diversity is an important feature of image translation models. As shown in Figure 6, we demonstrate that CycleNet can generate a variety of images that accurately satisfy the specified translation task in the text prompts, while maintaining consistency.
Figure 6: Examples of output diversity and zero-shot generalization to out-of-domain distributions.
5.4 Limitations: Trade-off between consistency and translation
There have been concerns that cycle consistency could be too restrictive for some translation task [54]. As shown in Figure 7, while CycleNet maintains a strong consistency over the input image, the quartered apple failed to be translated into its orange equivalence. In GAN-based methods, local discriminators have been proposed to address this issue [56], yet it remains challenging to keep global consistency while making faithful local edits for LDM-based approaches.
Figure 7: The trade-off between consistency and translation.
6.1 Conditional Image Manipulation with Diffusion Models
Building upon Diffusion Probabilistic Models [42, 13], pre-trained Diffusion Models (DMs) [38, 37, 39] have achieved state-of-the-art performance in image generation tasks. Text prompts are the most common protocol to enable conditional image manipulation with DMs, which can be done by fine-tuning a pre-trained DM [19, 20]. Mask-based methods have also been proposed with the help of user-prompted/automatically generated masks [32, 7, 1] or augmentation layers [2]. To refrain from employing additional masks, recent work has explored attention-based alternatives [10, 30, 25, 35].
They first invert the source images to obtain the cross-attention maps and then perform image editing with attention control. The promising performance of these methods is largely dependent on the quality of attention maps, which cannot be guaranteed in images with complicated scenes and object relationships, leading to undesirable changes. Very recently, additional image-conditioning has been explored to perform paired image-to-image translation, using a side network [51] or an adapter [31]. In this work, we follow this line of research and seek to enable unpaired image-to-image translation with pre-trained DMs while maintaining a satisfactory level of consistency.
6.2 Unpaired Image-to-Image Translation
Image-to-image translation (I2I) is a fundamental task in computer vision, which is concerned with learning a mapping across images of different domains. Traditional GAN-based methods [16] require instance-level paired data, which are difficult to collect in many domains. To address this limitation, the unpaired I2I setting [55, 27] was introduced to transform an image from the source domain  $\mathcal{X}$  into one that belongs to the target domain  $\mathcal{Y}$ , given only unpaired images from each domain. Several GAN-based methods [55, 50, 27] were proposed to address this problem. In recent years, DPMs have demonstrated their superior ability to synthesize high-quality images, with several applications in I2I translation [40, 5]. With the availability of pre-trained DMs, SDEdit [29] changes the starting point of generation by using a noisy source image that preserves the overall structure. EGSDE [53] combines the merit of ILVR and SDEdit by introducing a pre-trained energy function on both domains to guide the denoising process. While these methods result in leading performance on multiple benchmarks, it remains an open challenge to incorporate pre-trained DMs for high-quality image generation, and at the same time, to ensure translation consistency.
6.3 Cycle Consistency in Image Translation
The idea of cycle consistency is to regularize pairs of samples by ensuring transitivity between the forward and backward translation functions [41, 33, 17]. In unpaired I2I translation where explicit correspondence between source and target domain images is not guaranteed, cycle consistency plays a crucial role [55, 21, 27]. Several efforts were made to ensure cycle consistency in diffusion-based I2I translation. UNIT-DDPM [40] made an initial attempt in the unpaired I2I setting, training two DPMs and two translation functions from scratch. Cycle consistency losses are introduced in the translation functions during training to regularize the reverse processes. At inference time, the image generation does not depend on the translation functions, but only on the two DPMs in an iterative manner, leading to sub-optimal performance. Su et al. [43] proposed the DDIB framework that exact cycle consistency is possible assuming zero discretization error, which does not enforce any cycle consistency constraint itself. Cycle Diffusion [48] proposes a zero-shot approach for image translation based on Su et al. [43]'s observation that a certain level of consistency could emerge from DMs, and there is no explicit treatment to encourage cycle consistency. To the best of our knowledge, CycleNet is the first to guarantee cycle consistency in unpaired image-to-image translation using pre-trained diffusion models, with a simple trainable network and competitive performance.
The paper introduces CycleNet that incorporates the concept of cycle consistency into text-guided latent diffusion models to regularize the image translation tasks. CycleNet is a practical framework for low-resource applications where only limited data and computational power are available. Through extensive experiments on unpaired I2I translation tasks at scene, object, and state levels, our empirical studies show that CycleNet is promising in consistency and quality, and can generate high-quality images for out-of-domain distributions with a simple change of the textual prompt.
Future Work This paper is primarily concerned with the unpaired I2I setting, which utilizes images from unpaired domains during training for domain-specific applications. Although CycleNet demonstrates robust out-of-domain generalization, enabling strong zero-shot I2I translation capabilities is not our focus here. We leave it to our future work to explore diffusion-based image manipulation with image conditioning and free-form language instructions, particularly in zero-shot settings.
This work was supported in part by NSF IIS-1949634, NSF SES-2128623, LG AI Research, and by the Automotive Research Center at the University of Michigan. The authors would like to thank Yinpei Dai and Yuexi Du for their valuable feedback, and extend their special appreciation to Michael de la Paz for granting us permission to use his photograph of summer Yosemite<sup>3</sup> in the opener.
[1] Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Xi Yin. Spatext: Spatio-textual representation for controllable image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[2] Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text-driven layered image and video editing. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV, pages 707-723. Springer, 2022.
[3] Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 22560-22570, October 2023.
[4] Luen C Chan and Peter Whiteman. Hardware-constrained hybrid coding of video imagery. IEEE Transactions on Aerospace and Electronic Systems, (1):71-84, 1983.
[5] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 14347-14356. IEEE Computer Society, 2021.
[6] Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11472-11481, 2022.
[7] Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion-based semantic image editing with mask guidance. In The Eleventh International Conference on Learning Representations, 2023.
[8] Xiaolin Fang, Caelan Reed Garrett, Clemens Eppner, Tomás Lozano-Pérez, Leslie Pack Kaelbling, and Dieter Fox. Dimsam: Diffusion models as samplers for task and motion planning under partial observability. arXiv preprint arXiv:2306.13196, 2023.
[9] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In Neural Information Processing Systems, 2014.
[10] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or. Prompt-to-prompt image editing with cross-attention control. In The Eleventh International Conference on Learning Representations, 2022.
[11] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7514-7528, 2021.
[12] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
<sup>3</sup>A hyperlink to the original photo on Flickr. All rights reserved by the artist.
[45] Ashish Vavwani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[46] Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, and Jiwen Lu. Drivedreamer: Towards real-world-driven world models for autonomous driving. arXiv preprint arXiv:2309.09777, 2023.
[47] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4): 600-612, 2004.
[48] Chen Henry Wu and Fernando De la Torre. A latent space of stochastic diffusion models for zero-shot image editing and guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7378-7387, 2023.
[49] Yue Yang, Wenlin Yao, Hongming Zhang, Xiaoyang Wang, Dong Yu, and Jianshu Chen. ZLaVI: Zero-shot language solver fueled by visual imagination. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1186-1203, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[50] Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image-to-image translation. In Proceedings of the IEEE international conference on computer vision, pages 2849-2857, 2017.
[51] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[52] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[53] Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. Egsde: Unpaired image-to-image translation via energy-guided stochastic differential equations. Advances in Neural Information Processing Systems, 35:3609-3623, 2022.
[54] Yihao Zhao, Ruihai Wu, and Hao Dong. Unpaired image-to-image translation using adversarial consistency loss. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX 16, pages 800-815. Springer, 2020.
[55] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223-2232, 2017.
[56] Xianhui Zong, Zhehan Chen, and Dadong Wang. Local-cyclegan: a general end-to-end network for visual enhancement in complex deep-water environment. Applied Intelligence, 51:1947-1958, 2021.
A CycleNet Details
A.1 Algorithmatic Details
The pseudocode for training is given as follows.
Algorithm 1: CycleNet Training Framework
1 Input: Source image  $x_0$ , source domain prompt  $c_x$ , target domain prompt  $c_y$ , constants  $\lambda_{1,2,3}$ ;  
2 repeat  
3  $x_0 \sim q(x_0)$ ;  
4  $t \sim [1,T]$ ;  
5  $\varepsilon_x, \varepsilon_y \sim \mathcal{N}(0,I)$ ;  
6  $x_t = \sqrt{\bar{\alpha}_t} x_0 + \sqrt{1 - \bar{\alpha}_t} \varepsilon_x$ ;  
7  $\varepsilon_\theta^x = \varepsilon_\theta (x_t, c_x, x_0)$ ;  
8  $\varepsilon_\theta^{xy} = \varepsilon_\theta (x_t, c_y, x_0)$ ;  
9  $\bar{y}_0 = (x_t - \sqrt{1 - \bar{\alpha}_t} \varepsilon_\theta^{xy}) / \sqrt{\bar{\alpha}_t}$ ;  
10  $\varepsilon_\theta^y = \varepsilon_\theta (x_t, c_y, \bar{y}_0)$ . detach();  
11  $y_t = \sqrt{\bar{\alpha}_t} \bar{y}_0 + \sqrt{1 - \bar{\alpha}_t} \varepsilon_y$ ;  
12  $\varepsilon_\theta^{yx} = \varepsilon_\theta (y_t, c_x, x_0)$ ;  
13 Take gradient descent step on  
 $\nabla_\theta [\lambda_1||\varepsilon_\theta^x - \varepsilon_x||_2^2 + \lambda_2||\varepsilon_\theta^{xy} - \varepsilon_\theta^y||_2^2 + \lambda_3||\varepsilon_\theta^{xy} + \varepsilon_\theta^{yx} - \varepsilon^x - \varepsilon^y||_2^2]$ ;  
14 until converged;
A.2 Proof of Cycle Consistency Regularization
$x$ -Reconstruction Loss Dropping the variances,  $\mathcal{L}_{x\to x} = \mathbb{E}_{x_0,\varepsilon_x}\left||\varepsilon_\theta (x_t,c_x,x_0) - \varepsilon_x\right||_2^2$
$$
\begin{array}{l} \mathcal {L} _ {x \rightarrow x} = \mathbb {E} _ {x _ {0}, \varepsilon_ {x}} | | x _ {0} - G (x _ {t}, c _ {x}, x _ {0}) | | _ {2} ^ {2} \\ = \mathbb {E} _ {x _ {0}, \varepsilon_ {x}} | | x _ {0} - \left[ x _ {t} - \sqrt {1 - \bar {\alpha} _ {t}} \varepsilon_ {\theta} (x _ {t}, c _ {x}, x _ {0}) \right] / \sqrt {\bar {\alpha} _ {t}} | | _ {2} ^ {2} \\ = \mathbb {E} _ {x _ {0}, \varepsilon_ {x}} | | x _ {0} - \left[ \sqrt {\bar {\alpha} _ {t}} x _ {0} + \sqrt {1 - \bar {\alpha} _ {t}} \varepsilon_ {x} - \sqrt {1 - \bar {\alpha} _ {t}} \varepsilon_ {\theta} (x _ {t}, c _ {x}, x _ {0}) \right] / \sqrt {\bar {\alpha} _ {t}} | | _ {2} ^ {2} \\ = \mathbb {E} _ {x _ {0}, \varepsilon_ {x}} | | x \sigma - x \sigma - \sqrt {1 - \bar {\alpha} _ {t}} [ \varepsilon_ {x} - \varepsilon_ {\theta} (x _ {t}, c _ {x}, x _ {0}) ] / \sqrt {\bar {\alpha} _ {t}} | | _ {2} ^ {2} \\ = \mathbb {E} _ {x _ {0}, \varepsilon_ {x}} \sqrt {\frac {1 - \bar {\alpha} _ {t}}{\bar {\alpha} _ {t}}} | | \varepsilon_ {\theta} (x _ {t}, c _ {x}, x _ {0}) - \varepsilon_ {x} | | _ {2} ^ {2} \\ \end{array}
$$
$y$ -Reconstruction Loss Dropping the variances,  $\mathcal{L}_{y\to y} = \mathbb{E}_{x_0,\varepsilon_x,\varepsilon_y}\| \varepsilon_\theta (y_t,c_y,\bar{y}_0) - \varepsilon_y\| _2^2$
$$
\begin{array}{l} \mathcal {L} _ {y \rightarrow y} = \mathbb {E} _ {x _ {0}, \varepsilon_ {x}, \varepsilon_ {y}} | | \bar {y} _ {0} - G (y _ {t}, c _ {y}, \bar {y} _ {0}) | | _ {2} ^ {2} \\ = \mathbb {E} _ {x _ {0}, \varepsilon_ {x}, \varepsilon_ {y}} | | \bar {y} _ {0} - \left[ y _ {t} - \sqrt {1 - \bar {\alpha} _ {t}} \varepsilon_ {\theta} (y _ {t}, c _ {y}, \bar {y} _ {0}) \right] / \sqrt {\bar {\alpha} _ {t}} | | _ {2} ^ {2} \\ = \mathbb {E} _ {x _ {0}, \varepsilon_ {x}, \varepsilon_ {y}} | | \bar {y} _ {0} - \left[ \sqrt {\bar {\alpha} _ {t}} \bar {y} _ {0} + \sqrt {1 - \bar {\alpha} _ {t}} \varepsilon_ {y} - \sqrt {1 - \bar {\alpha} _ {t}} \varepsilon_ {\theta} (y _ {t}, c _ {y}, \bar {y} _ {0}) \right] / \sqrt {\bar {\alpha} _ {t}} | | _ {2} ^ {2} \\ = \mathbb {E} _ {x _ {0}, \varepsilon_ {x}, \varepsilon_ {y}} | | \bar {y} 0 - \bar {y} 0 - \sqrt {1 - \bar {\alpha} _ {t}} [ \varepsilon_ {y} - \varepsilon_ {\theta} (y _ {t}, c _ {y}, \bar {y} _ {0}) ] / \sqrt {\bar {\alpha} _ {t}} | | _ {2} ^ {2} \\ = \mathbb {E} _ {x _ {0}, \varepsilon_ {x}, \varepsilon_ {y}} \sqrt {\frac {1 - \bar {\alpha} _ {t}}{\bar {\alpha} _ {t}}} | | \varepsilon_ {\theta} (y _ {t}, c _ {y}, \bar {y} _ {0}) - \varepsilon_ {y} | | _ {2} ^ {2} \\ \end{array}
$$
Cycle Consistency Loss Dropping the variances,  $\mathcal{L}_{x\to y\to x} = \mathbb{E}_{x_0,\varepsilon_x,\varepsilon_y}\| \varepsilon_\theta (y_t,c_x,x_0) + \varepsilon_\theta (x_t,c_y,x_0) - \varepsilon_x - \varepsilon_y\| _2^2.$