Title: CycleNet: Rethinking Cycle Consistency in Text-Guided Diffusion for Image Manipulation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Image to image translation

latent diffusion models

conditional diffusion models
Abstract: Diffusion models (DMs) have enabled breakthroughs in image synthesis tasks but lack an intuitive interface for consistent image-to-image (I2I) translation. Various methods have been explored to address this issue, including mask-based methods, attention-based methods, and image-conditioning. However, it remains a critical challenge to enable unpaired I2I translation with pre-trained DMs while maintaining satisfying consistency. This paper introduces Cyclenet, a novel but simple method that incorporates cycle consistency into DMs to regularize image manipulation. We validate Cyclenet on unpaired I2I tasks of different granularities. Besides the scene and object level translation, we additionally contribute a multi-domain I2I translation dataset to study the physical state changes of objects. Our empirical studies show that Cyclenet is superior in translation consistency and quality, and can generate high-quality images for out-of-domain distributions with a simple change of the textual prompt. Cyclenet is a practical framework, which is robust even with very limited training data (around 2k) and requires minimal computational resources (1 GPU) to train. Project homepage: https://cyclenetweb.github.io/

CycleNet: Rethinking Cycle Consistency in Text-Guided Diffusion for Image Manipulation

Abstract

Diffusion models (DMs) have enabled breakthroughs in image synthesis tasks but lack an intuitive interface for consistent image-to-image (I2I) translation. Various methods have been explored to address this issue, including mask-based methods, attention-based methods, and image-conditioning. However, it remains a critical challenge to enable unpaired I2I translation with pre-trained DMs while maintaining satisfying consistency. This paper introduces CycleNet, a novel but simple method that incorporates cycle consistency into DMs to regularize image manipulation. We validate CycleNet on unpaired I2I tasks of different granularities. Besides the scene and object level translation, we additionally contribute a multi-domain I2I translation dataset to study the physical state changes of objects. Our empirical studies show that CycleNet is superior in translation consistency and quality, and can generate high-quality images for out-of-domain distributions with a simple change of the textual prompt. CycleNet is a practical framework, which is robust even with very limited training data (around 2k) and requires minimal computational resources (1 GPU) to train.

Input

CycleNet (Ours)

P2P + NullText

Summer → Winter

SDEdit

1 Introduction

Recently, pre-trained diffusion models (DMs) [38, 37, 39] have enabled an unprecedented breakthrough in image synthesis tasks. Compared to GANs [9] and VAEs [22], DMs exhibit superior stability and quality in image generation, as well as the capability to scale up to open-world multimodal data. As such, pre-trained DMs have been applied to image-to-image (I2I) translation, which is to acquire a mapping between images from two distinct domains, e.g., different scenes, different

1

objects, and different object states. For such translations, text-guided diffusion models typically require mask layers [32, 2, 7, 1] or attention control [10, 30, 25, 35]. However, the quality of masks and attention maps can be unpredictable in complex scenes, leading to semantic and structural changes that are undesirable. Recently, researchers have explored using additional image-conditioning to perform paired I2I translations with the help of a side network [51] or an adapter [31]. Still, it remains an open challenge to adapt pre-trained DMs in unpaired I2I translation with a consistency guarantee.

We emphasize that consistency , a desirable property in image manipulation, is particularly important in unpaired I2I scenarios where there is no guaranteed correspondence between images in the source and target domains. Various applications of DMs, including video prediction and infilling [14], imagination-augmented language understanding [49], robotic manipulation [18, 8] and world models [46], would rely on strong consistency across the source and generated images.

To enable unpaired I2I translation using pre-trained DMs with satisfactory consistency, this paper introduces CycleNet, which allows DMs to translate a source image by conditioning on the input image and text prompts. More specifically, we adopt ControlNet [51] with pre-trained Stable Diffusion (SD) [38] as the latent DM backbone. Motivated by cycle consistency in GAN-based methods [55], CycleNet leverages consistency regularization over the image translation cycle. As illustrated in Figure 2, the image translation cycle includes a forward translation from x 0 to ¯ y 0 and a backward translation to ¯ x 0 . The key idea of our method is to ensure that when conditioned on an image c img that falls into the target domain specified by c text , the DM should be able to reproduce this image condition through the reverse process.

We validate CycleNet on I2I translation tasks of different granularities. Besides the scene and object level tasks introduced by Zhu et al. [55], we additionally contribute ManiCups , a multi-domain I2I translation dataset for manipulating physical state changes of objects. ManiCups contains 6k images of empty cups and cups of coffee, juice, milk, and water, collected from human-annotated bounding boxes. The empirical results demonstrate that compared to previous approaches, CycleNet is superior in translation faithfulness, cycle consistency, and image quality. Our approach is also computationally friendly, which is robust even with very limited training data (around 2k) and requires minimal computational resources (1 GPU) to train. Further analysis shows that CycleNet is a robust zero-shot I2I translator, which can generate faithful and high-quality images for out-of-domain distributions with a simple change of the textual prompt. This opens up possibilities to develop consistent diffusionbased image manipulation models with image conditioning and free-form language instructions.

2 Preliminaries

We start by introducing a set of notations to characterize image-to-image translation with DMs.

Diffusion Models Diffusion models progressively add Gaussian noise to a source image z 0 ∼ q ( z 0 ) through a forward diffusion process and subsequently reverse the process to restore the original image. Given a variance schedule β 1 , . . . , β T , the forward process is constrained to a Markov chain q ( z t | z t -1 ) := N ( z t ; √ 1 -β t z t -1 , β t I ) , in which z 1: T are latent variables with dimensions matching z 0 . The reverse process p θ ( z 0: T ) is as well Markovian, with learned Gaussian transitions that begin at z T ∼ N (0 , I ) . Ho et al. [13] noted that the forward process allows the sampling of z t at any time step t using a closed-form sampling function (Eq. 1).

in which α t := 1 -β t and ¯ α t := ∏ t s =1 α s . Thus, the reverse process can be carried out with a UNet-based network ε θ that predicts the noise ε . By dropping time-dependent variances, the model can be trained according to the objective in Eq. 2.

Eq. 2 implies that in principle, one could estimate the original source image z 0 given a noised latent z t at any time t . The reconstructed ¯ z 0 can be calculated with the generation function:

For simplicity, we drop the temporal conditioning t in the following paragraphs.

̅

x

0

x

t

ε

x

ε

Ө

(x

t

, c

y

,      )

y

0

y

t

x

0

y

0

ε

Ө

(x

t

, c

y

,      )

ε

Ө

(x

t

, c

x

,      )

ε

Ө

(y

t

, c

x

,      )

x

0

x

0

x

0

'

ε

y

y

0

SD Encoder

SD Encoder

(Trainable Copy)

z

t

ε

Ө

c

img

Conditional

prompt

+

Zero Conv

Zero Conv

Unconditional

Prompt

ε

SD Decoder

🔒🔒

z

0

Sampling

+

z

0

̅

(a) The translation cycle in diffusion-based I2I translation.

̅

(b) CycleNet adopts Stable Diffusion (SD) as the backbone and ControlNet for conditioning.

Conditioning in Latent Diffusion Models Latent diffusion models (LDMs) like Stable Diffusion [38] can model conditional distributions p θ ( z 0 | c ) over condition c , e.g., by augmenting the UNet backbone with a condition-specific encoder using cross-attention mechanism [45]. Using textual prompts is the most common approach for enabling conditional image manipulation with LDMs. With a textual prompt c z as conditioning, LDMs strive to learn a mapping from a latent noised sample z t to an output image z 0 , which falls into a domain Z that is specified by the conditioning prompt. To enable more flexible and robust conditioning in diffusion-based image manipulation, especially a mixture of text and image conditioning, recent work obtained further control over the reverse process with a side network [51] or an adapter [31]. We denote such conditional denoising autoencoder as ε θ ( z t , c text , c img ) , where c img is the image condition and the text condition c text . Eq. 3 can thus be rewritten as:

The text condition c text contains a pair of conditional and unconditional prompts { c + , c -} . A conditional prompt c + guides the diffusion process towards the images that are associated with it, whereas a negative prompt c -drives the diffusion process away from those images.

Consistency Regularization for Unpaired Image-to-Image Translation The goal of unpaired image-to-image (I2I) translation is to learn a mapping between two domains X ⊂ R d and Y ⊂ R d with unpaired training samples { x i } for i = 1 , . . . , N , where x i ∈ X belongs to X , and { y j } for j = 1 , . . . , M , where y j ∈ Y . In traditional GAN-based translation frameworks, the task typically requires two mappings G : X → Y and F : Y → X . Cycle consistency enforces transitivity between forward and backward translation functions by regularizing pairs of samples, which is crucial in I2I translation, particularly in unpaired settings where no explicit correspondence between images in source and target domains is guaranteed [55, 27, 50, 44]. To ensure cycle consistency, CycleGAN [55] explicitly regularizes the translation cycle, bringing F ( G ( x )) back to the original image x , and vice versa for y . Motivated by consistency regularization, we seek to enable consistent unpaired I2I translation with LDMs. Without introducing domain-specific generative models, we use one single denoising network ε θ for translation by conditioning it on text and image prompts.

3 Method

In the following, we discuss only the translation from domain X to Y due to the symmetry of the backward translation. Our goal, at inference time, is to enable LDMs to translate a source image x 0 by using it as the image condition c img = x 0 , and then denoise the noised latent y t to y t -1 with text prompts c text = c y . To learn such a translation model ε θ ( y t , c y , x 0 ) , we consider two types of training objectives. In the following sections, we describe the cycle consistency regularization to ensure cycle consistency so that the structures and unrelated semantics are preserved in the generated images, and the self regularization to match the distribution of generated images with the target

̅

domain, As illustrated in Figure 2, the image translation cycle includes a forward translation from a source image x 0 to ¯ y 0 , followed by a backward translation to the reconstructed source image ¯ x 0 .

3.1 Cycle Consistency Regularization

We assume a likelihood function P ( z 0 , c text ) that the image z 0 falls into the data distribution specified by the text condition c text . We consider a generalized case of cycle consistency given the conditioning mechanism in LDMs. If P ( c img , c text ) is close to 1, i.e., the image condition c img falls exactly into the data distribution described by the text condition c text , we should expect that G ( z t , c text , c img ) = c img for any noised latent z t . With the translation cycle in Figure 2, the goal is to optimize (1) L x → x = E x 0 ,ε x || x 0 -G ( x t , c x , x 0 ) || 2 2 ; (2) L y → y = E x 0 ,ε x ,ε y || ¯ y 0 -G ( y t , c y , ¯ y 0 ) || 2 2 ; (3) L x → y → x = E x 0 ,ε x ,ε y || x 0 -G ( y t , c x , x 0 ) || 2 2 ; and (4) L x → y → y = E x 0 ,ε x || ¯ y 0 -G ( x t , c y , ¯ y 0 ) || 2 2 .

Proposition 1 (Cycle Consistency Regularization) . With the translation cycle in Figure 2, a set of consistency losses is given by dropping time-dependent variances:

We leave the proof in Section A.2. Proposition 1 states that pixel-level consistency can be acquired by regularizing the conditional denoising autoencoder ε θ . Specifically, the reconstruction loss L x → x and L y → y ensures that CycleNet can function as a LDM to reverse an image similar to Eq. 2. The cycle consistency loss L x → y → x serves as the transitivity regularization, which ensures that the forward and backward translations can reconstruct the original image x 0 . The invariance loss L x → y → y requires that the target image domain stays invariant under forward translation, i.e., given a forward translation from x t to ¯ y 0 conditioned on x 0 , repeating the translation conditioned on ¯ y 0 would reproduce ¯ y 0 .

3.2 Self Regularization

In the previous section, while x 0 is naturally sampled from domain X , we need to ensure that the generated images fall in the target domain Y , i.e., the translation leads to G ( x t , c y , x 0 ) ∈ Y . Our goal is therefore to maximize P (¯ y 0 , c y ) , or equavilently to minimize

Assumption 1 (Domain Smoothness) . For any text condition, P ( · , c text ) is L -Lipschitz.

Proposition 2 (Self Regularization) . Let ε ∗ θ denote the denoising autoencoder of the pre-trained text-guided LDM backbone. Let x t = S ( x 0 , ε x ) be a noised latent. A self-supervised upper bound of L LDM is given by:

Lipschitz assumptions have been widely adopted in diffusion methods [53, 48]. Assumption 1 hypothesizes that similar images share similar domain distributions. A self-supervised upper bound L self can be obtained in Proposition 2, which intuitively states that if the output of the conditional translation model does not deviate far from the pre-trained LDM backbone, the outcome image should still fall in the same domain specified by the textual prompt. We leave the proof in Section A.3.

3.3 CycleNet

In practice, L self can be minimized from the beginning of training by using a ControlNet [51] with pre-trained Stable Diffusion (SD) [38] as the LDM backbone, which is confirmed through preliminary

experiments. As shown in Figure 2, the model keeps the SD encoder frozen and makes a trainable copy in the side network. Additional zero convolution layers are introduced to encode the image condition and control the SD decoder. These zero convolution layers are 1D convolutions whose initial weights and biases vanish and can gradually acquire the optimized parameters from zero. Since the zero convolution layers keep the SD encoder features untouched, L self is minimal at the beginning of the training, and the training process is essentially fine-tuning a pre-trained LDM with a side network.

The text condition c text = { c + , c -} contains a pair of conditional and unconditional prompts. We keep the conditional prompt in the frozen SD encoder and the unconditional prompt in the ControlNet, so that the LDM backbone focuses on the translation and the side network looks for the semantics that needs modification. For example, to translate an image of summer to winter, we rely on a conditional prompt l x = ' summer ' and unconditional prompt l y = ' winter '. Specifically, we use CLIP [36] encoder to encode the language prompts l x and l y such that c x = { CLIP( l x ) , CLIP( l y ) } and c y = { CLIP( l y ) , CLIP( l x ) } .

We also note that L y → y can be omitted, as L x → x can serve the same purpose in the symmetry of the translation cycle from Y to X , and early experiments confirmed that dropping this term lead to significantly faster convergence. The simplified objective is thus given by:

Consider both translation cycle from X ↔ Y , the complete training objective of CycleNet is:

The pseudocode for training is given in Algo. 1.

3.4 FastCycleNet

Similar to previous cycle-consistent GAN-based models for unpaired I2I translation, there is a trade-off between the image translation quality and cycle consistency. Also, the cycle consistency loss L x → y → x requires deeper gradient descent, and therefore more computation expenses during training (Table 6). In order to speed up the training process in this situation, one may consider further removing L x → y → x from the training objective, and name this variation FastCycleNet. Through experiments, FastCycleNet can achieve satisfying consistency and competitive translation quality, as shown in Table 1. Different variations of models can be chosen depending on the practical needs.

4 Experiments

4.1 Benchmarks

Scene/Object-Level Manipulation We validate CycleNet on I2I translation tasks of different granularities. We first consider the benchmarks used in CycleGAN by Zhu et al. [55], which contains:

(Scene Level) Yosemite summer ↔ winter : We use around 2k images of summer and winter Yosemite, with default prompts ' summer ' and ' winter ';

(Object Level) horse ↔ zebra : We use around 2.5k images of horses and zebras from the dataset with default prompts ' horse ' and ' zebra ';

(Object Level) apple ↔ orange : We use around 2k apple and orange images with default prompts of ' apple ' and ' orange '.

State Level Manipulation Additionally, we introduce ManiCups 1 , a dataset of state-level image manipulation that tasks models to manipulate cups by filling or emptying liquid to/from containers, formulated as a multi-domain I2I translation dataset for object state changes:

(State Level) ManiCups : We use around 5.7k images of empty cups and cups of coffee, juice, milk, and water for training. The default prompts are set as ' empty cup ' and ' cup of <liquid> '. The task is to either empty a full cup or fill an empty cup with liquid as prompted.

ManiCups is curated from human-annotated bounding boxes in publicly available datasets and Bing Image Search (under Share license for training and Modify for test set). We describe our three-stage data collection pipeline. In the image collection stage, we gather raw images of interest from MSCOCO [26], Open Images [23], as well as Bing Image Search API. In the image extraction stage, we extract regions of interest from the candidate images and resize them to a standardized size. Specifically, for subsets obtained from MSCOCO and Open Images, we extract the bounding boxes with labels of interest. All bounding boxes with an initial size less than 128 × 128 are discarded, and the remaining boxes are extended to squares and resized to a standardized size of 512 × 512 pixels. After this step, we obtained approximately 20k extracted and resized candidate images. We then control the data quality through a filtering and labeling stage. Our filtering process first discards replicated images using the L2 distance metric and remove images containing human faces, as well as cups with a front-facing perspective with a CLIP processor. Our labeling process starts with an automatic annotation with a CLIP classifier. To ensure the accuracy of the dataset, three human annotators thoroughly review the collected images, verifying that the images portray a top-down view of a container and assigning the appropriate labels to the respective domains. The resulting ManiCups dataset contains 5 domains, including 3 abundant domains (empty, coffee, juice) with more than 1K images in each category and 2 low-resource domains (water, milk) with less than 1K images to facilitate research and analysis in data-efficient learning.

To our knowledge, ManiCups is one of the first datasets targeted to the physical state changes of objects, other than stylistic transfers or type changes of objects. The ability to generate consistent state changes based on manipulation is fundamental for future coherent video prediction [14] as well as understanding and planning for physical agents [49, 18, 8, 46]. For additional details on data collection, processing, and statistics, please refer to Appendix B.

4.2 Experiment Setup

Baselines We compare our proposed models FastCycleNet and CycleNet to state-of-the-art methods for unpaired or zero-shot image-to-image translation.

GAN-based methods: CycleGAN [55] and CUT [34];

Mask-based diffusion methods: Direct inpainting with CLIPSeg [28] and Text2LIVE [2];

Mask-free diffusion methods: ControlNet with Canny Edge [51], ILVR [5], EGSDE [53], SDEdit [29], Pix2Pix-Zero [35], MasaCtrl [3], CycleDiffusion [48], and Prompt2Prompt [10] with null-text inversion [30].

Training We train our model with a batch size of 4 on only one single A40 GPU. 2 Additional details on the implementations are available in Appendix C.

Sampling As shown in Figure 3, CycleNet has a good efficiency at inference time and more sampling steps lead to better translation quality. We initialize the sampling process with the latent noised input image z t , collected using Equation 1. Following [29], a standard 50-step sampling is applied at inference time with t = 100 for fair comparison.

#Step = 1

#Step = 2

#Step = 5

Input (

winter,apple

)

summer

orange

→

→

4.3 Qualitative Evaluation

We present qualitative results comparing various image translation models. Due to the space limit, additional examples will be available in Appendix E. In Figure 4, we present the two unpaired translation tasks: summer → winter , and horse → zebra . To demonstrate the image quality, translation quality, and consistency compared to the original images, we provide a full image for each test case

Input

CycleNet

Summer → Winter

P2P +

NullText

Cycle

Diffusion

Fast

CycleNet

Horse → Zebra

SDEdit

ILVR

EGSDE

CUT

CycleGAN

Inpaint

+ClipSeg

Pix2Pix

Zero

MasaCtrl

Text2LIVE

ControlNet

+Canny

and enlarge the boxed areas for detailed comparisons. As presented with the qualitative examples, our methods are able to perform image manipulation with high quality like the other diffusion-based methods, while preserving the structures and unrelated semantics.

In Figure 5, we present qualitative results for filling and emptying a cup: coffee ↔ empty and empty ↔ juice . As demonstrated, image editing tasks that require physical state changes pose a significant challenge to baselines, which struggle with the translation itself and/or maintaining strong consistency. CycleNet, again, is able to generate faithful and realistic images that reflect the physical state changes.

4.4 Quantitative Evaluation

We further use three types of evaluation metrics respectively to assess the quality of the generated image, the quality of translation, and the consistency of the images. For a detailed explanation of these evaluation metrics, we refer to Appendix C.4.

Image Quality . To evaluate the quality of images, we employ two metrics: The naïve Fréchet Inception Distance (FID) [12] and FID clip [24] with CLIP [36];

Input

CycleNet

(Empty)

Cup of Coffee

→ Empty Cup

Cycle

Diffusion

SDEdit

Text2LIVE

Inpaint

+ClipSeg

(Fill)

Empty Cup

→ Cup of Juice

(Fill)

Empty Cup

→Cup of Coffee

(Empty)

Cup of Juice

→ Empty Cup

P2P +

NullText

COOVFE

nocsunhtu

Translation Quality . We use CLIPScore [11] to quantify the semantic similarity of the generated image and conditional prompt;

Translation Consistency . We measure translation consistency using four different metrics: L2 Distance, Peak Signal-to-Noise Ratio (PSNR) [4], Structural Similarity Index Measure (SSIM) [47], and Learned Perceptual Image Patch Similarity (LPIPS) [52].

The evaluations are performed on the reserved test set. As demonstrated in the quantitative results from Table 1, we observe that some baselines display notably improved consistency in the ManiCups tasks. This could possibly be attributed to the fact that a considerable number of the test images were not successfully translated to the target domain, as can be seen in the qualitative results presented in Figure 5. Overall, CycleNet exhibits competitive and comprehensive performance in generating high-quality images in both global and local translation tasks, especially compared to the mask-based diffusion methods. Meanwhile, our methods ensure successful translations that fulfill the domain specified by text prompts while maintaining an impressive consistency from the original images.

5 Analysis and Discussions

5.1 Ablation study

Recall that FastCycleNet removes the cycle consistency loss L x → y → x from CycleNet. We further remove the invariance loss L x → y → y to understand the role of each loss term. For better control, we initialize the sampling process with the same random noise ε rather than the latent noised image z t . Table 2 shows our ablation study on winter → summer task with FID, CLIP score, and LPIPS. When both losses are removed, the model can be considered as a fine-tuned LDM backbone (row 4), which produces a high CLIP similarity score of 24.35. This confirms that the pre-trained LDM backbone can already make qualitative translations, while the LPIPS score of 0.61 implies a poor consistency from the original images. When we introduced the consistency constraint (row 3), the model's LPIPS score improved significantly with a drop of the CLIP score to 19.89. This suggests a trade-off between cycle consistency and translation quality. When we introduced the invariance constraint (row 2), the model achieved the best translation quality with fair consistency. By introducing both constraints (row 1), CycleNet strikes the best balance between consistency at a slight cost of translation quality.

Consistency Only

CycleNet

Input

None

Invariance Only

summer

→

winter

FID

↓

CLIP

↑

LPIPS

↓

CycleNet

77.16

24.15

0.15

Invariance Only

76.23

25.13

0.23

Consistency Only

84.18

19.89

0.14

None

211.26

24.35

0.61

5.2 Zero-shot Generalization to Out-of-Distribution Domains

CycleNet performs image manipulation with text and image conditioning, making it potentially generalizable to out-of-distribution (OOD) domains with a simple change of the textual prompt. As illustrated in Figure 6, we demonstrate that CycleNet has a remarkable capability to generate faithful and high-quality images for unseen domains. These results highlight the robustness and adaptability of CycleNet to make the most out of the pre-trained LDM backbone to handle unseen scenarios. This underscores the potential to apply CycleNet for various real-world applications and paves the way for future research in zero-shot learning and OOD generalization.

5.3 Translation Diversity

Diversity is an important feature of image translation models. As shown in Figure 6, we demonstrate that CycleNet can generate a variety of images that accurately satisfy the specified translation task in the text prompts, while maintaining consistency.

Source

Target

Out-Of-Domain Generalization

winter

summer

fall

fire

minecraft

horse

zebra

sky

desert

unicorn

5.4 Limitations: Trade-off between consistency and translation

There have been concerns that cycle consistency could be too restrictive for some translation task [54]. As shown in Figure 7, while CycleNet maintains a strong consistency over the input image, the quartered apple failed to be translated into its orange equivalence. In GAN-based methods, local discriminators have been proposed to address this issue [56], yet it remains challenging to keep global consistency while making faithful local edits for LDM-based approaches.

Input

CycleNet

P2P + NullText

Cycle Diffusion

SDEdit

6 Related Work

6.1 Conditional Image Manipulation with Diffusion Models

Building upon Diffusion Probabilistic Models [42, 13], pre-trained Diffusion Models (DMs) [38, 37, 39] have achieved state-of-the-art performance in image generation tasks. Text prompts are the most common protocol to enable conditional image manipulation with DMs, which can be done by fine-tuning a pre-trained DM [19, 20]. Mask-based methods have also been proposed with the help of user-prompted/automatically generated masks [32, 7, 1] or augmentation layers [2]. To refrain from employing additional masks, recent work has explored attention-based alternatives [10, 30, 25, 35].

They first invert the source images to obtain the cross-attention maps and then perform image editing with attention control. The promising performance of these methods is largely dependent on the quality of attention maps, which cannot be guaranteed in images with complicated scenes and object relationships, leading to undesirable changes. Very recently, additional image-conditioning has been explored to perform paired image-to-image translation, using a side network [51] or an adapter [31]. In this work, we follow this line of research and seek to enable unpaired image-to-image translation with pre-trained DMs while maintaining a satisfactory level of consistency.

6.2 Unpaired Image-to-Image Translation

Image-to-image translation (I2I) is a fundamental task in computer vision, which is concerned with learning a mapping across images of different domains. Traditional GAN-based methods [16] require instance-level paired data, which are difficult to collect in many domains. To address this limitation, the unpaired I2I setting [55, 27] was introduced to transform an image from the source domain X into one that belongs to the target domain Y , given only unpaired images from each domain. Several GAN-based methods [55, 50, 27] were proposed to address this problem. In recent years, DPMs have demonstrated their superior ability to synthesize high-quality images, with several applications in I2I translation [40, 5]. With the availability of pre-trained DMs, SDEdit [29] changes the starting point of generation by using a noisy source image that preserves the overall structure. EGSDE [53] combines the merit of ILVR and SDEdit by introducing a pre-trained energy function on both domains to guide the denoising process. While these methods result in leading performance on multiple benchmarks, it remains an open challenge to incorporate pre-trained DMs for high-quality image generation, and at the same time, to ensure translation consistency.

6.3 Cycle Consistency in Image Translation

The idea of cycle consistency is to regularize pairs of samples by ensuring transitivity between the forward and backward translation functions [41, 33, 17]. In unpaired I2I translation where explicit correspondence between source and target domain images is not guaranteed, cycle consistency plays a crucial role [55, 21, 27]. Several efforts were made to ensure cycle consistency in diffusion-based I2I translation. UNIT-DDPM [40] made an initial attempt in the unpaired I2I setting, training two DPMs and two translation functions from scratch. Cycle consistency losses are introduced in the translation functions during training to regularize the reverse processes. At inference time, the image generation does not depend on the translation functions, but only on the two DPMs in an iterative manner, leading to sub-optimal performance. Su et al. [43] proposed the DDIB framework that exact cycle consistency is possible assuming zero discretization error, which does not enforce any cycle consistency constraint itself. Cycle Diffusion [48] proposes a zero-shot approach for image translation based on Su et al. [43]'s observation that a certain level of consistency could emerge from DMs, and there is no explicit treatment to encourage cycle consistency. To the best of our knowledge, CycleNet is the first to guarantee cycle consistency in unpaired image-to-image translation using pre-trained diffusion models, with a simple trainable network and competitive performance.

7 Conclusion

The paper introduces CycleNet that incorporates the concept of cycle consistency into text-guided latent diffusion models to regularize the image translation tasks. CycleNet is a practical framework for low-resource applications where only limited data and computational power are available. Through extensive experiments on unpaired I2I translation tasks at scene, object, and state levels, our empirical studies show that CycleNet is promising in consistency and quality, and can generate high-quality images for out-of-domain distributions with a simple change of the textual prompt.

Future Work This paper is primarily concerned with the unpaired I2I setting, which utilizes images from unpaired domains during training for domain-specific applications. Although CycleNet demonstrates robust out-of-domain generalization, enabling strong zero-shot I2I translation capabilities is not our focus here. We leave it to our future work to explore diffusion-based image manipulation with image conditioning and free-form language instructions, particularly in zero-shot settings.

References

Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Xi Yin. Spatext: Spatio-textual representation for controllable image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023.

Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text-driven layered image and video editing. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV , pages 707-723. Springer, 2022.

Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 22560-22570, October 2023.

Luen C Chan and Peter Whiteman. Hardware-constrained hybrid coding of video imagery. IEEE Transactions on Aerospace and Electronic Systems , (1):71-84, 1983.

Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 14347-14356. IEEE Computer Society, 2021.

Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11472-11481, 2022.

Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusionbased semantic image editing with mask guidance. In The Eleventh International Conference on Learning Representations , 2023.

Xiaolin Fang, Caelan Reed Garrett, Clemens Eppner, Tomás Lozano-Pérez, Leslie Pack Kaelbling, and Dieter Fox. Dimsam: Diffusion models as samplers for task and motion planning under partial observability. arXiv preprint arXiv:2306.13196 , 2023.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In Neural Information Processing Systems , 2014.

Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or. Prompt-to-prompt image editing with cross-attention control. In The Eleventh International Conference on Learning Representations , 2022.

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 7514-7528, 2021.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Tobias Höppe, Arash Mehrjou, Stefan Bauer, Didrik Nielsen, and Andrea Dittadi. Diffusion models for video prediction and infilling. Transactions on Machine Learning Research , 2022. ISSN 2835-8856.

Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised imageto-image translation. In Proceedings of the European conference on computer vision (ECCV) , pages 172-189, 2018.

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1125-1134, 2017.

Zdenek Kalal, Krystian Mikolajczyk, and Jiri Matas. Forward-backward error: Automatic detection of tracking failures. In 2010 20th international conference on pattern recognition , pages 2756-2759. IEEE, 2010.

Ivan Kapelyukh, Vitalis Vosylius, and Edward Johns. Dall-e-bot: Introducing web-scale diffusion models to robotics. IEEE Robotics and Automation Letters , 2023.

Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6007-6017, 2023.

Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2426-2435, 2022.

Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning to discover cross-domain relations with generative adversarial networks. In International conference on machine learning , pages 1857-1865. PMLR, 2017.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations , 2014.

Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International Journal of Computer Vision , 128(7):1956-1981, 2020.

Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fréchet inception distance. In The Eleventh International Conference on Learning Representations , 2023.

Senmao Li, Joost van de Weijer, Taihang Hu, Fahad Shahbaz Khan, Qibin Hou, Yaxing Wang, and Jian Yang. Stylediffusion: Prompt-embedding inversion for text-based editing. arXiv preprint arXiv:2303.15649 , 2023.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

Ming-Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. Advances in neural information processing systems , 30, 2017.

Timo Lüddecke and Alexander Ecker. Image segmentation using text and image prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7086-7096, 2022.

Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations , 2021.

Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6038-6047, 2023.

Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08453 , 2023.

Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning , pages 16784-16804. PMLR, 2022.

Pan Pan, Fatih Porikli, and Dan Schonfeld. Recurrent tracking using multifold consistency. In Proceedings of the Eleventh IEEE International Workshop on Performance Evaluation of Tracking and Surveillance , volume 3, 2009.

Taesung Park, Alexei A Efros, Richard Zhang, and Jun-Yan Zhu. Contrastive learning for unpaired image-to-image translation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX 16 , pages 319-345. Springer, 2020.

Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu. Zero-shot image-to-image translation. In ACM SIGGRAPH 2023 Conference Proceedings , pages 1-11, 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo-Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems , 2022.

Hiroshi Sasaki, Chris G Willcocks, and Toby P Breckon. Unit-ddpm: Unpaired image translation with denoising diffusion probabilistic models. arXiv preprint arXiv:2104.05358 , 2021.

Ishwar K. Sethi and Ramesh Jain. Finding trajectories of feature points in a monocular image sequence. IEEE Transactions on Pattern Analysis and Machine Intelligence , PAMI-9(1):56-73, 1987. doi: 10.1109/TPAMI.1987.4767872.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256-2265. PMLR, 2015.

Xuan Su, Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image-to-image translation. In International Conference on Learning Representations , 2022.

Dmitrii Torbunov, Yi Huang, Haiwang Yu, Jin Huang, Shinjae Yoo, Meifeng Lin, Brett Viren, and Yihui Ren. Uvcgan: Unet vision transformer cycle-consistent gan for unpaired imageto-image translation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 702-712, 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, and Jiwen Lu. Drivedreamer: Towards real-world-driven world models for autonomous driving. arXiv preprint arXiv:2309.09777 , 2023.

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing , 13(4): 600-612, 2004.

Chen Henry Wu and Fernando De la Torre. A latent space of stochastic diffusion models for zero-shot image editing and guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7378-7387, 2023.

Yue Yang, Wenlin Yao, Hongming Zhang, Xiaoyang Wang, Dong Yu, and Jianshu Chen. ZLaVI: Zero-shot language solver fueled by visual imagination. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 1186-1203, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.

Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image-to-image translation. In Proceedings of the IEEE international conference on computer vision , pages 2849-2857, 2017.

Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3836-3847, 2023.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018.

Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. Egsde: Unpaired image-to-image translation via energy-guided stochastic differential equations. Advances in Neural Information Processing Systems , 35:3609-3623, 2022.

Yihao Zhao, Ruihai Wu, and Hao Dong. Unpaired image-to-image translation using adversarial consistency loss. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX 16 , pages 800-815. Springer, 2020.

Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision , pages 2223-2232, 2017.

Xianhui Zong, Zhehan Chen, and Dadong Wang. Local-cyclegan: a general end-to-end network for visual enhancement in complex deep-water environment. Applied Intelligence , 51:19471958, 2021.

A CycleNet Details

A.1 Algorithmatic Details

The pseudocode for training is given as follows.

Algorithm 1: CycleNet Training Framework

A.2 Proof of Cycle Consistency Regularization

x -Reconstruction Loss Dropping the variances, L x → x = E x 0 ,ε x || ε θ ( x t , c x , x 0 ) -ε x || 2 2 .

Proof.

y -Reconstruction Loss Dropping the variances, L y → y = E x 0 ,ε x ,ε y || ε θ ( y t , c y , ¯ y 0 ) -ε y || 2 2 .

Proof.

Cycle Consistency Loss Dropping the variances, L x → y → x = E x 0 ,ε x ,ε y || ε θ ( y t , c x , x 0 ) + ε θ ( x t , c y , x 0 ) -ε x -ε y || 2 2 .

Proof.

Invariance Loss Dropping the variances, L x → y → y = E x 0 ,ε x || ε θ ( x t , c y , x 0 ) -ε θ ( x t , c y , ¯ y 0 ) || 2 2 .

Proof.

A.3 Proof of Self Regularization

Let ε ∗ θ denote the denoising autoencoder of the pre-trained text-guided LDM backbone with a generation function G ∗ . Note that x t = S ( x 0 , ε x ) .

Proof.

Note that E x 0 ,ε x ∣ ∣ ∣ 1 -P [ G ∗ ( x t , c y ) , c y ] ∣ ∣ ∣ := E ∗ LDM is the translation error likelihood of the pre-trained LDM backbone. According to the domain smoothness in Assumption 1, we have:

B Scientific Artifacts and Licenses

B.1 The ManiCups Dataset

We introduce ManiCups , a dataset that tasks image editing models to manipulate cups by filling or emptying liquid to/from containers, curated from human-annotated bounding boxes in publicly available datasets and Bing Image Search (under Share license for training and Modify for test set). The dataset consists of a total of 5714 images of empty cups and cups of coffee, juice, milk, and water. We describe our three-stage data collection pipeline in the following paragraphs.

In the Image Collection stage, we gather raw images of interest from MSCOCO [26], Open Images [23], as well as Bing Image Search API. For the MSCOCO 2017 dataset, we specifically searched for images containing at least one object from the categories of ' bottle ,' ' cup ,' and ' wine glass .' Regarding the Open Images v7 dataset, our search focused on images with at least one object falling under the categories of "coffee," ' coffee cup ,' ' mug ,' ' juice ,' ' milk ,' and ' wine glass .' To conduct the Bing Image Search, we utilized API v7 and employed queries with the formats 'empty <container> ' and ' <container> of <liquid> .' The <container> category encompasses cups, glasses, and mugs, while the <liquid> category includes coffee, juice, water, and milk. We obtained approximately 30,000 image candidates after this step.

During the Image Extraction stage, we extract regions of interest from the candidate images and resize them to a standardized size of 512 × 512 pixels. Specifically, for subsets obtained from MSCOCO and Open Images, we extract the bounding boxes with labels of interest. To ensure a comprehensive representation, each bounding box is extended by 5% on each side, and the width is adjusted to create a square window. In cases where the square window exceeds the image boundaries, we shift the window inwards. If, due to resizing or shifting, a square window cannot fit within the image, we utilize the cv2.BORDER_REPLICATE function to replicate the edge pixels. All bounding boxes with an initial size less than 128 × 128 are discarded, and the remaining images are resized to a standardized size of 512 × 512 pixels. The same approach is applied to images obtained from the Bing Search API, utilizing cv2.BORDER_REPLICATE for edge pixel replication. Following this step, we obtained approximately 20,000 extracted and resized images.

In the Filtering and Labeling stage, our focus is on controlling the quality of images and assigning appropriate labels. Our filtering process begins by identifying and excluding replicated images using the L2 distance metric. Subsequently, we leverage the power of CLIP to detect and remove images containing human faces, as well as cups with a front-facing perspective. Additionally, we use CLIP to classify the remaining images into their respective domains. To ensure the accuracy of the dataset, three human annotators thoroughly review the collected images, verifying that the images portray a top-down view of a container and assigning the appropriate labels to the respective domains. The resulting ManiCups dataset contains 5 domains, including 3 abundant domains (empty, coffee, juice) with more than 1K images in each category and 2 low-resource domains (water, milk) with less than 1K images to facilitate research and analysis in data-efficient learning.

To our knowledge, ManiCups is one of the first datasets targeted to the physical state changes of objects, other than stylistic transfers or type changes of objects. The ability to generate consistent state changes based on manipulation is fundamental for future coherent video prediction [14] as well as understanding and planning for physical agents [49 ? ]. We believe that ManiCups is a valuable resource to the community.

B.2 Dataset Statistics

We present the statistics of the ManiCups in Table 3, and other scene/object level datasets in Table 4.

B.3 Licenses of Scientific Artifacts

We present a complete list of references and licenses in Table 5 for all the scientific artifacts we used in this work, including data processing tools, datasets, software source code, and pre-trained weights.

C Experiment Details

C.1 Computational Resources

Table 6 illustrates the training performance of CycleNet and FastCycleNet on a single NVIDIA A40 GPU under 256 × 256 with batch size of 4. The table provides information on the training speed in seconds per iteration and the memory usage in gigabytes for both models. FastCycleNet exhibits a faster training speed of 1.1 seconds per iteration while consuming 24.5 GB of memory. On the other hand, CycleNet demonstrates a slightly slower training speed of 1.8 seconds per iteration, and it requires 27.9 GB of memory.

C.2 Hyper-parameter Decisions

We include the major hyper-parameter tuning decisions for reproducibility purposes. In the training of CycleNet, the weights of our three loss functions are respectively set as λ 1 = 1 , λ 2 = 0 . 1 , and λ 3 = 0 . 01 . We train the model for 50k steps. Following [29], we initialize the sampling process with the latent noised input image z t , collected using Equation 1. A standard 50-step sampling is applied at inference time with t = 100 . Our configuration is as follows:

For more details, please refer to the supplementary codes.

C.3 Baseline Implementations

CycleGAN : We used some of the results provided in [34].

CUT : We used the official code 4 provided by the authors.

Inpainting + ClipSeg : we modified from the gradio 5 provided by the community.

Text2LIVE : We used the official code 6 provided by the authors.

ILVR : We first pre-train the diffusion model using P2-weighting [6], and then generated output using the official code 7 provided by the authors.

EGSDE : We first pre-train the diffusion model using P2-weighting [6], and then generated output using the official code 8 provided by the authors.

SDEdit : We used the community implementation 9 of SDEdit based on stable diffusion.

CycleDiffusion : We modified from the official gradio 10 provided by the authors.

Prompt2Prompt + NullText : We used the official code 11 provided by the authors.

Masactrl : We used the official code 12 provided by the authors.

Pix2pix-zero : We used the official code 13 provided by the authors, and we generated the summer2winter direction assets following the scripts provided by the authors.

C.4 Evaluation Metrics Explained

Image Quality To evaluate the quality of images, we employ two metrics.

Fréchet Inception Distance (FID) [12] is a widely used metric in image generation tasks. A lower FID score indicates better image quality and more realistic samples.

FID clip [24] combines the FID metric with features extracted with a CLIP [36] encoder, providing better assessment of image quality. Similar to FID, a lower FID clip score represents better image quality.

Translation Quality To measure to what extent is the translation successful, we use the CLIP Score [11], i.e., the CLIP model [36] to obtain the latent representations of images and prompts, and then calculate the cosine similarity between them. A higher CLIP score indicates a stronger similarity between the generated image and the text prompt, thus better translation.

Translation Consistency We measure translation consistency using four different metrics.

L2 Distance is a measure of the Euclidean distance between two images. A lower L2 distance indicates higher similarity and better translation consistency.

Peak Signal-to-Noise Ratio (PSNR) [4] measures the ratio between the maximum possible power of a signal and the power of corrupting noise. A higher PSNR score indicates better translation consistency.

Structural Similarity Index Measure (SSIM) [47] is a metric used to compare the structural similarity between two images. A higher SSIM score suggests higher similarity and better translation consistency.

Learned Perceptual Image Patch Similarity (LPIPS) [52] is a comprehensive evaluation metric for the perceptual similarity between two images. A lower LPIPS score indicates higher perceptual similarity.

D Broader Impact

While CycleNet holds great promise, it is essential to address potential broader impacts, including ethical, legal, and societal considerations. One significant concern is copyright infringement. As an image translation model, CycleNet can potentially be used to create derived works from artists' original images, raising the potential for copyright violations. To safeguard the rights of content creators and uphold the integrity of the creative economy, it is imperative to prioritize careful measures and diligently adhere to licensing requirements. Another critical aspect to consider is the potential for fabricated images to contribute to deception and security threats. If misused or accessed by malicious actors, the ability to generate realistic fake images could facilitate misinformation campaigns, fraud, and even identity theft. This underscores the need for responsible deployment and robust security measures to mitigate such risks. CycleNet leverages pre-trained latent diffusion models, which may encode biases that lead to fairness issues. It is worth noting that the proposed method is currently

purely algorithmic, devoid of pre-training on web-scale datasets itself. By acknowledging and actively addressing these broader impacts, we can work towards harnessing the potential of CycleNet while prioritizing ethics, legality, and societal well-being.

E Addendum to Results

E.1 Choice of Pre-trained LDM Backbone

For all experiments in the main paper, we use Stable Diffusion 2.1 14 as the pre-trained LDM backbone. We additionally attach a quantitative comparison of CycleNet using Stable Diffusion 1.5, 15 which indicates marginal differences in practice.

E.2 Image Resolution

We notice that the Stable Diffusion [38] backbone is pre-trained in multiple stages, initially at a resolution of 256 × 256 and followed by another stage at a resolution of 512 × 512 or beyond. This could potentially lead to the under-performance of zero-shot diffusion-based methods on summer → winter and horse → zebra , which are at a resolution of 256 × 256. We repeat the experiment on these two tasks at a generation resolution of 512 × 512 and report the results in Table 8. It's important to acknowledge that this particular setting presents an unfair comparison with considerable challenges for our methods, primarily because the training images are set at a resolution of 256 × 256, yet our model is expected to adapt to a higher resolution. Still, we observe a competitive performance of our models, especially in summer → winter . The diminished effectiveness of our method in the horse → zebra task can be attributed to the fact that the zebra patterns, initially acquired at a resolution of 256 × 256, lose realism and become overly dense when scaled up to 512 × 512 (see Example 3, Figure 11). This limitation can potentially be addressed by scaling to images with multiple resolutions.

E.3 Sudden Convergence

As shown in Figure 8, CycleNet can translate an input image of summer to winter at the beginning of training with no consistency observed. Similar to ControlNet [51], CycleNet also demonstrates the sudden convergence phenomenon, which usually happens around 3k to 8k iterations of training.

E.4 Addtional Quantitative Results

In Table 9, we present the complete numerical performance of the state-changing tasks on ManiCups . In general, we found that emptying a cup is a more challenging image editing task for most of

0

1000

2000

3000

4000

Input

10000

50000

the existing computation models, compared to filling a cup with liquid. CycleNet shows superior performance, especially in domains with abundant training samples (coffee and juice). We also observe that the performance is marginally less competitive in low-resource domains (milk and water). We leave it to future work to explore more data-efficient models that fine-tune pre-trained latent diffusion models with cycle consistency granted.

E.5 Additional Qualitative Results

We present additional qualitative examples in Figure 9 and 10.

E.6 Additional High-resolution Qualitative Results

We present additional high-resolution examples in Figure 11.

E.7 Additional Out-of-Domain Examples

We present additional OOD examples in Figure 12.

cup of

juice

cup of

milk

cup of

water

cup of

coffee

Empty Cup

→

Input

CycleNet

Summer → Winter

P2P + NullText

CUT

CycleGAN

Cycle Diffusion

FastCycleNet

Text2LIVE

Apple → Orange

Horse → Zebra

SDEdit

Inpaint + ClipSeg

ILVR

EGSDE

Control + Canny

MasaCtrl

Pix2Pix-Zero

Input

CycleNet

P2P + NullText

Cycle Diffusion

SDEdit

Apple → Orange

Horse → Zebra

Winter → Summer

Source

Target

Out-Of-Domain Generalization

summer

winter

spring

sunset

desert

apple

orange

baseball

glass

christmas

zebra

horse

robot

arctic

bear

orange

apple

basketball

CD

moon

Peer reviews:
comments: SUMMARY:
The paper proposes CycleNet, a method for unpaired image-to-image translation using pretrained diffusion models. The main idea is to reconstruct the conditional images through a reverse process. The idea seems reasonable, and the paper is well-organized, with sufficient experimental results.

STRENGTHS:
The idea seems reasonable, and the paper is well-organized, with sufficient experimental results.

WEAKNESSES:
1. The concept of cycle construction has been extensively studied in the field, and this work follows a similar idea, which weakens its contribution.
2. In CycleGAN, the reconstruction constraint $L_{x \rightarrow y  \rightarrow x}$ plays a vital role. However, in FastCycleNet, this constraint is omitted, yet the performance does not show significant drops. It would be helpful if the authors could provide an explanation for this observation.
3. For some variable notation (such as ($\bar{x}_o$), authors had better provide their definitions, It would be beneficial for readers to quickly understand these notations.

QUESTIONS:
Please see in weakness

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces CycleNet, a new method that enhances image manipulation by incorporating cycle consistency into diffusion models. The paper addresses the challenge of unpaired image-to-image translation and aims to provide a consistent and intuitive interface for this task.\n\nSTRENGTHS:\n1.Originality: The paper introduces CycleNet, a new method that incorporates cycle consistency into diffusion models for image manipulation. This approach combines the concept of cycle consistency with text-guided diffusion models. \n\n2. Quality: The paper provides a comprehensive evaluation of CycleNet's performance using various quantitative metrics such as FID, CLIP Score, LPIPS, PSNR, SSIM, and L2.  The results also support the claims of the paper.\n\n3. Clarity: The paper is well-written and presents the concepts, methodology, and findings.\n\nWEAKNESSES:\n1.Novelty is limited. The concept of combing cycle-consistency constraint and diffusion model for image-to-image translation has been widely explored by other works, such as UNIT-DDPM, Dual Diffusion Implicit Bridges (DDIBs), cyclediffusion and so on.\n\n2.Lack of Comparative Analysis between the proposed method with other existing cycle-consistency based diffusion models. The part of 6.3 is not sufficient to illustrate the key difference and novelty of the proposed method.\n\n3.The results are low-quality and far from satisfaction. For the classic image-to-image translation tasks, i.e., winter-to-summer and horse-to-zebra, I do not find any improvement over existing ui2i methods. In addition, the results of CUT aand CycleGAN are questionable, especially for winter-to-summer task, they are too bad compared with the results from their original papers.\n\n4.The results of coffee-to-empty also do not convince me as it seems to only transfer the rgb image to gray image. Similar results also happen in the horse-to-zebra.\n\nQUESTIONS:\n1. Why there lack the results of strong baseline CycleGAN in Table 1?\n\n2. Could the authors consider conducting a user study to evaluate the subjective quality and usability of CycleNet? This would provide insights into how users perceive the generated images and the overall user experience, further validating the practicality and effectiveness of CycleNet.\n\n3. Could the authors provide more information about the computational efficiency of CycleNet? Specifically, details about the training and inference times, as well as the hardware specifications used, would be helpful in understanding the practicality of CycleNet for real-world applications.\n\nLIMITATIONS:\n1. Novelty is limited.\n\n2. Poor results are far from satisfaction as the strong generation ability of the diffusion model.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
This work aims at addressing the task of unpaired I2I translation with pre-trained diffusion models. Inspired by CycleGAN, authors incorporate cycle consistency into diffusion model to regularize process of image translation and proposed CycleNet. In addition it also contributes a multi-domain I2I translation dataset with object state change. Extensive experiments demonstrate that the proposed method can produce high-quality translation results with good consistency and is able to generalize to out-of-domain generation tasks by changing text prompts.

STRENGTHS:
1. It is interesting and reasonable to introduce cycle consistency into diffusion models for training unpaired I2I translation models.
2. A new multi-domain I2I translation dataset for object state change with text prompt is collected.
3. Extensive experiments demonstrate the effectiveness of the proposed method.
4. Capbility to generalize to out-of-domain data by simply changing text prompt.

WEAKNESSES:
1. I wonder how CycleNet compares with CycleGAN. More comparison between CylceNet and CycleGAN should be provided. If their performance is similar, then why use a diffusion-based method instead of GAN-based for unpaired I2I translation task?
2. Authors are suggested to experiment with more diffusion models such as T2I Adapter to show its generalization on models.
3. Any explanation on why FastCycleNet perform better in several metrics? The coffee-to-empty translation result does not seem good, not only changing the color but also failing to become empty.

QUESTIONS:
See weakness.

LIMITATIONS:
Authors have clearly showed failure cases of the proposed method in the main text. But they do not address potential negative socital impact of this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This paper presents a method that tackles the challenge of consistent image-to-image (I2I) translation using diffusion models (DMs) and unpaired image conditioning and text prompts. The proposed approach incorporates Cycle Consistency Regularization to ensure cycle consistency and Self-Regularization to generate images that align with the target domain. Furthermore, the paper introduces a multi-domain I2I translation dataset that encompasses object state changes.

STRENGTHS:
a)	From the visual results, the proposed CycleNet can well preserving the structures of the input image.

b)	The author introduces a multi-domain I2I translation dataset for object state changes.

WEAKNESSES:
a)	The experimental results are quite strange. 

i.	In Figure 2, Figure 3, Figure 5, and many images in the supplementary materials, The results of CycleNet show severe hue shifting. This is a serious issue, and I suggest the author analyze and address this problem.

b)	The author's comparison methods are not comprehensive. Many editing methods based on diffusion have been proposed, such as pix2pix-zero, masactrl, etc. I suggest the author compare these methods.

c)	Although CycleNet aims to handle unpaired I2I translation tasks, it still requires paired datasets for training. So, how well does CycleNet generalize to unseen domains (such as variations of other unseen species, etc.)?

QUESTIONS:
Please refer to the weaknesses part.

LIMITATIONS:
Please refer to the weaknesses part.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces cycle consistency in diffusion models to achieve regularization in image manipulation. It allows out of domain image generation with text prompt modification, and can be trained with very little data (~2K) with minimal computational requirements (1GPU). When we apply pre-trained DM models on I2I tasks, it is difficult or unpredictable to control the mask and attention maps to achieve the desired results. Cyclic consistency regularization allows us to address this. Experiments are conducted on many datasets such as Yosemiti summer-winter, horse-zebra, etc. On standard metrics such as CLIP, FID, and LPIPS, the proposed method outperforms other baselines such as CycleGAN, TEX2LIVE, etc.\n\nSTRENGTHS:\n1) There has not been much regularization in diffusion models except for guidance driven by the conditional text. It is nice to see the celebrated Cycle-Consistency loss developed for diffusion models. \n\n2) The paper clearly motivates and states the problem. It is nice to see that the proposed method nicely inherits the controlnet architecture to incorporate the cycle-consistency loss function during training.\n\nWEAKNESSES:\n1) While enforcing cyclic consistency we are feeding the actual input image as the conditional input. This seems a bit strong and it appears that the diffusion model may be overfitting to the conditional input. \n\n2) It is not clear how one obtains the negative prompt c^{-1}. The method takes a prompt c_{text}=(c^{+}, c^{-}), where c^{+} is the prompt that drives the diffusion process towards the images that are associated with it, and the negative prompt c^{-} drives the process away from these images. The paper only mentions this once without sufficient details on how this is done. \n\n3) The proposed method heavily relies on controlNet and uses the original image to impose the Cyclic-Consistency loss. It may be worth just testing with standard ControlNet with canny edge-map as conditional input and appropriate text prompt.\n\nQUESTIONS:\nOverall, the paper looks at an important problem and brings cyclic consistency to diffusion models. Please see my concerns above.\n\nLIMITATIONS:\nNo concerns.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}