CycleNet: Rethinking Cycle Consistency in Text-Guided Diffusion for Image Manipulation
Authors: Sihan Xu
Ziqiao Ma
Yidong Huang
Honglak Lee
Joyce Chai
Venue: NeurIPS 2023 poster (2023)
Keywords: Image to image translation
latent diffusion models
conditional diffusion models
TL;DR: We propose a simple neural network called CycleNet that incorporates cycle consistency into conditional diffusion models for text-guided image manipulation.

Abstract: Diffusion models (DMs) have enabled breakthroughs in image synthesis tasks but lack an intuitive interface for consistent image-to-image (I2I) translation. Various methods have been explored to address this issue, including mask-based methods, attention-based methods, and image-conditioning. However, it remains a critical challenge to enable unpaired I2I translation with pre-trained DMs while maintaining satisfying consistency. This paper introduces Cyclenet, a novel but simple method that incorporates cycle consistency into DMs to regularize image manipulation. We validate Cyclenet on unpaired I2I tasks of different granularities. Besides the scene and object level translation, we additionally contribute a multi-domain I2I translation dataset to study the physical state changes of objects. Our empirical studies show that Cyclenet is superior in translation consistency and quality, and can generate high-quality images for out-of-domain distributions with a simple change of the textual prompt. Cyclenet is a practical framework, which is robust even with very limited training data (around 2k) and requires minimal computational resources (1 GPU) to train. Project homepage: https://cyclenetweb.github.io/

CycleNet: Rethinking Cycle Consistency in Text-Guided Diffusion for Image Manipulation
Sihan Xul* Ziqiao Ma |M Yidong Huang' Honglak Lee Joyee Chail
University of Michigan, LLG Al Research
(sihanxu,marstingg owenhji honglak, chaijy) @umich edu
Abstract
Diffusion models (DMs) have enabled breakthroughs in image synthesis tasks but lack intuitive interface for consistent image-to-image (I2I) translation. Various methods have been explored to address this issue, including mask-based meth- ods, attention-based methods, and image-conditioning. However, it remains a critical challenge to enable unpaired 12I translation with pre-trained DMs while maintaining satisfying consistency. This paper introduces CycleNet, novel but simple method that incorporates cycle consistency into DMs to regularize image manipulation. We validate CycleNet on unpaired 12I tasks of different granulari- ties. Besides the scene and object level translation, we additionally contribute a multi-domain 12I translation dataset to study the physical state changes of objects. Our empirical studies show that CycleNet is superior in translation consistency and quality, and can generate high-quality images for out-o--domain distributions with a simple change of the textual prompt. CycleNet is practical framework, which is robust even with very limited training data (around 2k) and requires minimal computational resources (1 GPU) to train.
Input
CycleNet (Ours)
P2P + NullText
Cycle Diffusion
SDEdit
Summer > Winter
Figure A high-resolution example of CycleNet for diffusion- based image-to-image translation compared to other iuffusion-basedd methods. CycleNet produces high-quality translations with satisfactory consistency. The areas in the boxes are enlarged for detailed comparisons.
Introduction
Recently, pre-trained diffusion models (DMs) [38 37 39 have enabled an unprecedented break- through in image synthesis tasks. Compared to GANs and VAEs [22], DMs exhibit superior stability and quality in image generation, as well as the capability to scale up to open-world multi- modal data. As such, pre-trained DMs have been applied image-to-image (12I) translation, which is to acquire a mapping between images from two distinct domains, e.g., different scenes, different
Equul contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

objects, and different object states. For such translations, text-guided diffusion models typically re- quire mask layers [32 7 or attention control [O
25 351. However, the quality of masks and attention maps can be unpredictable in complex scenes, leading to semantic and structural changes that are undesirable. Recently, researchers have explored using additional maagecconditionin to perform paired 12I translations with the help of a side network [51] or an adapter [31]. Still, it remains an open challenge to adapt pre-trained DMs in unpaired 12I translation with a consistency guarantee.
30
We emphasize that consistency, desirable property in image manipulation, is particularly impor- tant in unpaired 12I scenarios where there no guaranteed correspondence between images in the source and target domains. Various applications of DMs. including video prediction and infill- ing [14), imagination-augie
language understanding [49], robotic manipulation [[8 g and
world models [46), would rely on strong consistency across the source and generated images.
To enable unpaired 12T translation using pre-trained DMs with satisfactory consistency, this paper introduces CycleNet, which allows DMs to translate a source image by conditioning on the input image and text prompts. More specifically, we adopt ControlNet [51] with pre-trained Stable Diffusion (SD) [38] as the latent DM backbone. Motivated by cycle consistency in GAN-based methods (55), CycleNet leverages consistency regularization over the image translation cycle. As illustrated in Figure 2 the image translation cycle includes a forward translation from To to Do and a backward translation to Tg. The key idea of our method ensure that when conditioned on an image Cing that falls into the target domain specified by Cuext: the DM should be able to reproduce this image condition through the reverse process.
We validate CycleNet on 12I translation tasks of different granularities. Besides the scene and object level tasks introduced by Zhu et al. (55). we additionally contribute Mani ManiCups, multi-domain 121 translation dataset for manipulating physical state changes of objects. Mani Cuni contains 6k images of empty cups and cups of coffee, juice, milk, and water, collected from human-annotated bounding boxes. The empirical results demonstrate that compared to previous approaches, CycleNet is superior in translation faithfulness, cycle consistency, and image quality. Our approach is also computationally friendly, which is robust even with very limited training data (around 2k) and requires minimal computational resources (1 GPU) to train. Further analysis shows that CycleNet is robust zero-shot 121 translator, which can generate faithful and high-qualis images for out-of of-domain distributions with a simple change of the textual prompt. This opens up possibilities to develop consistent diffusion- based image manipulation models with image conditioning and free-form language instructions.
2 Preliminaries
We start by introducing a set of notations to characterize image-to-image translation with DMs.
Diffusion Models Diffusion models progressively add Gaussian noise to a source image 2g ~ q( =n) through forward diffusion process and subsequently reverse the process to restore the original image. Given a variance schedule B1...., ,r,, the forward process is constrained to a Markov chain q< (1) := ((z /I B 1, (,I),, in which 21.7 are latent variables with dimensions matching z0. The reverse process pa (Z):T) is as well Markovian, with learned Gaussian transitions that begin øт (O, I). Ho et al. [13] noted that the forward process allows the sampling of 2, at any time step / using a closed-form sampling function (Eq. D.
2 S(20, e, = Vārzo vI āze, NOOO I) and [1,
(1)
in which 0e := 1 3r and ā. II as.. Thus, the reverse process can be carried out with a UNet-based network eg that predicts the noise E. By dropping time-dependent variances, the model can be trained according to the objective in Eq.
min E E. ||s Ea e 1)
(2)
Eq. 2 implies that in principle, could estimate the original source image 2o given noised latent Z, at any time i. The reconstructed än can be calculated with the generation function:
G( t) 2 v1 ārEa(Er /
For simplicity, we drop the temporal conditioning t in the following paragraphs.

---

20
8
Ging
e
Sampling
Zero Conv
X
e fx, o c. x
x
yo
Conditional prompt
Unconditional Prompt
4
SD Encoder
SD Encoder
(Trainable Copy)
x
e e 02.%)
y
yo
SD Decoder
Zero Conv
Zo
(b) CycleNet adopts Stable Diffusion (SD) as the backhone and ControlNet conditioning.
(a) The translation cycle in diffusion-basee 12I translation.
Figure 2: The image translation eycle includes forward translation from Ty to To and backward translation to To. The key idea of our method is to ensure that when conditioned on an image Cimg that falls into the target domain specified by Ctext, the LDM should reproduce this image condition through the reverse process. The dashed lines indicate the regularization in the loss functions.
Conditioning in Latent Diffusion Models Latent diffusion models (LDMs) like Stable Diffu- sion [38] can model conditional distributions pe pe(zale c) over condition c.g., by augmenting the UNet backbone with condition-spectt sppecifc encoder using cross-attention mechanism [45). Using textual prompts is the most common approach for enabling conditional image manipulation with LDMs. With a textual prompt 6₂ as conditioning, LDMs strive to learn mapping from latent noised sample 2, to an output image 20, which falls into domain Z that is specified by the conditioning prompt. image manipulation, especially
To enable more flexible and robust conditioning diffusion- mixture of text and image conditioning, recent work obtained further control over the reverse process with a side network (51] or an adapter 331. We denote such conditional denoising autoencoder as Eo Ez(,,, Crext, Cing), where Ging is the image condition and the text condition Crext- rewritten as: Ey G( Czext, Cung) : [2
Eq. 3
can thus be
ārEg Cuext Cing / /
The text condition Gtext contains a pair of conditional and unconditional prompts (ct,c-h, c_ A conditional prompt et guides the diffusion process towards the images that are associated with it, whereas a negative prompt e- drives the diffusion process away from those images.
Consistency Regularization for Unpaired Image-to-Image Translation The goal of unpaired image-to-image (I21) translation is to learn mapping between two domains X' C 12d and y C Rd with unpaired training samples {ææ} for
I.....N, .,, where x; E X belongs to X, and {ıy} for j = 1,..,, M, where Yj E y. In traditional GAN-based translation frameworks, the task typically requires two mappings G : X + Y and F y + X. Cycle consistency enforces transitivity between forward and backward translation functions by regularizing pairs of samples, which is crucial in 12I translation, particularly unpaired settings where no explicit correspondence between images in source and target domains is guaranteed [55 27
50 44]. To ensure cycle consistency,
CycleGAN (55) explicitly regularizes the translation cycle, bringing F(G())) back to the original image and vice versa for y. Motivated by consistency regularization, we seek to enable consistent unpaired I2I translation with LDMs. Without introducing domain -specffc generative models, we use one single denoising network sq for translation by conditioning it on text and image prompts.
3 Method
In the following, we discuss only the translation from domain X' to y due to the symmetry of the backward translation. Our goal, at inference time, to enable LDMs translate a source image Ty by using it as the image condition Cing = Tu, and then denoise the noised latent ye to yt-1 with text prompts Ctext = Gy. To learn such a translation model Ep (Yt, Cy. Wo), we consider two types of training objectives. In the following sections, we describe the cycle consistency regularization to ensure cycle consistency so that the structures and unrelated semantics are preserved the generated images, and the self regularization to match the distribution of generated images with the target
3

---

domain, As illustrated in Figure 2 the image translation cycle includes a forward translation from source image 2n yo- followed by backward translation the reconstructed source image Fp-
3.1 Cycle Consistency Regularization
We assume a likelihood function P(zo, Crext) that the image 2o falls into the data distribution specified by the text condition Ctext- We consider generalized case of cycle consistency given the conditioning mechanism in LDMs. If P(Cing, Ctext) is close to 1, i... the image condition Cung falls exactly into the data distribution described by the text condition Cexx. we should expect that G(Z Crext: Cing) = Cin for any noised latent With the translation cycle in Figure 2 the goal is to
optimize (1) L2-+0 Ex,,s= 100 - G Cz. 11: (2) Cy-v E 11 - G Cy. Bo) (3) L2- = Exo, 100 G(ye, Cz. zo)l and (4) CN- Eza.e. 11 G( Cy. Bo) 2. Proposition 1 (Cycle Consistency Regularization), With the translation cycle in Figure 2
a set of
consistency losses is given by dropping time deeendennd variances:
b = Exo.c, || Cz. vo) -xx1
(5)
Cy-ty = EmgLEzis (VE cy. H0) Eyl12
(6)
Cz- ExoLErIs | Cos xo) + Eo (Ze, Cy. x0) - Ex Eyl12
(7)
E Exo.E | Cz. Fo) - Eg (xre Cy. Bo)1
(8)
We leave the proof in Section A.2 Proposition states that pixel-level consistency can be acquired by regularizing the conditional denoising autoencoder Eg. Specifically, the reconstruction loss Lu-ew and Ly-oy ensures that CycleNet can function LDM to reverse image similar to Eq. 2 The cycle consistency loss Lz-y-= serves as the transitivity regularization, which ensures that the forward and backward translations
reconstruct the original image 2g. The invariance loss Le-oy-y requires that the target image domain stays invariant under forward translation, i.e., given forward translation from 2; to Ju conditioned on 2o, repeating the translation conditioned on jo would reproduce .0-
3.2
Self Regularization
In the previous section, while zo is naturally sampled from domain X, we need to ensure that the generated images fall in the target domain ), i.e., the translation leads to G( Cy. 2o) E y Our goal is therefore maximize P(yo, Cy), equavilently to minimize
LLDM -Ezg.., P (SG(S e). Cy. mo); cyl
Assumption (Domain Smoothness). For any text condition, (text) I-Lipschitz
3 ] <00,, 00, [P(=): Crext) - P( Crext)) S L - =ill2
(10)
Proposition (Self Regularization). Let E denote the denoising autoencoder of the pre-trained text- text-guided LDM backbone. Let 2, S(zo, Ex) be noised latent. A self-supervised upper bound of CLDM is given by:
Cuelf Exn.sr
L
ā āi
|e Cy. ma) (x cy)lla
+ const
(11)
I Lipschitz assumptions have been widely adopted in diffusion methods 53
48
Assumption
hypothesizes that similar images share similar domain distributions. A self-supervised upper bound Ceetr can be obtained in Proposition 2 which intuitively states that if the output of the conditional translation model does not deviate far from the pre-trained LDM backbone, the outcome image should still fall in the same domain specified by the textual prompt. We leave the proof in Section A.3
3.3 CycleNet
In practice, Cueff can be minimized from the beginning of training by using ControlNet [5T] with pre-trained Stable Diffusion (SD) [38] the LDM backbone, which is confirmed through preliminary

---

experiments. As shown in Figure 2 the model keeps the SD encoder frozen and makes trainable copy in the side network. Additional zero convolution layers are introduced to encode the image condition and control the SD decoder. These 7ero convolution layers are ID convolutions whose initial weights and biases vanish and can gradually acquire the optimized parameters from zero. Since the zero convolution layers keep the SD encoder features untouched, Leer is minimal at the beginning of the training, and the training process is essentially fine-tuning a pre-trained LDM with a side network.
The text condition Ctext = {c [[T) ,c contains pair of conditional and unconditional prompts. We keep the conditional prompt in the frozen SD encoder and the unconditional prompt in the ControlNet, so that the LDM backbone focuses on the translation and the side network looks for the semantics that needs modification. For example, to translate an image of summer to winter, we rely on conditional prompt lg = "ssummer and unconditional prompt ly = "winter". Specifically, we use CLIP [36] encoder to encode the language prompts l and ly such that = {CIII(()), (L,)((,)) and Cy = (CTIP(L,,) CL.I((+)).
We also note that Ly-y can be omitted, as La-sæ can serve the same purpose in the symmetry of the translation cycle from y to X, and early experiments confirmed that dropping this term lead to significantly faster convergence. The simplified objective is thus given by:
c. = Ao + X2h 1ohz-+--v - λzZZ
(12)
Consider both translation cycle from X' ++ V, the complete training objective of CycleNet is:
LCycleNet = Lx + Cyy
(13)
The pseudocode for training is given in Algo.
3.4 FastCycleNet
Similar to previous cycle-consistent GAN-based models for unpaired 12I translation, there trade-off between the image translation quality and cycle consistency. Also, the cycle consistency loss f requires deeper gradient descent, and therefore more computation expenses during training (Table 6). In order speed up the training process in this situation, one may consider further removing C--g-v from the training objective, and name this variation FastCycleNet. Through experiments, FastCycleNet can achieve satisfying consistency and competitive translation quality, as shown in Table Different variations models can be chosen depending on the practical needs.
4 Experiments
4.1 Benchmarks
Scene/Object-Lvell Manipulation We validate CycleNet on 121 translation tasks of different granularities. We first consider the benchmarks used in CycleGAN by Zhu et al. (55), which contains:
(Scene Level) Yosemite sumerr+winter. We use around 2k images of summer and winter Yosemite, with default prompts "summer" and "winter":
(Object Level) horset-zebra: We use around 2.5k images of horses and zebras from the dataset with default prompts "horse" and "zebra";
(Ohject Level) applet-orange: We use around 2k apple and orange images with default prompts of "apple" and "orange".
State Level Manipulation Additionally, we introduce Mani dataset of state-level image manipulation that tasks models manipulate cups by filling emptying liquid to/from containers, formulated as a multi-domain I2I translation dataset for object state changes:
(State Level) ManiCups: We use around 5.7k images empty cups and cups of coffee, juice, milk. and water for training. The default prompts are set as "empty cup" and "cup of <liquid>". The task is to either empty a full cup or fill an empty cup with liquid as prompted.
'Our data is available at httttprr/waggnggaa....///ttttttt http:///ssseeedmmmmac//aaiiiia

---

ManiCups is curated from human aanoottte bounding boxes in publicly available datasets and Bing Image Search (under Share license for training and Modify for test set). We describe our three-stage data collection pipeline. In the image collection stage, we gather raw images of interest from MSCOCO [26], Open Images [33,, as well as Bing Image Search API. the image extraction stage, we extract regions of interest from the candidate images and resize them to a standardized size. Specifically, for subsets obtained from MSCOCO and Open Images, we extract the bounding boxes with labels of interest. All bounding boxes with an initial size less than 128x x 128 are discarded, and the remaining boxes are extended to squares and resized to a standardized size of 512 512 pixels. After this step, we obtained approximately 20k extracted and resized candidate images. We then control the data quality through filtering and labeling stage. Our filtering process first discards replicated images using the L2 distance metric and remove images containing human faces, well as cups with a front-facing perspective with a CLIP processor. Our labeling process starts with an automatic annotation with CLIP classifier. To ensure the accuracy of the dataset, three human annotators thoroughly review the collected images, verifying that the images portray a top-down view of container and assigning the appropriate labels the respective domains. The resulting ManiCups dataset contains 5 domains, including abundant domains (empty, coffee, juice) with more than IK images in each category and 2 low-resource domains (water, milk) with less than IK images to facilitate research and analysis in data-efficient learning.
To our knowledge, ManiCups is one of the first datasets targeted to the physical state changes of objects, other than stylistic transfers or type changes of objects. The ability to generate consistent state changes based on manipulation is fundamental for future coherent video prediction [14 as well as understanding and planning for physical agents 49
18
8 46). For additional details on data
collection, processing, and statisties, please refer to Appendix B
4.2 Experiment Setup
Baselines
We compare our proposed models FastCycleNet and CycleNet to state-of-the fe-thhe-attt methods for unpaired or zero-shot image-to-image translation.
GAN-based methods:
CycleGAN (55] and CUT 34:
diffusion methods: Direct inpainting with CLIPSeg 28
and Tex12LIVE 2
Mask-free diffusion methods: ControlNet with Canny Edge 51
ILVR (5)
EGSDE 53
SDEdit [29], Pix2Pix-Zero 355 MasaCtrl
3 CycleDiffusion
48 and Prompt2Prompt
10
with null-text inversion 30
Training We train our model with batch size of 4 on only one single A40 GPU Additional
details on the implementations are available in Appendix
Sampling As shown in Figure 3 CycleNet has good efficiency at inference time and more
sampling steps lead to better translation quality. We initialize the sampling process with the latent noised input image 2,, collected using Equation [ Following 22) standard 50-step sampling is applied at inference time with t = 100 for fair comparison.
Input (winter, apple)
#Step= = 1 #Step 2
#Step= 5 #Step-
10 #Step 20
#Step = 50
Forang 1
Figure 3: Step skipping during sampling. The source image is from MSCOCO 26
4.3
Qualitative Evaluation
We present qualitative results comparing various image translation models. Due to the space limit, additional examples will be available in Appendix [. In Figure we present the two unpaired trans- lation tasks: Hummer-/wintee and horse >zebra. To demonstrate the image quality, translation quality, and consistency compared to the original images, we provide a full image for each test case
Our code is available at uulattps://githim www. wwt.c/ooo/scceeeeee.e http:///ooodddddooo//////oooleee groee/

---

Cycketei
Cyciebies
Dilluvion
NallTo
MisaCir
Zune
SDBdi:
EOSDE
TeeiuLive
cClppic
Controllles CCunss
OUT
QydkOAN
Summera
Figure 4: Qualitative comparison in scene/object- tasks. CycleNet produces high high-quality transla- tions with satisfactory consistency. The areas in the boxes are enlarged for detailed comparisons.
SUEBOZ ipinter (Scene level, x
horso r2ebra (Object level, 2560
Metrics
FIDJ
FID,ap
CIJP1
[PMPS]
PSNR1
SSIM
12"10
FID
FID,spd
CLIP
[.PIS]]
PSNR1
SSIMI
1.2
GAN.based MethosS
CydeGAN
133.16
18.85
22.07
0.20
16.27
0.39
3.62
77.18
27.09
28.07
0.25
18.55
0.67
1.39
CUT
180.09
23.45
24.21
0.19
20.05
0.71
1.15
45.30
21.00
29.15
0.46
13.71
0.35
2.44
Mask
bused Diffusson Methods
Impaint e Chipee Tex
246.56
79.70
21.85
0.57
12.63
0.19
2.83
187.65 6.
40.00
26.32
0.30
15:45
0.43
2.31
100.63
22.59
26.03
0.22
16.51
0.67
128.22 21
24.46
30.51
0.14
21.00 05
0.81
1.03
Misk:
Eree Diflusios Methods
ConcooNee + Canny
338.24
$3.26
21.77
0.59
6.03
0.00
13.30
397.21
77.68
23.88
0.61
7.37
0.07
3.89
ILVR
105.19
37.24
22.91
0.59
10.06
0.16
3.62
188.05
40.80
25.95
0.57
10.26
0.17
3.57
EGSDE
131.00
18.74
22.96
0.44
[7.68
0.27
1.53
97.61
27.79
27.31
0.41
18.05
0.29
1.46
SDEdi
330.98
79.70
21 21.85
0.57
12.63
0.19
2.83
60
53.21
24.17
0.66
9.75
0.11
4.01
Pie2Pis-Zemo
311.03
$1.54
0.57
14:31
0.30
44
24.37
0.67
11:18
0.19
MisaCral
106.91
52.38
20.79
0.36
16.22
0.36
3:71
333.17
68.31
21.35
0.40
16.31
0.37
1.85
y2P NullText
160.00
41.12
23.31
0.37
16.84
0.39
1.75
287.45
48.93
13.91
0.36
17.20
0.41
1.68
CycleDifrasion
243.98
$2.96
22.32
0.44
15.06
0.31
2:20
347:27
06.80
25.04
0.57
11.51
0.21
3.45
Fist PydCCclieeet
0.14
12.45
0..77
0.32
19.29
0.51
CydlNNa
82.52
17.54
23.32
0.33
22.42
0.57
81,69
38.11
28.91
0.27
20.42
0152
wopty-ccoffee (Statc lexel, 612)
nof coffeetwxyyyy (Stnt lexel, 517)
Metrics
FIDJ
FID
CLJP1
[.PISS]
PSNR1
SSIMT
12"1
FDD
FID
CLIP*
[.PISS]
PSNR1
SSIM1
1
Mast bosed Dictusion Mediods
Iapaint a Chipsez
94.14
22.96
27.12
0.29
14.1
0.65
4.81
148.11
36.18
25.95
0.33
12.82
0.37
5.32
Tex2LIVE
106.07
28.11
28.37
0.13
20.1
0.35
2.5
142.50
89
39.39
29.31
0.11
20.82
0.88
2.17
Misk-frre Dillusius Methads
SDEdi
74,08
20.61
27.75
0.38
16.82
0..61
3.32
134.87
33.38
26.04
0.15
15.85
0.67
3.48
PIP NullTest
103.97
24.53
25.67
0.14
24.92
0.83
1.38
138.13
31.19
25.65
0.11
24.31
0.83
1.46
CycleDifrision
$7.39
17.59
27.39
0.18
23.36
0.81
1.67
131.25
32.52
25.73
0.10
26.47
0.85
1.13
CycleNet
105.52
16.36
27.45
0.17
21332
0.77
1.99
95.34
18.79
27.54
0.14
21:85
0.78
1.92
Tasks
eopty-njuice (State level. 512 512)
juice-vempty (State level. 512)
Metrics
1DDapp
CLIP
LPIPS]
PSNR1
SSIMI
12K10 1
FTD
F10
CLIP
LPIPS]
PSNR1
SSIMI
12
Moodd Di on Melhish
Iqpiitt a Clipbeg
124:55
35.75
26.60
0.27
14.3
0.67
4.57
163.33 35
36.01
24.30
0.34
13.21
0.58
5.27
TeuZLIVE
116.14
$1.44
29.18
0.35
16.52
0.79
3.55
157.45
45.41
26.47
0.39
18.04
0.78
3.05
Mrak-frr Diffusion Methods
SDFdda
148.44
30.53
36.45
0.78
13.36
0.59
4.51
135.00 91
36.05
25
0.38
16.64
0.59
3.97
PIP + NollTesa
148.77
$7.74
36.28
0.33
1782
0.69
2.99
10
36.48
23.57
0.14
22.68
0.52
1.71
CycleDilaasion
139.76
33.41
25.78
0.36
23.99
0.50
191
159.39
42.80
23.16
0.15
24.15
0.78
1.71
CycleNet
79.02
23.42
27.75
0.37
20.18
0.56
227
114.33
28.79
26.17
0.17
19.78
034
237
Table 1: A quantitative comparison of various image translation models for the summer twinter, zebra, empty->coffee and puice--empty Performances are quantified in terms of FID, CLIP Score, LPIPS, PSNR, SSIM, and L2.
and enlarge the boxed areas for detailed comparisons. As presented with the qualitative examples, our methods are able to perform image manipulation with high quality like the other diffusion- methods, while preserving the structures and unrelated semantics.
In Figure S we present qualitative results for filling and emptying a cup: coffee+ and empty++juu juice. As demonstrated, image editing tasks that require physical state changes pose significant challenge to baselines, which struggle with the translation itself and/or maintaining strong consistency. CycleNet, again, is able to generate faithful and realistic images that reflect the physical state changes.
4.4 Quantitative Evaluation
We further three types of evaluation metrics respectively assess the quality of the generated image, the quality of translation, and the consistency of the images. For a detailed explanation of these evaluation metrics, we refer to Appendix C.4
Image Quality. To evaluate the quality of images, we employ two metrics: The naive Fréchet Inception Distance (FID) 12 and FID 24 with CLIP (3);;
7

---

Cycle Diffusion
P2P
Inpaint ClipSeg
Input
CycleNet
SDEdit
Tex:2LIVE
NullText
(Empty)
Cup of Coffee + Empty Cup
(Fill)
Empty Cup
+Cup of Coffee
(Empty)
Cup of Juice Empty Cup
(Fill) Empty Cup
Cup of Juice
Figure 5: Qualitative comparison coffee and empty++juice tasks.
Translation Quality. We use CLIPScore
quantify the semantic similarity of the generated
image and conditional prompt:
Translation Consistency. We measure translation consistency using four different metrics: L2 Distance, Peak Nignal-to-Noise Ratio (PSNR) (4, Structural Similarity Index Measure (SSIM) 47
and Learned Perceptual Image Patch Similarity (LPIPS) [52]
The evaluations are performed on the reserved test set. As demonstrated in the quantitative results from T we observe that some baselines display notably improved consistency in the Mani ManiCups tasks. This could possibly be attributed to the fact that considerable number of the test images were successfully translated the target domain, can seen in the qualitative results presented in Figure Overall, CycleNet exhibits competitive and comprehensive performance in generating high-quality images both global and local translation tasks, especially compared to the mask- based diffusion methods. Meanwhile, our methods ensure successful translations that fulfiil the domain specified by text prompts while maintaining an impressive consistency from the original images.
5
Analysis and Discussions
5.1 Ablation study
Recall that FastCycleNet removes the cycle consistency loss Le-ty+z from CycleNet. We further remove the invariance loss Le-sy-y to understand the role of each loss term. For better control, we initialize the sampling process with the same random noise rather than the latent noised image task with FID, CLIP score, and LPIPS. When
Table shows our ablation study on Winter--summe both losses are removed, the model can be considered as fine-tuned LDM backbone (row 4), which produces high CLIP similarity score of 24.35. This confirms that the pre-trained LDM backbone can already make qualitative translations, while the LPIPS score of 0.61 implies poor consistency from original images. When we introduced the consistency constraint (row 3), the model's LPIPS score improved significantly with drop of the CLIP score 19.89. This suggests trade-off between cycle consistency and translation quality. When we introduced the invariance constraint (row 2), the model achieved the best translation quality with fair consistency. By introducing both constraints (row 1), CycleNet strikes the best balance between consistency at a slight cost of translation quality.
Inpur
CycleNet
Invariance Only
Cunsistency Only
None
summer
twinter
FID J
CLIP t
LPIPS
CycleNet
77.16
24.15
0.15
Invariance Only
76.23
25.13
0.23
Consistency Only
84.18
0.14
None
211.26
24.35
0.61
Table 2: Ablation study over the cycle consistency loss and invariance loss.

---

5.2 Zero-shot Generalization to Out-of-Distribution Domains
CycleNet performs image manipulation with text and image conditioning, making potentially generalizable to
(OOD) domains with simple change of the textual prompt. As we demonstrate that CycleNet has remarkable capability generate faithful and high-quality images for unseen domains. These results highlight the robustness and adaptability of CycleNet to make the most out of the pre-trained LDM backbone handle unseen scenarios. This underscores the potential to apply CycleNet for various real-world applications and paves the way for future research in zero-shot learning and OOD generalization.
illustrated in Figure [
5.3 Translation Diversity
Diversity is an important feature of image translation models. As shown in Figure 6 we demonstrate that CycleNet can generate variety of images that accurately satisfy the specified translation task in the text prompts, while maintaining consistency.
Source
Target
Out-Of-Domain Generalization
fall
fire
minecraft
horse
zabra
sky
desert
Figure 6: Examples of output diversity and zero-shot generalization to out-of-domain distributions.
5.4 Limitations: Trade-off between consistency and translation
There have been concerns that cycle consistency could be too restrictive for some translation task [54) As shown in Figure 7 while CycleNet maintains strong consistency over the input image. the quartered apple failed to be translated into its orange equivalence. In GAN-based methods, local discriminators have been proposed to address this issue (56), yet it remains challenging to keep global consistency while making faithful local edits for LDM-based approaches.
Input
CycleNet
P2P + Null
Cycle Diffusion
SDEdit
Figure 7: The trade-off between consistency and translation.
6
Related Work
6.1
Conditional Image Manipulation with Diffusion Models
Building upon Diffusion Probabilistic Models 42 13,, pre-trained Diffusion Models (DMs) B8
37 have achieved state-of-the performance in image generation tasks. Text prompts are the most common protocol enable conditional image manipulation with DMs, which can be done by fine-tuning pre-trained DM 19 20 Mask-based methods have also been proposed with the help of
userprronpted/atoomaticllll generated masks [32 or augmentation layers (2). To refrain from
employing additional masks, recent work has explored attention- based alternatives [10 30 25 35

---

They first invert the source images to obtain the cross- cross-attention maps and then perform image editing with attention control. The promising performance of these methods largely dependent on the quality of attention maps, which cannot be guaranteed in images with complicated scenes and object relationships, leading to undesirable changes. Very recently, additional mmage-conditionin has been explored to perform paired image-to-image translation, using side network [5T] or an adapter [31]. In this work, we follow this line of research and seek to enable unpaired image-to-image translation with pre-trained DMs while maintaining a satisfactory level of consistency.
6.2 Unpaired Image-to-Image Translation
Image-to-ime to-image translation (121) is fundamental task in computer vision, which is concerned with learning a mapping across images of different domains. Traditional GAN-based methods [16] require instance-level paired data, which are difficult to collect in many domains. To address this limitation, the unpaired I2I setting (55 27 was introduced to transform an image from the source domain X' into one that belongs to the target domain V'. given only unpaired images from each domain. Several GAN-based methods [55] 50 27 were proposed to address this problem. In recent years, DPMs have demonstrated their superior ability to synthesize high-quality images, with several applications in I2I translation [4 5) With the availability of pre-trained DMs, SDEdit [29] changes the starting point of generation by using a noisy source image that preserves the overall structure. EGSDE (53) combines the merit of ILVR and SDEdit by introducing a pre- treitra energy function on both domains to guide the denoising process. While these methods result in leading performance multiple benchmarks, it remains an open challenge to incorporate pre-trained DMs for high-quality image generation, and at the same time, to ensure translation consistency.
6.3 Cycle Consistency in Image Translation
The idea of cycle consistency is to regularize pairs of samples by ensuring transitivity between the forward and backward translation functions [41] 33 17). In unpaired 12I translation where explicit correspondence between source and target domain images is not guaranteed, cycle consistency plays a crucial role [1 (7]) Several efforts were made to ensure cycle consistency in diffusion-based 121 translation. UNIT-DDPM 44] made an initial attempt in the unpaired 121 setting, training two DPMs and two translation functions from scratch. Cycle consistency losses are introduced in the translation functions during training to regularize the reverse processes. At inference time, the image generation does not depend on the translation functions, but only on the two DPMs in an iterative manner, leading to sub-optimal performance. Su et al. 43 proposed the DDIB framework that exact cycle consistency is possible assuming zero discretization error, which does not enforce any cycle consistency constraint itself. Cycle Diffusion 48) proposes a zero-shot approach for image translation based on Su et al. 43 's observation that a certain level of consistency could emerge from DMs, and there is no explicit treatment to encourage cycle consistency. To the best of our knowledge, CycleNet is the first to guarantee cycle consistency in unpaired image-to-iman translation using pre-trained diffusion models, with a simple trainable network and competitive performance.
7 Conclusion
The paper introduces CycleNet that incorporates the concept of cycle consistency into text-guided latent diffusion models to regularize the image translation tasks. CycleNet is practical framework for low-resource applications where only limited data and computational power are available. Through extensive experiments on unpaired 121 translation tasks at scene, object, and state levels, our empirical studies show that CycleNet is promising in consistency and quality, and can generate high-quality images for out-of-domain distributions with simple change of the textual prompt.
Future Work This paper is primarily concerned with the unpaired 121 setting, which utilizes images from unpaired domains during training for domain-specific applications. Although CycleNet demon- strates robust out-of-domain generalization, enabling strong zero-shot 121 translation capabilitiess is not our focus here. We leave to our future work to explore diffusion- based image manipulation with image conditioning and free-form language instructions, particularly in sero-shot settings.
10

---

Acknowledgements
This work was supported in part by NSF I18-1949634, NSF SES-2128623, LG Al Research, and by the Automotive Research Center the University of Michigan. The authors would like to thank Yinpei Dai and Yuexi Du for their valuable feedback, and extend their special appreciation to Michael de la Paz for granting permission to use his photograph of summer Yosemite in the teaser.
References
[1] Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Xi Yin. Spatext: Spatio-textual representation for controllable image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[2] Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text-driven layered image and video editing. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV, pages 707-723. Springer, 2022.
[3] Mingdeng Cao. Xintao Wang, Zhongang Qi. Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 22560, October 2023.
[4] Luen Chan and Peter Whiteman. Hardware-constrainee hybrid coding of video imagery. IEEE Transactions on Aerospace and Electronic Systems, (I):71-84, 1983.
[5] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models. In 202/ IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pages 14347-14356 IEEE Computer Society, 2021.
[6] Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11472-1 1481, 2022.
[7] Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion- based semantic image editing with mask guidance. The Eleventh International Conference on Learning Representations, 2023.
[8] Xiaolin Fang, Caelan Reed Garrett, Clemens Eppner, Tomás Lozano-Péree Leslie Pack Kael- bling, and Dieter Fox. Dimsam: Diffusion models as samplers for task and motion planning under partial observability. urXiv preprint arXiv:2306. 13 196, 2023.
[9] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In Neural Information Processing Systems, 2014.
[10] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or. Prompt-to-prompt image editing with cross-attention control. In The Eleventh International Conference on Learning Representations, 2022.
[11] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7514-7528, 2021.
[12] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
A hyperlink to the original photo on Flickr. All rights reserved by the artist.

---

[13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 1568440-855,, 2020.
[14] Tobias Höppe, Arash Mehrjou, Stefan Bauer, Didrik Nielsen, and Andrea Dittadi. Diffusion models for video prediction and infilling. Transactions on Machine Learning Research, 2022.
ISSN 2835-8856.
[15] Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image- to-image translation. In Proceedings of the European conference on computer vision (ECCV), pages 172-189, 2018
[16] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. image-to-image translation with conditional adversarial networks. In Proceedings the IEFE conference on computer vision and pattern recognition, pages 1125-11344 2017.
[17] Zdenek Kalal, Krystian Mikolajezyk, and Jiri Matas. backwrd enror. Automatic detection of tracking failures. In 2010 20th international conference on pattern recognition, pages 2756-2759. IEEE, 2010.
[18) Ivan Kapelyukh, Vitalis Vosylius, and Edward Johns. Dall-e-bot: Introducing web-scale diffusion models to robotics. IEEE Robotics and Automation Letters, 2023.
[19] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6007-6017, 2023.
[20] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2426-24422 2435, 2022.
[21] Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning to discover cross-domain relations with generative adversarial networks. In International conference on machine learning, pages 1857-1865. PMLR, 2017.
[22] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
[23] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont- -ust,, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International Journal of Computer Vision, 128(7):196-11981 2020.
[24] Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fréchet inception distance. The Eleventh International Conference on Learning Representations, 2023.
[25] Senmao Li, Joost van de Weijer, Taihang Hu, Fahad Shahbaz Khan, Qibin Hou, Yaxing Wang. and Jian Yang. Stylediffusion: Prompt-embedding inversion for text-based editing. arXiv preprint arXiv:2303. 15649, 2023.
[26] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[27] Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. Advances neural information processing systems, 30, 2017.
[28] Timo Lüddeekke and Alexander Ecker. Image segmentation using text and image prompts. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 7086-7096, 2022.
12

---

[29] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2021.
[30] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6038-6047, 2023.
[31] Chong Mou, Xintao Wang. Lianghin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.084. 2023.
[32] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image genera- tion and editing with text-guided diffusion models. In International Conference on Machine Learning, pages 16784-16804, PMLR, 2022.
[33] Pan Pan, Fatih Porikli, and Dan Schonfeld. Recurrent tracking using multifold consistency. In Proceedings of the Eleventh IEEE International Workshop on Performance Evaluation of Tracking and Surveillance, volume 3, 2009.
[34] Taesung Park, Alexei A Efros, Richard Zhang, and Yan Zhu. Contrastive learning for unpaired image-to-image translation. In Computer Vision-ECCV 2020: 16th European Confer- ence, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX 16, pages 319-345. Springer, 2020.
[35] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun- Zhu. Zero-shot image-to-image translation. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1-11, 2023.
[36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763.873.3663 PMLR, 2021.
[37] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint D.XXi:2204406125. 2022.
[38] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 10684-10695 2022.
[39] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo-Loper Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems, 2022.
[40] Hiroshi Sasaki, Chris G Willcocks, and Toby P Breckon. Unit-ddpm: Unpaired image translation with denoising diffusion probabilistic models. arXiv preprint 0rXiv:2104.05338 2021.
[41] Ishwar K. Sethi and Ramesh Jain. Finding trajectories of feature points in a monocular image sequence. IEEE Transactions Pattern Analysis and Machine Intelligence, PAM5-9(1):56-73. 1987. doi: 10.1109/TPAMII
TWW.TTAAAM.99...... MTT.MM..987.766777 (8777776787..
[42] Jascha Sohl- Dickssic,,,,,,, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2265. PMLR, 2015.
[43] Xuan Su, Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image-to-image translation. In International Conference on Learning Representations, 2022.
[44] Dmitrii Torbunov, Yi Huang, Haiwang Yu, Jin Huang, Shinjae Yoo, Meifeng Lin, Brett Viren, and Yihui Ren. Uvegan: Unet vision transformer cycle-consistent gan for unpaired image- to-image translation. In Proceedings of the IEFF/CVF Winter Conference on Applications of Computer Vision, pages 702-712, 2023.
13

---

[45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[46] Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, and Jiwen Lu. Drivedreamer: Towards wrrl-werld world models for autonomous driving. arXiv preprint arXiv:2309.09777, 2023.
[47] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4): 600-612, 2004.
[48] Chen Henry Wu and Fernando De la Torre. A latent space of stochastic diffusion models for zero-shot image editing and guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7378-7387, 2023.
[49] Yue Yang. Wenlin Yao, Hongming Zhang, Xiaoyang Wang. Dong Yu, and Jianshu Chen. Z.- LaVI: Zero-shot language solver fueled by visual imagination. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1186-1203, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[50] Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image-to-image translation. In Proceedings of the IEEE international conference on computer vision, pages 2849-288 2857, 2017.
[51] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[52] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[53] Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. Egsde: Unpaired to-image translation via energy-guides stochastic differential equations. Advances in Neural Information Processing Systems, 35:3609-3623, 2022.
[54] Yihao Zhao, Ruihai Wu, and Hao Dong. Unpaired image-to-image translation using adversarial consistency loss. In Computer Vision-ECCV 2020: 16th European Conference, Glaxgow, UK, August 23-28, 2020, Proceedings, Part IX 16, pages 800-815. Springer, 2020.
[55] Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-ing image translation using cycle-consistent adversarial networks, In Proceedings of the IEEE international conference computer vision, pages 2223-2232. 2017.
[56] Xianhui Zong, Zhehan Chen, and Dadong Wang. Local-cyclegan: general end-to-end network for visual enhancement in complex deep-water environment. Applied Intelligence, 51:1947- 1958, 2021.
14

---

A CycleNet Details
A.1 Algorithmatic Details
The pseudocode for training given as follows.
Algorithm 1: CycleNet Training Framework
Input: Source image Zo, source domain prompt Ca. target domain prompt cy. constants 21,2,3 repeat
Zo ~ q(20)
~~ T];
Eas Ey ~ NN(00 I):
It Vā + vI -I- ChEsi
Eg E0 Cz. Cz, zo):
zy Eœ( Cy, vo);
Eg
Jo (It VI ()Eg)) / تy
10
e Eo( cy. 10 .eeach)))
Y1 vā,yo + /1 مرEسi
12
eo Mz (Yt, Ca. Ca, vo);
Take gradient descent step on 13
Vo |All= - Ex 112 + >211= <0112 + >all= + = -E- - -
14
until converged;
A.2 Proof of Cycle Consistency Regularization
x- Reconstruction Loss
Dropping the variances, Ce-xx = Exc.es (5, - 1x
Proof.
Lx-xx
Ezoiez
|1:0
Gi Ca. xo)l
Exnnes ||:00
It - v1 aico( C2
Exn,s
||z0
[Va + /1 a,E, v1 Ca
Exoits
1|:0 Po - v1 ai Cn - e0(x,, Co, 2o) Co)
1 αt 11eo(0t, Ca, zo) Ex l
y- o-RReonstruction Loss
Dropping the variances, Cy-ty = Erasensy (1E Cy. Fo) -1,
Proof.
Ly-oy = Eugg,ene
11oo
Gly,, Cy. 0o)1
Ezg.enis
11yo
Ut vI arsolye, Cy. Bb)l///..cobb//aaalla/
Exg.Enr
11ã0
Vāryu + -I- uicy vI C5 Cy. Bo)] Varll?
Exg.Errs
11
H6
vI 2e [Ey cy.
Cy. FoD] Va
[| cy. Fo) - =
α
15

---

Cycle Consistency Loss
Dropping the variances, C2-ty-z
E Exo. C2. 2o) +
Eo (T( To) - Ex - Eyl12.
Proof.
LE--y--
Exn.
| Glyi, Cz: 2o
Exp.ea.y
I y vI CeEolye Cz. 2o.
Expres
| - [Vānūo + vIV /1E v1 - Ca. Co.
Zo Yo 1 ai
[Eo(y)) C2, 2o) - sy 2
Ex
/
| -- at
(aa Cr. x0) Eyg |/
v0 G(a,, Cy. 20)
[ Cy. mo) Eo co(Ue: Cr, vo) - [g] <0]1[
F0 0t/Von +
04
40 (Varso + vI a1s,) //v + +1 -1 [ea Cy. 20) + sw Ea( Cz; 20) - Eyy
2
Emo.e2:5x
Öt
- at
[co(Ve, Cx, 20) + 60(01, Cy. mo) - Ex - -2y] 1
2 - Pr +
āt
EExo,e2:fx
|100 To) + Eol mn) - Ex -By
04
Invariance Loss
Dropping the variances, La--y-y = Bergce |100(1,, Cy. vo) c0 Co. 1o
Proof.
Lxz-g-t = Enoos
|1ã0 G( Cy. To)113
Exoies
|[([z,, Sy, 2o) G( Cy. 11
Exorea
||[58
āreg (x Cy. 20)] / / [es vI azeo(Er, Cyp Do) /o)o/v//ll
V= 1 - at
Exowes
(1e, 20) - eo (ve, Cy. Cy. To)l
αt
A.3 Proof of Self Regularization
Let Eg denote the denoising autoencoder of the pre-trained text-guided LDM backbone with generation function G*. Note that x, = 5(x) Eg).
Proof.
LLDM = -Ezne, P.
-
Erooez
- P(G( Cy. 20), -1
= -Exo.e
(p cyl + (Z cy), - P(G((z (y, cul -1
< EErosss
1- (r,, + Eroccs
16

---

Note that Ex
1 P-G cy), c/] == LLDM the translation error likelihood of the pre-trained
LDM backbone.
According to the domain smoothness in Assumption 1
we have:
LLDM I E Exa.e
(z."
P(C zo), + FLDM
L
G (1x, (y) ( Cy: m0)
+RDDM
-Ergse
L
pf v1 arg cy)) Vv:== - 1 v1 Crso Cy, /vā + ELDM -
1 at W
-Eagsfx
|1 Cy, 2o) - (2e BİDMM -1
04
L
B Scientific Artifacts and Licenses
B.1 The ManiCups Dataset
We introduce Mani Cups, a dataset that tasks image editing models to manipulate cups by filling or emptying liquid to/from containers, curated from human-annotated bounding boxes in publicly available datasets and Bing Image Search (under Share license for training and Modify for test set). The dataset consists of total of 5714 images of empty cups and cups of coffee, juice, milk, and water. We describe our three-stage data collection pipeline in the following paragraphs.
In the Image Collection stage, we gather raw images of interest from MSCOCO [26.. Open Im- ages [23], as well as Bing Image Search API. For the MSCOCO 2017 dataset, we specifically searched for images containing at least one object from the categories of "bottle," "cup," and "wine glass." Regarding the Open Images dataset, our search focused on images with least one object falling under the categories of "coffee," " "coffee cup," " "mug." "juice," "milk "milk," and "wine glass." To conduct the Bing Image Search, we utilized API v7 and employed queries with the formats "empty <container>" and "<container>> of <liquid>> The <container> category encompasses cups, glasses, and mugs, while the <liquid> category includes coffee, juice, water, and milk. We obtained approximately 30,000 image candidates after this step.
During the Image Extraction stage, we extract regions of interest from the candidate images and resize them to standardized size of 512x512 pixels. Specifically, for subsets obtained from MSCOCO and Open Images, we extract the bounding boxes with labels of interest. To ensure a comprehensive representation, each bounding box is extended by 5% on each side, and the width is adjusted to create a square window. cases where the square window exceeds the image boundaries, we shift the window inwards. If, due to resizing shifting, a square window cannot fit within the image, we utilize the cv2. BORDER, REPLICATE function to replicate the edge pixels. All bounding boxes with an initial size less than 128 128 are discarded. and the remaining images are resized to a standardized size of 512 x 512 pixels. The same approach is applied to images obtained from the Bing Search API, utilizing cv2. BORDER RENDDERTEEEIIATEE for edge pixel replication. Following this step, we obtained approximately 20,000 extracted and resized images.
In the Filtering and Labeling stage, our focus is on controlling the quality of images and assigning appropriate labels. Our filtering process begins by identifying and excluding replicated images using the L2 distance metric. Subsequently, we leverage the power of CLIP to detect and remove images containing human faces, as well as cups with a front-facing perspective. Additionally, we use CLIP to classify the remaining images into their respective domains. To ensure the accuracy of the dataset, three human annotators thoroughly review the collected images, verifying that the images portray a top-down view of container and assigning the appropriate labels to the respective domains. The resulting Mani Cups dataset contains domains, including 3 abundant domains (empty, coffee, juice) with more than 1K images in each category and 2 low-resource domains (water, milk) with less than 1K images to facilitate research and analysis in data-efficient learning.
To our knowledge, ManiCups is one of the first datasets targeted to the physical state changes of objects, other than stylistic transfers or type changes objects. The ability to generate consistent state changes based on manipulation is fundamental for future coherent video prediction [14] as well as understanding and planning for physical agents (49 J. We believe that Mani Cups is a valuable resource to the community.
17

---

B.2 Dataset Statistics
We present the statisties of the Mani Mani Table 3 and other scene/object level datasets Table
Domains
Train
Test
Total
Domains
Train
Test
Total
Empty Cup
1256
160
1416
Summer
1231
309
1540
Cup of Coffee
15500
100
1650
Winter
962
238
1200
Cup of Juice
1754
100
1854
Horse
1067
118
1194
Cup of Water
801
50
851
Zebra
1334
140
1474
Cup of Milk
353
50
403
Apple
995
256
1251
Total
5714
460
6174
Orange
1019
248
1267
Table The statistics of the ManiCups dataset, with abundant domains and 2 low-
The statistics of the Yosemite summer-winter, horse<+zebra, and applet>orange datasets.
Table 4:
resource domains.
B.3 Licenses of Scientific Artifacts
We present a complete list of references and licenses in Table all the scientific artifacts we used in this work, including data processing tools, datasets, software source code, and pre-trained weights.
Data Sources
URL
License
Fnf (Tool)
Link
Apache v2.0
Sunmer2Winter Yosemite
Link
ImageNet, See Link
Horse27ebra
Link
ImageNet, See
Link
Apple2Orange
Link
ImageNet, See Link CC BY 4.0
MSCOCO 2017
Link
Open Images v7
Link
Apache v2.0 Share (training) & Modiiy (test)
Bing Image Search API v7
Link
Software Code
URL
License
Stable Diffusion vl
Link
CreativeML Open RAIL-M CreativeML Open RAIL++M
Stable Diffusion v2
Link
ControlNet
Link
Apache v2.0
CUT
Link
Mixed, See Link
Prompt2Prompt + Null'ext
Link
Apache v2.0
Stable Diffusion Inpainting
Link
MIT license
Text2LIVE
Link
MIT license
Stable Diffusion SDEdit
Link
Apache v2.0
Cycle diffusion
Link
Apache v2.0
ILVR
Link
MIT license
EGSDE
Link
N/A
P2-weighting
Link
MIT license
Pix2pix-zero
Link
MIT license
Masactrl
Link
Apache v2.0
Metric Implementations
URL
License
FID
Link
Apache v2.0
FCD
Link
MIT license
CLIP Score
Link
Apache v2.0
PSNR & SSIM
Link
BSD-3-Clause
L2 Distance
Link
BSD-3-Clause
Table 5: License information for the scientific artifacts used.
18

---

C Experiment Details
C.1 Computational Resources
Tllustaslss the training performance of CycleNet and FastCycleNet on single NVIDIA A40 GPU under 256 256 with batch size of 4. The table provides information the training speed in seconds per iteration and the memory usage in gigabytes for both models. FastCycleNet exhibits faster training speed of 1.1 seconds per iteration while consuming 24.5 GB of memory. On the other hand, CycleNet demonstrates slightly slower training speed of 1.8 seconds per iteration, and it requires 27.9 .9 GB of memory.
Training
Train Speed (sec/iteration)
Mem Use (GB)
FastCycleNet CycleNet
1.1
24.5
1.8
27.9
Table 6: Speed of CycleNet and FastCycleNet
C.2 Hyper-parameter Decisions
We include the major hyper-panee parameter tuning decisions for reproducibility purposes. the training of CycleNet, the weights of our three loss functions respectively set Àι 1, A2 0.1, and A3 = 0.01. We train the model for 50k steps. Following (29), we initialize the sampling process with the latent noised input image 24, collected using Equation T A standard 50-step sampling is applied
at inference time with t 100. Our configuration is as follows:
nodel:
parans:
Aineer_start: 0.88685
nun som_ttinestes_conde
tinesteps: 1809
image size: 64
channels:
cond stage, trainable: false
noniton: val/loss sisppplppeeem_
scale_factor: 0.18215
False
only_niddcontrol: False recon_weight: disc weight: 0.1
#lanbda1
#laxbda2
cycle_weight: 0.01
Rlanbda3
disc node:
weight: 0.1
For more details, please refer to the supplementary codes.
C.3 Baseline Implementations
CycleGAN:
We used some the results provided in 34
CUT:
We used the official codef] provided by the authors.
Inpainting ClipSeg:
modified from the g
by the community.
Text2LIVE: We used the official code code" provided by the authors.
ILVR: We first pre-train the diffusion model using P2-weighting
6 and then generated output
using the official code" provided by the authors.
https: http:///waatataaooooooo///ssslls
www.dffffuussoonnaaniiiiinnn inpainting
http:://gihhiib.ccm//mmmm//////p
http:://gtthb.cccm/ycco111111111
19

---

pre-train train the diffusion model using P2- P2-weighting [6, and then generated output using the official code[provided" by the authors.
EGSDE:
We first pre-
SDEdit: We used the community implementation SDEdit based on stable diffusion.
CycleDiffusion: We modified from the official gradic provided by the authors.
Prompt2Prompt Null'eet: We used the official code provided by the authors.
Masactrl: We used the official code provided by the authors.
Pix2pix-zero: We used the official code
13 provided by the authors, and we generated the sum- mer2winter direction assets following the scripts provided by the authors.
C.4
Evaluation Metries Explained
Image Quality
To evaluate the quality of images, we employ two metrics.
Fréchet Inception Distance (FID) 12
widely used metric image generation tasks. lower FID score indicates better image quality and more realistic samples.
FIDetip [24] combines the FID metric with features extracted with a CLIP [36] encoder, providing
better assessment of image quality. Similar to FID, lower FiD score represents better image quality.
Translation Quality To measure to what extent the translation successful, use the CLIP Score [T i.e., the CLIP model [36] to obtain the latent representations of images and prompts, and then calculate the cosine similarity between them. higher CLIP score indicates a stronger similarity between the generated image and the text prompt, thus better translation.
Translation Consistency We measure translation consistency using four different metrics.
1.2 Distance is measure of the Euclidean distance between two images. A lower 1.2 distance indicates higher similarity and better translation consistency.
Peak Signal-to-Noise Ratio (PSNR) 4 measures the ratio between the maximum possible power of signal and the power of corrupting noise. A higher PSNR score indicates better translation consistency.
Structural Similarity Index Measure (SSIM) 47 a metric used to compare the structural sim- ilarity between two images. A higher SSIM score suggests higher similarity and better translation consistency.
Learned Perceptual Image Patch Similarity (LPIPS) 52 is a comprehensive evaluation metric for the perceptual similarity between two images. A lower LPIPS score indicates higher perceptual similarity.
D Broader Impact
While CycleNet holds great promise, it is essential to address potential broader impacts, including ethical, legal, and societal considerations. One significant concern is copyright infringement. As an image translation model, CycleNet can potentially be used to create derived works from artists' original images, raising the potential for copyright violations. To safeguard the rights content creators and uphold the integrity of the creative economy, it is imperative to prioritize careful measures and diligently adhere to licensing requirements. Another critical aspect to consider the potential for fabricated images contribute deception and security threats. If misused accessed by malicious actors, the ability to generate realistic fake images could facilitate misinformation campaigns, fraud, and even identity theft. This underscores the need for responsible deployment and robust security measures to mitigate such risks. CycleNet leverages pre-trained latent diffusion models, which may encode biases that lead to fairness issues. It is worth noting that the proposed method is currently
20

---

purely algorithmic. devoid of pre- pre-training on web-scale datasets itself. By acknowledging and actively addressing these broader impacts, we can work towards hameessing the potential of CycleNet while prioritizing ethics, legality, and societal well-being.
E
Addendum to Results
E.1
Choice of Pre-trained LDM Backbone
For all experiments in the main paper, we use Stable Diffusion 2. 2 as the pre-trained LDM backbone. We additionally attach a quantitative comparison of CycleNet using Stable Diffusion 1.5 15 indicates marginal differences in practice.
which
Tasks
-vinter (512 512)
bozee-izebra (512
512)
Metrics
PID
FiD
CLIP
LPIPS,
PSNRT
SSIMI
12"
PID]
F1D
CLIP
LPIPS)
PSNR1
SSIM
1,211²
SD
83.45
15.21
23.56
0.17
25.40
0.74
1.39
73.34
25.42
27.77
0:25
20.96
0.64
0.79
79,79
15.39
24.12
0.15
25,88
0.89
1,33
76.85
24.78
25.27
0.08
26.21
0.74
0.59
Table 7: A quantitative comparison using Stable Diffusion 1.5 and 2.1.
E.2
Image Resolution
We notice that the Stable Diffusion 38 backbone is pre-trained in multiple stages, initially a reso- lution of 256x256 and followed by another stage at resolution of 512x512 or beyond. This could potentially lead to the under--erformance of zero-shot diffusion-based methods on summer-iwinte and horse->zebra, which are at a resolution of 256x 256. We repeat the experiment on these two tasks at a generation resolution of 512x512 and report the results in Table 8 It's important to ac- knowledge that this particular setting presents an unfair comparison with considerable challenges for our methods, primarily because the training images are set at a resolution of x 256,256, yet our model is expected to adapt to a higher resolution. Still, we observe a competitive performance of our models, especially in summer->u nummer-winter. The diminished effectiveness of our method the horse- task can be attributed to the fact that the zebra patterns, initially acquired at resolution of 256 256, lose realism and become overly dense when scaled up to 512 (see Example Figure This limitation can potentially be addressed by scaling to images with multiple resolutions.
Tasks
SUEEN Ousnt-vvittr -vinter (Gilbbl. 012 012)
Serve Serve-izwbew (Local, 512 x 512)
Metries
FID.ap.
CLIP
LPIPS|
PSNRT
SSIMI
1.2 10"
FlD
CLIP
LPIPS|
PSNRT
SSIM1
12
Mask Diffusoon Methods
Impaint Chnbeg
168.43
46.43
26.57
0.65
9,01
0.15
14
26.05
28.71
0.29
16.39
0.60
1.95
$6.12
18.30
25.98
0.27
16.83
0.68
1.67
103.14
22.71
31.55
0.16
20.96
0.81
2.08
Misk Diflusion Methods
ComcoNNet
179.17
44.71
25.01
0.61
112.65
68.31
27.22
0.6
8.52
0.06
ILVR
101.26 26
27.72
21.71
0.58
9.57
0.18
191.92
45.44
23.90
056
10.06
0.24
1.27
EGSDE
108.71
35.89
21.50
0.37
19.84
0.39
2.40
161.26 26126
39.94
25.52
0.36
20.67
0.40
2.15
SDCdi
90.51
21.23
23.26
0.30
18.59
0.43
63.04
22.65
27.97
0.33
18449
0.44
2.96
Pre2Fis-7em
88.79
83.78
23.63
0.29
20.91
0.62
2.15
115:52
79.74
27.42
0.37
18:18
0.57
3,14
MasaCCrl
0.37
14.66
0.43
0.41
PIP NullTest
92.65
24.82
0.24
20.19
105.85
26.57
0.21
21.45
CycleDiffision
84.52
24.40
0.24
29.89
0.29
0.61
Fiot FyteNee
78.43
14.99
34.33
0.36
25.81
11.66
1.24
72.68
25.34
24.42
0.13
26.74
0.72
1.12
CycleNet
79.79
15.30
34.12
0.35
25.88
0.60
1.23
76.83
34.78
25.27
0
26.21
0.74
0.59
Table 8: A quantitative comparison of various image translation models for the summer summer-wintee twinter and
at 512x512.
E.3 Sudden Convergence
As shown in Figure CycleNet can translate an input image summer to winter at the beginning of training with consistency observed. Similar to ControlNet [51]. CycleNet also demonstrates the sudden convergence phenomenon, which usually happens around 3k to Sk iterations of training.
E.4 Addtional Quantitative Results
In Table 9 we present the complete numerical performance of the state-changing tasks on Mani Cups. In general, we found that emptying a cup more challenging image editing task for most of
21

---

1000
2000
3000
4000
10000
50000
Input
Figure 8: Sudden convergence appears at around 3k 8k in our experiments.
the existing computation models, compared to filling a cup with liquid. CycleNet shows superior performance, especially in domains with abundant training samples (coffee and juice). We also observe that the performance is marginally less competitive in low-resource domains (milk and water). We leave it to future work explore more data-efficient models that fine-tune pre- t-airrindd latent diffusion models with cycle consistency granted.
Taoks
eeoty- 6512
512)
coffee coffee-rexpty (522 x 522)
Menrcs
FIDL
FiD
CLIFT
LPIPS)
PSNRT
SSIMI
L2*00"
HID,
FID,
CLIPT
LPIPS)
PSNK"
SSIM
L2-31
Misk-based Dilluiio Methods
Inpaist CligSog
94.14
22.96
27.12
0.29
14.1
0.65
4,51
145.
36.15
25.95
0.33
12.82
0.57
5,52
Tes221
106.07
28.11
28.37
0.13
20.4
0.85
2.3
142.89
30.89
29.31
0.11
20.82
0.88
2,17
Mask-Fron Diflision Methods
SDEdi
74.08
20.61
21.75
0.28
16.82
0.61
3.32
134.87
33.56
26.04
15.83
0,67
3.48
P2P NollText
100.97
24.53
25.67
0.14
24.92
0.83
1.38
138.13
31.19
25.65
24.31
0.81
1,46
CycleDiffusion
87.39
17.59
27.39
0.18
23.36
0.81
1.67
131.25
30.52
25733
0.10
26.47
0.85
1,13
CycleNer
105.52
16.36
27.45
0.17
21.32
0.77
1.99
95.24
28.79
27.54
21.85
0,78
Tasks
capty-11use0 1512
512)
jxxpe- (502
512)
FIDJ
FIDa,4
CLIP
LPIPS)
PSNRT
SSIMI
HDD
RDDual
CLIP + LPIPS)
PSNKT
SSIM;
L2
Dittusion
ClipSez
0.58
116.14
31.44
29.18
0.79
26.47
0.78
Mask-fize
145.64
NolfTest
148.77
37.74
26.28
23.57
0.82
CycleDiffusion
139.76
33:41
25.78
0.16
23.99
0.80
1.91
159.39
42.89
23.16
0.15
24.18
0.78
1.71
CycleNer
79.02
23.42
27.75
0.17
0.76
26.17
19.78
wepty-im mw Reweree 512
513)
xi1k-wwppyy (1
Resmrren, 512 812)
Metrics
FIDL
FiDaap
CLIP
LPIPS]
PSNRT
SSIMI
L2
FIDE
PIDagl
CLIP 7 LPIPS)
PSNK
SSIMI
L2
Misk: based Diffusion Metheds
Inpoine ClipSez
138,81
30.13
27.11
0.28
14.57
0.06
4.47
183 92
41.37
26.27
0.35
13.09
0.57
5.59
TesE2LIVE
130.73
30.69
30.75
0.13
20.11
0.35
2.37
166.72
49.97
28.12
0.17
19.15
0.83
2.65
Mask free Diffusiom Methods
SDEda
125.75
28.57
28.38
0.38
16.91
0.61
3.39
142.41
39.09
26,88
0.36
16.95
0.42
3.35
F2P NolfText
125.54
29.57
25.51
0.13
25.15
0.34
1.34
147.65
40.16
24.99
0.12
22.45
0,83
1.76
CyeleDiffision
130.24
27.27
26.71
0.16
24.21
0.81
1.75
151.01
38.14
25.36
0.15
24.19
0.81
1.96
CycleNer
97.07
25,84
27.61
0.19
22.58
0.77
1.74
121.99
32.21
26.29
0.15
21.65
0.79
1.93
Tanks
Mapty rvatez (Low Resoures, 502
012)
Nater
(Low Resousce 012
512)
FIDL
FID..4
CLIP
LMIPS)
PSNRE
12"00"
FIDL
FIDawl
LPIPS)
SSIM)
Maskobusel
Methods
ClipSez
135.67
32.87
26.48
0.63
Tes
133.04
30.25
Diftusion
25.08
132.89
CycleDitusion
146.11
31.40
24.43
0.74
157.81
35.95
CycleNer
Table 9: The complete quantitative comparison of the state change tasks on ManiCups.
E,5 Additional Qualitative Results
We present additional qualitative examples in Figure and 10
E.6
Additional High-resolution Qualitative Results
We present additional high-resolution examples Figure
E.7 Additional Out-of-Domain Examples
We present additional OOD examples in Figure 12
22

---

cup of coffee
cup of juice
cup of milk
cup of water
Empty Cup >
Figure 9: Additional qualitative results using CycleNet on Manicups dataset.
23

---

Sattn Wite
Figure 10: Additional qualitative comparison our method with other baselines.
24

---

Input
CycleNet
P2P NullText
Cycle Diffusion
SDEdit
Winter > Summer
Apple 2 Orange
Zebra
Horse
Figure 11: Additional high-resolution examples from Yosemite twinter HD 15 and
MSCOCO (26].
25

---

Source
Target
Out-Of-Domain Generalization
summer r
winter
spring
sunset
desert
zebra
horse
bear
robot
arctic
orange
apple
basketball
moon
CD
apple
orange
baseball
glass
christmas
Figure 12: Additional examples of output diversity in the target domains and zero-shot generalization out-of out-of-domain distributions.
26


Reviews:
[{"review_id": "u4jKsO2mwu", "reviewer": "Reviewer_rhuF", "summary": "This paper introduces cycle consistency in diffusion models to achieve regularization in image manipulation. It allows out of domain image generation with text prompt modification, and can be trained with very little data (~2K) with minimal computational requirements (1GPU). When we apply pre-trained DM models on I2I tasks, it is difficult or unpredictable to control the mask and attention maps to achieve the desired results. Cyclic consistency regularization allows us to address this. Experiments are conducted on many datasets such as Yosemiti summer-winter, horse-zebra, etc. On standard metrics such as CLIP, FID, and LPIPS, the proposed method outperforms other baselines such as CycleGAN, TEX2LIVE, etc. ", "questions": "Overall, the paper looks at an important problem and brings cyclic consistency to diffusion models. Please see my concerns above. ", "limitations": "No concerns. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) There has not been much regularization in diffusion models except for guidance driven by the conditional text. It is nice to see the celebrated Cycle-Consistency loss developed for diffusion models. \n\n2) The paper clearly motivates and states the problem. It is nice to see that the proposed method nicely inherits the controlnet architecture to incorporate the cycle-consistency loss function during training. \n", "weaknesses": "1) While enforcing cyclic consistency we are feeding the actual input image as the conditional input. This seems a bit strong and it appears that the diffusion model may be overfitting to the conditional input. \n\n2) It is not clear how one obtains the negative prompt c^{-1}. The method takes a prompt c_{text}=(c^{+}, c^{-}), where c^{+} is the prompt that drives the diffusion process towards the images that are associated with it, and the negative prompt c^{-} drives the process away from these images. The paper only mentions this once without sufficient details on how this is done. \n\n3) The proposed method heavily relies on controlNet and uses the original image to impose the Cyclic-Consistency loss. It may be worth just testing with standard ControlNet with canny edge-map as conditional input and appropriate text prompt. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our novelty (“there has not been much regularization in diffusion models”) and contributions (out-of-domain generalization, minimal data, and compute). We appreciate their constructive feedback on our efforts.\n\n### Question 1\n> Q1: While enforcing cyclic consistency we are feeding the actual input image as the conditional input. This seems a bit strong and it appears that the diffusion model may be overfitting to the conditional input.\n\n**Response to Q1**: We first note that our main focus and contribution is to enable consistency in text-guided manipulation with pre-trained diffusion models (DMs). While pre-trained DMs are widely known for their unprecedented capability in high-quality text-to-image synthesis, pixel-level consistency remains to be challenging when there exists image conditioning, as can be shown in the constantly changing backgrounds in image and video synthesis tasks.\n\nWe acknowledge this trade-off between **translation consistency** and **translation quality** in image translation [52], as has been discussed in Section 5.4 (Limitations). While we are aware of the current limitation, we firmly recognize the significance of addressing this matter in future research endeavors. In this context, methodologies such as employing local discriminators [54] hold great promise as inspiring solutions to tackle this concern in the future, which lies beyond the scope of this work.\n\n### Question 2\n> Q2: It is not clear how one obtains the negative prompt c^{-1}. The method takes a prompt c_{text}=(c^{+}, c^{-}), where c^{+} is the prompt that drives the diffusion process towards the images that are associated with it, and the negative prompt c^{-} drives the process away from these images. The paper only mentions this once without sufficient details on how this is done.\n\n**Response to Q2**: Thank you for raising this clarity issue! We hope the following paragraphs can ensure there is no misunderstanding regarding the **source**, **implementation** and **mechanisms** of conditional (positive) and unconditional (negative) prompts.\n\n- **Regarding the source**: We kindly point to lines 151-155 where we described how we obtain conditional and unconditional prompts. In the context of unpaired I2I translation, the conditional prompt is a piece of textual description of the target domain, and the unconditioned prompt is that of the source domain. For example, to translate a summer image to winter, we can simply use a conditional prompt “summer” and an unconditional prompt “winter”. These prompts are pre-specified during training (see lines 174-179) and are provided during inference time. We will be sure to be clearer about this.\n- **Regarding the implementation**: We encode these prompts using a CLIP encoder similar to prior work (lines 156-157).\n- **Regarding the mechanism**: Negative prompts are a common interface in pre-trained DMs to remove undesirable semantic features in the unconditional sampling. We point to this tutorial [A1] for illustrative explanations. In CycleNet, we encode the conditional prompt in the frozen SD backbone. The unconditional prompt and the image condition are encoded in the adapter, so that the adapter learns to attend to regions that need to be modified. \n\n[A1] How does negative prompt work? https://stable-diffusion-art.com/how-negative-prompt-work/\n\n### Question 3\n> Q3: The proposed method heavily relies on controlNet and uses the original image to impose the Cyclic-Consistency loss. It may be worth just testing with standard ControlNet with canny edge-map as conditional input and appropriate text prompt.\n\n**Response to Q3**: We thank the reviewer for pointing out this valid baseline to consider. We hereby provide the experimental results as requested, quantitatively in the table below and qualitatively attached in the PDF.\n\n- For Summer-to-Winter:\n\n|             | **FID** | **FID CLIP** | **CLIP Sim** | **LPIPS** | **PSNR** | **SSIM** | **L2** |\n|-----------|-----------|-------------------|-------------------|---------------|---------------|--------------|----------|\n|ControlNet+Canny| 338.24 (+258.14) | 83.26 (+69.12)  | 21.77 (-3.35) | 0.59 (+0.46)  | 6.05 (-16.4) | 0.09 (-0.48) | 11.3 (+10.4) |\n\n- For Horse-to-Zebra:\n\n|             | **FID** | **FID CLIP** | **CLIP Sim** | **LPIPS** | **PSNR** | **SSIM** | **L2** |\n|-----------|-----------|-------------------|-------------------|---------------|---------------|--------------|----------|\n|ControlNet+Canny| 397.71 (+316.42) | 77.68 (+51.65)  | 23.88 (-4.29) | 0.61 (+0.34)  | 7.37 (-13.46) | 0.07 (-0.45) | 4.89 (+3.75) |\n\nAs can be seen from the results, standard ControlNet plus canny edge-map does not guarantee faithful and qualitative translation. This method might suffer from a significant loss of semantic information in the canny edge-maps, which only present the structural information. \n"}, {"review_id": "WhPMTW34WS", "reviewer": "Reviewer_Tt5a", "summary": "This paper presents a method that tackles the challenge of consistent image-to-image (I2I) translation using diffusion models (DMs) and unpaired image conditioning and text prompts. The proposed approach incorporates Cycle Consistency Regularization to ensure cycle consistency and Self-Regularization to generate images that align with the target domain. Furthermore, the paper introduces a multi-domain I2I translation dataset that encompasses object state changes.", "questions": "Please refer to the weaknesses part.", "limitations": "Please refer to the weaknesses part.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "a)\tFrom the visual results, the proposed CycleNet can well preserving the structures of the input image.\n\nb)\tThe author introduces a multi-domain I2I translation dataset for object state changes.", "weaknesses": "a)\tThe experimental results are quite strange. \n\ni.\tIn Figure 2, Figure 3, Figure 5, and many images in the supplementary materials, The results of CycleNet show severe hue shifting. This is a serious issue, and I suggest the author analyze and address this problem.\n\nb)\tThe author's comparison methods are not comprehensive. Many editing methods based on diffusion have been proposed, such as pix2pix-zero, masactrl, etc. I suggest the author compare these methods.\n\nc)\tAlthough CycleNet aims to handle unpaired I2I translation tasks, it still requires paired datasets for training. So, how well does CycleNet generalize to unseen domains (such as variations of other unseen species, etc.)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our dataset contributions and our model’s capability to preserve the structures of input images. We are happy that much of the constructive feedback has been integrated into our paper!\n\n### Question 1\n> Q1: The experimental results are quite strange. In Figure 2, Figure 3, Figure 5, and many images in the supplementary materials, The results of CycleNet show severe hue shifting. This is a serious issue, and I suggest the author analyze and address this problem.\n\n**Response to Q1**: We thank the reviewer for pointing out this hue-shifting issue, and kindly redirect the reviewer to **General response 1** in the General Response for our solution and updated results. In short, we address this with a simple but effective solution at inference time, and the updated results are as follows.\n\nPreviously, when we are sampling the image at inference time, we start from a **random noise** $\\epsilon$. In our updated method, we initialize the process with the **latent noised input image** $z_t$, collected using equation (1). This is in fact a commonly used inference time trick, which is also adopted in some of the baselines [28, 46] to improve consistency in image translation.\n\nOur updated experimental results show that this issue of **hue shifting can be addressed with minimal trade-off in image quality and translation quality**. We sincerely hope that the reviewer can re-evaluate our work.\n\n### Question 2\n> Q2: The author's comparison methods are not comprehensive. Many editing methods based on diffusion have been proposed, such as pix2pix-zero, masactrl, etc. I suggest the author compare these methods.\n\n**Response to Q2**: We thank the reviewer for pointing out these concurrent work to us. We will for sure give a detailed discussion and evaluation in the next iteration. We hereby provide the experimental results as requested, quantitatively in the table below and qualitatively attached in the PDF.\n\n- For Summer-to-Winter:\n\n|                    | **FID** | **FID CLIP** | **CLIP Sim** | **LPIPS ** | **PSNR** | **SSIM** | **L2** |\n|-----------------|-----------|-------------------|-------------------|---------------|---------------|--------------|----------|\n| pix2pix-zero| 311.03 (+231.02) | 81.54 (+67.4)  | 22.03 (-3.09) | 0.57 (+0.44)  | 14.31 (-8.14) | 0.32 (-0.24) | 5.08 (+4.18) |\n|    masactrl   | 106.91 (+26.9) | 52.38 (+38.24)  | 20.79 (-4.33) | 0.36 (+0.23)  | 16.22 (-6.23) | 0.36 (-0.21) | 3.71 (+2.81) |\n\n- For Horse-to-Zebra:\n\n|                    | **FID** | **FID CLIP** | **CLIP Sim** | **LPIPS ** | **PSNR** | **SSIM** | **L2** |\n|-----------------|-----------|-------------------|-------------------|---------------|---------------|--------------|----------|\n| pix2pix-zero| 377.44 (+296.69) | 86.21 (+60.18)  | 24.37 (-4.54) | 0.67 (+0.4)  | 11.18 (-9.65) | 0.19 (-0.33) | 3.85 (+2.71) |\n|    masactrl   | 333.17 (+252.42) | 68.31 (+42.28)  | 21.15 (-7.76) | 0.40 (+0.13)  | 16.31 (-4.52) | 0.37 (-0.15) | 1.83 (+0.69) |\n\nAs can be seen from the results, CycleNet is able to outperform these newly proposed baselines by a large margin over all metrics and dimensions.\n\n### Question 3\n> Q3: Although CycleNet aims to handle unpaired I2I translation tasks, it still requires paired datasets for training. So, how well does CycleNet generalize to unseen domains (such as variations of other unseen species, etc.)?\n\n**Response to Q3**: We are afraid that there might be a factual misunderstanding. We hope the following paragraphs can ensure there is no misunderstanding regarding the **setting** or **performance** on the generalizability of CycleNet to unseen domains.\n\n- **Regarding the setting**: To the best of our understanding, paired datasets typically refer to datasets containing image pairs at the **instance-level**, representing images before and after translation [26, 53]. We first kindly emphasize that CycleNet does **NOT** require such paired datasets for training, while ControlNet does. \n- **Regarding the performance**: However, if the reviewer's remark pertains to **domain-level** pairs, we would also love to kindly remind the reviewer of our experiments of the CycleNet’s generalization ability in Section 5.2 in the main paper. Figure 5 shows that CycleNet can leverage the pre-trained diffusion backbone, and generalize well to unseen domains with a simple change of the textual prompt.\n"}, {"review_id": "qEBzdbeguG", "reviewer": "Reviewer_aEiE", "summary": "This work aims at addressing the task of unpaired I2I translation with pre-trained diffusion models. Inspired by CycleGAN, authors incorporate cycle consistency into diffusion model to regularize process of image translation and proposed CycleNet. In addition it also contributes a multi-domain I2I translation dataset with object state change. Extensive experiments demonstrate that the proposed method can produce high-quality translation results with good consistency and is able to generalize to out-of-domain generation tasks by changing text prompts.", "questions": "See weakness.", "limitations": "Authors have clearly showed failure cases of the proposed method in the main text. But they do not address potential negative socital impact of this work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It is interesting and reasonable to introduce cycle consistency into diffusion models for training unpaired I2I translation models.\n2. A new multi-domain I2I translation dataset for object state change with text prompt is collected.\n3. Extensive experiments demonstrate the effectiveness of the proposed method.\n4. Capbility to generalize to out-of-domain data by simply changing text prompt.", "weaknesses": "1. I wonder how CycleNet compares with CycleGAN. More comparison between CylceNet and CycleGAN should be provided. If their performance is similar, then why use a diffusion-based method instead of GAN-based for unpaired I2I translation task?\n2. Authors are suggested to experiment with more diffusion models such as T2I Adapter to show its generalization on models.\n3. Any explanation on why FastCycleNet perform better in several metrics? The coffee-to-empty translation result does not seem good, not only changing the color but also failing to become empty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are happy that the reviewer found it “interesting and reasonable to introduce cycle consistency into diffusion models”. We thank the reviewer for recognizing our novelty, our “extensive experiment”, and our contributions (out-of-domain generalization, new multi-domain dataset). We appreciate their constructive feedback on our efforts.\n\n### Question 1\n> Q1.1: I wonder how CycleNet compares with CycleGAN. More comparison between CylceNet and CycleGAN should be provided.\n\n**Response to Q1.1**: We thank the reviewer for reminding us of this missing baseline. The qualitative comparisons to CycleGAN are already provided in the main paper. We apologize for missing this baseline in quantitative comparison and hereby provide the experimental results as requested.\n\n- For Summer-to-Winter:\n\n|             | **FID** | **FID CLIP** | **CLIP Sim** | **LPIPS** | **PSNR** | **SSIM** | **L2** |\n|-----------|-----------|-------------------|-------------------|---------------|---------------|--------------|----------|\n| CycleGAN | 133.16 (+53.15) | 18.85 (+4.71)  | 22.07 (-3.05) | 0.2 (+0.07)  | 16.27 (-6.18) | 0.39 (-0.18) | 3.62 (+2.72) |\n\n- For Horse-to-Zebra:\n\n|             | **FID** | **FID CLIP** | **CLIP Sim** | **LPIPS** | **PSNR** | **SSIM** | **L2** |\n|-----------|-----------|-------------------|-------------------|---------------|---------------|--------------|----------|\n| CycleGAN | 77.18 (-3.57) | 27.69 (+1.66)  | 28.07 (-0.84) | 0.25 (+0.02)  | 18.53 (-2.3) | 0.67 (+0.15) | 1.39 (+0.25) |\n\nIn the quantitative evaluation, we observe that CycleNet outperforms CycleGAN in most of the metrics.\n\n> Q1.2: If their performance is similar, then why use a diffusion-based method instead of GAN-based for unpaired I2I translation task?\n\n**Response to Q1.2**: Diffusion-based methods are superior in terms of image quality, enable multi-resolution generation, and are more scalable toward large multimodal datasets. Additionally, our method also enables strong generalization to unseen domains, where CycleGAN can hardly perform. We also show in Figure 3 and 10 (Appendix) that the GAN-based methods are more prone to assigning zebra patterns to non-horse objects, indicating their limited understanding of the entities in the scene.\n\n### Question 2:\n> Q2: Authors are suggested to experiment with more diffusion models such as T2I Adapter to show its generalization on models.\n\n**Response to Q2**: We were a bit lost by what the reviewer meant by “more diffusion models such as T2I Adapter”, so we hope to make sure to address the reviewer’s concern from the aspects of **diffusion models** or **diffusion model adapters**.\n- **More diffusion models**: To our knowledge, T2I-Adapter [30] is a plug-in-and-play adapter on pre-trained diffusion models (DMs) like stable diffusion (SD), rather than a diffusion model itself. If the reviewer is looking for additional experiments on more latent diffusion backbones, we would love to kindly point to Section E.1 in the Appendix, where we compare CycleNet on different versions of SD backbone, and confirm that our framework can generalize across different DM backbone.\n- **More diffusion models adapters**: We agree that our theoretical framework could potentially generalize to different adapters, not restricted to ControlNet. We would love to present additional experiments to support this view, but unfortunately, the training configurations are not available yet in the T2I-Adapter official GitHub, leading to a reproducibility issue. Also, T2I-Adapter does not take uncondition prompts, and we need non-trivial work to adapt it to our framework. Given the current status, we are unable to present quantitative experiments during the discussion period. We will be happy to update our paper once the training config of T2I-Adapter is publicly available to the community.\n\n### Question 3\n> Q3.1: Any explanation on why FastCycleNet perform better in several metrics? \n\n**Response to Q3.1**: Great question! A quick explanation: There is a known trade-off between translation quality and translation consistency [52]. As FastCycleNet does not enforce the cycle consistency loss $L_{x-y-x}$, it becomes better at image quality and translation quality, compared to the full CycleNet that enforces this loss.\n\nWe want to kindly refer the reviewer to Section 5.3 in the main paper, where we presented an ablation study. In this section, FastCycleNet is identical to the Invariance-Only version of CycleNet, which achieves better translation performance, as there is less trade-off between translation and consistency when the cycle consistency loss is removed. On the other hand, FastCycleNet suffers from relatively lower performance in terms of consistency metrics. \n\n> Q3.2 The coffee-to-empty translation result does not seem good, not only changing the color but also failing to become empty.\n\n**Response to Q3.1**: We hope to address the reviewer’s concern in the following aspects:\n- **Addressing the problem**: We kindly redirect the reviewer to **General response 1** in the General Response for our solution and updated results. In short, we address this with a simple but effective solution at inference. Our updated experimental results show that this issue of **hue shifting can be addressed with a minimal trade-off in image quality and translation quality**. We sincerely hope that the reviewer can re-evaluate our work.\n- **Emphasizing our dataset contribution**: We would like to emphasize our contribution to identify a challenging problem in image translation with object state changes, and to provide the first possible dataset to study this problem. We have observed that the task has posed a formidable challenge for all the models we tested. While our model performs satisfactorily in “Empty-to-Full” tasks, we acknowledge that the “Full-to-Empty” task remains challenging for our current solution. We hope the community will allocate more attention to this problem of image translation with object state changes.\n"}, {"review_id": "ce4KnOuKGI", "reviewer": "Reviewer_yCsT", "summary": "The paper introduces CycleNet, a new method that enhances image manipulation by incorporating cycle consistency into diffusion models. The paper addresses the challenge of unpaired image-to-image translation and aims to provide a consistent and intuitive interface for this task.", "questions": "1. Why there lack the results of strong baseline CycleGAN in Table 1?\n\n2. Could the authors consider conducting a user study to evaluate the subjective quality and usability of CycleNet? This would provide insights into how users perceive the generated images and the overall user experience, further validating the practicality and effectiveness of CycleNet.\n\n3. Could the authors provide more information about the computational efficiency of CycleNet? Specifically, details about the training and inference times, as well as the hardware specifications used, would be helpful in understanding the practicality of CycleNet for real-world applications.", "limitations": "1. Novelty is limited.\n\n2. Poor results are far from satisfaction as the strong generation ability of the diffusion model.", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.Originality: The paper introduces CycleNet, a new method that incorporates cycle consistency into diffusion models for image manipulation. This approach combines the concept of cycle consistency with text-guided diffusion models. \n\n2. Quality: The paper provides a comprehensive evaluation of CycleNet's performance using various quantitative metrics such as FID, CLIP Score, LPIPS, PSNR, SSIM, and L2.  The results also support the claims of the paper.\n\n3. Clarity: The paper is well-written and presents the concepts, methodology, and findings. ", "weaknesses": "1.Novelty is limited. The concept of combing cycle-consistency constraint and diffusion model for image-to-image translation has been widely explored by other works, such as UNIT-DDPM, Dual Diffusion Implicit Bridges (DDIBs), cyclediffusion and so on.\n\n2.Lack of Comparative Analysis between the proposed method with other existing cycle-consistency based diffusion models. The part of 6.3 is not sufficient to illustrate the key difference and novelty of the proposed method.\n\n3.The results are low-quality and far from satisfaction. For the classic image-to-image translation tasks, i.e., winter-to-summer and horse-to-zebra, I do not find any improvement over existing ui2i methods. In addition, the results of CUT aand CycleGAN are questionable, especially for winter-to-summer task, they are too bad compared with the results from their original papers.\n\n4.The results of coffee-to-empty also do not convince me as it seems to only transfer the rgb image to gray image. Similar results also happen in the horse-to-zebra.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are happy that the reviewer finds our paper “well-written”, and that “the results also support the claims of the paper”. We are more than grateful for the reviewer’s feedback, many of which has been integrated into our paper! We sincerely hope there will be engaged communication during our discussion, and that with our clarifications and updated results, the reviewer would consider re-evaluating our work!\n\n### Weakness 1 and 2\n> W1: Novelty is limited, and Lack of Comparative Analysis.\n\n**Response to W1 and W2**: We notice that there is an inconsistent view of our novelty across reviewers. We believe this might be addressed with more clarity and better comparisons to related efforts. We kindly redirect the reviewer to **General response 2** in the General Response for our clarifications. In short, we hope to bring it to reviewers’ attention that CycleNet presents the first effort that **explicitly enforces cycle consistency constraint** in unpaired image-to-image translation using pre-trained diffusion models.\n\nDDIB is the theoretic foundation for various diffusion-based image translation models, and CycleDiffusion is an extension of the vanilla DDIB [46]. We thus compared with CycleDiffusion in our main paper. Unfortunately, UNIT-DDPM does not have any public implementation, and can not be reproduced given the limited information in the paper. \n\n### Weakness 3\n\n> W3.1: The results are low-quality and far from satisfaction. For the classic image-to-image translation tasks, i.e., winter-to-summer and horse-to-zebra, I do not find any improvement over existing ui2i methods. \n\n**Response to W3.1**: We provide additional experiments on high-resolution examples in Appendix. Our observation is that CycleNet performs well, especially on complex scenes with multiple objects.\n\n> W3.2: In addition, the results of CUT aand CycleGAN are questionable, especially for winter-to-summer task, they are too bad compared with the results from their original papers. The results of coffee-to-empty also do not convince me as it seems to only transfer the rgb image to gray image. Similar results also happen in the horse-to-zebra.\n\n**Response to W3.2**: We use exactly the same images presented in the original papers without modifications. The quantitative results for horse-to-zebra are generated using the official model weights released by the authors.\n\n### Question 1\n> Q1: Why there lack the results of strong baseline CycleGAN in Table 1?\n\n**Response to Q1**: We thank the reviewer for reminding us of this missing baseline. The qualitative comparisons to CycleGAN are already provided in the main paper. We apologize for missing this baseline in quantitative comparison and hereby provide the experimental results as requested.\n\n- For Summer-to-Winter:\n\n|             | **FID** | **FID CLIP** | **CLIP Sim** | **LPIPS** | **PSNR** | **SSIM** | **L2** |\n|-----------|-----------|-------------------|-------------------|---------------|---------------|--------------|----------|\n| CycleGAN | 133.16 (+53.15) | 18.85 (+4.71)  | 22.07 (-3.05) | 0.2 (+0.07)  | 16.27 (-6.18) | 0.39 (-0.18) | 3.62 (+2.72) |\n\n- For Horse-to-Zebra:\n\n|             | **FID** | **FID CLIP** | **CLIP Sim** | **LPIPS** | **PSNR** | **SSIM** | **L2** |\n|-----------|-----------|-------------------|-------------------|---------------|---------------|--------------|----------|\n| CycleGAN | 77.18 (-3.57) | 27.69 (+1.66)  | 28.07 (-0.84) | 0.25 (+0.02)  | 18.53 (-2.3) | 0.67 (+0.15) | 1.39 (+0.25) |\n\nIn the quantitative evaluation, we observe that CycleNet outperforms CycleGAN in most of the metrics.\n\n### Question 2\n> Q2: Could the authors consider conducting a user study to evaluate the subjective quality and usability of CycleNet? This would provide insights into how users perceive the generated images and the overall user experience, further validating the practicality and effectiveness of CycleNet.\n\n**Response to Q2**: We thank the reviewer for this suggestion and hereby provide the human evaluation results as requested. **Our human study has been approved by the Institution’s Institutional Review Board (IRB)**, who considered this study exempt from ongoing review.\n\n- **Setup**: We recruited 12 participants to vote for 30 sets of generated images (15 from summer to winter and 15 from horse to summer).\n> In this task, you'll assess an AI model's performance in translating a summer Yosemite photo into a winter version. You'll review the provided summer input photo and a set of images generated by different AI models. Your evaluation will focus on two aspects: Translation Quality: Assess the image's translation quality, considering its visual quality, depiction of a winter scene, and adherence to common expectations. Consistency is not a priority here, so changes in structure or missing objects can be overlooked. Translation Consistency: Evaluate the image's consistency in translation. Check if it faithfully maintains similarity to the original photo in terms of object count, types, patterns, and textures. The translation quality can be overlooked in this evaluation. You will not be limited to choosing only 1 out of all images, but please restrict your number of choices to under 3 for each set.\n- **Results**: Our human study results show that our model is preferred by human users, both in terms of consistency and quality. Due to the limited space, we will update the tabled results in the next iteration.\n\n### Question 3\n\n> Q3: Could the authors provide more information about the computational efficiency of CycleNet? Specifically, details about the training and inference times, as well as the hardware specifications used, would be helpful in understanding the practicality of CycleNet for real-world applications.\n\n**Response to Q3**: The resulting algorithm is also practical in applications, relying on very little data (~2K) with minimal computational requirements (1GPU). We refer to Appendix B for more details."}, {"review_id": "1XIabV11Ro", "reviewer": "Reviewer_sYAZ", "summary": "The paper proposes CycleNet, a method for unpaired image-to-image translation using pretrained diffusion models. The main idea is to reconstruct the conditional images through a reverse process. The idea seems reasonable, and the paper is well-organized, with sufficient experimental results.", "questions": "Please see in weakness", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "The idea seems reasonable, and the paper is well-organized, with sufficient experimental results.", "weaknesses": "1. The concept of cycle construction has been extensively studied in the field, and this work follows a similar idea, which weakens its contribution.\n2. In CycleGAN, the reconstruction constraint $L_{x \\rightarrow y  \\rightarrow x}$ plays a vital role. However, in FastCycleNet, this constraint is omitted, yet the performance does not show significant drops. It would be helpful if the authors could provide an explanation for this observation.\n3. For some variable notation (such as ($\\bar{x}_o$), authors had better provide their definitions, It would be beneficial for readers to quickly understand these notations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are happy that the reviewer found our paper \"well-organized, with sufficient experimental results.\". We appreciate their constructive feedback on our efforts.\n\n### Question 1\n> Q1: The concept of cycle construction has been extensively studied in the field, and this work follows a similar idea, which weakens its contribution.\n\n**Response to Q1**: We notice that there is an inconsistent view of our novelty across reviewers. We believe this might be addressed with more clarity and better comparisons to related efforts. We kindly redirect the reviewer to **General response 2** in the General Response for our clarifications. In short, we hope to bring it to reviewers’ attention that CycleNet presents the first effort that guarantees cycle consistency in **unpaired** image-to-image translation using **pre-trained diffusion models**, which is theoretically motivated and empirically promising.\n\n### Question 2\n> Q2: In CycleGAN, the reconstruction constraint Lx-y-x plays a vital role. However, in FastCycleNet, this constraint is omitted, yet the performance does not show significant drops. It would be helpful if the authors could provide an explanation for this observation.\n\n**Response to Q2**: We believe that the reconstruction loss $L_{x-x}$ can still help the model to maintain a certain level of consistency. The Stable Diffusion is originally trained on reconstruction with text conditions but without image conditions, losing its consistency when image conditions are introduced. In FastCycleNet, we explicitly train the diffusion model to reconstruct an image given on the image condition. Therefore, the performance does not show significant drops, while there is a noticeable gap according to our ablation study.\n\n### Question 3\n> Q3: For some variable notation (such as $\\bar{x_0}$), authors had better provide their definitions, It would be beneficial for readers to quickly understand these notations.\n\n**Response to Q3**: $\\bar{x_0}$ refers to the reconstructed input image $x_0$ from back translation. We thank the reviewer for the suggestions! We will make sure to provide more concrete definitions for the notations we introduced.\n"}]

Author rebuttal:
### General response 1: Hue-Shifting and Sub-Optimal Performance on ManiCups

We thank the reviewers for pointing out this hue-shifting issue. We address this with a simple but effective solution at inference time and the updated results are as follows.

**Method**: Previously, when we are sampling the image at inference time, we start from a **random noise** $\epsilon$. In our updated method, we initialize the process with the **latent noised input image** $z_t$, collected using equation (1). This is in fact a commonly used trick at the inference time, which is also adopted in some of the baselines [28, 46] to improve consistency in image translation.

**Results**: Following [46], we set $t=100$ to collect the noised latent for a fair comparison. We label these updated models with “(Init)” in the tables. We hereby provide updated experimental results, quantitatively in the table below and qualitatively in the attached PDF. 

- For Summer-to-Winter and Horse-to-Zebra

|                    | **FID** | **FID CLIP** | **CLIP Sim** | **LPIPS** | **PSNR** | **SSIM** | **L2** |
|-----------------|-----------|-------------------|-------------------|---------------|---------------|--------------|----------|
| Summer-to-Winter, CycleNet (Init) | 82.52 (+0.73) | 17.54 (+3.4) | 23.32 (-1.32) | 0.13 (-0.07)  | 22.42 (+4.07) | 0.57 (+0.07) | 0.9 (-0.52) |
| Summer-to-Winter, FastCycleNet (Init) | 82.48 (+2.47) | 17.61 (+3.38) | 23.62 (-1.5) | 0.14 (-0.1) | 22.45 (+5.24) | 0.57 (+0.11) | 0.91 (-0.53) |
| Horse-to-Zebra, CycleNet (Init) | 81.69 (-0.43) | 28.11 (-0.1) | 28.91 (+0.96) | 0.27 (-0.04) | 20.42 (+1.57) | 0.52 (+0.02) | 1.14 (-0.09) |
| Horse-to-Zebra, FastCycleNet (Init) | 80.75 (-9.71) | 27.23 (+1.2) | 27.36 (-0.81) | 0.32 (0) | 19.29 (-1.54) |  0.51 (+0.05) | 1.31 (-0.02) |

- For CycleNet (Init) on Coffee-to-Empty and Empty-to-Juice:

|                    | **FID** | **FID CLIP** | **CLIP Sim** | **LPIPS** | **PSNR** | **SSIM** | **L2** |
|-----------------|-----------|-------------------|-------------------|---------------|---------------|--------------|----------|
| Coffee-to-Empty |  95.24 (-27.68) | 28.79 (0) | 27.54 (+0.88) | 0.14 (-0.04) | 21.85 (-0.32) | 0.78 (+0.08) | 1.92 (-0.09) |
| Empty-to-Juice | 79.02 (-47.83) | 23.42 (-9.31) | 27.75 (+1.29) | 0.17 (-0.04) | 20.18 (-1.58) | 0.76 (+0.06) | 2.27 (+0.19) |

The qualitative examples (in PDF) have shown that this simple initialization trick can satisfyingly mediate the hue-shifting issue, resulting in improved performance, especially in ManiCups tasks. In the quantitative evaluation, except for a very marginal increase in FID and decrease in CLIPScore in some cases, we observe better performance in most of the metrics (especially in terms of ManiCups tasks and the consistency metrics), compared to previous non-initialized CycleNets. 

Overall, we show that this issue of **hue shifting can be addressed with a minimal trade-off in image quality and translation quality**. We sincerely hope that the reviewer can re-evaluate our work.

### General response 2: Novelty, and Cycle Consistency in Diffusion Models

We notice that there are different views in terms of the novelty of this work across reviewers. Especially, Reviewer yCsT mentioned that:
> “The concept of **combing cycle-consistency constraint** and **diffusion model** for image-to-image translation has been widely explored by other works”

We believe there is a factual misunderstanding. To the best of our knowledge, this is the first effort that **explicitly integrates cycle consistency constraint** in unpaired image-to-image translation using pre-trained diffusion models. 

**Q: Has cycle-consistency constraint been explicitly integrated into diffusion models?**
Not yet, to the best of our knowledge.

We expand on what we have discussed in Section 6.3 regarding the several existing efforts to study cycle consistency in diffusion-based I2I translation.
- UNIT-DDPM ([39], Apr 2021) made an initial attempt in the unpaired I2I setting with cycle consistency. At inference time, the image generation does not depend on the cycle-consistent translation functions, leading to sub-optimal performance. Also, this preprint does not have any public implementation, and can not be reproduced given the limited information in the paper.
- DDIB (Mar 2022): The key proposition states that exact cycle consistency is possible **assuming zero discretization error**. DDIB itself does not enforce any cycle consistency constraint:
> “DDIBs first obtain latent encodings for source images with the source diffusion model, and then decode such encodings using the target model to construct target images.” 
- CycleDiffusion ([46], Nov 2022) proposes a zero-shot approach for image translation based on DDIB’s observation that a certain level of consistency could emerge from DMs. There is again no explicit treatment to encourage cycle consistency.
> “The method is based on the observation that cycle consistency naturally emerges from diffusion models, i.e., the same random noise leads to similar images. Therefore, there is no training objective that explicitly encourages cycle consistency.” (https://github.com/ChenWu98/cycle-diffusion/issues/16)

If there is any similar prior work the reviewers are aware of, please let us know.

**Q: Do we benefit from enforcing it as a constraint? Why is it non-trivial to do so in diffusion models?**

We echo the comments of reviewer rhuF:
> “When we apply pre-trained DM models on I2I tasks, it is difficult or unpredictable to control the mask and attention maps to achieve the desired results.”

While exact cycle consistency is theoretically possible, the assumption generally doesn’t hold true. Consistency remains to be challenging when there exists image conditioning. Unlike GAN-based models, diffusion-based methods are iterative and lack such a simple and intuitive translation cycle. Also, for text-guided diffusion models, it is difficult to design a discriminator.

Meta-review: The paper received borderline scores slightly leaning positive. All the reviewers acknowledged the key idea of the paper---regularize diffusion models with cycle consistency. The reviewers had concerns with the evaluation, in particular, reviewer Tt5a mentioned the hue-shifting problem visible in the results. In their response the authors addressed this and other concerns. The reviewer yCsT maintained a strongly negative score as they believe the literature contains a lot of related works in which similar ideas are explores. The authors disagreed with that statement. This concern also didn't sway the other reviewers to the negative side. Hence, while there is merit in yCsT's comments, the AC tends to agree with the positive reviewers and decides to recommend acceptance of the manuscript.
Decision: Accept (poster)