Novel Object Synthesis via Adaptive Text-Image Harmony
Authors: Zeren Xiong
Ze-dong Zhang
Zikun Chen
Shuo Chen
Xiang Li
Gan Sun
Jian Yang
Jun Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Text-to-image Generation; Diffusion Model; Object Editing; Combination
Primary area: machine_vision
TL;DR: ATIH

Abstract: In this paper, we study an object synthesis task that combines an object text with an object image to create a new object image. However, most diffusion models struggle with this task, \textit{i.e.}, often generating an object that predominantly reflects either the text or the image due to an imbalance between their inputs. To address this issue, we propose a simple yet effective method called Adaptive Text-Image Harmony (ATIH) to generate novel and surprising objects.
First, we introduce a scale factor and an injection step to balance text and image features in cross-attention and to preserve image information in self-attention during the text-image inversion diffusion process, respectively. Second, to better integrate object text and image, we design a balanced loss function with a noise parameter, ensuring both optimal editability and fidelity of the object image. Third, to adaptively adjust these parameters, we present a novel similarity score function that not only maximizes the similarities between the generated object image and the input text/image but also balances these similarities to harmonize text and image integration.  
Extensive experiments demonstrate the effectiveness of our approach, showcasing remarkable object creations such as colobus-glass jar. https://xzr52.github.io/ATIH/

Novel Object Synthesis via Adaptive Text-Image Harmony
Zeren Xiong', Zedong Zhang's Zikun Chen', Shuo Chen',
Xiang Li", Gan Sun Jian Yang' Jun Li
School of Computer Science and Engineering.
Nanjing University of Seience and Technology, Nanjing, 210094, China
RIKEN,,
College of Computer Science, Nankai University. Tianjing, 300350, 300350, China CCollege of Automation Science and Engineering.
South China University of Technology, Guangzhou, 510640, China
(zandyz, cajyang, juml justt..mjjuuteeeeeeee edu. cn, (xzr3312, zikunchence sungan14 0gmaaa1
xiang. 1i. wmplus@nankai.ccmmmmmmmmm edu.. cn,
shuo chen. meeniyarreen..)
Bang penquin
bullrount
Jee Heats
proteiE
Eopyelion Eoppetiorrcmm cade
ayeey press
Cancontie
percupine
bald enggle
Figure 1: We propose straightforward yet powerful approach generate combinational objects from a given object text-image pair for novel object synthesis. Our algorithm produces these combined object images using the central image and its surrounding text inputs, such glass jar (image) and porcupine (text) in the left picture, and horse (image) and bald eagle (text) in the right picture.
Abstract
In this paper, we study an object synthesis task that combines an object text with an object image to create a new object image. However, most diffusion models struggle with this task, i.e. often generating an object that predominantly reflects either the text or the image due to an imbalance between their inputs. To address this issue, we propose simple yet effective method called Adaptive Tmxt-Imaae Harmony (ATIH) to generate novel and surprising objects. First, we introduce scale factor and an injection step to balance text and image features in cross- attention and to preserve image information in self-attention during the text-image inversion diffusion process, respectively. Second, better integrate object text and image design balanced loss function with noise parameter, ensuring both
"Correspondigg author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

SDXL Turbo dupttp
Ours
PuPInv expeciang!
Ours
Inaage
Imveye
lest
Tost
Image
Imagee
Taxt
Tied.
Whiee
Figure 2: Imbalances between text and image in diffusion models. Using SDXL-Turbo [56] (left)
and PnPinv [27] (right), the top pictures show a tendency for generated objects to align with textual content (green circles), while the bottom pictures tend to align with visual aspects (orange circles). In contrast, our approach achieves a more harmonious integration of both object text and image.
optimal editability and fidelity of the object image. Third, to adaptively adjust these parameters, we present novel similarity score function that not only maximizes the similarities between the generated object image and the input text/image but also balances these similarities to harmonize text and image integration. Extensive experiments demonstrate the effectiveness of our approach, showcasing remarkable object creations such as colobus- glass jar Fig. Project Page
1 Introduction
Image synthesis from text or/and image using diffusion models such as Stable Diffusion [51].
SDXL (56). and DALL-E3 (43 has gained considerable attention due to their impressive generative capabilities and practical applications, including editing 75) and inversion [24 61]. Many of these methods focus on object-centric diffusion, utilizing textual descriptions manipulate objects within images through operations like composition [58]. addition [36 15] removal [63], replacement (7]. movement [29]. and adjustments in size, shape, action, and pose
In contrast, we study an object synthesis task that creates a new object image by combining an object text with an object image. For instance, combining kingfisher (image) and terrier (text) results a new and harmonious terrier-like kingfisher object, as shown in the right-side of Fig.
To implement object text-image fusion. most diffusion models, such as SDXL-Turbo (56), often use cross-attention [24] to integrate the input text and image. However, the cross-attention frequently results in imbalanced outcomes, as evidenced by the following observations. On the left side of Fig. 2 when inputting an axolot! (image) and a toucan (text), SDXL -urbo only generates an image of a toucan, showing bias towards the toucan text (green circles). Conversely, when inputting rooster (image) and an iron (text), it produces an image of a rooster, which closely resembles the original rooster image (orange circles). These observations reveal that the text (or image) feature often suppresses the influence of the image (or text) feature during the diffusion process, leading to failed fusion. To mitigate the image degeneration, Plug-and-Play [61] can inject the guidance image features into self-attention. Unfortunately, even with the application of the best inversion editing method, PnPinv (27), which incorporates the plug-and-p and-plad inversion into diffusion based editing methods for improved performance, still observe similar imbalances, as shown on the right-side of Fig. This arises an important problem: how can we balance object text and image integration? ?
To address this problem, we propose an Adaptive Text-Image Harmony (ATIH) method for novel object synthesis, as shown in Fig. 3 First, during the inversion diffusion process, we introduce scale factor to balance text and image features in cross-attention, and an injection step i to preserve image information in self-attention for adaptive adjustment. Second, the inverted noise maps adhere to the statistical properties of uncorrelated Gaussian white noise, which increases editability [46]. However, they are preferable for approximating the feed-forward noise maps, thereby enhancing fidelity. To better integrate object text and image, we treat sampling noise a parameter in designing

---

a balanced loss function, which strikes a balance between reconstruction and Gaussian white noise approximation, ensuring both optimal editability and fidelity of the object image. Third. we present a novel similarity loss that considers both i and a. This loss function not only maximizes the similarities between the generated object image and the input text/image but also balances these two similarities to harmonize text and image integration. Furthermore, we employ the Golden Section Search [47] algorithm to quickly find the optimal parameters a and i. Therefore, our ATIH method is capable of generating novel object combinations. For instance, an iron-like rooster is produced by merging the image rooster with the text iron, resulting in a rooster image with an iron texture, as shown in Fig. 2
Overall, our contributions can be summarized as follows: (1) To the best of our knowledge, we are the first to propose an adaptive text-image harmony method for generating novel object synthesis. The key idea is to achieve a balanced blend of object text and image by adaptively adjusting scale factor and an injection step the inversion diffusion process, ensuring their effective harmony. (2) We introduce novel similarity score function that incorporates the scale factor and injection step. This aims to balance and maximize the similarities between the generated image and the input text/image, achieving a harmonious integration of text and image. (3) Experimental results on PIE-bench [26) and ImageNet [53] demonstrate the effectiveness of our method. Our approach shows superior performance in creative object combination compared to state-of-the-art image-editing and creative mixing methods. Examples of these creative objects, such as sea lion- lion-glass jar, African chameleon-bing birdd and corgi-cock are shown in Figs. 6 and g
2
Related Work
Text-to-Image Generation The rapid development generative models based on diffusion processes has advanced the state-of-the art for tasks 12,
21
33 like text-to-image synthesis [22] 31. image
editing 664 2) and style transfer
655
23
35
Large-scale models such as Stable Diffusion [5]].
Imagen 55 and DALL-E
49 have demonstrated remarkable capabilities. Sdxlturbo [56] introduced
a distillation method that further enhances efficiency by reducing the steps needed for high-quality image generation. Our method utilizes Sdxlturbo for adaptive and innovative object fusion, preserving the original image's layout and details while requiring only the textual description of the target object.
Text Guided Image Editing. Diffusion models have garnered significant attention for their success in text-to-image generation and text-driven image editing using natural language descriptions. Early studies [ 54 70 40), such as SDEdit (40). balanced authenticity and fidelity by adding noise, while Prompt2Prompt 24 and Brug-and-Play (PNP) [6] enhanced editing through attention mechanisms. Further research, including MasaCtrl [5). Instructpix2pix 4. and InfEdit [69], explored non-rigid editing, specialized image editing models, and rapid editing via consistency sampling. Advances in image inversion and reconstruction (20 have focused on diffusion-based denoising process inversion, categorized into deterministic and non-deteriiiistic sampling (28.. Deterministic methods, such as Null-text inversion using DDIM sampling [4]. precisely recover original images but require lengthy optimization; -on-deteriiiisicc methods, such as DDPM inversion (25 and CycleDiffusion [67]. achieve precision by storing variance noise. PnPinv (26) simplifies the process by accurately replacing latent features during denoising, achieving perfect reconstruction but with weaker editability, We propose framework for creative object synthesis using object textual descriptions for effective fusion and a regularization technique to enhance PnPinv editability.
Object Composition. Compositional Text-to-Image synthesis and multi-image subject blending 59 aim to create novel images by integrating various concepts, including object interactions, colors, shapes, and attributes. Numerous methodologies 8 24 52, 55
methods (37 19 58
70
have been developed focusing on object combinations, context integration, segmentation, and text descriptions. However, these methods often merely assemble components without effectively melding inter-object relationships, resulting in compositions that, while accurate, lack deeper integration and interaction. This limitation is particularly evident in image editing, where multiple objects in single image fail to achieve cohesive synthesis. Our method addresses this by harmoniously fusing two objects to create novel entities, thereby enhancing creativity and imagination.
Semantic Mixing. The breadth of creativity spans diverse fields, from scientific theories to culinary recipes, driving advancements in AI as highlighted by scholars 3 39 and recent researchers (62 (32. This creativity has led to significant innovations in AI, particularly through generative models. Creative Adversarial Networks [16] push traditional art boundaries, producing norm-defying works
3

---

Rabbit (Image)
Noise E
L(2 2 cD
Self-A + Cross A tzz
2+
sim (a,,t)
Inject M
Step
Balance
InSelf-Athh un @ ScCross-Are
B 88
(Ct,
Peacock (lexl)
88
Peacock (lesc))
Scale aVF
Foctor a
daptivv Text-Image Harmony
Figure 3: Framework of our object synthesis incorporating a scale factor a, an injection step and noise & in the diffusion process. We design balance loss for optimizing the noise e, balance object editability and fidelity. Using the optimal noise c,, we introduce an adaptive harmony mechanism to adjust a and i, balancing text (Peacock) and image (Rabbit) similarities.
while maintaining artistic connections. Efforts to adapt AI for novel engineering designs 11 further exemplify this technological creativity. MagicMix S introduced semantic mixing task,u traditional style transfer methods 73
60 I0] which blending two concepts into photo-realistic
object while retaining the original image's layout and geometry, but often resulting in biased images and less harmonious fusion. ConceptLab [50] uses diffusion models generate unique concepts, like new types of pets, but requires time-consuming optimization and struggles to semantically blend real images. Our method operates at the attention layer of diffusion models for harmonious semantic fusion and proposes an adaptive fast search to quickly produce balanced, fused images, ensuring novel and cohesive integration of semantic concepts.
3 Methodology
Let O₂ and Of be an object image and an object text, respectively, used as inputs for diffusion models. Our goal is to create novel object image O by combining O₂ with Or during the diffusion process. To achieve this goal, we develop an adaptive mext-image harmony (ATIH) method in our object synthesis framework, shown in Fig. In subsection 3.1 we introduce text-image diffusion model with scale factor a, an injection step and noise a, In subsection 3..2 we present to optimize the noise to balance object editability and fidelity. In subsection 3.3] we propose a simple yet effective ATIH method to adaptively adjust a and for barmonizing text and image.
3.1 Tmxt- Diffusion Model (TIDM)
Here, we construct a Text-Image Diffusion Model (TIDM) by utilizing the pre-trained SDXL Turbo (56). The key components include dual denoising branches: inversion for inverting the input object image, and fusion for fusing the object text and image. Following the latent diffusion model [11, the input latent codes are defined 20 E(O,) ) for object image Oy and E(Or) for object text Or, using a pre-trained image/text encoder E(.) TN E(Ox) denotes as a null-text embedding. The latent denoising process is described follows:
Inversion Denoising. The inversion denoising process predicts the latent code at the previous noise level, --1, based the current noisy data 2, This process is defined as:
2 n + (se 6, + 7efr,
where Vt, and Yt are sampler parameters, 6 is sampled noise, and co t, 7) a pre-trained U-Net model [56] with self-attention and cross-attention layers. The self-attention is implemented as:
Self-Attn (0i Ri Vi:) M₁ V. M Softmax (Q:(R:)//Va)
(2)
where Qi Rg and Vi are the query, key and value features derived from the representation E, and the dimension of projected keys and queries. The cross cross-attention is to control the synthesis process

---

through the imput null-text embedding ON. implemented as follows: Cross-Attn @ Ry. Vt)
Mg. Vy, where Mj = Softmax (Qi(R;) Qi the query feature derived from the output of the self-attention layer, Ky and Ve are the key and value features derived from TN-
Fusion Denoising. Similar the inversion denoising branch, we redefine the self-attention and cross- attention for easily adjusting the balance between the image latent code bn and the text embedding The fusion denoising process is redefined as:
2t-1 U + b,
where Vt, Bt, "hr and Et are defined Eq. ① and eo (2 E, ax i) is also the pre-trained U-Net model [56] with injected self-attention and scale cross-attention layers. The injected self-attention with an adjustable injection step i(0 S i < 7') is implemented as:
M
if f i
InSelf-Attn (M₁, V(*) Mi- V2, M;-
(4)
Softmax (Qi(K?)
otherwise
where Qi, Kÿ and V are the query, key and value features derived from the representation 2,. Unlike the approach of injecting Ki and Vi2 from Eq. into K2 and V in MasaCtrl (5] we focus on adjusting the injection step by injecting M; from Eq. (2) into M The scale cross-attention with
an adjustable factor E [0, 2] is to control the synthesis process through the input text embedding implemented as follows:
(5) where Qi is the query feature derived from the output of the self-attention layer, Kū and Vi are the key and value features derived from the text embedding Unlike the non-adjustable scale attention map approach in Prompt-to-Prompp [24), we introduce a factor, a, to adjust the value feature. This allows for better balancing of the text and image features, even though they share the same form. Using this fusion denoising process, the generation of a new object image is denoted as O.
ScCross-Alts Attn (Qi, KG, V() = MC a. VE, M = Softmax (O((K)"//V))
Following the ReNoise inversion technique [20). based on the denoising Eq. T and the approximation cs( c, t, = c0(21-1t,, [1, the noise addition process is reformulated as:
z = { - (2,, -(61)) /vc.
(6)
3.2 Balance fidelity and editability by optimizing the noise e, inversion process
In this subsection, our goal is to achieve better fidelity and editability of the object image during the inversion process. We observe that increasing the Gaussian white noise of the denoising latent
code 2 enhance editability [46). while reducing the difference between the denoising latent
code _ and the standard path noise code E in Eq. (6) improve fidelity (25] 67 However, these two objectives are contradictory. To address this, we treat the sampling noise in Eq. as a learnable parameter. We define reconstructed & loss between أ, and E/_1. (E))
|| (142, 8:co 81co T) + HH)]]., and KL divergence loss between 6, and Gaussian distribution, (G) (L()) (p, I))), to simultaneously handle fidelity and editability. (P)((())))(((((
Based on Eqs.(] and (6) we design balance loss function follows:
[(E,) (E.) (Cn()
where À represents the weight to balance C, and C,, and in this paper, we set to X 6- = 125.
C.
Since the parameter e; is sampled from a standard Gaussian distribution during the noise addition process, C. is used solely to balance L, and its gradient is not computed for optimization.
3.3 Text-image harmony by adaptively adjusting injection step i and scale factor
Using the optimal noise e,, a fused object image O(a, i) can generated by the TIDM with an initial scale factor ag = 1 and injection step io [T/2] from the input object image O, and object text
5

---

1.0
lundat
Adjest Injection
Mdjot Booke
Tavilo)
hyperbulla
0.8
eliisee
0.7
Similarity
Kmue la
Rmaz 2 k 2 Ranin
Bele Bugi
Bigiin leyf Tan
0.4-
0.3
Lo (4) A A
0.1
0.0
0.2
0.4%1
0.6 Oo 0.8
1.2
Figure 5: The adjusted process of our ATIH with three initial points and
Factor a
Figure 4: Laim and Tesm with a E [0, 1.4).
sim (a) k (ax) F(a)).
Or. Here, we adaptively adjust E [0, 2] and (0 S < T) by introducing an Adaptive Text-Image Harmony (ATTH) method. We denote the similarity between the image O, and the fused image ((a,i) as [a, i) d(Or, (a, i)), and the similarity between the text Or and the fused image O(a,i) 1) as Tsim (a, i) d(Or, (a,,)) i)), where -) represents the similarity distance between text/image and image. In this paper, we compute the similarities ssi (a, i) and T'sm (a, i) using the DINO features [44) and the CLIP features [48), respectively, based on a cosine distance d. Our key idea is to balance and maximize both sim ( i) and T'm (a, i) for optimal text-image fusion.
Adjust injection step to balance fidelity and editability. Before achieving the idea, we first enable the object image to be smoothly editable by adjusting the injection step in the injected self- We denote Lam (a) = Isim (Ag. i) for for convenience. In the inversion process. it is generally observed that more injections lead to less editability. When all injections are applied (i 7)) an ideal fidelity achieved. We observe that when (1) < ymin
sim the fused image deviates significantly from the input image, resulting in a loss of fidelity. Conversely, when T (i) > Tmax, the fused image is too similar to the input image, resulting in no editability. To balance fidelity and editability, I (2) must satisfy m < sim 2 m in Fig. Therefore, initializing i [T/2], i is adjusted as follows:
i
1,
(im < Imin
i
I miin sim
S -ssim (t) < I
(8)
sim
i+
Isum >
sim
where sim and sim are set to 0.45 and 0.85 in this paper, respectively, based on observations from Fig. 17 After using Eq. (8), this adaptive approach can obtain an injection step i" to smooth the fusion process while maintaining balance between fidelity and editability. Fixing the injection step i next we use abbreviations, Lsi (a) = sim (a, and (a) = Tim (a,,)
A Adaptively adjust the scale factor for harmonizing text and image. To implement our key idea, we design exquisite score function with a as:
max F(a) (a) + Tsim (a) (a) k Tsim (a))
maximize similarities tellipse) balance similarities (hyperboln)
where 3 is weighting factor, and the parameter is introduced to mitigate inconsistencies in scale between high Taim(x)) and low Tsim (ax due to differences in text and image modalities, ensuring their scale balance. As shown in Fig. sam (a) decreases and Tam (a) increases as a increases, and vice versa. Based on these observations, we set k- = 2.3 and 3 = 1 in this paper.
In Eq. (9), the left-hand side represents the sum of the text and image similarities, forming an ellipse, while the right-hand side represents the absolute value of the difference between the text and image similarities, forming hyperbola. A larger sum value indicates that the generated image integrates more information from the input text and image. Conversely, a smaller absolute value signifies better balance between the text and image similarities. Additionally, given that Isim (a) E [0. 1] and Tam (a) E [0, 1]. their sum is greater than or equal to the absolute value of their difference, leading to F(a) 2 O. Therefore, our objective to maximize F(a:) to simultaneously enhance and balance both (an and (a)(a). Maximizing F(a)) is easily implemented by the Golden Section Search 447 algorithm, and we get the optimal a". Fig. deppics a schematic diagram to adjust both ii and a. Overall, our novel object synthesis, detailed in Algorithm I is presented in Appendix F

---

European fire
Text:
Bighorn
Bowling ball
Triceratops
Peacock
African chameleon
Original image
Our
a:0.811
a.0.
dz..392
1:3
020. 580
Masactti
Infodit
Instrnct pix
Figure Comparisons with different image editing methods. We observe that InfEdit (69 MasaCtrl [5] and InstructPix2Pix [4] struggle to fuse object images and texts, while method successfully implements new object synthesis, such as bowling ball-fawn in the second row.
Experiments
4.1
Experimental Settings
We constructed an object text-image fusion (OTIF) dataset consisting of 1,800 text-image pairs, derived from 60 texts and 30 images in Appendix C Images, selected from various classes in PIE-bench 26 include 20 animal and 10 non-animal categories. Texts were chosen from the 1,000 classes ImageNet (53] with ChatGPT 42 filtering out 40 distinct animals and 20 non-animals.
Datasets.
Details. We implemented our method on SDXLturbo [56] only taking ten seconds. For image editing, set the source prompt p, empty string "Null" and the target prompt Pt the target object class name. During sampling, we used the Ancestral-Eulee sampler (28 with four denoising steps. All input images experiments. Our experiments were conducted using two NVIDIA GeForce RTX 4090 GPUs.
uniformly scaled to 512 512 pixels to
consistent resolution in all the
Metrics. To comprehensively evaluate the performance of our method, we employed four key metrics: aesthetic score (AES) (57. CLIP text-image similarity (CLIP-T) [88, Dinov2 image similarity (Dino-I) 44], and human preference score (HPS) (68). Following the Eq. (9), Fscore and balance similarities (Bsim) with & = 2.3 are used to measure the text-image fusion effect.
Main Results
We conducted comprehensive comparison of our ATIH model with three image-editing models (i.e., MasaCtrl [5) InfEdit (69), and InstructPix2pix (4), two mixing models (i.e., MagicMix (34 and Conceptl.a abb (50) ) and ControlNet (72.. Notably, MagicMix and Conceptl ab share similar objective with ours to fuse object text/image, while Conceptl.at only accepts two text prompts as its
inputs. Due available code for MagicMix, utilized its unofficial implementation [1].
7

---

Iuaaee
Injection Sup
0.34
0,44
0.54
Facter
Scale
2.0
Original Image
7,3
2.0
2,5
Obfect
4,0
4,5
3,0
Cook
2.0
6,5
Figure 7: Comparisons with InstructPix2Pix using image/text strength variations.
Pencock
King pangoin
Zucokini
Grocory bug
Trioerstop
Orighnl iwage
Owr
DJ 090
622 0:0.472
122 1L.0.464
8:0.966
Mapicate
Conceptte
PEDEP
rubbi
Figure 8: Comparisons with different creative mixing methods. We observe that results surpass those of MagicMix (34). For ConceptLab (50), we exclusively examine its fusion results without making good or bad comparisons, as it distinct approach to creative generation.
Comparisons with image-editing methods. For a fair comparison, we uniformly set the editing text prompt in all methods photo of an (image category) creatively fused with a (lext category) to achieve the fusion of two objects. Fig. 6 visualizzes some combinational objects, with additional results available in Appendix H Our observations preserve the original image's details better during editing, as seen in examples like sheep-triceratops In contrast, InstructPix2Pix tends to alter the image more significantlly making closer to the edited text description. Secondly, different methods exhibit varying degrees of distortion when fusing two objects during the image editing process. For instance, in the case of African chameleon-bite method performs better by minimizing distortions and maintaining the harmony and high quality of the image. Thirdly, our method shows significant advantages in enhancing the editability of images. For the European fire salamander- glass jar example, other methods often result in only color changes and slight deformations, failing to effectively merge the two objects. In contrast, our method harmoniously integrates the colors and shapes of both the glass jar and the European fire salamander, significantly improving the editing effect and operability. Specially, Fig. 7 shows the results of InstructPix2Pix with manually adjusted image strengths (1.0, 1.5, 2.0) and text strengths (ranging
follows: Firstly, MasaCtrl and InfEdit generally
our
8

---

Original twag
Oote
ConnnNVe-rieeeereeu Object
Figure 9: Comparisons with ControlNet-depth and ControlNet-edge 72 using a description that "A photo of an (object image) creatively fused with (object text )".
from 1.5 to 7.5). At optimal settings of image strength 1.5 and text strength 5.0, InstructPix2Pix produced its best fusion, though some results were unnatural, like replacing the rabbit's 's ears with rooster's head. In contrast, our method created novel and natural combinations of the rabbit and rooster by automatically achieving superior visual synthesis without manual adjustments.
Comparisons with the mixing methods. Fig. g illustrates the results of text-image object synthesis. We observe that both MagicMix and ConceptLab tend to overly bias towards one class, such as zucchini-ow/ and corgi-cock. Their generated images often lean more towards one category. In contrast, our method achieves a more harmonious balance between the features of the two categories. Moreover, the fusion images produced by MagicMix frequently exhibit insufficiently smooth feature blending. For instance, in the fusion a rabbit and an emperor penguin, the rabbit's facial features nearly disappear. Conversely, our method seamlessly merges the facial features of both the penguin and the rabbit in the head region, preserving the main characteristics of cach.
Comparisons with ControlNet. We rigorously compared our method with ControlNet to assess their performance in complex text-image fusion tasks, as shown in Fig. | Our results highlight notable differences: ControlNet preserves structure well from depth or edge maps but struggles with semantic integration, especially with complex prompts, often failing to achieve seamless blending. In contrast, our method leverages full RGB features, including color and texture, alongside structural data.
Table 1: Quantitative comparisons on our TIF dataset.
Models
DI 44
CLIP-TT 48
AES
[77]
HPST [68]
Fscore]
fismm]
Our ATTH
0.756
0.296
0.383
1.362
0.075
MagicMix
14
0.587
0.328
5,786
0.373
1.174
0.167
InfEdit (8)
0.817
0.255
6.080
0.367
1.173
0.230
MasaCirl 5
0.815
0.234
5.684
0.343
1.077
0.277
InstructPixPPxx 4
0.384
0.394
5,881
0.375
0.768
0.522
Table 2: H-statistics (1) (a-velue (L)) between ATTH and other methods under different metrics.
Methods
DINO-1 [4]
CLIP [B
AES 5Y
HPS 68
Pscore
Dsim
MagacMix 3d
665.20 (1.1 (iy
248.35 (8.08e "1
43300 (LO1e "
252.
(1.45e
633.89
(7.186
792.72 (2.00e 14
IntEdit [9
402.36 (1.68e-19)
477.31 (8.23e (8.226-100) 100)
3.70 (5 (5.45e-93
114.02 (1 2.220-22
504.53
(9181e
917.99
2003
MosdCli
1
404.87(5-51eeMM)
943. 37.33333330 07.3-77----00
80 (2.27e
654.6 62. (2.21le 144y
991.48
(1.28e-217)
1183.59 (2 96-9---20 210)
IostrccPacPic
1565.38 (0.000000)
1891.69 (0.000000)
268.57 (2.036 -1)
(3.00e
1421.64 (4.1% 1311
1997.67000000000
displays the quantitative results, illustrating that our method achieves state-of-the-art performance in AES, HPS, Fscore and Bsim, surpassing other methods. These results indicate that our approach excels in enhancing the visual appeal and artistic quality of images, while also aligning more closely with human preferences and understanding in terms of object fusion. Moreover, when dealing with text-image inconsistencies scale k=2.3, our method achieves superior text-image similarity and balance, demonstrating superior fusion capability. Despite achieving the best D and CLIP-T scores under inconsistencies, InfEdit and InstructPix2Pix perform worse than our method in terms of AES, HPS, F Fscore and Bsim, and their visual results remain sub-optimal. These inconsistencies ultimately lead to the failure of integrating object text and image. contrast, our approach achieves better text-image balance similarities. Furthermore, Table presents the H-statistics (30] and P-values (66) assessing the statistical significance of performance differences between our ATIH and other methods across various metries. Compared to Instructpix2pix, for
Quantitative Results. Table 1

---

instance, our method shows significant differences, with H-statistics of 268.57 for AES and 39.63 for HPS, indicating potential improvements in both aesthetic quality and human preference scoring.
User Study. We conducted two user studies to assess intuitive human perception of results presented in Table Table and Appendix G Each participant evaluated 6 image-editing sets and 6 fusion sets. In total, these studies garnered 570 votes from 95 participants. Our method received the highest ratings in both studies, capturing 74.03% and 79.47% of the total votes, respectively. Among the image-editing methods, InfEdit [69] garnered 14.7% of votes for its superior editing performance, while InstructPix2Pix and MasaCtrl SS] received only 8% and 2.8%. respectively. In the fusion category, ConceptLab
50 received 12.28% of votes, while MagicMix [34] received 8%.
Table 3: User study with image editing methods.
Table 4: User study with mixing methods.
Models Our MasaCirl InstroctPixPPix
69
Models
Our ATIH
MugicMix [4
ConceptLab
Vote
422
16
84
Vote f
70
4.3 Parameter Analysis and Ablation Study
Parameter analysis. Our primary parameters include X in 7 rmin and sim in 88 and & in ( A balances the editability and fidelity of our model. We determined the specific value through personal observation combined with the changes in AES, CLIP-T, and Dino-I values at different À settings. Ultimately, we set to 125. To address the inconsistency between image similarity and text similarity scales, we approximated the scale k. Initially, measured the variations in image similarity and text similarity with changes in a, and identified the balanced similarity regions in the fusion results. As shown in Fig. 4 the optimal range for & found to be between [0.21, 0.27]. Based on these observations and experimental experience, we ultimately set to 0.23. As shown in Fig. 17 of Appendix D we observe that when the similarity between the image and the original exceeds 0.85, the images become too similar, making edits with different class texts less effective and necessitating decrease in i. Conversely, when the similarity is below 0.45, the images overly favor the text, making them excessively editable, requiring an increase in injection steps. Therefore, we set ymin to 0.45 and yua 0.85, More discussions are provided Appendix D
Ablation Study. In Figs.
PriPirv
w/ Ialance
w Wadspiixinjeet
w/ wadhptieeseke
10 and 18 in Appendix E we visualize the results with and without the balance loss in Eq. 7 the adaptive in- jection in Eq. (8) and the adaptive selection a in Eq. p within our object synthesis framework. Pn- Pinv, used for direct inver- sion and prompt editing, re- sulted in some distortion and blurriness. Compared to PnPinv, the balance loss significantly enhances image fidelity, improving details, textures, and editability. The adaptive injection enables a smooth transition from Corgi to Fire Engine in Fig. 18 Without this injection, the transformation is too abrupt, lacking seamless fusion process. Finally, the adaptive selection achieves a balanced image that harmoniously integrates the original and target features. Note that for limitations, please refer to Appendix B
"broxcni""
"bald augle"
Figure 10: Ablation study of the balance loss, adaptive injection ii and adaptive selection from the third column to the fifth column.
5 Conclusion
In this paper, we explored a novel object synthesis framework that fuses object texts with object images create unique and surprising objects. We introduced a simple yet effective difference loss to optimize sampling noise, balancing image fidelity and editability. Additionally, we proposed an adaptive text-image harmony module to seamlessly integrate text and image elements. Extensive experiments demonstrate that our framework excels at generating wide array of impressive object combinations. This capability particularly advantageous for crafting innovative and captivating animated characters in the entertainment and film industry. Broader impact, please Appendix A
10

---

Acknowledgements
This work partially supported by the National Science Fund of China, Grant Nos. 62072242 and 62361 166670. We sincerely appreciate the valuable feedback provided by the anonymous reviewers.
References
[1] Johannes Ackermann and Minjun Li. High-resolution image editing via multi-stage blended diffusion. In Proceedings of the NeurlPS Workshop on Machine Learning for Creativity and Design, 2022.
[2] David Bau, Jiaming Song, Xinlei Chen, David Belanger. Jonathan Ho, Andrea Vedaldi, and Bolei Zhou. Editing implicit assumptions in lext-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2023.
[3] Margaret A, Boden. The creative mind- and mechanisms. Taylor Francis e-I e-Liirrry, 2004.
[4] Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 18,992-18402, 2023.
[5] Mingdeng Cao, Xintao Wang. Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 22560-22570, October 2023.
[6] Goirik Chakrabarty, Aditya Chandrasckar, Ramya Hebbalaguppe, and Prathosh AP. Lomoe: Localized multi-object editing via multi-diffusion. arXiv preprint arXiv: 200402443.004377, 2024.
[7] Haoxing Chen, Zhuoer Xu, Zhangxuan Gu, Jun Lan, Xing Zheng, Yaohui Li, Changhua Meng, Huijia Zhu, and Weiqiang Wang. Diffute: Universal text editing diffusion model. In Proceedings of the Thirty-seventh Conference Neural Information Processing Systems (NeurIPS), 2023.
[8] I. Chen, Y. Wang, and X. Zhao. Multi-concept customization of text-to-image diffusion. In Procoedings of the IRFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[9] Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming-Wer Wei Chang, and William W. Cohen. Subject-driven text-to-image generation via apprenticeship learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[10] Y. Chen, H. Wu, and G. Li. Neural preset for color style transfer. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[11] Celia Cintas, Payel Das, Brian Quanz, Girmaw Abebe Tadesse, Skyler Speakman, and Pin- Yu Chen. Towards creativity characterization of generative models via group-based subset scanning. arXiv preprint arXiv:2203.0m 000523,,..00....... 2022.
[12] Yuqin Dai, Wanlu Zhu, Ronghui Li, Zeping Ren, Xiangzheng Zhou, Xiu Li, Jun Li, and Jian Yang. Harmonious group choreography with mmjectory-conttollablli diffusion. arXiv preprint arXiv:2403.061 2024.
[13] Partho Das. magiemix.
January 2022.
[14] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), pages 8780 0794,8 2021.
[15] Xiaoyue Dong and Shumin Han. Prompt tuning inversion for text-driven image editing using diffuusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7430-7440, 2023.
[16] Ahmed Elgammal, Bingchen Liu, Mohamed Elhoseiny, and Marian Mazzone. Can: Creative adversarial networks: Generating "art" by Jearning about styles and deviating from style norms. In Proceedings of the International Conference on Computational Creativity (ICCC), pages 96-103, 2017.
[17] Dave Epstein, Allan Jabri, Ben Poole, Alexei Efros, and Aleksander Holynski. Diffusion self guidanee for controllable image generation. In Advances in Neural Information Processing Systems 36 (NeurIPS). 2023.
[18] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-inat -magemaaeeeaa generation using textual inversion. In Proceedings of the International Conference Learning Representations (ICLR), 2023.
11

---

[19] Vidit Goel, E. Peruzzo. Yifan Jiang, Dejia Xu, N. Sebe. Trevor Darrell, Zhangyang Wang, and Humphrey Shi. Pair-diffusion: Object-level image editing with aauucture-and-apeaaaul paired diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[20] Chaoqun Gong. Yugin Dai, Ronghui Li, Achun Bao. Jun Li. Jian Yang, Yachao Zhang, and Xiu Li. Renoise: Real image inversion through iterative noising. arXiv preprint arXiv:2401 007110 2024.
[21] Chaoqun Gong, Yuqin Dai, Ronghui Li, Achun Bao, Jun Li, Jian Yang, Yachao Zhang, and Xiu Li. Text2avatar: Text to 3d human avatar generation with codehnok-driven body controllable attribute. arXiv preprint arXiv:2401.00 100711, 2024.
[22] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, and Bo Zhang. Vector quantized diffusion model for text-lo-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10696-10706, 2022.
[23] Mark Hamazaspyan and Shant Navasardyan. Diffusion-enhaan enhanced patchmatch: A framework for arbitrary style transfer with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 797-805, 2023.
[24] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or Or. Prompt- to-prompt image editing with cross attention control. In Proceedings of the International Conference on Learning Representations (ICLR), 2023.
[25] Inbar Huuberman-Spiegelllaan Vladimir Kulikov, and Tomer Michaeli. An edit friendly ddpm noise space: Inversion and manipulations. In Proceedings of the IECE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[26] Xuan Ju. Ailing Zeng, Yuxuan Bian, Shaoteng Liu, and Qiang Xu. Direct inversion: Boosting bbased editing with 3 lines of code. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
[27] Xuan Ju, Ailing Zeng, Yuxuan Bian, Shanteng Liu, and Qiang Xu. Pup inversion: Boosting diffusion-based editing with 3 lines of code. In Proceedings of the International Conference Learning Representations (ICLR), 2024.
[28] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Proceedings of the Advances Neural Information Processing Systems (NeurIPS), volume 35, pages 26565 26577, 2022.
[29] Bahjjat Kawar, Shiran Zada. Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6007-6017 2023.
[30] William H. Kruskal and W. Allen Wallis. Use of ranks in one-criterion variance analysis. Journal of the American Statistical Association, 533666) 1952.
[31] Chongxuan Li, Jun Zhu, Bo Zhang. Zhenghao Peng, and Yong Jiang. Shifted diffusion for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10707-10716, 2022.
[32] Jun Li, Zedong Zhang, and Jian Yang. Tp2o: Creative text pair-to-object generation using balance swwp-wwwppppppppse samplinng In Proceedings of the European Conference on Computer Vision (ECCV), 2024.
[33] Ronghui Li, Yuqin Dai, Yachao Zhang. Jun Li, Jian Yang. Jie Guo, and Xiu Li. Exploring multi-modal control in music-driven dance generation. arXiv preprint Xixiv:2401 00 01382, 2024.
[34] Jun Hao Liew, Hanshu Yan, Daquan Zhou, and Jiashi Feng. Magicmix: Semantic mixing with diffusion models. arXiv preprint arXiv:2210. 16056, 2022.
[35] Chenxi Liu, Gan Sun, Wenqi Liang, Jiahua Dong, Can Qin, and Yang Cong. Museummaker: Continual style customization without catastrophic forgetting, arXiv preprint arXiv:2404.16612, 2024.
[36] Shilin Lu, Yanzhu Liu, and Adams Wai-Kin Kong. Tf-icon: Diffusion- training- free cross-domain image composition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2294-2305. 2023.
[37] Wuyang Luo, Su Yang, Xinjian Zhang, and Weishan Zhang. Siedob: Semantic image editing by disen- tangling object and background. In Proceedines of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
2023.
12

---

[38] Jian Ma, Junhao Liang, Chen Chen, and Haonan Lu. Subject-diffusion: Open domain personalized text-to-image generation without test-time fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[39] Mary Lou Maher. Evaluating creativity in humans, computers, and collectively intelligent systems. In Proceedings of the 1st DESIRE Network Conference Creativity and Innovation in Design, pages 22-28,
2010.
[40] Chenlin Meng, Yutong He, Yang Song, Jiamiing Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on .ccrming Representations (ICIRR, 2022.
[41] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the TEEF/CVF Conference Computer Vision and Pattern Recognition(CVPRR) pages 6038 9088-607, 2023.
[42] OpenAI. Chatgpt: Optimizing language models for dialogue. 2023, Accessed: 2023-05-07.
[43] OpenAL. Dall 3: Ai system for generating images from text. https://www..or wwtp:://wwwooppnai....o cow coo/dall-ee dall-e-3
2024. Accessed: 2024-05-16.
[44] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez. Daniel Haziza. Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran. Nicolas Ballas, Wojciech Galuba, Russell Howes, Yao Huang, Shang-Wer Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research (TMLR), 2024.
[45] Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, and Furu Wei. Kosmos-g: Generating images in context with multimodal large language models. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2023.
[46] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun- Yan Zhu. Zero-shot image-to-image translation. In Proceedings of the ACM SIGGRAPH, pages 1-11, 2023.
[47] William H. Press, William T. Vetterling, Saul A. Teukolsky, and Brian P. Flannery. Numerical recipes. Citeseer, 1988.
[48] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020, 2021.
[49] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Proceedings of the International Conference on Machine Learning (ICML), pages 8821-8831, 2021.
[50] Elad Richardson, Kfir Goldberg, Yuval Alaluf, and Daniel Cohen-Or. Conceptlab: Creative concept generation using vim-guided diffusion prior constraints. ACM Transactions on Graphics (TOG), 43(2), 2024.
[51] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10644106695 2022.
[52] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). 2022.
[53] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, ct al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115:211-252, 2015.
[54] Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho, Tim Salimans, David J. Flect, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings. 2021.
[55] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text to- image diffusion models with deep language understanding, Proceedings of Advances in Neural Information Processing Systems, 35:36479 -6494, 2022.
13

---

[56] Axel Sauer. Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint arXiv:2311.7742,, 2023.
[57] Christoph Schuhmann. pesthetie-predietus http://pwggggaaaa.../////./////a January 2022.
[58] X. Song. Y. Zhang, and L. Wang. Objectstirch: Object compositing with diffusion model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
[59] Gan Sun, Wenqi Liang. Jiahua Dong, Jun Li, Zhengming Ding, and Yang Cong. Create your world: Lifelong text-to-image diffussion IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[60] H. Tang, L. Yu, and J. Song. Master: Meta style transformer for controllable zero-shot and few-shot artistic style transfer. In Proceedings of the IEEF/VV Conference Computer Vision and Pattern Recognition (CVPR), 2023.
[61] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text- driven image-to-image translation. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1921-1930, June 2023.
[62] Renke Wang. Guimin Que, Shuo Chen, Xiang Li, Jun Li, and Jian Yang. Creative birds: Self-supervised single-view 3d style transfer. In Proceedings of the IEEF/CVP International Conference on Computer Vision, pages 777448744, 2023.
[63] Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont- Tuset, Shai Noy. Stefano Pellegrini. Yasumasa Onoe, Sarah Laszlo. David Fleet, Radu Soricut, Jason Baldridge. Mohammad Norouzi. Peter Anderson. and William Chan. Imagen editor and editbench: Advancing and evaluating text-guided image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18359-18388 2023.
[64] Xin Wang. Yang Song. Shuyang Gu, Jianmin Bao, Dong Chen, Han Hu, and Baining Guo. Zero-shot spatial layout conditioning for rext-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10717-10726, 2022.
[65] Zhizhong Wang, Lei Zhao, and Wei Xing. Stylediffusion: Controllable discntangled style transfer via diffusion models. In Proceedings of the IEFF/CVF International Conference on Computer Vision (ICCV), pages 7677-7689, 2023.
[66] Ronald L. Wasserstein and Nicole A. Lazar. The asa's statement on p-values: context, process, and purpose. The American Statistician, 70(2):129-133 2016.
[67] Chen Heary Wu and Fernando De la Torre. Unifying diffusion models' latent space. with applications to cyclediffusion and guidance. arXiv preprint arXiv=2210005559, 2022.
[68] Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen. Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.00 003411 2023.
[69] Sihan Xu, Yidong Huang, Jiayi Pan, Ziqiao Ma. and Joyce Chai. Inversion-free free image editing with natural language. In Proceedings of the IERECVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[70] Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Fxemplar- based image editing with diffusion models. In Proceedings of the IEFR/VVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18381-18391, 2022.
[71] I.. Zhang, R. Chen, and M. Zhao. Sine: Single image editing with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[72] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 3813-3824, 2023.
[73] Y. Zhang. L. Wang. and H. Li. inversion-based style transfer with diffusion models. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[74] Yufan Zhou, Ruiyi Zhang, Jiuxiang Gu, and Tong Sun. Customization assistant for text-to to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[75] Ye Zhu, Yu Wu, Zhiwei Deng. Olga Russakovsky, and Yan Yan. Boundary guided learning free semantic control with diffusion models. In Advances in Neural Information Processing Systems 36 (NeurIPS). 2023.
14

---

A
Broader Impact
Our model's capability to fuse images and text to generate new and creative object images holds significant potential across various fields, including entertainment, design, and education. However, it also raises important considerations regarding content safety and ethical In particular, if the input image iapppooprite or offensive material, the generated images may similarly be inappropriate, leading to potentially unplcasant experiences for users.
contains
To mitigate these risks, it crucial to implement robust NSFW (Not Safe For Work) content detection mecha- nisms. While existing methods can address some cases of inappropriate content, we acknowledge the need for continuous improvement in this area. As part of our future work, we will incorporate advanced NSFW checking models to ensure the generated content adheres to safety standards and ethical guidelines. This proactive approach to safeguard users promote responsible use our image generation technology.
B Limitation
Our method relies on the semantic correlation between the original and transformed content within the diffusion feature space. When the semantic match between two categorics is weak, our method tends to produce mere texture changes rather than deeper semantic transforma- tions. This limitation suggests that approach may struggle with transformations between categories with weak semantic associations. Future work could focus on enhancing semantic matching between different cat- egories to improve the generalizability and applicability of method.
original images
our results
+ African chameleon
There still some failure cases in model, shown Fig. [I]] These failures can categorized types. The first row illustrates that when the content of the image is significantly different from the text prompt, the changes become implicit. The second row demon- strates that in certain cases, adaptive function results changes only affect texture of the original image. In our future work, we will investigate these sit- uations further and analyze the specific items that do not yield satisfactory results.
two
'Komondor"
Figure 11: Failure results of our ATIH model.
C Text and Image Categories.
We selected 60 texts, as detailed in Table and categorized them into 7 distinct groups. The 30 selected images are shown in Fig [2 with each image corresponding to similarly categorized texts, outlined in Table Our model is capable of fusing content between any two categories, showeasing its strong generalization ability.
Table 5: List Text Items by Object Category.
Category
Items
fox, Siberian husky, Australian terrier, badger, Egyplian cat, cougar, gazelle, porcupine, sea lion, bison, komondor, oller, siamang, skunk, giant panda, zebra, hog, hippopotamus, bighorn, colobus, tiger cat, impala, coyote, mongoose
Mammals
Birds
king penguin, indigo bunting, hald eagle, cock, ostrich, peacock Komodo dragon, African chameleon, African crocodile, European fire salamander, tree frog, mud turtle
Reptiles and
Amphibians
Fish and Marine Life
anemone fish. white shark, brain coral
Plants
braccoli, acorn
Fruits
strawberry, orange, pineapple, zucchini, butternut squash
triceratops, beach wagon, beer glass, bowling ball, bruss, airship. digital
Objects
clock, espresso maker, fire engine, gas pump, grocery bag, harp, parking
meter, pill bottle
15

---

Table 6: Original Object Image Categories.
Category
Items
Sea lion, Dog (Corgi), Horse, Squirre,, Sheep, Mouse, Panda, Koala, Rabbit, Fox, Giraffe, Cat,
Mammals
Wolf, Bear
Birds
Owl, Duck, Bird
Insects
Ladybug
Plants
Tree, Hlower
Fruits
Vegeta-
Red pepper, Apple
bles
Cup ol collee, Jar, Church, Birthday cake
Objects
Human
Man in suit
Lion illustration, Deer illustration,
Artwork
Twitter logo
Figure 12: Original Object Image Set.
D Parameter Analysis.
max editability
balanced
max constructability
(coconstructoonn "deer"
"indigo bunting
original image
"husdy""
Image variations under different À values. The first displays the reconstructed images. The middle and bottom rows show the results of editing with different prompts, demonstrating variations in maximum editability, a balanced approach, and maximum constructability
Figure 13:
Analysis Here, we provide a detailed explanation the determination of A. As shown in Fig. [3 we use the ratio A Le to balance editability and fidelity. We iteratively adjust this ratio in the range of [o, 400) with intervals of 10, measuring the Dino-I score between the reconstructed and original images, well the CLIP-T and AES with the inverse latent values at different ratios. These experi- ments were conducted on the class fusion dataset, using fusion text for direct image editing. Figs. 14 15 T6]nddiie that the ratio increases, image editability improves, peaking a ratio of around 260, but with decrease in quality. At ratio of 125, both image fidelity and the AES score achieve an optimal balance. Therefore, we set À to 125.
Table 7: Quantitative comparison results with different X.
In
AFS
CL.PP- f
Dino-I 1
0.413
0.927
for images directly edited
125
6.153
0.417
0.902
260
6.012
0.419
0.760
Analysis of k. The experimental analysis of parameter range was set [0, 4], and for each value a was iterated from 0 in steps of 0.02 to observe changes in the fused image. The averaged experimental results produced a smooth curve, as shown in Fig Based on these observations, the optimal range for & was determined to be between [2.1, 2.7J. In our experiments, set the value of to 2.3.
conducted using sdxlturbo the base model. The
16

---

6.15
6.10
0.80
0.85
0.418
0.417
6.05
0.75
6.00
5.95
0.65
0.414
5.90
0.60
0.413
5.85
100
200
300
0
100
200
300
100
200
300
A
Figure 14: Dino-I changing with Figure 15: CLIP-T score chang- Figure 16: AES changing with
ing with À
Analysis of Tmin and /mm As shown in Fig. 17 visualized several specific node images generated during the variation of different factor values. When the image similarity with the original image exceeds 0.85, the images become overly similar.
example, in the dog-zebra fusion experiment, the dog' texture remains largely unchanged, and no zebra features are visible. Conversely, when the image similarity falls below 0.45, the images overly conform to the text description. In this case, the entire head of the image turns into zebra, representing formmraonooomtioon phenomenon. Based these observations, threshold unn 0.45 and maximum similarity threshold !mm 0.85. This range helps us achieve good balance between retaining original image information and integrating text features.
set the minimum similarity
Origin
sim:0.8523
sim:0.6727
stm: stm:0.446 0.4422
Tust Zehaa
sim: 0.255.
0.277
T. sim:0.3951
Figure 17: Illustrates the visual results of images at different similarity levels.
E
Ablation Study.
We present another set of ablation study results in Fig. 18 where the two rows represent the cases without (w/o) and with (w) attention projection. The input image is Corgi. and the text is Fire engine. The output images display the different transformations attention projection, resulting in sudden transition from Corgi fire engine. In contrast, with attention projection (bottom row), change is smoother, achieving desired blending result in middle.
varies. The top row shows the abrupt change in appearance without
0.2
0.26
We infertion
Fheengin
w (njuusoonnuuss)
0.50
enring
Figure 18: Results changing in Iteration w/ and w/o attention injection.
Algorithm.
Overall, our novel object synthesis comprises three key components: optimizing the noise e, through balance of fidelity and editability loss, adaptively adjusting the injection step and dynamically modifying the factor
17

---

Algorithm Novel Object Synthesis
Input: An initial image latent 20, target prompt Or, the number of inversion steps inject
step i, sampled noise scale factor a, F(a) is Eq.
2: Output: Object Synthesis O
3: {2r.-... 2 1:--. 20 f- scheduler_ (nneeleee)_
4: for tort- = to I do
5: 2 t st))(E,)
6: Ean[1] + Bulaacefbbilyycciiiiiiiyeeaaaaa
7: end for
8: Vinit T/2
9: Ffimal & Adjust-Inject(2r: fall, OT, finit)
agood Golden-SSccion--Sarrdd
O'miny Omax)
11: O <- DM fall; Or,, ffimal, Orgood)
12: return O
13: function BALANCE- BLLANCE-DDLLLY-DDAABBLLYC...LL.L 2, E)
14: while Ln/Ln > do
15:
6 V., f, -
end while
17: return
end function
19:
function GOLDENNSIECIIN-SSAARRIISS 1tvá
Golden ratio
21:
c+ b-a
d a b-2
+
23:
while (b a
> E do
24:
if f(c) b
f(d) then
25:
26:
else
27:
a c
28:
end if
29:
c b-
a + 31: end while
32: return bit 2
33:
end function
34:
function ADJUST-INJECTEE,,C Cau, Or)
35:
ite +0
36: while iter 2
do
37:
Isim + model Jae (ZT, eall, i, Or)
38:
if ii Iauin then
39:
i +i+t
else Imin
< 1si
mmax then
41:
i f i
break
43:
else
iti-1
45:
end if
iter +- iter + 1
47:
end while
48:
return /
49: end function
18

---

These processes are detailed in Algorithm[] [ Additionally, utilize the Golden Section Search method to
identify optimal sufficiendly good value for that maximizes the score function F((x) in Eq. ( This approach operates independent the function's derivative, enabling rapid iteration towards achieving optimal harmony. The key steps of the Golden Section Search algorithm are outlined as follows:
b
b
b
where @ (approximately 1.618) is the golden ratio, and a and are the current search bounds for a. During cach iteration, we compare F(a1) and ((a2), and adjust the search range accordingly:
if F(a1) F(oz) then
else
This process continues until the length of the search interval 66 al is less than predefined tolerance, indicating convergence local maximum.
G
User Study.
In this section, we delve into our two user studies in greater detail. The image results are illustrated in Figs. and while the outcomes of the user studies for both tasks are presented in Figs. 19 and 20 In total, collected 570 votes from 95 participants
8
both studies. The specific responses for each question are detailed in
Tables and
Notably, the fourth question in user study corresponding to our editing method, the example of peacock and cat fusion is shown in Fig 6 the number of votes for InfEdit (69) slightly exceeded ours. However, upon examining the image results, it becomes evident that their approach Icans towards disjointed fusion, where one half of object is spliced with the corresponding half of another object, rather than directly generating new object method does.
-3 Given original conssponding
EE. Pieose neled your prefemme fured Image the teto oipects from below.
Rabbe
King Benque (oxt)
Figure 19: An example of a user study compar- compar-Figure 20: An example of a user study compar- ing various mixing methods.
ing various image-editing methods.
Table 8: User study with image editing methods.
options(Models)
A(Our ATIH)
B(MasaCurl)
C(nstrrcctPxx2Pix D(InfFddit)
image-prompp
glass jar- salamanadd giraffe-bowling ball wolf-bighorn cat- cat-peacock
77.89 %
1.05%
16.84%
4.21%
89.74 %
2.11%
2.11%
6.32%
84.21 %
1.05%
10.53%
4.21%
40 %
3.16%
5.26%
51.58%
sheep-triceraptors
78.95 %
3.16%
11.58%
6.32%
bird-African chameleon
73.68 %
6.32%
4.21%
15.79%
19

---

Table User study with mixing methods.
options(Models)
(prompt) image-prompt
A(Our ATTH)
B(MagicMix)
C(ConeeptLaab)
Dog-white shark
81.05%
2.11%
16.84%
Rabbit-king penguin
83.16%
11.58%
5,26%
horse-microwave
71.58%
9,47%
18.95%
camel-candelabra
86.32%
6.32%
7.37%
airship-espresso maker
71.58%
11.58%
16.84%
jeep-anemone fish
83.16%
8.42%
8.42%
H
More results.
In this section, we present additional results from our model. Fig. 21 showaases fürther generation results using ATTH model. We experimented with four different images, each edited with four distinct text prompts. Fig. provides further examples showcasing the effectiveness of our method in complex text-driven fusion tasks. Specifically, our approach excels in extreme cases by accurately extracting prominent features, such as color and basic object forms, from detailed textual descriptions. For instance, Fig. [22] shows a well-defined edge structure for the fawn image and the text 'Green triceratops with rough, scaly skin and massive frilled head.' Additionally, Fig. 23 illustrates model'ss versatility with multiple prompts, emphasizing its capability for continuous editing.
22
Komodo dragon
original image
broccoli
salamander
skunk
Figure 21: More visual Results.
I More Comparisons
this section, we present additional results from our model compare its performance against other methods.
Fig. we compare our results with those from state-of the-the-rt art model DALL- E-3 assisted by Copilot. Our model shows superior performance when handling complex descriptive prompts for image editing. We observe that the competing model struggles to achieve results comparable to ours, particularly in maintaining the original structure and layout of images, despite adequate prompts.
20

---

Orgginal Snoge
mantee Poled heod
pultty randy
Frother
Figure 22: More visual results using complex prompt fusion.
original image
+ hiippoootamus
+ zebra
+ African chameleon
+ hog
M
Figure 23: Fused results using three prompts.
In Figs. 25 and 26 we present additional comparison results with mixing methods. We observed that both MagicMix and ConceptLab tend to overly favor category, seen in examples like Tricerutops- Teddy Bear Toy and Anemone fish-Car Their generated images often lean towards single category.
Recently, subject-driven text-to-image generation focuses creating highly customized images tailored target subject 18 9 (4), These methods often address the task, such as multiple concept composition, style transfer and action editing 38 g (45). In contrast, our approach aims to generate novel and surprising object images by combining object text with object images. Kosmos G (45) utilize a single image input and creative prompt to merge with specified text objects. The prompt is structured as "<i> creatively fuse with object text," guiding the synthesis to innovatively blend image and text elements. Our findings indicate that Kosmos-G sometimes struggle to maintain balanced integration original image features and text-driven attributes. In Fig. 27 the images generated by Kosmos-O often exhibit a disparity in feature integration.
21

---

original image
OHPS
bing(DALLEE3)
complex prompt
"Trunsform the image of a mujestic tion with gulden mane into on image of fioree eugle with vivid red and orange feathers. Change the lion" 's facial features to resemble on eagle, including beak eyes, while maintaining the dy dynamic, stylized design."
"cock"
"Transform the image of a supireel into "pineapple squirrel." Change its für texture to resemble pineopple skin and add pineoppie-tike tufts on its ears. Adjust background match the ouldoor setting with and sky."
"pineapple"
"Transform image of flowers into strawberry- like flowers. Change petala to resemble the texturee and color of strawberries. maintaining the overall shope Adjust the colors include vibrant reds and greens, while keeping the arrangement.
flowers.
and
"strawberry"
"Transform image of a hurse into shark- horse. Change horse's body to have smooth, gray skin and while keeping overall shope similar. Adjust the head to resemble shark's with sharp teuth and dorsal fin. Maintain the outdoor setting with grassy background.
"shark"
Figure 24: Comparisons with complex prompt editing.
Text:
Anemune fish
Candelabra
Micro-wave
White shark
Expresso maker
Generated image
Mate
Our
Magicoizx
Cuncopriab
Jeep
Camel
Horse
Dog
Airship
Figure 25: Comparison results of mixing methods using text-generated images.
22

---

Text:
Mud turtle
Congur
Strawberry
Porcupine
Triceratops
Originul imuge
Onr
Id 0:1.05
62 0:0.81
1:2 0:0.52
66:0.46
Magicmix
Concopilab
Figure 26: Further comparisons with mixing methods.
Original Image
Ours
Object
text
Figure 27: Comparisons with Subject-driven method.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, refer to Scctionrefsec:intro Introduction.
Guidelines:
The answer NA means that abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match thcorctical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Yes, refer to Appendix refsec:limitation Limitation.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors are encouraged create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (e.g.. independence assumptions, noiseless settings, model well rell-specifications asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should he articulated.
The authors should reflect the factors that inffuence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: the paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
24

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
Formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: refer to Subsection 4.1 Experimental Settings and Subsection 4.3 Parameter Analysis.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For novel architecture, describing the architecture fully might suffice,
example, if the contribution or the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the of large language model), releasing of model checkpoint, or other means that are appropriate to research performed.
While NeurIPS does not require relcasing code, the conference does require submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a way to access this model for reproducing the results a way to reproduce model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
new model (e.g.. large language model), then there should either be
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to deseribe the particular way they provide for reprodueibilitty In the case of closcd-sourc source models, it may be that access to the model is limited some way (c.g., to registered users), but it should be possible for other rescarchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described supplemental material?
Answer:
[Yes]
Justification: for data we used in paper, we introduced in Appendix Cefsec:T[Categoriee and will provide source code when this paper is published.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurTPS code and data submission guidelines ptttpe://nips.cccpublii htt////ooeeueeeiieeceeec
for more details.
While we encourage release of code and data, we understand that this might not possible,
so "Na" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source henchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: : :/nnis///////////////::::eee cc/public/ quiidess/CodeSuubmiiessoooooo... for more details.
25

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reprodueible, they should state which ones are omitted from script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Detain Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand results?
Answer: [Yes]
Justification: refer to Subsection cisubscc:exp,etting ef ef parameter, and Apeedaiiireeeeeeeee supparamGruidelins:
7. The answer NA means that the paper does not include experiments.
8. The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
9. The full details can be provided either with the code, in appendix, or as supplemental material.
Experiment Statistical Significance
Question: Daes the paper report bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: refer to SubsectioneeSuuseecmai,, esM Ren.teienst..Suiiiee Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard of the mean.
It is OK to report I-sigma стror bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI. if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative emrrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: refer Subsection efsihscceep.cttinguulddiiisss
The NA that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider. including relevant memory and storage.
26

---

The paper should provide the amount of compute required for each of the individual experimental runs well as estimate the total compute.
The paper should disclose whether the full research project roquired more compute than the experiments reported the paper (e.g.. preliminary failed experiments that didn't make into the paper).
Code Of Ethies
Question: Does the rescarch conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://wwppp.....o.oo//////aaaaa
Answer: [Yes]
Justification:
We conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If authors answer No, they should explain special circumstances that require deviation from Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consideration due to laws or regulations in their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of work performed?
Answer: [Yss]
Justification: refer Appendix refsecsupp.mpprrudeerrr htwp://rrrrreememettteee..eeeeee www.aaa.eeeeeeeeeeeeeeeeee Guidelines :
The answer NA means that there societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (c.g., disinforma- tion, generating fake profiles, surveillance), fairness considcrations (c.g.. deployment of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point out. For example, is legitimate to point out that improvement in the quality the other hand, it
generative models could used to generate deepfakes for disinformation. not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology being used as intended and functioning correctly, harms that could arisc when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative socictal impacts, authors could also discuss possible mitigation strategies (c.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators. or scraped datasets)?
Answer: [NA]
Justification: this paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
27

---

Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that adhere usage guidelines restrictions access model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, hut we encourage authors to take this into account and make hest faith effort.
Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite the original papers of the open open-sourcee sourced data and codes.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For popular datasets, paperswithcode. com/ datasets com/datasets
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
this paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
this paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid least the minimum wage the country of the data collector.
28

---

Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were dis- closed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: this paper does not involve crowdsoureing nor rescarch with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. Depending on the country in which research is conducted. IRB approval (or equivalent) may required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper:
We recognize that procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "kw1kr5jKOO", "reviewer": "Reviewer_xaqc", "summary": "The paper introduces an innovative method to generate new object images by combining textual descriptions with corresponding images. Addressing the common imbalance in diffusion models between text and image inputs, the authors propose the Adaptive Text-Image Harmony (ATIH) method. This method balances text and image features during cross-attention, ensuring a harmonious integration.\nKey to this method are the scale factor (α) and injection step (i), which adjust the influence of text features and preserve image information. A novel similarity score function maximizes and balances the similarities between the generated image and the input text/image, while a balanced loss function with a noise parameter optimizes the trade-off between editability and fidelity of the synthesized image.\nValidated through extensive experiments on datasets like PIE-bench and ImageNet, ATIH outperforms state-of-the-art techniques in creative object synthesis. The method showcases remarkable results, such as creating a dog-lobster or a rooster with an iron texture, demonstrating its innovative and high-quality image generation capabilities. The framework includes a Text-Image Diffusion Model (TIDM) using a pre-trained SDXL Turbo model with dual denoising branches. By treating sampling noise as a learnable parameter and designing a balance loss function, the method enhances image fidelity and editability. The ATIH method adaptively adjusts the scale factor and injection step to balance text and image similarities, using a similarity score function and the Golden Section Search algorithm to find optimal parameters.", "questions": "1. Please include a detailed description of the selection criteria for the text-image pairs used in your experiments. It is important to demonstrate the diversity and representativeness of these pairs to ensure the robustness of your Adaptive Text-Image Harmony (ATIH) method.\n\n2. It would be beneficial to expand the comparative analysis section with more detailed results, including statistical significance tests. This would provide a clearer picture of how your method improves over existing methods, underlining its novelty and effectiveness.\n\n3. Please discuss the performance of your method under non-ideal conditions, such as complex or abstract text descriptions. Identifying and describing limitations observed in such scenarios could guide future improvements and research.\n\n4. Including comprehensive implementation details, particularly regarding parameter settings and platform-specific optimizations, would significantly aid in the reproducibility and independent verification of your results.\n\n5. Identifying specific edge cases and failure modes encountered during testing and development would help in understanding the practical limitations of your model. This information is crucial for setting realistic expectations and guiding future research directions.", "limitations": "The limitations have not been addressed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Originality: The paper introduces a novel method for combining textual descriptions with corresponding images to generate new object images. The originality lies in the Adaptive Text-Image Harmony (ATIH) method, which effectively balances text and image features during the cross-attention mechanism in diffusion models. This innovative approach addresses a significant challenge in existing methods— the imbalance between text and image inputs— and offers a robust solution to achieve harmonious integration. The introduction of a scale factor (α) and an injection step (i) to fine-tune this balance further highlights the creativity and novelty of the approach.\n\n2. Quality: The paper demonstrates high quality through comprehensive experiments and rigorous validation. The authors provide detailed methodological explanations and present clear experimental results on datasets such as PIE-bench and ImageNet. The inclusion of a balanced loss function with a noise parameter to optimize the trade-off between editability and fidelity showcases the thoroughness of the approach. The comparison with state-of-the-art techniques further establishes the effectiveness and superiority of the ATIH method. The experimental results, including examples like a dog-lobster and a rooster with iron texture, illustrate the method's capability to produce high-quality, innovative, and coherent images.\n\n3. Clarity: The paper is well-written and clearly structured. The authors provide a concise and comprehensive introduction to the problem, followed by a detailed explanation of the proposed ATIH method. The use of figures and diagrams, such as the framework of the Text-Image Diffusion Model (TIDM) and the results of the experiments, aids in understanding the methodology and its impact. The step-by-step breakdown of the methodology, along with the clear presentation of experimental results, ensures that readers can easily follow the proposed approach and its benefits. The balanced loss function and similarity score function are well-explained, contributing to the overall clarity of the paper.\n\n4. Significance: The proposed method's capability to outperform existing techniques and produce high-quality, innovative results underscores its importance and impact on the broader field of text-to-image synthesis.", "weaknesses": "1.  While the paper discusses a novel algorithm for text-image harmony, the experimental validation focuses heavily on visual outcomes without substantial quantitative backing or comparisons against baselines using metrics relevant to the text-image synthesis field. Including more diverse and robust statistical evaluations could strengthen the claims about the algorithm's effectiveness.\n\n2. The experiments primarily utilize a specific set of images and texts, which might not adequately represent the variety of real-world scenarios where such an algorithm could be applied. Expanding the dataset to include a wider range of text and image pairs, especially challenging ones, would help in understanding the algorithm's limitations and strengths better.\n\n3. The paper could benefit from a deeper exploration of the robustness of the proposed method, particularly in how it handles edge cases or unusual text-image pairs. This includes testing the method's performance on non-ideal or adversarial inputs to gauge its resilience and adaptability.\n\n4. More detailed comparative analysis with existing methods, especially recent advancements in text-to-image synthesis and image editing technologies, would provide a clearer picture of where the proposed method stands in terms of innovation and improvement. Specific examples of where it outperforms and underperforms can guide future development.\n\n5. The methodology section could be expanded to include more technical details about the implementation, which would aid in reproducibility. For instance, details on parameter settings, algorithmic steps that were particularly effective, and potential pitfalls in the implementation could be highlighted.\n\n6. The paper mentions user studies but does not delve deeply into how the feedback from these studies was integrated into the algorithm refinement. Elaborating on this process could provide insights into the user-centric development of the algorithm and its practical usability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: More quantitative validation**\n\n**A1:** Thank you for your thoughtful feedback. We recognize the importance of quantitative metrics in validating our algorithm's effectiveness.\n\nWe use four widely-used metrics: CLIP-T \\[39\\], Dino-I \\[36\\], AES \\[46\\], and HPS \\[56\\]. Additionally, we propose two new metrics: Fscore and balance similarity (Bsim), to evaluate our ATIH method.\n\nCLIP-T and Dino-I measure similarity between the generated image and the object text or image, indicating how well the generated object incorporates characteristics of both sources. The balance similarity metric (Bsim) assesses the equilibrium between text and image fusion. The Fscore considers both higher CLIP-T and Dino-I, and the best balance. AES and HPS evaluate the aesthetic quality and human preference for the generated object.\n\nTable 1 shows that we achieve relatively high CLIP-T and Dino-I scores, though not the highest single scores. However, we obtain the best Bsim, Fscore, AES, and HPS. We also conducted a comprehensive user study to gather qualitative feedback on the perceived quality and creativity of the synthesized images.\n\n**Q2: Expanding dataset.**\n\n**A2:** Thank you for your valuable feedback regarding the dataset. We agree that testing the algorithm across a broader range of scenarios can provide more insights. Our dataset is created using an outer product on 60 different object texts and 30 different object images, representing widely-used categories. For example, the porcupine-bottle in Figure 1 illustrates the diverse text-image pairs.\n\nWe selected 30 images from the PIE-bench dataset, representing a diverse set of distinct categories, each with a clear and identifiable subject in Figure 10 and Table 5, including categories such as Corgi, jar, and man.\n\nFor the text categories, we utilized ImageNet and employed ChatGPT to filter and select 40 distinct animal categories and 20 non-animal categories, in Table 4. These include categories such as kit fox, peacock, acorn, and zucchini. This selection was intended to cover a broad range of typical scenarios while ensuring that the algorithm could effectively handle both common and varied text-image combinations.\n\n**Q3: Handling edge cases and unusual text-image pairs**\n\n**A3:** Thank you for your insightful comments on our method's performance with edge cases and unusual text-image pairs.\n\nTo address edge cases, our method effectively captures prominent features like colors and basic object forms from complex textual descriptions. Figure R1 shows a well-defined edge structure for the fawn image and the text 'Green triceratops with rough, scaly skin and massive frilled head.'\n\nHandling unusual text-image pairs is challenging. Our method relies on semantic correlations within the diffusion feature space. When the semantic match is weak, it tends to produce mere texture changes rather than deeper transformations. This suggests our approach may struggle with transformations between categories with weak semantic associations. Future work could enhance semantic matching between categories to improve generalizability. See Appendix B.\n\n**Q4: More detailed comparative analysis with existing methods.**\n\n**A4:** Thank you for your insightful feedback. The core of our method is an adaptive text-image harmony approach that balances the similarities between the generated object and the given text-image pair. Unlike existing methods like MasaCtrl, InfEdit, ConceptLab, and MagicMix, our approach ensures better fusion, especially in challenging cases such as the African chameleon-bird, minimizing distortions and maintaining high image quality.\n\nEven with manual adjustments to match our controls, InstructPix2Pix produced unnatural results. In Figure R3, even at optimal settings (image strength 1.5, text strength 5.0), InstructPix2Pix's fusion was still unnatural, such as replacing the rabbit's ears with the rooster's head.\n\n**Q5: Technical details of our method.**\n\n**A5:** Thank you for your suggestion.The technical details of our method are provided in Section 3 and Subsection 4.1, and summarized in Algorithm 1 in Appendix F.\n\nWe implemented our ATIH method using the SDXLturbo, achieving a processing time of ten seconds per sample. The input consists of a subject-specific object image and a fused object text.\n\nKey parameters in our experiments include:  \n- The balance parameter λ is set to 125.  \n- The scale factor k is set to 2.3.  \n- The similarity thresholds $I _{sim}^{min}$ and  $I _{sim}^{max}$ are set to 0.45 and 0.85, respectively. \n\n**Q6: User feedback integrated into algorithm refinement.**\n\n**A6:** Thank you for emphasizing the importance of user studies in our research. We clarify that user studies are used solely to evaluate the performance of our ATIH method, not to refine it.\nUser studies aim to evaluate and validate the performance of our ATIH method. These studies were designed to gather qualitative feedback on the novelty, coherence, and visual appeal of the generated images, rather than for algorithm refinement. We plan to incorporate it into subsequent versions to enhance usability and customization.\n\n**Q7: Statistical significance tests.**\n\n**A7:** Thank you for your suggestion to provide a more detailed comparative analysis. We conducted H tests to determine the statistical significance of performance differences between our method and other methods across various metrics. The results are presented in Table R1. For example, AES and HPS: Against Instructpix2pix, our method demonstrates statistically significant differences with H statistics of 268.57$(p < 10^{-59})$ for AES and 39.63 $(p < 10^{-9})$ for HPS, indicating potential improvements in aesthetic quality and human preference scoring. These results validate the effectiveness of our approach, demonstrating its superiority across multiple critical metrics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JI1lubz4eL", "reviewer": "Reviewer_yinU", "summary": "The paper proposes a new method for harmonized image generation conditioned on both text and image conditions, leading to better performance than baselines mentioned in the paper.", "questions": "Can the proposed method generate object whose shape is guided by the text prompt, while appearance/texture is guided by a reference image?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The idea of using scale factor to combine different conditions is straightforward and reasonable. It is also \n\nThe qualitative results shows better image fidelity compared to some previous image editing methods.", "weaknesses": "The optimal combination between image and text condition is subjective. In the paper, all the generate examples have similar depth/layout/structure with input image condition, while with modified texture/appearance guided by the text. Assume the user want to generate \"an image of a shark\" with color/pattern similar to a given dog image, i.e. which is different from the example in Figure 2. It is not clear whether the proposed method can achieve that.\n\nFurthermore,  some important baselines are missing:\n\n1. The depth/structure information from the input image can be extracted and injected into the generation process with a ControlNet-like model [1]. This baseline is important because there is little structural change shown in the results of proposed  methods, which is the scenario that the well-known ControlNet is good at;\n\n2. A naive way to generate with two conditions is using revised classifier-free guidance as [2]. The guidance strength from image and text can be controlled by hyper-parameters.\n\n3. Another naive baseline is to only use image guidance before timestep t , and use text guidance after t. Results with different t will be interesting.\n\n\nSubject-driven image generation and editing methods [3,4,5,6,7,8, 9] need to be discussed or at least mentioned, as these methods also aim to generate image guided by the reference images and text prompts. \n\n\n[1]. Adding Conditional Control to Text-to-Image Diffusion Models. Lvmin Zhang, Anyi Rao, Maneesh Agrawala.\n\n[2]. InstructPix2Pix: Learning to Follow Image Editing Instructions. Tim Brooks, Aleksander Holynski, Alexei A. Efros.\n\n[3]. An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion. Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, Daniel Cohen-Or.\n\n[4]. DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation. Nataniel Ruiz Yuanzhen Li Varun Jampani Yael Pritch Michael Rubinstein Kfir Aberman.\n\n[5]. Subject-driven Text-to-Image Generation via Apprenticeship Learning. Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming-Wei Chang, William W. Cohen.\n\n[6]. Customization Assistant for Text-to-image Generation. Yufan Zhou, Ruiyi Zhang, Jiuxiang Gu, Tong Sun.\n\n[7]. Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuning. Jian Ma, Junhao Liang, Chen Chen, Haonan Lu.\n\n[8]. Multi-Concept Customization of Text-to-Image Diffusion. Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, Jun-Yan Zhu.\n\n[9]. Kosmos-G: Generating Images in Context with Multimodal Large Language Models. Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, Furu Wei.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Results by the different assumption.**\n\n**A1:** Thank you for your suggestion. Inspired by \\[ref-1\\], we only inject the late self-attention layers in our method while keeping other settings the same. This simple adjustment enables our model to effectively modify the visual style of generated images while maintaining the desired structural features. In Figure R3, our method can create an image where the shape is guided by the text (\"shark\"), and the appearance (color/pattern) is influenced by the referenced image of a dog.\n\nThere are some key differences between our method and Visual Style Prompting (VSP) \\[ref-1\\]. First, our method introduces a new task that combines an object text with an object image to create a new object image, while VSP is a novel approach that guides the desired style using a reference image. Second, we propose ATIH to achieve a creative and harmonious fusion of two distinct objects, resulting in surprising and imaginative combinations. VSP aims to produce a diverse range of images while maintaining specific style elements and nuances. Third, while VSP primarily adjusts the visual style, our results demonstrate the novel combination of object text and object image in Figure R3.\n\n\\[ref-1\\] Visual Style Prompting with Swapping Self-Attention. arXiv preprint arXiv:2402.12974.\n\n**Q2: Results by using ControlNet-like model to inject structural information.**\n\n**A2:** Thank you for the opportunity to clarify the performance of our method compared to ControlNet, particularly in terms of achieving a balanced and semantically coherent image synthesis.\n\n**Comparison with ControlNet:**  \nWe rigorously compared our method with ControlNet to evaluate their capabilities in handling complex text-image fusion tasks in Figure R2. Our findings indicate significant differences in how both approaches manage the fusion process. ControlNet tends to maintain the structure from depth or edge maps well but struggles with semantic integration, especially when faced with complex prompts, failing to achieve a seamless blend. By contrast, our method utilizes the full spectrum of RGB image features, including color and texture, alongside structural data.\n\n**Q3: Results by controlling image and text strengths.**\n\n**A3:** Thank you for highlighting the relevance of adjusting the guidance strength in image and text synthesis, as seen in methods like InstructPix2Pix.\n\n**Comparison with InstructPix2Pix:**  \nWe manually adjusted the image and text strengths for InstructPix2Pix to match the controls of our method. We varied the image strength (e.g., 1.0, 1.5, 2.0) and text strength from 1.5 to 7.5 and observed the outcomes. In Figure R3, at optimal settings of image strength 1.5 and text strength 5.0, InstructPix2Pix produced the best fusion, but it is unnatural, such as replacing the rabbit's ears with the rooster's head. In contrast, our results are new and natural combinations of the rabbit and the rooster, demonstrating superior visual synthesis compared to InstructPix2Pix.\n\nMore importantly, our method can automatically iterate to the optimal fusion of image and text, eliminating the need for manual parameter adjustments. This automation significantly enhances the usability and efficiency of our synthesis process, allowing for seamless generation of harmonious images that meet both aesthetic and semantic criteria.\n\n**Q4: Baseline using only image guidance before t, text guidance after t.**\n\n**A4:** Thank you for suggesting the exploration of different guidance strategies.\n\n**Results and Analysis:**  \nFigure R6 illustrates the transition and effects of this guidance strategy: At \\(t=0\\), the output is purely influenced by the text, which in this case describes an ostrich. Starting at \\(t=1\\), image guidance is introduced, leading to a more complex synthesis process where the features of the deer image begin to merge with the previously established text-based ostrich features.\n\nFigure R6 shows the results that the integration is not fully balanced, as the distinctive features of the ostrich—such as its neck and feather texture—are not as prominent as desired. This indicates a need for more refined control over the balance and timing of text and image guidance.\n\n**Q5: Discussing subject-driven generation methods.**\n\n**A5:** Thank you for mentioning the subject-driven methods. Recently, subject-driven text-to-image generation focuses on creating highly customized images tailored to a target subject \\[3,4,5,6\\]. These methods often address the task, such as multiple concept composition, style transfer, and action editing \\[7,8,9\\]. In contrast, our approach aims to generate novel and surprising object images by combining object text with object images.\n\n**Comparison with Kosmos-G:**  \nKosmos-G \\[9\\] utilizes a single image input and a creative prompt to merge with specified text objects. The prompt is structured as “\\<i\\> creatively fuse with {object text},” guiding the synthesis to innovatively blend image and text elements. Our findings indicate that Kosmos-G can sometimes struggle to maintain a balanced integration of original image features and text-driven attributes. In Figure R4, the images generated by Kosmos-G often exhibit a disparity in feature integration.\n\n**Q6: Can the method use text for shape and image for texture?**\n\n**A6:** Thanks. Altering the shape of a given object image is quite challenging. To the best of our knowledge, there is no method that performs this task well. Fortunately, our method can generate a new object with slight deformations, such as the axolotl's mouth in Figure 2. However, our method still cannot significantly deform the shape of the object image, as we incorporate all self-attention to preserve image information during the diffusion process. We will study this large deformation in the future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "guYmCPq3Ey", "reviewer": "Reviewer_EP4q", "summary": "This paper aims to generate novel objects based on a reference image and a conditional text prompt (e.g., a bottle (image) with a penguin (text) outlook). To this end, a method called Adaptive Text-Image Harmony (ATIH) is proposed to better align the conditional image and text. Experiments show that ATIH yields better results compared to existing state-of-the-art methods.", "questions": "It'd be interesting if the authors can discuss how to make a better evaluation for this task. Overall, I think this task is ambiguous and difficult to evaluate (e.g., if you want to turn a cat into a peacock, is it better to (1) completely replace the cat with the peacock, or (2) yield a combination of the cat and the peacock?) How can we define which is a 'better alignment' with the user's intention? (This is an open-ended question and will not affect my recommendation for this paper).", "limitations": "The authors did briefly discuss the limitations in the Supplementary.\n\nAt this moment, I'd recommend this paper for acceptance, as I believe the task introduced by this paper outweighs the limitations in experiments and ablation studies.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Overall, the paper is easy to follow.\n- Experiments in the main paper and supplementary materials try to be as thorough as possible to cover the most promising frameworks (e.g., ConceptLab) and ablation studies.\n- On a high-level idea of fine-grained controllable image generation, I think this topic will attract interest in the image editing/image manipulation community.\n- While the idea of manipulating attention layers of diffusion models is not new, I think this paper finds a way to (1) preserve the structure of the reference image as much as possible while (2) editing the image toward the given text -- which I consider a main contribution of this paper.", "weaknesses": "- As far as I understand, the image for text information in Figure 2 (e.g., \"iron\", \"white shark\") is for visualization purposes only. Actually, I think this might mislead readers (e.g., models can take two reference images as input).\n- I think one thing the authors must consider is adjusting the strengths of existing baselines (e.g., InstructPix2Pix). If baselines allow for controlling the strength of the edit, it would be fairer if we had comparisons to that (e.g., similar to Figure 16, but for baselines).\n- While in the main paper, the Golden Section Search algorithm seems to play an important role, it's unclear how important this algorithm is (e.g., can we simply replace this with simple average values). In other words, I think the current paper lacks an ablation study for this module.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Remove the misleading images.**\n\n**A1:** Thank you for your feedback regarding the potential misunderstanding in Figure 2. We acknowledge that the image for text information was intended for visualization purposes only, and we agree that it could mislead readers. Therefore, we will remove the images and present only the text in the same location.\n\n**Q2: Adjusting baseline strengths of InstructPix2Pix.**\n\n**A2:** Thank you for your suggestion. We agree that adjusting the strengths of existing baselines, such as InstructPix2Pix, is essential. Using 'cock' (text) and 'rabbit' (image), we manually adjusted the image strength (e.g., 1.0, 1.5, 2.0) and text strength (e.g., 1.5, 2.0, 2.5, 4.0, 4.5, 5.0, 6.5, 7.0, 7.5) for InstructPix2Pix, and the results are shown in Figure R3. Using the optimal image and text strengths of 1.5 and 5.0, respectively, InstructPix2Pix generates unnatural combinations, such as the head of the cock replacing the ears of the rabbit, as seen on the left of Figure R3. In contrast, our method produces natural and novel combinations, with the head and feet of the rabbit fused correspondingly with the head and feet of the cock, in the right of Figure R3.\n\n**Q3: Ablation study of Golden Section Search.**\n\n**A3:** Thank you for your insightful feedback regarding the role of the Golden Section Search algorithm.\n\n**Role of Golden Section Search:**  \nIn our approach, we designed a score function to evaluate the quality of the text-image fusion. We utilize the Golden Section Search algorithm to efficiently find the optimal parameters that maximize our score function, allowing us to achieve a well-balanced and harmonious fusion of text and image features.\n\n**Alternative Search Methods:**  \nWhile the Golden Section Search is our chosen method for parameter optimization due to its efficiency and simplicity, we acknowledge that other methods, such as Ternary search \\[ref-1\\] or even Random search \\[ref-2\\], could potentially achieve similar results. The primary advantage of the Golden Section Search is its ability to converge to an optimal solution with relatively few iterations. In our task, it reaches convergence two iterations faster than the Ternary Search.\n\n**Importance and Flexibility:**  \nThe choice of search algorithm is flexible, and the primary contribution is the score function itself, which guides the search for optimal fusion parameters. The Golden Section Search provides a systematic approach, but its role can be substituted by other search methods if desired.\n\n**Ablation Study:**  \nWe conducted an ablation study comparing the Golden Section Search and Ternary Search, as shown in Figure R7. The images produced using the Golden Section Search exhibit a more novel fusion effect.\n\n\\[ref-1\\] Kiefer, J. (1953). Sequential Minimax Search for a Maximum. Proceedings of the American Mathematical Society, 4(3), 502-506.\n\n\\[ref-2\\] Solis, F. J., & Wets, R. J. B. (1981). Minimization by Random Search Techniques. Mathematics of Operations Research, 6(1), 19-30.\n\n**Q4: Discussing evaluation criteria for ambiguous tasks.**\n\n**A4:** Thank you for your understanding. We acknowledge that evaluating (1) complete replacement and (2) harmonious combination is challenging. For complete replacement, we can consider two metrics: the difference between the given object text and its generated object patch, and the harmony and naturalness of the entire image replaced by the given object text. To measure the difference, we can use CLIP text-image similarity (CLIP-T). For evaluating harmony and naturalness, we can employ the aesthetic score (AES) and the human preference score (HPS).\n\nFor harmonious combination, our goal is to generate an object that incorporates characteristics of both A and B. If the generated object is unbiased towards either A or B, it indicates a balance between them. To achieve this, we first use CLIP text-image similarity (CLIP-T) and Dinov2 image similarity (Dino-I) to ensure the generated object has high similarities with both A and B. We then introduce a balance similarity metric (\\(B\\)sim) to measure the equilibrium between A and B. If the generated object shows a bias towards either, we adjust it using a scale factor \\(k\\).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AjuC6hZDSO", "reviewer": "Reviewer_VGMU", "summary": "The paper works on novel object synthesis, namely, given an image and text prompt, the proposed method can generate a new image that contains the visual features from the given image and the textual information from the given prompt. The paper proposed a method names Adaptive Text-Image Harmony to tackle the task. The authors provide both qualitative and quantitate comparisons to show the effectiveness of the method.", "questions": "Please refer to the weakness section.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is easy to follow.\n\nThe author proposed an effective method that solves the challenging problem of novel object synthesis.\n\nThe paper provides extensive experiments, along with human evaluation, to justify the effectiveness of the approach.\n\nAuthors also provide detailed ablation for the design of the approach.", "weaknesses": "For the constructed dataset with 1,800 text-image pairs, will the authors release it? Could this dataset be used to fine-tune other baseline approaches to improve performance? If so, has this experiments been done?\n\nFor the baseline methods, e.g., InstructPix2Pix, did the authors directly use the released model or train a new one using the same text-to-image model as this work, which is SDXL-Turbo?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Dataset and fine-tune baselines.**\n\n**A1:** Thank you for your interest in our dataset and its potential applications.\n\n**Dataset Release:**  \nWe plan to release a dataset of 1,800 text-image pairs to the research community following acceptance. These pairs are created using an outer product on 60 different object texts and 30 different object images. Specifically, we selected 60 texts from the ImageNet dataset \\[43\\], detailed in Table 4, including categories such as kit fox, peacock, African chameleon, white shark, acorn, zucchini, and fire engine. The 30 images were chosen from the PIE-bench dataset \\[25\\], shown in Fig. 10, with each image corresponding to similarly categorized texts outlined in Table 5, such as Corgi, Duck, ladybug, flower vase, apple, jar, man, and Twitter logo. More details are provided in Appendix C. It should be noted that there are no ground truths for novel and surprising objects in this dataset created from text-image pairs.\n\n**No fine-tuning for all baselines and our ATIH method:**  \nDue to the absence of ground truths, this dataset cannot be used to train any baselines or our ATIH method. Instead, it is solely for evaluating performance using four key popular metrics: CLIP text-image similarity (CLIP-T) \\[39\\], Dinov2 image similarity (Dino-I) \\[36\\], aesthetic score (AES) \\[46\\] and human preference score (HPS) \\[56\\], and our two proposed metrics: Fscore and balance similarities (Bsim). Our ATIH method can generate novel and surprising objects by designing an adaptive balance between the provided object text and object image.\n\n**Q2: Using released or SDXL-Turbo-trained models.**\n\n**A2:** Thanks. We directly used the released models for the baseline methods, including InstructPix2Pix, due to the following reasons. First, our dataset lacks ground truths for novel and surprising objects created from text-image pairs, preventing us from training or fine-tuning these baseline models. Second, using the officially released models allows us to provide a direct and unbiased comparison, as these models have been trained and optimized by their developers. Third, we employed the officially released SDXL-Turbo \\[45\\] as a base model to create novel and surprising objects using our ATIH method, since no ground truths were used to train the SDXL-Turbo model.\n\nAdditionally, we manually adjusted the image and text scales of InstructPix2Pix, as shown in Figure R3. Our ATIH method demonstrates significantly better visualization compared to InstructPix2Pix, which displays objects with unnatural combinations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers and chairs for their time, constructive comments, and recognition of our work. We sincerely hope that all reviewers can support our work, as this paper proposes a novel and reasonable method (**Reviewers xaqc and yinU**) to solve an interesting and challenging task (**Reviewers EP4q and VGMU**) and produces extensive, high-quality, innovative results (**Reviewers xaqc, yinU, VGMU**). Special thanks to \( **Reviewers EP4q**) for recognizing that the significance of the task introduced by this paper outweighs the limitations in experiments and ablation studies.

Our **Main Contribution** lies in the following three parts. 

1. To the best of our knowledge, we are the first to propose an adaptive text-image harmony method for generating novel object synthesis. Our key idea is to achieve a balanced blend of object text and image by adaptively adjusting a scale factor and an injection step in the inversion diffusion process, ensuring effective harmony.

2. We introduce a novel similarity score function that incorporates the scale factor and injection step. This function aims to balance and maximize the similarities between the generated image and the input text/image, achieving a harmonious integration of text and image.

3. Our approach demonstrates superior performance in creative object combination compared to state-of-the-art image-editing and creative mixing methods. Examples of these creative objects include *sea lion-glass jar*, *African chameleon-bird*, and *mud turtle-car*.

Meta-review: The paper introduces the Adaptive Text-Image Harmony (ATIH) method for generating novel object images by balancing text and image inputs in diffusion models. Reviewers appreciated the introduction of a scale factor and similarity score function but raised concerns about the lack of theoretical justification and the limited generalizability of the experimental results. In response, the authors provided additional quantitative metrics, expanded the dataset description, and compared their method with more baselines, such as ControlNet and InstructPix2Pix. They also committed to releasing the full dataset for future research. With generally positive ratings (5, 5, 5, 6), the paper is recommended for acceptance, pending the suggested revisions and dataset release, provided the authors make the suggested revisions and release the dataset as stated.
Decision: Accept (poster)