Novel Object Synthesis via Adaptive Text-Image Harmony

Authors: zeren  xiong
zedong  zhang
zikun  chen
shuo  chen
xiang  li
gan  sun
jian  yang
jun  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 08:45:02

Abstract: In this paper, we study an object synthesis task that combines an object text with an object image to create a new object image. However, most diffusion models struggle with this task, \textit{i.e.}, often generating an object that predominantly reflects either the text or the image due to an imbalance between their inputs. To address this issue, we propose a simple yet effective method called Adaptive Text-Image Harmony (ATIH) to generate novel and surprising objects.
First, we introduce a scale factor and an injection step to balance text and image features in cross-attention and to preserve image information in self-attention during the text-image inversion diffusion process, respectively. Second, to better integrate object text and image, we design a balanced loss function with a noise parameter, ensuring both optimal editability and fidelity of the object image. Third, to adaptively adjust these parameters, we present a novel similarity score function that not only maximizes the similarities between the generated object image and the input text/image but also balances these similarities to harmonize text and image integration.  
Extensive experiments demonstrate the effectiveness of our approach, showcasing remarkable object creations such as colobus-glass jar. https://xzr52.github.io/ATIH/

Main paper: 
 introduction: 
 Image synthesis from text or/and image using diffusion models such as Stable Diffusion [51] , SDXL [56] , and DALL•E3 [43] has gained considerable attention due to their impressive generative capabilities and practical applications, including editing [6; 75] and inversion [24; 61] . Many of these methods focus on object-centric diffusion, utilizing textual descriptions to manipulate objects within images through operations like composition [58] , addition [36; 15] , removal [63] , replacement [7] , movement [29] , and adjustments in size, shape, action, and pose [17] . In contrast, we study an object synthesis task that creates a new object image by combining an object text with an object image. For instance, combining kingfisher (image) and terrier (text) results in a new and harmonious terrier-like kingfisher object, as shown in the right-side of Fig. 2 .To implement object text-image fusion, most diffusion models, such as SDXL-Turbo [56] , often use cross-attention [24] to integrate the input text and image. However, the cross-attention frequently results in imbalanced outcomes, as evidenced by the following observations. On the left side of Fig. 2 , when inputting an axolotl (image) and a toucan (text), SDXL-Turbo only generates an image of a toucan, showing a bias towards the toucan text (green circles). Conversely, when inputting a rooster (image) and an iron (text), it produces an image of a rooster, which closely resembles the original rooster image (orange circles). These observations reveal that the text (or image) feature often suppresses the influence of the image (or text) feature during the diffusion process, leading to a failed fusion. To mitigate the image degeneration, Plug-and-Play [61] can inject the guidance image features into self-attention. Unfortunately, even with the application of the best inversion editing method, PnPinv [27] , which incorporates the plug-and-play inversion into diffusion-based editing methods for improved performance, we still observe similar imbalances, as shown on the right-side of Fig. 2 . This arises an important problem: how can we balance object text and image integration?To address this problem, we propose an Adaptive Text-Image Harmony (ATIH) method for novel object synthesis, as shown in Fig. 3 . First, during the inversion diffusion process, we introduce a scale factor α to balance text and image features in cross-attention, and an injection step i to preserve image information in self-attention for adaptive adjustment. Second, the inverted noise maps adhere to the statistical properties of uncorrelated Gaussian white noise, which increases editability [46] . However, they are preferable for approximating the feed-forward noise maps, thereby enhancing fidelity. To better integrate object text and image, we treat sampling noise as a parameter in designing a balanced loss function, which strikes a balance between reconstruction and Gaussian white noise approximation, ensuring both optimal editability and fidelity of the object image. Third, we present a novel similarity loss that considers both i and α. This loss function not only maximizes the similarities between the generated object image and the input text/image but also balances these two similarities to harmonize text and image integration. Furthermore, we employ the Golden Section Search [47] algorithm to quickly find the optimal parameters α and i. Therefore, our ATIH method is capable of generating novel object combinations. For instance, an iron-like rooster is produced by merging the image rooster with the text iron, resulting in a rooster image with an iron texture, as shown in Fig. 2 .Overall, our contributions can be summarized as follows: (1) To the best of our knowledge, we are the first to propose an adaptive text-image harmony method for generating novel object synthesis. The key idea is to achieve a balanced blend of object text and image by adaptively adjusting a scale factor and an injection step in the inversion diffusion process, ensuring their effective harmony.(2) We introduce a novel similarity score function that incorporates the scale factor and injection step. This aims to balance and maximize the similarities between the generated image and the input text/image, achieving a harmonious integration of text and image. (3) Experimental results on PIE-bench [26] and ImageNet [53] demonstrate the effectiveness of our method. Our approach shows superior performance in creative object combination compared to state-of-the-art image-editing and creative mixing methods. Examples of these creative objects, such as sea lion-glass jar, African chameleon-bird, and corgi-cock are shown in Figs. 1, 6 , and 8. 
related work: 
 Text-to-Image Generation The rapid development of generative models based on diffusion processes has advanced the state-of-the-art for tasks [12; 21; 33] like text-to-image synthesis [22; 31] , image editing [64; 2] , and style transfer [65; 23; 35] . Large-scale models such as Stable Diffusion [51] , Imagen [55] , and DALL-E [49] have demonstrated remarkable capabilities. Sdxlturbo [56] introduced a distillation method that further enhances efficiency by reducing the steps needed for high-quality image generation. Our method utilizes Sdxlturbo for adaptive and innovative object fusion, preserving the original image's layout and details while requiring only the textual description of the target object.Text Guided Image Editing. Diffusion models have garnered significant attention for their success in text-to-image generation and text-driven image editing using natural language descriptions. Early studies [1; 54; 70; 40] , such as SDEdit [40] , balanced authenticity and fidelity by adding noise, while Prompt2Prompt [24] and Plug-and-Play (PNP) [61] enhanced editing through attention mechanisms. Further research, including MasaCtrl [5] , Instructpix2pix [4] , and InfEdit [69] , explored non-rigid editing, specialized image editing models, and rapid editing via consistency sampling. Advances in image inversion and reconstruction [20] have focused on diffusion-based denoising process inversion, categorized into deterministic and non-deterministic sampling [28] . Deterministic methods, such as Null-text inversion using DDIM sampling [41] , precisely recover original images but require lengthy optimization; non-deterministic methods, such as DDPM inversion [25] and CycleDiffusion [67] , achieve precision by storing variance noise. PnPinv [26] simplifies the process by accurately replacing latent features during denoising, achieving perfect reconstruction but with weaker editability.We propose a framework for creative object synthesis using object textual descriptions for effective fusion and a regularization technique to enhance PnPinv editability.Object Composition. Compositional Text-to-Image synthesis and multi-image subject blending methods [37; 19; 58; 70; 59] aim to create novel images by integrating various concepts, including object interactions, colors, shapes, and attributes. Numerous methodologies [8; 71; 24; 52; 55] have been developed focusing on object combinations, context integration, segmentation, and text descriptions. However, these methods often merely assemble components without effectively melding inter-object relationships, resulting in compositions that, while accurate, lack deeper integration and interaction. This limitation is particularly evident in image editing, where multiple objects in a single image fail to achieve cohesive synthesis. Our method addresses this by harmoniously fusing two objects to create novel entities, thereby enhancing creativity and imagination.Semantic Mixing. The breadth of creativity spans diverse fields, from scientific theories to culinary recipes, driving advancements in AI as highlighted by scholars [3] [39] and recent researchers [62] [32] . This creativity has led to significant innovations in AI, particularly through generative models. Creative Adversarial Networks [16] push traditional art boundaries, producing norm-defying works Figure 3 : Framework of our object synthesis incorporating a scale factor α, an injection step i and noise ϵ t in the diffusion process. We design a balance loss for optimizing the noise ϵ t to balance object editability and fidelity. Using the optimal noise ϵ t , we introduce an adaptive harmony mechanism to adjust α and i, balancing text (Peacock) and image (Rabbit) similarities.while maintaining artistic connections. Efforts to adapt AI for novel engineering designs [11] further exemplify this technological creativity. MagicMix [34] introduced semantic mixing task,unlike traditional style transfer methods [73; 60; 10] which blending two concepts into a photo-realistic object while retaining the original image's layout and geometry, but often resulting in biased images and less harmonious fusion. ConceptLab [50] uses diffusion models to generate unique concepts, like new types of pets, but requires time-consuming optimization and struggles to semantically blend real images. Our method operates at the attention layer of diffusion models for harmonious semantic fusion and proposes an adaptive fast search to quickly produce balanced, fused images, ensuring novel and cohesive integration of semantic concepts.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Yes, refer to Appendix refsec:limitation Limitation.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methodology: 
 Let O I and O T be an object image and an object text, respectively, used as inputs for diffusion models. Our goal is to create a novel object image O by combining O I with O T during the diffusion process. To achieve this goal, we develop an adaptive text-image harmony (ATIH) method in our object synthesis framework, as shown in Fig. 3 . In subsection 3.1, we introduce a text-image diffusion model with a scale factor α, an injection step i and noise ϵ t . In subsection 3.2, we present to optimize the noise ϵ t to balance object editability and fidelity. In subsection 3.3, we propose a simple yet effective ATIH method to adaptively adjust α and i for harmonizing text and image.Here, we construct a Text-Image Diffusion Model (TIDM) by utilizing the pre-trained SDXL Turbo [56] . The key components include dual denoising branches: inversion for inverting the input object image, and fusion for fusing the object text and image. Following the latent diffusion model [51] , the input latent codes are defined as z 0 = E(O I ) for object image O I and τ = E(O T ) for object text O T , using a pre-trained image/text encoder E(•). τ N = E(O N ) denotes as a null-text embedding. The latent denoising process is described as follows:Inversion Denoising. The inversion denoising process predicts the latent code at the previous noise level, z t-1 , based on the current noisy data z t . This process is defined as:EQUATIONwhere ν t , β t and γ t are sampler parameters, ϵ t is sampled noise, and ϵ θ ( z t , t, τ ) is a pre-trained U-Net model [56] with self-attention and cross-attention layers. The self-attention is implemented as:EQUATIONwhere Q s t , K s t and V s t are the query, key and value features derived from the representation z t , and d is the dimension of projected keys and queries. The cross-attention is to control the synthesis process through the input null-text embedding τ N , implemented as follows: Cross-AttnQ c t , K c t , V c t = M c t • V c t , where M c t = Softmax Q c t ( K c t ) T / √ d , Q ct is the query feature derived from the output of the self-attention layer, K c t and V c t are the key and value features derived from τ N . Fusion Denoising. Similar to the inversion denoising branch, we redefine the self-attention and crossattention for easily adjusting the balance between the image latent code z t and the text embedding τ . The fusion denoising process is redefined as:EQUATIONwhere ν t , β t , γ t and ϵ t are defined as Eq. ( 1), and ϵ θ (z t , t, τ, α, i) is also the pre-trained U-Net model [56] with injected self-attention and scale cross-attention layers. The injected self-attention with an adjustable injection step i(0 ≤ i ≤ T ) is implemented as:EQUATIONwhere Q s t , K s t and V s t are the query, key and value features derived from the representation z t . Unlike the approach of injecting K s t and V s t from Eq. (2) into K s t and V s t in MasaCtrl [5] , we focus on adjusting the injection step i by injecting M s t from Eq. (2) into M s t . The scale cross-attention with an adjustable factor α ∈ [0, 2] is to control the synthesis process through the input text embedding τ , implemented as follows:EQUATIONwhere Q c t is the query feature derived from the output of the self-attention layer, K c t and V c t are the key and value features derived from the text embedding τ . Unlike the non-adjustable scale attention map approach in Prompt-to-Prompt [24] , we introduce a factor, α, to adjust the value feature. This allows for better balancing of the text and image features, even though they share the same form. Using this fusion denoising process, the generation of a new object image is denoted as O. [20] , based on the denoising Eq. ( 1) and the approximation ϵ θ ( z t , t, τ ) ≈ ϵ θ ( z t-1 , t, τ ) [14] , the noise addition process is reformulated as:EQUATIONIn this subsection, our goal is to achieve better fidelity and editability of the object image during the inversion process. We observe that increasing the Gaussian white noise of the denoising latent code z t-1 can enhance editability [46] , while reducing the difference between the denoising latent code z t-1 and the standard path noise code z ′ t-1 in Eq. ( 6) can improve fidelity [25; 67] . However, these two objectives are contradictory. To address this, we treat the sampling noise ϵ t in Eq.( 1) as a learnable parameter. We define a reconstructed ℓ 2 loss between z t-1 and z′ t-1 , L r (ϵ t ) = ∥ z ′ t-1 -(ν t z t + β t ϵ θ ( z t , t, τ ) + γ t ϵ t )∥,and a KL divergence loss between ϵ t and a Gaussian distribution, L n (ϵ t ) = KL(q(ϵ t )||p(N (0, I))), to simultaneously handle fidelity and editability. Based on Eqs.( 1) and ( 6), we design a balance loss function as follows:EQUATIONwhere λ represents the weight to balance L r and L n , and in this paper, we set to λ = Lr Ln = 125. Since the parameter ϵ t is sampled from a standard Gaussian distribution during the noise addition process, L n is used solely to balance L r and its gradient is not computed for optimization.Using the optimal noise ϵ t , a fused object image O(α, i) can be generated by the TIDM with an initial scale factor α 0 = 1 and injection step i 0 = ⌊T /2⌋ from the input object image O I and object text In this paper, we compute the similarities I sim (α, i) and T sim (α, i) using the DINO features [44] and the CLIP features [48] , respectively, based on a cosine distance d. Our key idea is to balance and maximize both I sim (α, i) and T sim (α, i) for optimal text-image fusion.𝑘𝑘 𝑚𝑚𝑚𝑚𝑚𝑚 = ⁄ 𝐼𝐼 α 1 𝑇𝑇 α 1 𝑘𝑘 𝑚𝑚𝑚𝑚𝑚𝑚 = ⁄ 𝐼𝐼 α 2 𝑇𝑇 α 2 𝑘𝑘 𝑚𝑚𝑚𝑚𝑚𝑚 𝑘𝑘 𝑚𝑚𝑚𝑚𝑚𝑚 ≥ 𝑘𝑘 ≥ 𝛼𝛼 1 𝛼𝛼 2Adjust injection step i to balance fidelity and editability. Before achieving the idea, we first enable the object image to be smoothly editable by adjusting the injection step i in the injected self-attention. We denote I sim (i) = I sim (α 0 , i) for for convenience. In the inversion process, it is generally observed that more injections lead to less editability. When all injections are applied (i = T ), an ideal fidelity is achieved. We observe that when I sim (i) < I min sim , the fused image deviates significantly from the input image, resulting in a loss of fidelity. Conversely, when I sim (i) > I max sim , the fused image is too similar to the input image, resulting in no editability. To balance fidelity and editability, I sim (i) must satisfy I min sim ≤ I sim (i) ≤ I max sim , in Fig. 5 . Therefore, initializing i = ⌊T /2⌋, i is adjusted as follows:EQUATIONwhere I min sim and sim max are set to 0.45 and 0.85 in this paper, respectively, based on observations from Fig. 17 . After using Eq. ( 8), this adaptive approach can obtain an injection step i * to smooth the fusion process while maintaining a balance between fidelity and editability. Fixing the injection step i = i * , next we use abbreviations, I sim (α) = I sim (α, i * ) and T sim (α) = T sim (α, i * ).Adaptively adjust the scale factor α for harmonizing text and image. To implement our key idea, we design an exquisite score function with α as:EQUATION)where β is a weighting factor, and the parameter k is introduced to mitigate inconsistencies in scale between high I sim (α) and low T sim (α) due to differences in text and image modalities, ensuring their scale balance. As shown in Fig. 4 , I sim (α) decreases and T sim (α) increases as α increases, and vice versa. Based on these observations, we set k = 2.3 and β = 1 in this paper.In Eq. ( 9), the left-hand side represents the sum of the text and image similarities, forming an ellipse, while the right-hand side represents the absolute value of the difference between the text and image similarities, forming a hyperbola. A larger sum value indicates that the generated image integrates more information from the input text and image. Conversely, a smaller absolute value signifies a better balance between the text and image similarities. Additionally, given that I sim (α) ∈ [0, 1] and T sim (α) ∈ [0, 1], their sum is greater than or equal to the absolute value of their difference, leading to F (α) ≥ 0. Therefore, our objective is to maximize F (α) to simultaneously enhance and balance both I sim (α)and and T sim (α). Maximizing F (α) is easily implemented by the Golden Section Search [47] algorithm, and we get the optimal α * . Fig. 5 We observe that InfEdit [69] MasaCtrl [5] and InstructPix2Pix [4] struggle to fuse object images and texts, while our method successfully implements new object synthesis, such as bowling ball-fawn in the second row.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Justification: the paper does not include theoretical results.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experimental settings: 
 Datasets. We constructed an object text-image fusion (OTIF) dataset consisting of 1,800 text-image pairs, derived from 60 texts and 30 images in Appendix C. Images, selected from various classes in PIE-bench [26] , include 20 animal and 10 non-animal categories. Texts were chosen from the 1,000 classes in ImageNet [53] , with ChatGPT [42] filtering out 40 distinct animals and 20 non-animals.Details. We implemented our method on SDXLturbo [56] only taking ten seconds. For image editing, we set the source prompt p s as an empty string "Null" and the target prompt P t as the target object class name. During sampling, we used the Ancestral-Euler sampler [28] with four denoising steps. All input images were uniformly scaled to 512 × 512 pixels to ensure consistent resolution in all the experiments. Our experiments were conducted using two NVIDIA GeForce RTX 4090 GPUs.Metrics. To comprehensively evaluate the performance of our method, we employed four key metrics: aesthetic score (AES) [57] , CLIP text-image similarity (CLIP-T) [48] , Dinov2 image similarity (Dino-I) [44] , and human preference score (HPS) [68] . Following the Eq. ( 9), F score and balance similarities (Bsim) with k = 2.3 are used to measure the text-image fusion effect.We conducted a comprehensive comparison of our ATIH model with three image-editing models (i.e., MasaCtrl [5] , InfEdit [69] , and InstructPix2pix [4] ), two mixing models (i.e., MagicMix [34] and ConceptLab [50] ), and ControlNet [72] . Notably, MagicMix and ConceptLab share a similar objective with ours to fuse object text/image, while ConceptLab only accepts two text prompts as its inputs. Due to no available code for MagicMix, we utilized its unofficial implementation [13] . We observe that our results surpass those of MagicMix [34] . For ConceptLab [50] , we exclusively examine its fusion results without making good or bad comparisons, as it is a distinct approach to creative generation.Comparisons with image-editing methods. For a fair comparison, we uniformly set the editing text prompt in all methods as a photo of an {image category} creatively fused with a {text category} to achieve the fusion of two objects. Fig. 6 visualizes some combinational objects, with additional results available in Appendix H. Our observations are as follows: Firstly, MasaCtrl and InfEdit generally preserve the original image's details better during editing, as seen in examples like sheep-triceratops. In contrast, InstructPix2Pix tends to alter the image more significantly, making it closer to the edited text description. Secondly, different methods exhibit varying degrees of distortion when fusing two objects during the image editing process. For instance, in the case of African chameleon-bird, our method performs better by minimizing distortions and maintaining the harmony and high quality of the image. Thirdly, our method shows significant advantages in enhancing the editability of images. For the European fire salamander-glass jar example, other methods often result in only color changes and slight deformations, failing to effectively merge the two objects. In contrast, our method harmoniously integrates the colors and shapes of both the glass jar and the European fire salamander, significantly improving the editing effect and operability. Specially, Fig. 7 shows the results of InstructPix2Pix with manually adjusted image strengths (1.0, 1.5, 2.0) and text strengths (ranging from 1.5 to 7.5). At optimal settings of image strength 1.5 and text strength 5.0, InstructPix2Pix produced its best fusion, though some results were unnatural, like replacing the rabbit's ears with a rooster's head. In contrast, our method created novel and natural combinations of the rabbit and rooster by automatically achieving superior visual synthesis without manual adjustments.Comparisons with the mixing methods. Fig. 8 illustrates the results of text-image object synthesis. We observe that both MagicMix and ConceptLab tend to overly bias towards one class, such as zucchini-owl and corgi-cock. Their generated images often lean more towards one category. In contrast, our method achieves a more harmonious balance between the features of the two categories. Moreover, the fusion images produced by MagicMix frequently exhibit insufficiently smooth feature blending. For instance, in the fusion of a rabbit and an emperor penguin, the rabbit's facial features nearly disappear. Conversely, our method seamlessly merges the facial features of both the penguin and the rabbit in the head region, preserving the main characteristics of each.Comparisons with ControlNet. We rigorously compared our method with ControlNet to assess their performance in complex text-image fusion tasks, as shown in Fig. 9 . Our results highlight notable differences: ControlNet preserves structure well from depth or edge maps but struggles with semantic integration, especially with complex prompts, often failing to achieve seamless blending. In contrast, our method leverages full RGB features, including color and texture, alongside structural data. Quantitative Results. Table 1 displays the quantitative results, illustrating that our method achieves state-of-the-art performance in AES, HPS, F score and Bsim, surpassing other methods. These results indicate that our approach excels in enhancing the visual appeal and artistic quality of images, while also aligning more closely with human preferences and understanding in terms of object fusion. Moreover, when dealing with text-image inconsistencies at scale k=2.3, our method achieves superior text-image similarity and balance, demonstrating superior fusion capability. Despite achieving the best DINO-I and CLIP-T scores under inconsistencies, InfEdit and InstructPix2Pix perform worse than our method in terms of AES, HPS, F score and Bsim, and their visual results remain sub-optimal. These inconsistencies ultimately lead to the failure of integrating object text and image. In contrast, our approach achieves a better text-image balance similarities. Furthermore, Table 2 presents the H-statistics [30] and P -values [66] assessing the statistical significance of performance differences between our ATIH and other methods across various metrics. Compared to Instructpix2pix, for instance, our method shows significant differences, with H-statistics of 268.57 for AES and 39.63 for HPS, indicating potential improvements in both aesthetic quality and human preference scoring.User Study. We conducted two user studies to assess intuitive human perception of results presented in Table 3 , Table 4 , and Appendix G. Each participant evaluated 6 image-editing sets and 6 fusion sets. In total, these studies garnered 570 votes from 95 participants. Our method received the highest ratings in both studies, capturing 74.03% and 79.47% of the total votes, respectively. Among the image-editing methods, InfEdit [69] garnered 14.7% of votes for its superior editing performance, while InstructPix2Pix [4] and MasaCtrl [5] received only 8% and 2.8%, respectively. In the fusion category, ConceptLab [50] received 12.28% of votes, while MagicMix [34] received 8%. Parameter analysis. Our primary parameters include λ in ( 7), I min sim and I max sim in (8) , and k in ( 9). λ = Lr Ln balances the editability and fidelity of our model. We determined the specific value through personal observation combined with the changes in AES, CLIP-T, and Dino-I values at different λ settings. Ultimately, we set λ to 125. To address the inconsistency between image similarity and text similarity scales, we approximated the scale k. Initially, we measured the variations in image similarity and text similarity with changes in α, and identified the balanced similarity regions in the fusion results. As shown in Fig. 4 , the optimal range for k was found to be between [0.21, 0.27]. Based on these observations and experimental experience, we ultimately set k to 0.23. As shown in Fig. 17 of Appendix D, we observe that when the similarity between the image and the original exceeds 0.85, the images become too similar, making edits with different class texts less effective and necessitating a decrease in i. Conversely, when the similarity is below 0.45, the images overly favor the text, making them excessively editable, requiring an increase in injection steps. Therefore, we set I min sim to 0. Ablation Study. In Figs.10 and 18 in Appendix E, we visualize the results with and without the balance loss in Eq. ( 7), the adaptive injection ii in Eq. ( 8), and the adaptive selection α in Eq. ( 9) within our object synthesis framework. Pn-Pinv, used for direct inversion and prompt editing, resulted in some distortion and blurriness. Compared to PnPinv, the balance loss significantly enhances image fidelity, improving details, textures, and editability. The adaptive injection enables a smooth transition from Corgi to Fire Engine in Fig. 18 . Without this injection, the transformation is too abrupt, lacking a seamless fusion process. Finally, the adaptive selection achieves a balanced image that harmoniously integrates the original and target features. Note that for limitations, please refer to Appendix B.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
conclusion: 
 In this paper, we explored a novel object synthesis framework that fuses object texts with object images to create unique and surprising objects. We introduced a simple yet effective difference loss to optimize sampling noise, balancing image fidelity and editability. Additionally, we proposed an adaptive text-image harmony module to seamlessly integrate text and image elements. Extensive experiments demonstrate that our framework excels at generating a wide array of impressive object combinations. This capability is particularly advantageous for crafting innovative and captivating animated characters in the entertainment and film industry. Broader impact, please see Appendix A.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: for data we used in paper, we introduced in Appendix refsec:TICategories, and we will provide source code when this paper is published.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details.The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: refer to Subsection efsubsec:expsetting, ef subsec : parameter, andAppendixref sec : supparamGuidelines :7. The answer NA means that the paper does not include experiments.8. The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.9. The full details can be provided either with the code, in appendix, or as supplemental material. 
: 
 Figure 1 : We propose a straightforward yet powerful approach to generate combinational objects from a given object text-image pair for novel object synthesis. Our algorithm produces these combined object images using the central image and its surrounding text inputs, such as glass jar (image) and porcupine (text) in the left picture, and horse (image) and bald eagle (text) in the right picture.In this paper, we study an object synthesis task that combines an object text with an object image to create a new object image. However, most diffusion models struggle with this task, i.e., often generating an object that predominantly reflects either the text or the image due to an imbalance between their inputs. To address this issue, we propose a simple yet effective method called Adaptive Text-Image Harmony (ATIH) to generate novel and surprising objects. First, we introduce a scale factor and an injection step to balance text and image features in crossattention and to preserve image information in self-attention during the text-image inversion diffusion process, respectively. Second, to better integrate object text and image, we design a balanced loss function with a noise parameter, ensuring both Figure 2: Imbalances between text and image in diffusion models. Using SDXL-Turbo [56] (left) and PnPinv [27] (right), the top pictures show a tendency for generated objects to align with textual content (green circles), while the bottom pictures tend to align with visual aspects (orange circles). In contrast, our approach achieves a more harmonious integration of both object text and image. optimal editability and fidelity of the object image. Third, to adaptively adjust these parameters, we present a novel similarity score function that not only maximizes the similarities between the generated object image and the input text/image but also balances these similarities to harmonize text and image integration. Extensive experiments demonstrate the effectiveness of our approach, showcasing remarkable object creations such as colobus-glass jar in Fig. 1 . Project Page. 
 
 Appendix: Our model's capability to fuse images and text to generate new and creative object images holds significant potential across various fields, including entertainment, design, and education. However, it also raises important considerations regarding content safety and ethical use. In particular, if the input image or text contains inappropriate or offensive material, the generated images may similarly be inappropriate, leading to potentially unpleasant experiences for users.To mitigate these risks, it is crucial to implement robust NSFW (Not Safe For Work) content detection mechanisms. While existing methods can address some cases of inappropriate content, we acknowledge the need for continuous improvement in this area. As part of our future work, we will incorporate advanced NSFW checking models to ensure the generated content adheres to safety standards and ethical guidelines. This proactive approach aims to safeguard users and promote responsible use of our image generation technology.+"Komondor"+"African chameleon" original images our results Figure 11 : Failure results of our ATIH model.Our method relies on the semantic correlation between the original and transformed content within the diffusion feature space. When the semantic match between two categories is weak, our method tends to produce mere texture changes rather than deeper semantic transformations. This limitation suggests that our approach may struggle with transformations between categories with weak semantic associations. Future work could focus on enhancing semantic matching between different categories to improve the generalizability and applicability of our method.There are still some failure cases in our model, as shown in Fig. 11 . These failures can be categorized into two types. The first row illustrates that when the content of the image is significantly different from the text prompt, the changes become implicit. The second row demonstrates that in certain cases, our adaptive function results in changes that only affect the texture of the original image. In our future work, we will investigate these situations further and analyze the specific items that do not yield satisfactory results.C Text and Image Categories.We selected 60 texts, as detailed in Table 5 , and categorized them into 7 distinct groups. The 30 selected images are shown in Fig. 12 , with each image corresponding to similarly categorized texts, as outlined in Table 6 . Our model is capable of fusing content between any two categories, showcasing its strong generalization ability. Analysis of λ. Here, we provide a detailed explanation of the determination of λ. As shown in Fig. 13 , we use the ratio λ = Lr Ln to balance editability and fidelity. We iteratively adjust this ratio in the range of [0, 400] with intervals of 10, measuring the Dino-I score between the reconstructed and original images, as well as the CLIP-T and AES scores for images directly edited with the inverse latent values at different ratios. These experiments were conducted on the class fusion dataset, using fusion text for direct image editing. Figs. 14, 15, and 16 indicate that as the ratio increases, image editability improves, peaking at a ratio of around 260, but with a decrease in quality. At a ratio of 125, both image fidelity and the AES score achieve an optimal balance. Therefore, we set λ to 125.Analysis of k. The experimental analysis of parameter k was conducted using sdxlturbo as the base model. The range for i was set to [0, 4], and for each value of i, α was iterated from 0 to 2.2 in steps of 0.02 to observe changes in the fused image. The averaged experimental results produced a smooth curve, as shown in Fig. 4 . Based on these observations, the optimal range for kk was determined to be between [2.1,2.7][2.1, 2.7]. In our experiments, we set the value of kk to 2.3. We present another set of ablation study results in Fig. 18 , where the two rows represent the cases without (w/o) and with (w) attention projection. The input image is a Corgi, and the text is Fire engine. The output images display the different transformations as α varies. The top row shows the abrupt change in appearance without attention projection, resulting in a sudden transition from a Corgi to a fire engine. In contrast, with attention projection (bottom row), the change is smoother, achieving the desired blending result in the middle. F Algorithm.Overall, our novel object synthesis comprises three key components: optimizing the noise ϵt through a balance of fidelity and editability loss, adaptively adjusting the injection step i, and dynamically modifying the factor Algorithm 1 Novel Object Synthesis 1: An initial image latent z 0 , a target prompt O T , the number of inversion steps T , inject step i, sampled noise ϵ t , scale factor α, F (α) is Eq.( 9) 2: Output:Object Synthesis O 3: {z T , • • • , z ′ t-1 , • • • , z 0 } ← scheduler_inverse(z 0 ) 4: for t = 1 to T do 5: z t-1 ← step( z t ) 6: ϵ all [t] ← Balance-fidelity-editability( z t-1 , z ′ t-1 , z t , ϵ t ) 7: end for 8: i init ← T /2 9: i final ← Adjust-Inject(z T , ϵ all , O T , i init ) 10: α good ← Golden-Section-Search(F, α min , α max) 11: O ← DM(z T , ϵ all , O T , i final , α good ) 12: return O 13: function BALANCE-FIDELITY-EDITABILITY( z t-1 , z t-1 , z ′ t-1 , ϵ t ) 14: while L r /L n > λ do 15: ϵ t ← ϵ t -∇ ϵt L r ( z t-1 , z ′ t-1 , ϵ t , z t ) 16:end while I sim ← model Isim (z T , ϵ all , i, O T ) 38:if I sim < I min sim then 39:i ← i + 1 return i 49: end function These processes are detailed in Algorithm 1. Additionally, we utilize the Golden Section Search method to identify an optimal or sufficiently good value for α that maximizes the score function F (α) in Eq. ( 9). This approach operates independent of the function's derivative, enabling rapid iteration towards achieving optimal harmony. The key steps of the Golden Section Search algorithm are outlined as follows:α1 = b - b -a ϕ , α2 = a + b -a ϕ ,where ϕ (approximately 1.618) is the golden ratio, and a and b are the current search bounds for α. During each iteration, we compare F (α1) and F (α2), and adjust the search range accordingly:if F (α1) > F (α2) then b = α2 else a = α1.This process continues until the length of the search interval |b -a| is less than a predefined tolerance, indicating convergence to a local maximum.In this section, we delve into our two user studies in greater detail. The image results are illustrated in Figs. 6 and 8 , while the outcomes of the user studies for both tasks are presented in Figs. 19 and 20 . In total, we collected 570 votes from 95 participants across both studies. The specific responses for each question are detailed in Tables 8 and 9 .Notably, for the fourth question in the user study corresponding to our editing method, the example of peacock and cat fusion is shown in Fig. 6 , the number of votes for InfEdit [69] slightly exceeded ours. However, upon examining the image results, it becomes evident that their approach leans towards a disjointed fusion, where one half of an object is spliced with the corresponding half of another object, rather than directly generating a new object as our method does. H More results.In this section, we present additional results from our model. Fig. 21 showcases further generation results using our ATIH model. We experimented with four different images, each edited with four distinct text prompts. Fig. 22 provides further examples showcasing the effectiveness of our method in complex text-driven fusion tasks. Specifically, our approach excels in extreme cases by accurately extracting prominent features, such as color and basic object forms, from detailed textual descriptions. For instance, Fig. 22 shows a well-defined edge structure for the fawn image and the text 'Green triceratops with rough, scaly skin and massive frilled head.' Additionally, Fig. 23 illustrates our model's versatility with multiple prompts, emphasizing its capability for continuous editing. In this section, we present additional results from our model and compare its performance against other methods. Fig. 24 , we compare our results with those from the state-of-the-art T2I model DALL• E•3 assisted by Copilot. Our model shows superior performance when handling complex descriptive prompts for image editing. We observe that the competing model struggles to achieve results comparable to ours, particularly in maintaining the original structure and layout of images, despite adequate prompts. Recently, subject-driven text-to-image generation focuses on creating highly customized images tailored to a target subject [18; 52; 9; 74] . These methods often address the task, such as multiple concept composition, style transfer and action editing [38; 8; 45] . In contrast, our approach aims to generate novel and surprising object images by combining object text with object images. Kosmos-G [45] utilize a single image input and a creative prompt to merge with specified text objects. The prompt is structured as "<i> creatively fuse with object text," guiding the synthesis to innovatively blend image and text elements. Our findings indicate that Kosmos-G can sometimes struggle to maintain a balanced integration of original image features and text-driven attributes. In Fig. 27 , the images generated by Kosmos-G often exhibit a disparity in feature integration. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: refer to Subsectionrefsubsec:mainresM ainResults.Guidelines :The answer NA means that the paper does not include experiments.The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)The assumptions made should be given (e.g., Normally distributed errors).It should be clear whether the error bar is the standard deviation or the standard error of the mean.It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: refer to Subsection efsubsec:expsettingGuidelines :The answer NA means that the paper does not include experiments.The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.The paper should provide the amount compute required for each of the individual experimental runs as well as estimate the total compute.The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 


Decision: The paper introduces the Adaptive Text-Image Harmony (ATIH) method for generating novel object images by balancing text and image inputs in diffusion models. Reviewers appreciated the introduction of a scale factor and similarity score function but raised concerns about the lack of theoretical justification and the limited generalizability of the experimental results. In response, the authors provided additional quantitative metrics, expanded the dataset description, and compared their method with more baselines, such as ControlNet and InstructPix2Pix. They also committed to releasing the full dataset for future research. With generally positive ratings (5, 5, 5, 6), the paper is recommended for acceptance, pending the suggested revisions and dataset release, provided the authors make the suggested revisions and release the dataset as stated.

Reviews: review: paper_summary: paper_summary: The paper introduces an innovative method to generate new object images by combining textual descriptions with corresponding images. Addressing the common imbalance in diffusion models between text and image inputs, the authors propose the Adaptive Text-Image Harmony (ATIH) method. This method balances text and image features during cross-attention, ensuring a harmonious integration.
Key to this method are the scale factor (α) and injection step (i), which adjust the influence of text features and preserve image information. A novel similarity score function maximizes and balances the similarities between the generated image and the input text/image, while a balanced loss function with a noise parameter optimizes the trade-off between editability and fidelity of the synthesized image.
Validated through extensive experiments on datasets like PIE-bench and ImageNet, ATIH outperforms state-of-the-art techniques in creative object synthesis. The method showcases remarkable results, such as creating a dog-lobster or a rooster with an iron texture, demonstrating its innovative and high-quality image generation capabilities. The framework includes a Text-Image Diffusion Model (TIDM) using a pre-trained SDXL Turbo model with dual denoising branches. By treating sampling noise as a learnable parameter and designing a balance loss function, the method enhances image fidelity and editability. The ATIH method adaptively adjusts the scale factor and injection step to balance text and image similarities, using a similarity score function and the Golden Section Search algorithm to find optimal parameters.
strength_weakness: strength_weakness: 1.  While the paper discusses a novel algorithm for text-image harmony, the experimental validation focuses heavily on visual outcomes without substantial quantitative backing or comparisons against baselines using metrics relevant to the text-image synthesis field. Including more diverse and robust statistical evaluations could strengthen the claims about the algorithm's effectiveness.

2. The experiments primarily utilize a specific set of images and texts, which might not adequately represent the variety of real-world scenarios where such an algorithm could be applied. Expanding the dataset to include a wider range of text and image pairs, especially challenging ones, would help in understanding the algorithm's limitations and strengths better.

3. The paper could benefit from a deeper exploration of the robustness of the proposed method, particularly in how it handles edge cases or unusual text-image pairs. This includes testing the method's performance on non-ideal or adversarial inputs to gauge its resilience and adaptability.

4. More detailed comparative analysis with existing methods, especially recent advancements in text-to-image synthesis and image editing technologies, would provide a clearer picture of where the proposed method stands in terms of innovation and improvement. Specific examples of where it outperforms and underperforms can guide future development.

5. The methodology section could be expanded to include more technical details about the implementation, which would aid in reproducibility. For instance, details on parameter settings, algorithmic steps that were particularly effective, and potential pitfalls in the implementation could be highlighted.

6. The paper mentions user studies but does not delve deeply into how the feedback from these studies was integrated into the algorithm refinement. Elaborating on this process could provide insights into the user-centric development of the algorithm and its practical usability.
questions: questions: 1. Please include a detailed description of the selection criteria for the text-image pairs used in your experiments. It is important to demonstrate the diversity and representativeness of these pairs to ensure the robustness of your Adaptive Text-Image Harmony (ATIH) method.

2. It would be beneficial to expand the comparative analysis section with more detailed results, including statistical significance tests. This would provide a clearer picture of how your method improves over existing methods, underlining its novelty and effectiveness.

3. Please discuss the performance of your method under non-ideal conditions, such as complex or abstract text descriptions. Identifying and describing limitations observed in such scenarios could guide future improvements and research.

4. Including comprehensive implementation details, particularly regarding parameter settings and platform-specific optimizations, would significantly aid in the reproducibility and independent verification of your results.

5. Identifying specific edge cases and failure modes encountered during testing and development would help in understanding the practical limitations of your model. This information is crucial for setting realistic expectations and guiding future research directions.
limitations: limitations: The limitations have not been addressed.
paper_summary: paper_summary: The paper proposes a new method for harmonized image generation conditioned on both text and image conditions, leading to better performance than baselines mentioned in the paper.
strength_weakness: strength_weakness: The optimal combination between image and text condition is subjective. In the paper, all the generate examples have similar depth/layout/structure with input image condition, while with modified texture/appearance guided by the text. Assume the user want to generate "an image of a shark" with color/pattern similar to a given dog image, i.e. which is different from the example in Figure 2. It is not clear whether the proposed method can achieve that.

Furthermore,  some important baselines are missing:

1. The depth/structure information from the input image can be extracted and injected into the generation process with a ControlNet-like model [1]. This baseline is important because there is little structural change shown in the results of proposed  methods, which is the scenario that the well-known ControlNet is good at;

2. A naive way to generate with two conditions is using revised classifier-free guidance as [2]. The guidance strength from image and text can be controlled by hyper-parameters.

3. Another naive baseline is to only use image guidance before timestep t , and use text guidance after t. Results with different t will be interesting.


Subject-driven image generation and editing methods [3,4,5,6,7,8, 9] need to be discussed or at least mentioned, as these methods also aim to generate image guided by the reference images and text prompts. 


[1]. Adding Conditional Control to Text-to-Image Diffusion Models. Lvmin Zhang, Anyi Rao, Maneesh Agrawala.

[2]. InstructPix2Pix: Learning to Follow Image Editing Instructions. Tim Brooks, Aleksander Holynski, Alexei A. Efros.

[3]. An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion. Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, Daniel Cohen-Or.

[4]. DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation. Nataniel Ruiz Yuanzhen Li Varun Jampani Yael Pritch Michael Rubinstein Kfir Aberman.

[5]. Subject-driven Text-to-Image Generation via Apprenticeship Learning. Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming-Wei Chang, William W. Cohen.

[6]. Customization Assistant for Text-to-image Generation. Yufan Zhou, Ruiyi Zhang, Jiuxiang Gu, Tong Sun.

[7]. Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuning. Jian Ma, Junhao Liang, Chen Chen, Haonan Lu.

[8]. Multi-Concept Customization of Text-to-Image Diffusion. Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, Jun-Yan Zhu.

[9]. Kosmos-G: Generating Images in Context with Multimodal Large Language Models. Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, Furu Wei.
questions: questions: Can the proposed method generate object whose shape is guided by the text prompt, while appearance/texture is guided by a reference image?
limitations: limitations: Yes
paper_summary: paper_summary: This paper aims to generate novel objects based on a reference image and a conditional text prompt (e.g., a bottle (image) with a penguin (text) outlook). To this end, a method called Adaptive Text-Image Harmony (ATIH) is proposed to better align the conditional image and text. Experiments show that ATIH yields better results compared to existing state-of-the-art methods.
strength_weakness: strength_weakness: - As far as I understand, the image for text information in Figure 2 (e.g., "iron", "white shark") is for visualization purposes only. Actually, I think this might mislead readers (e.g., models can take two reference images as input).
- I think one thing the authors must consider is adjusting the strengths of existing baselines (e.g., InstructPix2Pix). If baselines allow for controlling the strength of the edit, it would be fairer if we had comparisons to that (e.g., similar to Figure 16, but for baselines).
- While in the main paper, the Golden Section Search algorithm seems to play an important role, it's unclear how important this algorithm is (e.g., can we simply replace this with simple average values). In other words, I think the current paper lacks an ablation study for this module.
questions: questions: It'd be interesting if the authors can discuss how to make a better evaluation for this task. Overall, I think this task is ambiguous and difficult to evaluate (e.g., if you want to turn a cat into a peacock, is it better to (1) completely replace the cat with the peacock, or (2) yield a combination of the cat and the peacock?) How can we define which is a 'better alignment' with the user's intention? (This is an open-ended question and will not affect my recommendation for this paper).
limitations: limitations: The authors did briefly discuss the limitations in the Supplementary.

At this moment, I'd recommend this paper for acceptance, as I believe the task introduced by this paper outweighs the limitations in experiments and ablation studies.
paper_summary: paper_summary: The paper works on novel object synthesis, namely, given an image and text prompt, the proposed method can generate a new image that contains the visual features from the given image and the textual information from the given prompt. The paper proposed a method names Adaptive Text-Image Harmony to tackle the task. The authors provide both qualitative and quantitate comparisons to show the effectiveness of the method.
strength_weakness: strength_weakness: For the constructed dataset with 1,800 text-image pairs, will the authors release it? Could this dataset be used to fine-tune other baseline approaches to improve performance? If so, has this experiments been done?

For the baseline methods, e.g., InstructPix2Pix, did the authors directly use the released model or train a new one using the same text-to-image model as this work, which is SDXL-Turbo?
questions: questions: Please refer to the weakness section.
limitations: limitations: Yes.

Comments: title: Response for authors
comment: Thank you for maintaining your positive rating. We sincerely appreciate your recognition of our work and your constructive feedback. We will incorporate the broader evaluation and discussion into the main paper as recommended.
Thanks authors for providing a detailed rebuttal.

I have read the rebuttal and other reviews. I have decided to keep my positive rating for this paper as "Borderline Accept," as before.

My justification for not giving a higher score is the lack of broader/thorough evaluation (as mentioned by other reviewers, which I agree). However, I find the task novel and interesting, so I am inclined to vote for acceptance.

Please incorporate the discussion/evaluation into the main paper. Good luck!
Thank you for raising the score. We sincerely appreciate your recognition of our work and effort.
Thank you for your valuable time and consideration. We would like to clarify the theoretical justification and broader comparative examples.

**Q1: Sufficient Theoretical Justification.**

**A1:** Our theoretical framework is grounded in a balance theory applied to multimodal data, specifically text and image, for the purpose of generating novel objects. Given an object described by text, $O_T$, and an object represented by an image, $O_I$, our goal is to synthesize a new object, $O(\alpha,i)$, parameterized by $\alpha$ and $i$. This synthesis aims to balance the distances (or similarities) between the new object and the original modalities. Ideally, the distance between $O(\alpha,i)$ and $O_T$ should be equal to the distance between $O(\alpha,i)$ and $O_I$, that is,

$$
d(O(\alpha,i),O_T)=d(O(\alpha,i),O_I),
$$

where $d(\cdot,\cdot)$ represents the similarity distance between text/image and image. For simplicity, we denote the similarity distance between the image $O_I$ and the created image $O(\alpha,i)$ as $I_{\text{sim}}(\alpha,i) = d(O_I, O(\alpha,i))$, and the similarity between the text $O_T$ and the created image $O(\alpha,i)$ as $T_{\text{sim}}(\alpha,i) = d(O_T, O(\alpha,i))$.

In practice, to address the inconsistencies between text and image modalities, we introduce a scaling factor, $k$, which mitigates these discrepancies and defines a balance function,

$$
F_\text{balance}=|I_{\text{sim}}(\alpha,i)-k\cdot T_{\text{sim}}(\alpha,i)|,
$$

where $|\cdot|$ is an absolute value. When $F_\text{balance}$ is small, there is a greater balance between $O(\alpha,i)$ and $O_T/O_I$. Additionally, we aim for the generated novel object to increasingly incorporate the information from both $O_T$ and $O_I$. This implies that higher values of $I_{\text{sim}}(\alpha,i)$ and $k\cdot T_{\text{sim}}(\alpha,i)$ correspond to more comprehensive information being retained. Thus, we define an information function,

$$
F_\text{information}=I_{\text{sim}}(\alpha,i)+k\cdot T_{\text{sim}}(\alpha,i),
$$

By combining the equations for $F_\text{balance}$ and $F_\text{information}$, we define a score function

$$
F(\alpha,i) = F_\text{information} - F_\text{balance} = \underset{\text{maximize similarities}}{I_{\text{sim}}(\alpha,i) + k \cdot T_{\text{sim}}(\alpha,i)} - \underset{\text{balance similarities}}{|I_{\text{sim}}(\alpha,i) - k \cdot T_{\text{sim}}(\alpha,i)|} 
$$


Finally, we use the score function $F(\alpha,i)$ to adaptively adjust the parameters, aiming to maximize the score, which corresponds to identifying the novel object $O$.

To further substantiate our theoretical claims, extensive experiments were conducted to demonstrate the effectiveness of our scoring function. The quantitative results, as shown in Table 1, indicate that our approach outperforms others in terms of Aesthetic Score (AES), Human Preference Score (HPS), Fscore, and Balance Similarity ( Bsim). These outcomes highlight our method's excellence in enhancing the visual appeal and artistic quality of images, while also aligning more closely with human preferences and understanding in object fusion.

**Q2: Broaden comparisons.**

**A2:** Thank you for your mention. Here we explain the broader range of comparative examples as follows.

**Broader Examples**: In our manuscript, supplementary materials, and rebuttal PDF, we have provided over 60 examples showcasing the versatility and robustness of our approach. These examples cover a wide array of categories, including *Cross-Species Fusions*, such as the combination of a corgi and a pineapple, and *Inanimate and Living Fusions*, like the fusion of a cup with a skunk.

**Complex Text-Driven Fusions**: Figure R1 in our rebuttal provides additional examples of our method’s performance with complex text-driven fusions.

**Demo and Code**: To facilitate further validation and exploration of our method's capabilities, we plan to release our code and a demo upon acceptance of the paper. This will enable academic and industry practitioners to replicate our results, ensuring transparency and reproducibility.

**Weaknesses**: Our method relies on the semantic correlation between the original and transformed content within the diffusion feature space. When the semantic match between two categories is weak, our method tends to produce mere texture changes rather than deeper semantic transformations. This limitation suggests that our approach may struggle with transformations between categories with weak semantic associations. This limitation is detailed in Appendix B and is also mentioned on line 294.

We thank you again for your feedback and hope that our explanation can make you better understand our contribution and efforts.

Best regards,

The authors
I have read the rebuttal and would like to thank the authors for the additional results, especially those on ControlNet and editing methods, which addressed some of my concerns. I have adjusted the score accordingly.
Thanks, I've reviewed the author's response. However, I still have several concerns regarding this paper. The primary contributions—a scale factor to balance text and image features in cross-attention and a novel similarity score function—while further clarified with specific parameter values, lack sufficient theoretical justification. Besides, the comparison of the proposed method with others, while claiming superiority in fusion quality, appears to rely on somewhat selective examples (e.g., African chameleon-bird). A broader range of comparative cases or more challenging scenarios would provide a more comprehensive understanding of the method's strengths and weaknesses.
## Dear Reviewer VGMU
Thank you for taking the time to review our submission and providing us with constructive comments and a favorable recommendation. We would like to know if our responses adequately addressed your earlier concerns. 

Additionally, if you have any further concerns or suggestions, please feel free to let us know. We eagerly await your response and look forward to hearing from you. 

Thank you for your valuable time and consideration!

Best regards,

The authors
## Dear Reviewer EP4q
Thank you for taking the time to review our submission and providing us with constructive comments and a favorable recommendation. We would like to know if our responses adequately addressed your earlier concerns. 

Additionally, if you have any further concerns or suggestions, please feel free to let us know. We eagerly await your response and look forward to hearing from you. 

Thank you for your valuable time and consideration!

Best regards,

The authors
## Dear Reviewer yinU
Thank you for taking the time to review our submission and providing us with constructive comments and a favorable recommendation. We would like to know if our responses adequately addressed your earlier concerns. 

Additionally, if you have any further concerns or suggestions, please feel free to let us know. We eagerly await your response and look forward to hearing from you. 

Thank you for your valuable time and consideration!

Best regards,

The authors
## Dear Reviewer xaqc
Thank you for taking the time to review our submission and providing us with constructive comments and a favorable recommendation. We would like to know if our responses adequately addressed your earlier concerns. 

Additionally, if you have any further concerns or suggestions, please feel free to let us know. We eagerly await your response and look forward to hearing from you. 

Thank you for your valuable time and consideration!

Best regards,

The authors