CustomNet: Object Customization with Variable-Viewpoints in Text-to-Image Diffusion Models

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 02:36:09

Abstract: Incorporating a customized object into image generation presents an attractive
feature in text-to-image (T2I) generation.
Some methods finetune T2I models for each object individually at test-time, which tend to be overfitted and time-consuming. 
Others train an extra encoder to extract object visual information for customization efficiently but struggle to preserve the object's identity. 
To address these limitations, we present CustomNet, a unified encoder-based object customization framework that explicitly incorporates 3D novel view synthesis capabilities into the customization process. 
This integration facilitates the adjustment of spatial positions and viewpoints, producing diverse outputs while effectively preserving the object's identity. 
To train our model effectively, we propose a dataset construction pipeline to better handle real-world objects and complex backgrounds. 
Additionally, we introduce delicate designs that enable location control and flexible background
control through textual descriptions or user-defined backgrounds. 
Our method allows for object customization without the need of test-time optimization, providing simultaneous control over viewpoints, location, and text. Experimental results show that our method outperforms other customization methods regarding identity preservation, diversity, and harmony.

Main paper: 
 related work: 
 Object customization with text-to-image diffusion models.With the promising progress of text-to-image diffusion models [11, 25, 28, 29, 31, 33, 34] , researches explore to capture the information of a reference object image and maintain its identity throughout the diffusion model generation process, i.e., object customization. These methods can be broadly classified into optimization-based techniques and encoder-based approaches. Optimization-based methods [5, 9, 20, 30] can achieve high-fidelity identity preservation; however, they are time-consuming and may sometimes result in overfitting. In contrast, current encoder-based methods [16, 18, 35, 42] enable zero-shot performance but may either lose the identity. To address this issue, several methods ELITE [40] , Subject-Diffusion [22] have been proposed to enhance detail preservation by introducing local features, which still only generate images that resemble the reference images in content and style and do not allow for versatile viewpoint control. This limitation makes it difficult to achieve harmonious results. In contrast, our proposed Custom-Net aims to preserve high fidelity while supporting controllable viewpoint variations, thereby achieving more diverse outcomes. Image harmonization. In image composition, a foreground object is typically integrated into a given background image to achieve harmonized results. Various image harmonization methods [3, 7, 10, 37] have been proposed to further refine the foreground region, ensuring more plausible lighting and color adjustments [4, 6, 41] . However, these methods focus on low-level modifications and cannot alter the viewpoint or pose of the foreground objects. In contrast, our proposed CustomNet achieves flexible background generation using user-provided images and offers additional viewpoint control and enhanced harmonization. 3D novel view synthesis aims to infer the appearance of a scene of novel viewpoints based on one or a set of images of a given 3D scene.Previous methods have typically relied on classical techniques such as interpolation or disparity estimation [26, 46] , as well as generative models [1, 36] . More recently, approaches based on Scene Representation Networks (SRN) [32] and Neural Radiance Fields (NeRF) [13, 23, 44] have been explored. Furthermore, diffusion models have been introduced into novel view synthesis [19, 39] . Zero-1to-3 [19] propose a viewpoint-conditioned diffusion model trained on large synthetic datasets, achieving excellent performance in single-view 3D reconstruction and novel view synthesis tasks. Our CustomNet incorporates the 3D capabilities to diffusion models for object customization, generating outputs with diverse viewpoints while preserving the identity. 
method: 
 Given a reference background-free 1 object image 𝑥 ∈ R 𝐻 ×𝑊 ×3 with height 𝐻 and width 𝑊 , we aim to generate a customized image x where the object of the same identity can be seamlessly placed in a desired environment (i.e., background) harmoniously with appropriate viewpoint and location variations. As illustrated in Fig. 3 , we propose CustomNet, a novel unified architecture designed to achieve this given object customization conditioned on the viewpoint 𝑅 (where 𝑅 represents a vector that contains the relative camera polar angle 𝜃 and azimuth angle 𝜙 of the object in desired viewpoints), object location 𝐿 (bounding box), and text prompt 𝑇 that describes the generated image:EQUATIONIn the following, we first introduce the dataset construction pipeline that helps to obtain such paired data: { x, 𝑥, 𝑅, 𝐿,𝑇 }, which facilitates harmonious customization with both 3D synthetic dataset and natural image dataset. Then based on LDM [29] , we further design the dual cross-attention and Unet input concatenation that makes our model support viewpoints, location, and text control simultaneously. Moreover, similar to other inpainting methods [18, 42] , CustomNet can also be extended to the inpainting-like 1 Background-free images can be easily obtained by segmentation methods, e.g., SAM [14] . customization that supports giving the reference background image instead of text prompts.As shown in Fig. 2 (a), we can acquire multi-view object images and their corresponding camera view parameters from existing 3D datasets, such as Objaverse [8] . However, these datasets only include object images without backgrounds (usually with a pure white background), and the objects are rendered only at the center of the images. This is not suitable for customization tasks. As a naive solution, we can first collect another background image and specify a random bounding box 𝐿 located in the background image, then we resize the object and perform mask-blending with the object image and background images in the bounding box. The blended image is the target image x and we use another view of the object and relative camera parameters from the 3D datasets as 𝑥 and 𝑅, respectively. We use BLIP-2 [17] to caption the textual descriptions of the blended images for the text prompts 𝑇 .However, since the composition between the object and background would be unreasonable (i.e., the object is placed into the background disharmoniously) and the blended target image is unrealistic, the model trained on them often generates a disharmonious customized image, e.g., the objects float over the background (see Sec. 4.3).To alleviate this problem, we propose a dataset construction pipeline that is the reverse of the above-mentioned way, i.e., directly utilizing natural images as the target image and extracting objects from the image as the reference. The specific pipeline is shown in Fig. 2 (b) . For a natural image x, we first use BLIP-2 to extract the foreground object with the instruction {"image": image, "prompt": "Question: What foreground objects are in the image? find them and separate them using commas. Answer:"}. Then we feed the object and its corresponding text to SAM. SAM receives text as input and outputs both the bounding box 𝐿 and the segmentation mask of the corresponding object. We use a bounding box and a segmentation mask to crop the object and synthesize a novel view of the object as 𝑥 by Zero-1-to-3 with randomly sampled relative viewpoints 𝑅. The textual description 𝑇 of the image can be also obtained using the BLIP-2 model. In this way, we can synthesize a large amount of data pairs from natural image datasets, like OpenImages [15] .To enable synthesizing a target customized image complied with the given viewpoint parameter 𝑅 ([𝜃, 𝑠𝑖𝑛(𝜙), 𝑐𝑜𝑠 (𝜙)]), we follow the view-conditioned diffusion method introduced by Zero-1-to-3.As shown in Fig. 3 (the left-top part), we first apply a pre-trained CLIP [27] image encoder to encode the reference background-free object image into an object embedding, containing high-level semantic information of the input object. Then the object embedding is concatenated with 𝑅 and passed through a trainable lightweight multi-layer perception (MLP). The fused object embedding further passes to the denoising UNet as a condition with the cross-attention mechanism to control viewpoints of the synthesized images. Even with multi-view datasets, it is worth noting that the explicit viewpoints control is of vital importance to perform object variation and ensure identity consistency, as discussed in 4.3.We further control the object location in the synthesized image by concatenating the reference object image with the desired location and size to the UNet input z 𝑡 -1 , where z 𝑡 -1 represents the noisy latent at the time step 𝑡-1. The process is illustrated in Fig. 3 (the left-bottom part). The desired location and size 𝐿, represented as a bounding box [𝑥, 𝑦, 𝑤, ℎ], is the object's desirable location in the target image. Then, we resize the reference object image into the size of [𝑤, ℎ] and place its left-top corner at the [𝑥, 𝑦] coordinate of a background-free image (this image is the same size as the target image being denoised but without background), which is represented as 𝑥 𝐿 . The additional concatenated reference object image helps the model synthesize the desired image while keeping the identity and the texture details [19, 29] . Note that Zero-1-to-3 directly concatenates the centrally-located reference object to the UNet input, which can only synthesize an image where the object is centered. Our method enables synthesizing the target object at the desired position with the proposed explicit location control.We first introduce how to incorporate text prompts to control the background generation, then extend it to inpainting-like customization with the reference background image. Text prompt Control. CustomNet is required to generate an appropriate background based on the textual description 𝑇 . Different from Zero-1-to-3, which solely accepts the object embedding without textual descriptions for background, we propose a dual cross-attention conditioning strategy that accepts both the fused object embedding with viewpoint control and textual descriptions for background. The dual cross-attention mechanism integrates the fused object embedding and the textual embedding through two distinct cross-attention modules. Specifically, we first employ the CLIP text encoder to obtain the textual embeddings and subsequently inject them into the denoising UNet, along with the fused object embedding, using the DualAttn:EQUATIONwhere the query features 𝑄 come from the UNet, while 𝐾 𝑜 , 𝑉 𝑜 are the object features projected from fused object embeddings with viewpoint control, and 𝐾 𝑇 , 𝑉 𝑇 are the textural embeddings, 𝑑 is the dimension of the aforementioned feature embeddings. This straightforward yet effective design enables us to achieve accurate background control without affecting the object viewpoint control. Background Image Inpainting. In many practical scenarios, users desire to seamlessly insert objects into pre-existing background images with specific viewpoints and locations. To achieve this, we extend the input channels of the UNet by concatenating the provided background image channel-wise, following the Stable Diffusion inpainting pipeline. Consequently, the diffusion model accepts [z 𝑡 -1 , object w/ loc, 𝑥 𝑏𝑔 ] as inputs. Note that in this mode, the textual description is optional, allowing for straightforward input of NULL to the text prompt. In comparison to existing image composition methods [42] which often struggle with identity loss issues, our method offers viewpoint and location control over objects and enhanced identity preservation in 4.2.Model Training. Given paired images (object image and target image) with corresponding relative camera viewpoint, resized object image in specified locations, and text prompt {𝑥, 𝑥 𝑡𝑔𝑡 , 𝑅, 𝑥 𝐿 ,𝑇 }, we train our diffusion model conditioned on these explicit controls. We adopt the latent diffusion model (LDM) [29] architecture, which contains a variational auto-encoder (VAE) with an encoder E, decoder D, and an UNet denoiser 𝜖 𝜃 . LDM uses VAE to convert the image from pixel-level space to latent space, and performs the diffusion-denoising process in the latent space rather than the pixel space for efficiency. The optimization objective is:EQUATIONOnce the denoising UNet 𝜖 𝜃 is trained, we can perform harmonious customization conditioned on the target viewpoint, location, and text prompt with CustomNet.Classifier-free diffusion guidance [12] is a method for adjusting the quality and diversity of diffusion generations. In CustomNet, we divide the conditions into two parts: image (𝑥, 𝑅, 𝑥 𝐿 , which is related to object viewpoint and location) and text (𝑇 for textual description). For sampling , we set two guidance scales (𝑆 𝐼 , 𝑆 𝑇 ) to control their influence respectively as follows:EQUATIONwhere ∅ * is set the * condition to null. During training, we randomly drop the image part by 5% and the text part by 15%. 
experiments 4.1 training datasets and implementation details: 
 As introduced in Sec. 3.5, We use multi-view synthetic dataset Objaverse [8] and additionally obtained background images from the web to synthesize blended images. Then We use natural image dataset OpenImages-V6 [15] filtered as BLIP-Diffusion [16] to construct data pairs with the pipeline. A total of (250+500)K data pairs are constructed for model training with sampling ratio 5% : 95%, respectivaley. We exploit the Zero-1-to-3 checkpoint as the model weight initialization. For training, we employ AdamW [21] optimizer with a constant learning rate 2×10 -5 for 500K optimization steps. The total batch size is 96, and about 6 days are taken to finish the training on 8 NVIDIA-V100 GPUs with 32GB VRAM.We compare our CustomNet to the optimization-based methods Textual Inversion [9] , Dreambooth [30] , and encoder-based method GLIGEN [18] , ELITE [40] , BLIP-Diffusion [16] , IP-Adapter [43] . We use their official implementation (for GLIDEN, ELITE, and BLIP-Diffusion, IP-Adapter) or the diffusers implementations [38] (for Textual Inversion, Dreambooth) to obtain the results. Note that Dreambooth requires several images of the same object to finetune. Figure 4 shows the images generated with different methods. We see that the methods GLIGEN, ELITE, BLIP-Diffusion, IP-Adapter and the optimization-based method Textual Inversion are far from the identity consistent with the reference object. Dreambooth and the proposed CustomNet achieve highly promising harmonious customization results, while our method allows the user to control the object viewpoint easily and obtain diverse results. In addition, our method does not require time-consuming model fine-tuning and textual embedding optimization.We also evaluate the synthesized results quantitatively. All methods apply 26 different prompts to perform customizations 3 times randomly on 50 objects, a total of above 3700 cases. Following [16] , we calculate the visual similarity with the CLIP image encoder and DINO encoder, denoted as CLIP-I and DINO-I, respectively. We measure the text-image similarity with CLIP directly, denoting CLIP-T. Tab. 1 shows the quantitative results, where CustomNet achieves better identity preservation (DINO-I and CLIP-I than other methods. Meanwhile, CustomNet shows comparable capacity to the state-of-the-art methods regarding textual control (CLIP-T). We also conducted a user study and collected 2700 answers for Identity similarity (ID), View variation (View), and Text alignment (Text), respectively. As shown in the right part of Tab. 1, most participants prefer CustomNet in all three aspects (76.11%, 56.67%, 64.67%). For the test-time-consuming comparison, CustomNet and other encoder-based methods are faster than optimization-based methods in general.Comparison to Inpainting-based Methods Existing inpaintingbased methods (SD-Inpainting model, Paint-by-Example [42] , GLI-GEN [18] ) can also place a reference object in the desired background in an inpainting pipeline. Given an object, the background can be inpainted with textual descriptions in the SD-Inpainting model, while this kind of method easily suffers from unreal and disharmonious results and cannot cast variations to the reference object. Our CustomNet can obtain more harmonious customization with diverse viewpoint control. Another line of methods Paint-by-Example and GLIGEN can inpaint the reference object to a given background image. From Fig. 5 , we see that they cannot maintain the identity and differ significantly from the reference object.Comparison to Zero-1-to-3 We show our improvements of Zero-1-to-3 and its limitations in Fig. 6 . In the 1st, and 2nd row, we compare with Zero-1-to-3 the ability to control the object location generation. As Zero-1-to-3 can only control polar angle 𝜃 , azimuth Table 1 : Quantitative Comparison. We compute DINO-I, CLIP-I, CLIP-T following [16] . We also conducted a user study to measure subjective metrics: ID, View, Text representing identity preservation, viewpoints variation, and text alignment, respectively. The last column compares the test-time consumption of different methods. angle 𝜙, and radius 𝑟 (the camera distance to the object) of the central object, users can not control the object location in their settings directly, so we apply our location control method on Zero-1-to-3. We resize the object and move it to the location in different bounding boxes (top left and top right in 1st row, bottom left and bottom right in 2nd row.). We can see that Zero-1-to-3 generates distortion in the generated images. This is because Zero-1-to-3 is only trained on the object-centered image, it can not really control the object location.↑ CLIP-I ↑ CLIP-T ↑ ID ↑ View ↑ Text ↑ Time (s) ↓ DreamBooth [In the 3rd, and 4th row, We compare with Zero-1-to-3 the ability of novel-view synthesis. We can see from the figure that Zero-1to-3 fails to generate reasonable geometry for the dog and cartoon character. Our CustomNet, training with our real-world constructed datasets, has a better comprehension of the object geometry. We can generate a normal dog and a reasonable cartoon character in multi-views. Moreover, CustomNet supports diverse text prompt condition generation that Zero-1-to-3 can not, which is more practical for customization. In all rows, our CustomNet generates harmonious customized images with the control of different text prompts.We conduct detailed ablation studies to demonstrate the effectiveness of each design in CustomNet and the necessity of explicit viewpoints control for identity preservation in harmonious customization. Explicit viewpoint control is the key for customization that enables simultaneous viewpoint alteration and object identity preservation. We conduct a comparison in terms of with and without explicit viewpoint control parameters 𝑅 on the original Zero-1-to-3 model. As shown in the left part of Fig. 7 , models trained without viewpoint conditions tend to generate images that cannot change the viewpoint or have undesired artifacts. This is the same on CustomNet. Specifically, as shown in the right part of Fig. 7 , without the explicit camera pose control, our model can only obtain copying-and-pasting effects, even though it is trained with the multi-view dataset. Note that in this setting, we also concatenate the object image into the UNet input, otherwise, it cannot preserve adequate identity. This phenomenon may indicate that even with multi-view data to train, an explicit additional view control is still necessary, which allows the model to better distinguish between the distribution of images with different views, rather than learning a mixed or average of different views.Pretraining with massive multi-view data is important to preserve identity in CustomNet. We adopt Zero-1-to-3 as the initialization, i.e., our CustomNet is pre-trained with massive multiview Objarverse data, so that view-consistency information has been already encoded into the model. When we train Custom-Net from the SD checkpoint (see the 2nd column in Fig. 8 ), the synthesized images cannot maintain view consistency from other viewpoints and suffer from quality degradation.Input concatenation helps better maintain texture details. Previous methods [22, 40] also try to integrate local features to maintain texture details in synthesized images. In our method, concatenating the reference object to the UNet input can also preserve textures. Without the concatenation (see the 3rd column in Fig. 8 ), the color, shape, and texture of the generated images differ significantly from the reference object image. We also note that the model would generate copying-and-pasting images without any view variations when we do not adopt explicit viewpoint control (see Fig. 7 ). This is to say, the combination of input concatenation and explicit view conditions enables precise and harmonious customization.Our data construction pipeline enables more harmonious outputs. We adopted a new data construction pipeline for utilizing the OpenImages dataset in Sec. 3.5. Without this design, the model trained with only the data constructed by the naive combination between multi-view object images in Objaverse and background images can result in unrealistic and unnatural customized results, usually leading to artifacts that object is 'floating' on the background (see the 4th column in Fig. 7 ).Dual cross-attention enables disentangled object and text controls. We introduce dual attention for the disentangled objectlevel control and text control. When directly concatenating the text embedding and fused object embedding as the condition to be injected into the attention layers of UNet, the model tends to learn a coupled control with viewpoint parameters and textual description. As a result, the viewpoint control capacity would degrade significantly and the model cannot generate the desired background (see 5th column in Fig. 8 ).Classifier-free guidance weights effects. CustomNet is controlled by both image-related and text conditions for generation using classifier-free guidance. In Tab. 2, we compare different classifierfree guidance weights effects and empirically choose 𝑆 𝐼 = 3, 𝑆 𝑇 = 7.5 for comparison experiments. 
conclusion: 
 We present CustomNet, a novel unified encoder-based diffusion object customization approach that explicitly incorporates 3D novel Table 2 : Classifier-free guidance weights effects. We compute DINO-I, CLIP-I CLIP-T following [16] to compare different classifier-free guidance weights effects. view synthesis for enhanced identity preservation and viewpoint control. We develop a dataset construction pipeline to handle realworld objects and complex backgrounds effectively. Moreover, we introduce extra necessary designs for location control and flexible background control through textual descriptions or provided background images, which improve the novel view synthesis model (e.g.Zero-1-to-3), and show potential for bridging the gap between 3D novel-view synthesis and text-to-image object customization.Our experiments show that CustomNet enables identity preserving and diverse object customization while controlling location, viewpoints, and background simultaneously. 
limitation and future works: 
 CustomNet may be limited to some complex prompts and rarely seen objects and scenery, and existing diffusion models trained with web-scale datasets would also encounter these problems. Current research shows that fine-tuning the model with specific small but high-quality datasets may make the generation better. We will explore it in future work.  
customnet: object customization with variable-viewpoints in: 
 Text-to-Image Diffusion Models users to incorporate objects from reference images into the generation while preserving the object identity. Pioneering works such as Dreambooth [30] and Textual Inversion [9] use a few images of the same object to finetune the parameters of diffusion models or learn concept textual embeddings through an iterative optimization process. Although these optimization-based techniques excel at maintaining object identity, they suffer from certain drawbacks, such as time-consuming optimization for each object and a tendency to be overfitted when only a single image is provided. Consequently, researchers have started exploring encoder-based methods [16, 18, 22, 40, 42] for efficient customization. These methods train an encoder to encode visual concepts of objects to an embedding. Once trained, users can use the encoder to get the object image embedding and send it into the denoising process during inference, achieving a speed comparable to the standard diffusion model sampling process. However, simply injecting an image into a compressed concept embedding often leads to inadequate identity preservation [16, 42] . Several methods further propose to enhance detail preservation by introducing local features [22, 40] , which still cannot preserve complex textures well.Besides identity-preserving and efficiency, a user may also want to simultaneously change the object viewpoints, which is hard for the customization methods mentioned above. Novel View Synthesis (NVS) diffusion models are capable of addressing viewpoint control. Recent methods [2, 39] regard NVS as an image-to-image translation task and train diffusion models in certain specific categories. Zero-1-to-3 [19] , which leverages the massive synthetic 3D object multiple view dataset Objaverse [8] to train a camera pose conditioned diffusion model, extending the generalization capability from a single category to the open world. However, directly applying an NVS diffusion model like Zero-1-to-3 into the object customization task poses several challenges due to its inherent limitations: 1) It is solely capable of generating centrally positioned objects, lacking the ability to place them in alternative locations in the synthesized image; 2) It can not generate diverse backgrounds, being restricted to a simplistic white background; 3) It also lacks the high-level semantic information control like text prompts. These constraints significantly hinder its applicability in object customization tasks.To efficiently generate harmonious customized images under the control of viewpoints, location, and text while ensuring object identity preservation, We introduce CustomNet. CustomNet is a unified encoder-based object customization framework that facilitates diverse viewpoints in text-to-image diffusion models. Unlike previous optimization-based and encoder-based methods that simply rely on text-image paired datasets, CustomNet needs to be trained on more complex data, i.e., a target image with text description and multi-view images of the objects which come from the image. Therefore, we design a dataset construction pipeline that effectively utilizes synthetic multi-view data and massive natural images to better handle real-world objects and complex background relationships for training. Moreover, based on latent diffusion model (LDM) [29] architecture, we design dual cross attention to support both viewpoints and text control for the spatial Transformer in the LDM UNet, and adjust object size and location by concatenating the transformed reference object image with the UNet input. We can further apply style editing to customization through the textcontrol branch, as the style condition derives from the text prompt in the data. We also extend CustomNet to reference background image inpainting applications that can receive multiple objects and background images for more flexible customization.Built upon those designs, CustomNet can achieve harmonious customization with identity preservation and diverse control without test-time optimization, as shown in Fig. 1 . We summarize our contributions as follows:  
 
 Appendix:  


Decision: 1. This paper presents a unified encoder-based diffusion object customization approach that explicitly integrates 3D novel view synthesis to enhance identity preservation and viewpoint control. 
It also introduces a well-designed dataset augmentation process and effectively incorporates multiple image generation control conditions using the constructed dataset, addressing shortcomings in previous approaches. 
The proposed method seamlessly merges multiple control conditions, with each condition's usage and integration being very intuitive. 
Experiments demonstrate its superior performance and efficiency compared to other methods. 
The paper is well-written with straightforward method flowcharts, providing a clear understanding of the approach.
 
2. However, there are some weaknesses in this paper. It lacks quantitative baseline and ablation experiment analyses, relying solely on visual results, which reduces its persuasive strength.
Additionally, it does not provide an analysis of why the proposed method is faster than other methods or address its limitations in pose or action editing.

Based on these reasons and all reviews rate borderline accept, I am inclined to accept this paper.

Reviews: review: paper_summary: paper_summary: This paper proposes an encoder-based object customization framework that enables view and location control without test-time optimization. They also propose a dataset construction pipeline to handle complex cases. The method shows a good ID preservation ability as well as controllability.
strength_weakness: strength_weakness: The method is efficient since it is encoder-based without need of test-time optimization. 
The qualitative and quantitative results show the advantages of the method over others.
limitations: limitations: I do have some questions regarding the method details.

1. About Dataset construction pipelines shown in Figure 2. It is said that Pipeline 1 generates disharmonious customized images, so the second pipeline is adopted to alleviate the problem. So my understanding is that these two pipelines are not complementary, instead the latter is an improved version over the former right? If so, how does the second pipeline solve the issue of disharmony? Using the synthesized object from Zero-1-to-3, we still have to put the object back to a background image. So how is the new pipeline better?

2. Line 327 says "We use BLIP-2 to caption the blended images for T". I am wondering how well BLIP-2 works on those disharmonious blended images? For example, the Target image in Figure 2(a), the head blended in a background image, because it is very unusual image.

3. In Figure 3, for UNet, there are two types of input, 1) concatenation of obj and z_{t-1} 2) concatenation of bd, obj and z_{t-1}. In order to feed these two types of input, do we need to adapt input/output channels of the UNet? (change the original 4 channels to 8 and 12?) If we do have to adapt the input/output channels, how does the method deal with these two different types with different channels (8 and 12)?

4. In Figure 4, compared with other encoder-based methods like BLIP-Diffusion, ELITE and IP-Adapter, the proposed method shows way better ID-preservation abilities. But as shown in Figure 8, we see the ID is preserved badly without concatenation of the object image. So I am wondering if the strong ID-preservation ability is simply from concatenation and whether it is a fair comparison over other methods. The concatenation technically leaks the object appearance information as test-time whereas the other encoder-based methods only use the high-level features from the image encoder. In other word, how well do other methods perform with concatenation of the object as well?  

To conclude, I think the main contribution is adding view and location control signals to the customization network. But I think the comparison is not very fair to truly show the model performance as stated in point 4. I also think the writing could be improved.
paper_summary: paper_summary: This paper presents a unified encoder-based diffusion object customization approach that explicitly integrates 3D novel view synthesis to improve identity preservation and viewpoint control.
strength_weakness: strength_weakness: 1. This work employs SAM and Zero-1-to-3 to construct image datasets from various viewpoints.
2. Introducing viewpoint information allows for controlling the target's perspective.
3. The experimental results in Table 1 appear to be highly effective.
limitations: limitations: 1. The paper lacks quantitative baseline and ablation experiment analyses. Relying solely on visual results for analysis lacks persuasive strength.
2. Despite not requiring fine-tuning of the entire pre-trained model, this method still demands a substantial amount of time for fine-tuning, i.e., 6 days on 8 V100 GPUs. Additionally, why CustomNet's test-time in Table 1 is faster compared to other methods remains unclear. It seems that the introduction of more complex conditional control information and dual attention mechanisms during the CustomNet training process may contribute to the speed improvement.
3. The authors' approach relies on pre-training weights from Zero-1-to-3, thus necessitating a quantitative comparison with Zero-1-to-3.
4. Were the experiments in Table 1 conducted under identical settings? Additional experimental details are required, including the base model, classifier-free guidance scale, and training data.
paper_summary: paper_summary: The paper introduces a new unified encoder framework named CustomNet for object customization in text-to-image (T2I) diffusion models, incorporating 3D novel viewpoint synthesis capabilities. It designs a well-suited dataset augmentation construction process and effectively integrates multiple image generation control conditions into the model using the built dataset, achieving the final output that can utilize various viewpoints, texts, and specified background images.
strength_weakness: strength_weakness: 1. The dataset construction process is intuitive and addresses shortcomings in previous approaches, resulting in an ideal dataset for the current task that includes multiple viewpoints and contains text and background data for image synthesis.
2. The proposed method effectively merges multiple control conditions, with each condition's usage and integration process being very intuitive. Experiments also demonstrate its superior performance and efficiency compared to other methods.
3. The paper is well-written with straightforward method flowcharts, allowing for a clear understanding of the ideas and model pipeline. The visualization of experimental results is impressive, effectively showcasing the method's capabilities.
limitations: limitations: 1. It appears that the ability of CustomNet to control viewpoints stems from training with multi-view images. However, it seems that there is no direct control over viewpoints during testing. Is it possible to explicitly control the object's angle through text?

2. Why is your inference speed faster than IP-Adapter? There doesn't seem to be any difference in your structures.

3. Although CustomNet can change the target's perspective, the target is still static and it seems that the corresponding action cannot be changed. Is it possible to use some avater method to get image generation that keeps the ID instead of just the static target itself?