When ControlNet Meets Inexplicit Masks: A Case Study of ControlNet on its Contour-following Ability

Authors: controlnet contour recall
controlnet  contour
controlnet -𝒎𝒎  𝟓𝟓
controlnet -𝒎𝒎  𝟏𝟏𝟎𝟎
controlnet -𝒎𝒎  𝟐𝟐𝟎𝟎
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-01 10:01:58

Abstract: ControlNet excels at creating content that closely matches precise contours in user-provided masks. However, when these masks contain noise, as a frequent occurrence with non-expert users, the output would include unwanted artifacts. This paper first highlights the crucial role of controlling the impact of these inexplicit masks with diverse deterioration levels through in-depth analysis. Subsequently, to enhance controllability with inexplicit masks, an advanced Shape-aware ControlNet consisting of a deterioration estimator and a shape-prior modulation block is devised. The deterioration estimator assesses the deterioration factor of the provided masks. Then this factor is utilized in the modulation block to adaptively modulate the model's contour-following ability, which helps it dismiss the noise part in the inexplicit masks. Extensive experiments prove its effectiveness in encouraging ControlNet to interpret inaccurate spatial conditions robustly rather than blindly following the given contours, suitable for diverse kinds of conditions. We showcase application scenarios like modifying shape priors and composable shape-controllable generation. Codes are soon available.

Main paper: 
 introduction: 
 Text-to-Image (T2I) generation techniques [5, 23, 25, 26, 29, 31] have greatly changed the content creation area with high-fidelity synthesized images. By generating content following user-provided guidance like contours and shapes, ControlNet [37] stands out for its prominent capability of spatial control over T2I diffusion models and has become an indispensable tool for creation. The shape guidance always takes the format of segmentation masks, where the details of the contours and positions are perfectly reserved as shown in Fig. 1 . However, such good contour-following ability 1 of ControlNet may cause artifacts when there exists noise in the mask, especially for non-expert users that have difficulty in providing accurate masks. Unfortunately, its performance on inexplicit masks with inaccurate contours remains under-explored, and studies on how to employ Figure 1 : ControlNet tends to preserve contours for spatial controllable generation over multi-modal control inputs, where green denotes recalled contours and blue denotes missing ones. However, inexplicit masks cause catastrophic degradation of image fidelity and realism. This paper largely enhances its robustness in interpreting inexplicit masks with inaccurate contours.inexplicit masks are ignored in current works. This issue hinders non-expert users from creating better images through ControlNet.To better assist non-expert users in generating satisfactory images, this paper focuses on utilizing inexplicit masks during the generation process. We first analyze the most important property, i.e., the contour-following ability, on the inexplicit mask-guided generation process through deteriorating masks and tuning hyperparameters. Specifically, we study this property quantitatively and reveal preliminary analyses on two key areas: 1) its performance on conditional masks of varying deterioration degrees, and 2) the influence of hyperparameters on this property. One of the main findings is that masks with inaccurate contours would cause artifacts due to strong contour instructions ( § 4.2). In other words, the contour-following ability implicitly assumes that the conditional inputs align with the shape priors of specific objects. Otherwise, it would cause artifacts or violate the spatial control in synthesized images as shown in Fig. 1 . Given that precise conditional images are typically image-oriented (e.g., human annotation or those extracted from reference images by offline detectors) or expert-provided (e.g., artists), it constrains creation within the boundaries of existing images and experts. However, obtaining precise control inputs can be cumbersome and challenging for non-experts. This fact largely restricts creation through human scribbles, thus hindering broad applications of ControlNet.While an intuitive solution is to adjust hyperparameters to achieve satisfactory results, finding the optimal setting can be challenging ( § 4.3) . Besides, this strategy usually entails a trade-off between image fidelity and spatial control. Based on this observation, we improve ControlNet with the awareness of shape priors, namely Shape-aware ControlNet ( § 5), which alleviates the impact of strong contour instructions and shows advances in robust interpretation of inexplicit masks ( § 6). Specifically, our improvements comprise a deterioration estimator and a shape-prior modulation block to automatically adjust the model's contour-following ability. The deterioration estimator assesses the deterioration ratio that depicts the similarity of the provided masks to explicit masks. Features from the Stable Diffusion (SD) [27] encoders are utilized during the estimation, as SD effectively encodes the shape priors of different objects. Then, we modulate this shape prior to the zero-convolution layers of ControlNet through the proposed shape-prior modulation block. This design enables extra control over the strength of contour instructions and adapts the model to the guidance of inaccurate contours. As Fig. 1 shows, our method follows object position and poses guided by inexplicit masks, keeping high fidelity and spatial control over T2I generation. It also adapts to other conditions beyond masks ( § 6.4). Moreover, though we only employ dilated masks for training, our method generalizes well to more realistic masks including user scribbles and programmatic TikZ sketches ( § 6.5).This paper primarily focuses on ControlNet for two reasons. Firstly, ControlNet is the most representative and widely adopted method for spatially controllable T2I generation. It prevails over concurrent works for higher image quality and better preservation of outlines owing to the inherited powerful SD encoder and extensive training data. ControlNet is influential and has been widely incorporated in tasks including image animation [34, 36] , video generation [8, 19] , and 3D generation [4] , etc. Secondly, Control-Net shares fundamental ideas and similar structures with existing adapter-based methods like T2I-Adapter [22] . Moreover, these methods exhibit similar properties (i.e., the contour-following ability) and face similar issues (i.e., performance degradation with inaccurate contours). More examples of ControlNet and T2I-Adapter are provided in Appendix Fig. S1 . Therefore, our conclusions and methods can be potentially extended to these methods.To summarize, our contributions are as follows:• We study the contour-following ability of ControlNet quantitatively by examining its performance on deteriorated masks of varying degrees under different hyperparameter settings. We reveal inexplicit masks would severely degrade image fidelity for strong shape priors induced by inaccurate contours. • We propose a novel deterioration estimator and a shape-prior modulation block to integrate shape priors into ControlNet, namely Shape-aware ControlNet, which realizes robust interpretation of inexplicit masks. • Our method adapts ControlNet to more flexible conditions like scribbles, sketches, and more condition types beyond masks. We showcase its application scenarios like modifying object shapes and creative composable generation with deteriorated masks of varying degrees. 
related works: 
 Training with Spatial Signals. An intuitive solution to introduce spatial control is training models with spatially aligned conditions from scratch. Make-A-Scene [6] employs scene tokens derived from dense segmentation maps during generation, enabling complex scene generation and editing. SpaText [1] extends it to open-vocabulary scenarios and introduces spatio-textual representations for sparse scene control. Composer [16] decomposes images to representative factors like edges and then trains a model to recompose the input from these factors. However, training models from scratch requires large-scale training data and high computational costs.Adapters for Spatial Control. Another kind of work focuses on the utilization of adapters to inject spatial control into the pretrained T2I models. For example, GLIGEN [18] involves gated self-attention layers to control the spatial layout during generation. T2I-adapter [22] and ControlNet encode the spatial guidance via lightweight adapters or duplicated UNet encoders, which are then fed into the decoder to inject spatial structures like shapes and contours. Uni-ControlNet [39] , UniControl [24] , and Cocktail [15] consolidate multi-modal conditions within a single framework. Since the base models are always frozen, these methods obtain better computational and data efficiency while retaining the generation ability of the base models. Despite these appealing characteristics, the impact of inexplicit masks is overlooked in these works, which may cause significant artifacts in generated images. Our proposed method can also be extended to these approaches to address this limitation. 
preliminary: 
 ControlNet [37] introduces a network as the adapter to control T2I generation models, i.e., Stable Diffusion (SD), with extra spatially localized, task-specific conditional images including Canny edges, Hough lines, fake scribbles, key points, segmentation masks, shape normals, depths, and so on. Supposing one encoder block 𝐹 (•; 𝜃 ) of SD parameterized by 𝜃 , it accepts the input feature 𝒙 and output 𝒚 = 𝐹 (𝒙; 𝜃 ). ControlNet freezes the block and makes a trainable copy 𝐹 (•; 𝜃 ′ ) with parameter 𝜃 ′ to inject additional condition 𝒄 with zero convolution, formulated as, 𝒚 𝒄 = 𝐹 (𝒙; 𝜃 ) + 𝜆 * 𝑍 (𝐹 (𝒙 + 𝑍 (𝒄; 𝑤 1 ); 𝜃 ′ ); 𝑤 2 ),where 𝑍 (•; 𝑤 1 ) and 𝑍 (•; 𝑤 2 ) are two 1 × 1 zero convolution layers with parameters 𝑤 1 , 𝑤 2 initialized with zeros for stable training. 𝒚 𝒄 is the output feature modulated by spatial control signals. 𝜆, namely conditioning scale, is introduced during inference to adjust condition strength. Since features are additive, it allows compositions of multiple conditions, also known as Multi-ControlNet [37] .Classifier-free Guidance (CFG) [14] is a widely employed technique in generative diffusion models to increase image quality. Given a latent noise 𝒛, CFG is conducted through mixing samples generated by an unconditional model 𝝐 𝜃 (•) and a jointly trained conditional model 𝝐 𝜃 (•, 𝒄), which is computed as,EQUATIONwhere ε𝜃 (𝒛, 𝒄) represents the final output under condition 𝒄. 𝜔 is a scaling factor for a trade-off between image quality and diversity, namely CFG scale. In this paper, the spatial conditions are injected into both 𝝐 𝜃 (•) and 𝝐 𝜃 (•, 𝒄) when conducting classifier-free guidance with ControlNet. 
visiting the contour-following ability: 
 This section studies the contour-following ability from two aspects, including the performance on deteriorated masks of varying degrees and its interaction with hyperparameters, i.e., CFG scale, conditioning scale, and condition injection strategy. We utilize the most common real-world control signal, i.e., object mask for experiments. The object mask is simple but effective in covering shape information. For simplicity, we focus on plain binary masks in experiments, which are the most straightforward userprovided signals in real-world applications. This choice also helps to exclude the impacts of color and occlusion.Datasets. Following Control-GPT [38] , our experiments involve COCO images [20] and corresponding captions [3] . We utilize instance masks from LVIS [9] , which offers sparse and precise humanannotated object masks of COCO images with over 1,200 object classes. We filter out images containing empty annotations, resulting in 114k image-caption-mask triplets for training and 4.7k for testing, namely the COCO-LVIS dataset.Implementation Details. We adopt SD v1.5 [28] as the base model, and ControlNet is trained from scratch on COCO-LVIS for ten epochs with a learning rate of 1𝑒 -5 for all experiments. We take 50% prompt dropping for classifier-free guidance while keeping the SD parameters frozen. We use UniPC [40] sampler with 50 sampling steps. To evaluate, we take ground-truth masks for control and generate four images per caption. CFG scale and conditioning scale are set to 7.5 and 1.0 if there is no extra illustration.Metrics. For evaluation, CLIP-Score (ViT-L/14) [11] and FID [12] are adopted as two basic metrics to measure the text-image alignment and image fidelity. In addition, as ControlNet tends to preserve all contour structures provided in the conditional images, we calculate the number of edge pixels retained in the generated images, namely Contour-Recall (CR), to measure the contour-following effects quantitatively. Supposing a conditional image 𝒄 and generated images 𝑿 = {𝒙 𝑖 } 𝑁 𝑖=1 , CR is defined as the recall of edge pixels in the control inputs, formulated as,EQUATIONwhere 𝐷 (•) is an edge detector that returns binary edge maps. We employ a max-pooling function 𝑀𝑎𝑥𝑃𝑜𝑜𝑙 (•, 𝜎) to tolerate an edge detection error of 𝜎-pixels. In our implementation, we utilize the Canny edge detector as 𝐷 (•) and set a tolerance as 𝜎 = 2. A higher CR score indicates stronger contour-following ability.To clarify concerns about ControlNet's performance on inexplicit masks, we first imitate inaccurate control signals by dilating object masks 𝒎 with progressively increasing radius 𝑟 (𝑟 ≥ 0). The dilated mask is denoted as 𝒎 𝑟 , where 𝒎 0 represents masks with precise contours. Considering that bounding-box masks only convey object position and size without identifiable contours, we take an intersection over the dilated mask 𝒎 𝑟 and the bounding-box mask denoted as 𝒃, to construct inexplicit masks of various degrees, formulated as,EQUATIONwhere the bounding-box mask 𝒃 can also be expressed as the extreme case 𝒎 ∞ (𝒃 ⊂ 𝒎 ∞ ). In the rest of the paper, we use 𝒎 ∞ to denote the bounding-box mask. As 𝑟 grows large, the masks lose detailed shape information gradually. We use the notation ControlNet-𝒎 𝒓 to denote ControlNet trained with deteriorated masks 𝒎 𝑟 , where the vanilla ControlNet trained with precise masks is ControlNet-𝒎 0 . We test these models on masks of varying deterioration degrees to explore the performance of Con-trolNet with inexplicit masks and its contour-following ability. Fig. 2 presents the experimental results. Here are two main observations. ❶ The vanilla ControlNet, i.e., ControlNet-𝒎 0 , strictly adheres to the provided outlines, where inexplicit masks would cause severe performance degradation. As shown in Fig. 1 , the generated images are faithful to the contours of the control inputs under various control modalities, which is the key to excellent spatial controllable generation. One underlying reason is supposed to be that edges provide a shortcut to reconstruct the input images. Therefore, it exhibits strong contour instructions during the controllable text-to-image generation process of ControlNet.However, we notice that such a contour-following ability is blind, which takes no account for inaccurate contours and maintains a high average CR of over 60% in Fig. 2 . While it allows minor editing by manipulating masks to some extent, this property imposes severe problems on image realism when inaccurate contours (e.g. user scribbles) are provided. Similar observations are also noted in [2, 32, 33] , but an in-depth analysis of this fact is neglected. For example, ControlNet tends to interpret rectangle masks as boards rather than animals specified in the prompt, causing incorrect spatial layouts and distorted objects. It is also reflected in a degradation of 1.35 CLIP-score and 8.62 FID in Fig. 2 . Thus, precise masks with accurate contours are necessary for high-quality image generation. However, obtaining masks with accurate contours is challenging, especially for non-expert users.❷ Training ControlNet with deteriorated masks also converges and exhibits high robustness on masks of varying deterioration degrees with a sacrifice of contour instructions. We further train a series of ControlNet, i.e. ControlNet-𝒎 𝑟 , on progressively dilated masks 𝒎 𝑟 . Surprisingly, ControlNet converges with all conditions, even on the extreme bounding-box masks. As depicted in Fig. 2 , training with deteriorated masks consistently improves ControlNet's robustness in interpreting inexplicit masks with better CLIP-score and FID at the cost of weak contour-following ability.However, ControlNet trained with dilated masks implicitly assumes the presence of dilation in the provided mask, even for precise masks. Fig. 3 Moreover, training ControlNet with severely deteriorated masks, e.g., bounding-box masks, yields poorer CLIP-score and FID with precise masks as shown in Fig. 2 . Examples in Fig. 7 illustrate that ControlNet-𝒎 ∞ probably misinterprets one whole object mask into multiple small objects. This reminds us that too weak contour instructions can also cause problematic images and deviate from user intentions. So, it is worthwhile to develop a method to make contour-following ability more controllable, thereby fulfilling robust interpretation with inexplicit masks of varying degrees.As a common practice, adjusting hyperparameters like CFG scales is a useful tool to achieve satisfactory generation results in T2I applications. Therefore, we investigate how hyperparameters impact the contour-following ability, especially when the conditional inputs are inexplicit, and reveal several guidelines. Specifically, we test the vanilla ControlNet (i.e., ControlNet-𝒎 0 ), under two extreme condition cases, i.e., precise mask 𝒎 0 and Effects of Classifier-free Guidance 𝜔. Under both conditions, ControlNet maintains high CR scores under all CFG-scale settings, showing little impact on contour-following ability. This is because conditioning signals are added to both 𝝐 𝜃 (•) and 𝝐 𝜃 (•, 𝒄) in Eq. 2. Moreover, performance on precise mask 𝒎 0 always surpasses that on 𝒎 ∞ , indicating precise control inputs improve image quality. The trends of CLIP-Score and FID curves align with common sense, where a relatively higher 𝜔 brings better fidelity and text faithfulness.Effects of Conditioning Scale 𝜆. The conditioning scale decides the strength of control signals as Eq. 1. A higher 𝜆 imposes stronger instructions, consistent with increasing CR scores under both conditions. However, the behavior diverges on CLIP-Score and FID curves when increasing 𝜆. While both CLIP-Score and FID almost consistently increase on 𝒎 0 , its performance on 𝒎 ∞ increases slightly at the very beginning but then declines dramatically, showing a large performance gap with 𝒎 0 . This reveals a trade-off between image quality and spatial control with bounding-box masks, where 𝜆 ∈ [0.5, 0.7] are recommended. Finding suitable 𝜆 for masks of varying deterioration degrees usually requires tens of attempts.Effects of Condition Injection Strategy. We divide the sampling steps into ten stages during the reverse sampling phase for experiments. As shown in Fig. 4(c ), all stages contribute to contour instructions during the generation process in both cases. The first 10% ∼ 40% steps hold the key, consistent with observations in T2I-Adapter [22] . In addition, a similar divergence of CLIP-Score and FID curves is observed between the two conditions, further declaring the performance gap between explicit and inexplicit masks.In summary, while the CFG scale 𝜔 has little impact on the contour-following ability, a smaller conditioning scale 𝜆 and discarding conditions at early reverse sampling stages help to relieve contour instructions. These strategies help to relieve the degradation of image fidelity and text faithfulness for inexplicit masks and achieve a trade-off between image quality and spatial control. However, subtle changes to such hyperparameters may lead to dramatic changes in image appearance (see Appendix Fig. S3 ). Hence, it is still tricky to search for the ideal combination of hyperparameters for each image, and it also takes risks of violating spatial control.To clarify the difference between explicit and inexplicit mask controls, we compare ControlNet-𝒎 0 with ControlNet-𝒎 ∞ on precise masks 𝒎 0 and bounding-box masks 𝒎 ∞ in Fig. 6 . By removing textual prompts, we find that the strong contourfollowing ability of vanilla ControlNet derives intense shape priors from contours, thus it tends to infer specific objects from the mask directly. Since inexplicit masks introduce incorrect shape priors, they will cause conflicts with textual prompts or violations of spatial control for generation. For example, bounding-box masks are misinterpreted as the door in Fig. 6 . This is why the vanilla Control-Net fails to interpret inaccurate contours with the correct content. VisorGPT [35] also validates that conditional generative models implicitly learn visual priors from the data, such as size and shape. If spatial conditions deviate from that prior, it would lead to artifacts and incorrect contents. In contrast, ControlNet-𝒎 ∞ relies less on the shape priors from the conditional mask and prompts object locations for spatial control. This leads to improved text alignment and robustness on inexplicit masks. However, ControlNet-𝒎 ∞ exhibits weak control with precise masks. So, it is necessary to develop a method to adapt ControlNet to deteriorated masks of varying degrees. 
shape-aware controlnet: 
 To enhance ControlNet's ability to interpret inexplicit masks robustly instead of blindly adhering to contours, we introduce a novel deterioration estimator and a shape-prior modulation block to ControlNet, namely Shape-aware ControlNet, as depicted in Fig. 5 . Our main insight is to introduce the shape prior explicitly to the model for better controllability.We first introduce the deterioration ratio 𝜌, which depicts the gap between the inexplicit masks and explicit shape priors of specific objects, formulated as,EQUATIONwhere 𝑆 (•) returns the area of masks, and 𝜌 ∈ [0, 1]. Eq. 5 measures the shape priors of masks, where 𝜌 = 0 indicates the exact object shape and 𝜌 = 1 means no identifiable shape.While computing the deterioration ratio 𝜌 based on precise object masks is straightforward, estimating 𝜌 from inexplicit masks is nontrivial. This motivates us to train an extra deterioration estimator. Since ControlNet-𝒎 𝑟 can imply the dilation radius 𝑟 (refer to § 4.2), we suppose ControlNet implicitly learns shape priors from deteriorated masks. So, we construct a deterioration estimator with stacked convolutional and linear layers with batch normalization, to predict the ratio from encoder features {𝐹 𝑖 𝑗 }, where 𝐹 𝑖 𝑗 denotes the 𝑗-th feature from the 𝑖-th encoder block. Such a design proves to provide accurate estimation as discussed in § 6.3. We train the estimator separately with an 𝐿2 loss and detach the gradients from the encoder to avoid negative effects on the convergence of ControlNet.Taking the shape prior 𝜌 as an extra condition, we design a shapeprior modulation block inspired by StyleGAN [17] . Specifically, we encode 𝜌 into Fourier embedding. Then, we employ a hypernetwork [10] to modulate shape priors to zero convolution layers in ControlNet as illustrated in Fig. 5(c ). Supposing a zero convolution 𝑍 (•; 𝑤) parameterized by 𝑤 and ControlNet encoder feature 𝐹 𝑖 𝑗 , the modulated feature 𝐹 ′ 𝑖 𝑗 is computed as,EQUATIONwhere Δ𝑊 = 𝐻 (𝜌), and 𝐻 (•) is a hypernetwork constructed by multiple linear layers with normalization. 
experiments 6.1 implementation details: 
 The baselines including ControlNet-𝒎 0 and ControlNet-𝒎 ∞ are implemented as § 4.1. We inherit the same setting to train our shapeprior modulation block on our COCO-LVIS dataset for 10 epochs from scratch with a learning rate of 1𝑒 -5. The dilation radius 𝑟 is uniformly sampled from 0 to 80. The deterioration estimator is trained for another 10 epochs individually with gradient-detached features from ControlNet.Metrics. We employ CLIP-Score and FID for evaluation. As the proposed CR metric would fail when the strict alignment of contours is violated, we propose two additional metrics inspired by SOA [13] , namely Layout Consistency (LC) and Semantic Retrieval (SR). Definitions are as follows. Definition 6.1 (Layout Consistency, LC). LC measures the spatial alignment between conditional masks and generated images. Supposing detected bounding boxes {𝑎 𝑖 } 𝑀 𝑖=1 and those of control masks {𝑏 𝑗 } 𝑁 𝑗=1 , LC is computed as follows:EQUATION)Definition 6.2 (Semantic Retrieval, SR). SR is a retrieval measurement to verify whether the semantic objects assigned in the prompt are generated. Supposing 𝑀 detected object categories 𝑆 = {𝑠 𝑖 } 𝑀 𝑖=1 over confidence threshold 𝑡 and 𝑁 assigned labels Specifically, we utilize an open-vocabulary object detector, i.e., OWL-ViT [21] following VISOR [7] . We set the confidence threshold 𝑡 = 0.1 and use instance labels as prompts.EQUATIONAs Tab. 1 depicts, our method achieves competitive or better performance on all metrics, demonstrating robustness under all dilation radius. Visualized results in Fig. 7 further illustrate the effectiveness of our method. ControlNet-𝒎 0 tends to blindly follow the outlines of control inputs, thus suffering from unnatural images or violation of user intentions, especially when it is conditioned on largely deteriorated masks. On the other hand, ControlNet-𝒎 ∞ struggles with precise masks with fine details like misinterpreting a single instance mask into multiple objects. In contrast, our method excels in understanding object orientations and shapes according to inexplicit masks, thanks to additional guidance of shape priors. More visualized examples are available in Appendix Fig. S4 .Comparison with random dilation augmentation. As augmentation with random dilated masks serves as an intuitive and straightforward solution to improve the robustness of deteriorated masks, we first compare this strategy with our Shape-aware ControlNet in Tab. 2. While this data augmentation strategy is also effective, our method achieves consistently higher performance on all metrics. Moreover, we highlight that our method provides users with the additional convenience of modifying the shape of generated objects through additional control over shape priors as presented in § 6.5.Accuracy of deterioration estimator. We validate the accuracy of our proposed deterioration estimator in Fig. 8 . The overall average 𝐿1 error under all dilation radius is 5.47%. Owning to the robustness of the modulation block, such an error shows minimal impact on the image fidelity as validated in Tab. 3. For page limitations, the complete error curves are presented in Appendix Fig. S5 . Robustness on shape priors. We further examine the robustness of our method on different shape priors. We take dilation radius 𝑟 = 20 and manually adjust shape prior 𝜌 to (𝜌 + Δ𝜌) ∈ [0, 1], where Δ𝜌 ∈ [0, 1]. As shown in Fig. 9 the text faithfulness and image quality. But lower 𝜌 encourages better adherence to conditional masks, resulting in slightly higher LC with the assigned control masks, and vice versa. Visualized examples of tuning 𝜌 can be found in Fig. 12 .It should also be noted that although the proposed method is designed for inexplicit masks, it can effortlessly adapt to other degraded conditions, demonstrating the versatility of our method in dealing with control signal deterioration. Here is an example with degraded edges and we leave other condition types for future works.Specifically, we take the deteriorated edges from masks in Eq. 4 for condition. The experimental settings are the same as § 6.1 and the only change is the condition used for training. As depicted in Fig. 10 , the vanilla ControlNet keeps adhering to the provided deteriorated edges blindly and struggles to generate high-quality images with inexplicit conditions, which are consistent with the observations on conditional masks. In contrast, our proposed method adapts well to degraded edges of varying degrees, proving its effectiveness in dealing with other types of degraded conditions. For page limitations, quantitative results are provided in Appendix Tab. S2. Here we showcase several applications of our method. For page limitations, additional examples are included in Appendix §A.Generation with TikZ sketches and scribbles. While our model is only trained with dilated masks, we find it generalizes well to other types of realistic inexplicit masks and exhibits robust performance. Fig. 11 shows the effectiveness of our method in handling programmatic sketches and human scribbles. Our method generates reasonable images under abstract or inexplicit mask conditions while keeping high fidelity and spatial control. Here we follow Control-GPT [38] to prompt GPT-4 to produce TikZ codes for object sketches. Human scribbles are converted from Sketchy [30] dataset.Shape-prior modification. While the deterioration estimator provides the deterioration ratio ρ for reference, we can manually set 𝜌 to control the shape prior. It determines how much the generated objects conform to the provided conditional masks as shown in Fig. 12 . We notice it is possible to extend ρ to a large range of other values, even though 𝜌 ∈ [0, 1] for training. A lower |𝜌 | empirically encourages stronger adherence to the contours of the provided masks.Composable shape-controllable generation. We realize composable shape control via a multi-ControlNet structure [37] , where we generate images by assigning different priors to each part of control masks. This enables strict shape control over specifically assigned masks while allowing T2I diffusion models to unleash creativity in imagining objects of diverse shapes within inexplicit masks. A vivid example is presented in Fig. 13 , where the parrots share fixed contours but hats are diverse in shapes, and vice versa. 
conclusion: 
 In this paper, we reveal several insights into the core traits of Con-trolNet, i.e., the contour-following ability. We quantitatively validate this property by distorting conditional masks and tuning hyperparameters, where we uncover severe performance degradation caused by inexplicit masks. In light of this, we propose a novel deterioration estimator and a shape-prior modulation block to endow ControlNet with shape-awareness to robustly interpret inexplicit masks with an extra shape-prior control. This exhibits the potential of employing ControlNet in more creative scenarios like user scribbles and diverse degraded condition types. 
 
 Appendix: The contour-following ability refers to the ability to preserve the contours of conditional inputs in this paper. 


Decision: All reviewers are positive on this paper, and the authors have provided a strong rebuttal to clear concerns/questions. AC believes this paper deserves clear acceptance, and encourages the authors to further improve the paper when preparing camera ready.

Reviews: review: paper_summary: paper_summary: This paper first highlights the crucial role of controlling the impact of these inexplicit masks with diverse deterioration levels through in-depth analysis. Subsequently, to enhance controllability with inexplicit masks, an advanced Shapeaware ControlNet consisting of a deterioration estimator and a shapeprior modulation block is devised. The deterioration estimator assesses the deterioration factor of the provided masks. Then this factor is utilized in the modulation block to adaptively modulate the model’s contour-following ability, which helps it dismiss the noise part in the inexplicit masks.
strength_weakness: strength_weakness: 1. Sufficient experimental validation: The article validates the effectiveness of the proposed method through a large number of experiments, including the performance under masks with different degradation levels and the comparison with existing methods. These experimental results show the significant improvement of Shape-Aware ControlNet in image quality and spatial control.
2. In-depth theoretical analysis: The article not only proposes a new method, but also provides an in-depth quantitative analysis of ControlNet's contour-following ability, revealing its limitations when dealing with imprecise masks. This theoretical analysis provides a solid foundation for the proposed new method.
3. Promising applications: This paper demonstrates the potential of Shape-Aware ControlNet in several application scenarios, such as shape a priori modification and composable shape control generation. These application scenarios demonstrate the broad prospects of the method in practical applications and increase the likelihood of its acceptance.
limitations: limitations: 1. Although the article conducts a large number of experiments, it mainly focuses on the COCO-LVIS dataset. The lack of validation on other datasets may affect the generalizability and persuasiveness of the method.

2. Certain technical details may not be described in sufficient detail, such as the specific implementation of the degeneracy estimator and the parameter settings of the shape a priori modulation module. This may affect the reader's understanding and reproduction of the method.
paper_summary: paper_summary: This paper attempt to address the issue of ControlNet's inability to generate high-quality images when using rough masks as prompt conditions. This paper designed  a deterioration estimator to assess the deterioration factor. They also proposed a shape-prior modulation block to integrate shape priors into ControlNet. The experiments shows better results when the input masks are rough.
strength_weakness: strength_weakness: 1. This paper addressed and, to some extent, resolved the current limitation of ControlNet's inability to generate high-quality images based on rough masks, which has practical implications.
2. The proposed framework made very lightweight modifications to ControlNet.
limitations: limitations: 1. If the masks are relatively rough, whether it will be better to utilize the bounding box as the control signal, as providing bounding box is much easier. The advantage of rough mask over bounding box should be emphasized.
2. According to Figure 8, the prediction of deterioration is not precise, whether it means the prediction will not significantly influence the synthesized images. Whether the designed modules are useful for this task is not convincing.
3. In Table 2, the original ControlNet with random augmentation shows almost comparable performance with the proposed methods, whether that means the framework design is not that useful.
paper_summary: paper_summary: The paper addresses a challenge faced by ControlNet. The authors highlight the need to manage the influence of such inexplicit masks through an in-depth analysis and propose an advanced version of ControlNet to enhance its robustness.
Shape-aware ControlNet introduces two key components: a deterioration estimator and a shape-prior modulation block. The deterioration estimator assesses the quality of the input masks, and the modulation block uses this assessment to adaptively adjust the model's contour-following ability. This adjustment allows the model to better handle noisy input, reducing the introduction of artifacts in the generated images.
strength_weakness: strength_weakness: - The proposed ControlNet is a novel enhancement method that has proven to be effective.
- The paper is well written and easy to follow.
- The paper includes many experiments and comparisons to assess the performance of the proposed method.
limitations: limitations: - How does the model determine when to follow a given contour and when not to? For example, in Figure 1, the user may wish for an object in the shape of a bucket to be repainted.
- Integrating additional components such as a deterioration estimator and a shape-prior modulation block could increase the computational complexity of the model. The computational costs and parameters associated with these modules are unclear.