ObjBlur: A Curriculum Learning Approach With Progressive Object-Level Blurring for Improved Layout-to-Image Generation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-02 11:11:51

Abstract: We present ObjBlur, a novel curriculum learning approach to improve layout-to-image generation models, where the task is to produce realistic images from layouts composed of boxes and labels. Our method is based on progressive object-level blurring, which effectively stabilizes training and enhances the quality of generated images. This curriculum learning strategy systematically applies varying degrees of blurring to individual objects or the background during training, starting from strong blurring to progressively cleaner images. Our findings reveal that this approach yields significant performance improvements, stabilized training, smoother convergence, and reduced variance between multiple runs. Moreover, our technique demonstrates its versatility by being compatible with generative adversarial networks and diffusion models, underlining its applicability across various generative modeling paradigms. With ObjBlur, we reach new state-of-the-art results on the complex COCO and Visual Genome datasets.

Main paper: 
 introduction: 
 Layout-to-image generation is a fundamental task in computer vision and graphics, bridging the gap between structured scene descriptions, such as layouts composed of bounding boxes and labels, and the generation of realistic images [10, 29, 39, 43] . It is a complex task, further compounded by intrinsic variations in the difficulty of learning to generate different object classes and their inherent diversity in shapes, sizes, and context [8] .Layout-to-image models are mainly based on GANs [9] and thus inherit their training stability issues, such as mode collapse and overfitting [22] . While data augmentation (DA) techniques have been proven to be effective in visual recognition models [25, 35] , training a GAN under similar augmentations leads to a leaking effect in which the generator learns to produce augmented (instead of clean) images. For example, if rotation is used as a DA, the generator will produce rotated images after training, an undesirable outcome. To mitigate this problem, consistency regularization [37, 41, 42] , invertibility [31] , and differential augmentation techniques [15, 40] have been proposed. Meanwhile, the machine learning community has been interested in curriculum learning (CL) strategies [1, 28, 32] to structure training examples in a meaningful order that gradually exposes the model to more complex concepts. It provides an intuitive approach to guiding models through progressively challenging training scenarios. Interestingly, their exploration remains relatively limited in the context of generative models [28, 32] and nonexistent in the domain of layout-to-image generation. For single-object generative image models, previous work proposed the use of multiple discriminators [7, 13, 24] , progressively growing the model [14] or ranking images by difficulty [27] . However, all previous work requires either changing the model, loss function, using a difficulty estimator, or a combination of them. To our knowledge, there is no previous work on using curriculum learning for layout-to-image generation.This paper introduces ObjBlur, a new approach to layout-toimage generation that utilizes curriculum learning by applying progressive object-level blurring to improve the image quality of layout-to-image models. Blurring is a natural image degradation operation because low frequencies are retained over higher frequencies. In fact, even human perception is more sensitive to low frequencies of an image [2, 23] . Strong blurring removes highfrequency details, resulting in a simpler signal without affecting the structural content of the image (as opposed to degradation alternatives such as additive noise). Decreasing the blur strength produces a more complex signal with high-frequency details, thus exposing the model to a more difficult task. Therefore, blurring offers an intuitive and powerful approach to incrementally adjust task difficulty, ensuring a smooth training progression.Our method can be realized by only modifying the data loader to apply a progressive blur to the images. As a result, it can be easily integrated into existing layout-to-image approaches and does not depend on difficulty estimators or changes in the model architecture and optimization protocol. By systematically applying varying degrees of blurring during training, starting with strong blurring and progressing to cleaner images, we stabilize training and ensure that the model learns to generate high-quality images.A crucial aspect of image quality is the appearance of foreground objects in relation to the background. Thus, we propose an objectlevel approach that randomly applies the blur to either the objects or the background. To demonstrate the benefits of ObjBlur, we perform extensive analysis on several layout-to-image generation models, including adversarial- [10, 29] , and diffusion-based [43] approaches. We also comprehensively analyze several design choices and their impact on performance and stability. Using LayoutDiffusion [43] as a backbone, our proposed ObjBlur schedule significantly improves the quality of generated images, offering a robust and versatile approach that leads to new state-of-the-art results. In terms of FID [11] , SceneFID [30] and CAS [20] , we reach relative improvements of 2.38%, 34.43%, 4.70% on COCO [3] , and 6.13%, 18.45%, 10.15% on Visual Genome [17] while only requiring changes to the dataloader. 
related work 2.1 layout-to-image: 
 The layout-to-image (L2I) task was first studied in [39] using a VAE [16] by composing object representations into a scene before producing an image. Adversarial approaches [29] produced higherresolution images and provided better control of individual objects by using a reconfigurable layout with separate latent style codes. Further developments studied better instance representations [30] and context awareness [10] . Recent developments have focused on using diffusion models by adjusting the self-attention mask to focus on the instances and adding prompt tokens [5] or by constructing a structural image patch with region information to facilitate multimodal fusion of image and layout [43] .The idea of monotonically increasing the difficulty of tasks is related to curriculum learning (CL) [1] which introduces more complex concepts gradually, instead of randomly presenting training data. CL is inspired by the teaching paradigm of organizing learning material in an orderly fashion. Although it has been successfully applied in a wide range of tasks [28, 32] , its application to generative models is minimal. In text generation, the length of character sequences can gradually be increased as training progresses [19] . To improve image generation, multiple discriminators are used in [7, 13, 24] , while the image resolution increases in [14] . While [14] is similar in spirit, it requires an entirely different implementation to grow both the generator and discriminator layers during training, a challenging and not generalizable procedure. In [4] , a CL strategy based on semantic difficulty determined by embedding distance is used for text-to-image synthesis. Several CL strategies are proposed in [27] based on ranking the training images according to their difficulty scores, and a smoothing schedule to intermediate CNN features is presented in [26] . To the best of our knowledge, we propose the first CL strategy for L2I models using a progressive object-level blurring schedule without requiring architectural changes to the model. Instead, our method only requires dataloader changes and can thus be seamlessly integrated into any method.Data augmentation (DA) techniques have played an essential role in the success of deep learning over the last decade by artificially expanding the data set and enabling the training of large models [25, 35] . Examples of model-free image augmentation can be categorized into geometrical transformations, color space augmentations, kernel filters, image and feature mixing, and random erasing. Unfortunately, training generative models under similar augmentations typically leads to a leaking effect where the generator learns to produce the augmented data distribution. Our method is inspired by CutBlur [36] , a data augmentation technique developed to improve the performance of super-resolution models. CutBlur pastes a low-resolution patch into the corresponding high-resolution image region and vice versa. On the contrary, our method is tailored to generate images from layouts and is a creative application of CL. It uses progressive object-level blurring as a CL strategy and solves the leaking issue by progressing from strong blur to clean images during training, thus guiding our model to produce clean images at inference time.Diffusion models achieve excellent sample quality but traditionally suffer from expensive sampling. GANs [9] can generate high-quality images in one step but are typically prone to training instability, such as overfitting and mode collapse [22] . Recently, approaches combining GANs with diffusion models have become popular to address this issue. In [34] , the diffusion model is parameterized by a multimodal conditional GAN to smoothen the data distribution and increase the sampling step size. In [33] , instance noise is injected using a diffusion timestep-dependent discriminator to stabilize training, augment the dataset, and ease the vanishing gradient problem. In contrast, we propose a CL strategy using an objectlevel blurring schedule during training to improve layout-to-image models.An interesting connection, especially when combining our idea with diffusion models, can be made with recent heat dissipation [21] and blurring diffusion [6, 12, 18] models. In [21] , images are generated by stochastically reversing the heat equation, corresponding to a blur operator. Meanwhile, [12] defines blurring through a diffusion process with non-isotropic noise, combining heat dissipation and additive noise. In [18] , each frequency component of an image is diffused at different speeds, resulting in a reverse process that gradually deblurs and removes noise. The pairing of blur with noise as the diffusion mechanism is also proposed in [6] . In contrast to previous research, we maintain the diffusion and sampling processes unchanged. Instead, we adapt the data loader to implement a curriculum learning strategy, facilitating a seamless transition from an easier to a more challenging task. 
method: 
 This section describes our curriculum learning strategy based on progressive object-level blurring using notation similar to that in CutBlur [36] . Our ObjBlur method is straightforward: Given a clean, high-resolution image x HR ∈ R 𝑊 ×𝐻 ×𝐶 and layout ℓ = {(𝑏 𝑖 , 𝑐 𝑖 ) x HR , ℓ ∼ D ⊲ Sample image and layout 3:𝑠 𝑡 = s(𝑡) ⊲ Get blur strength 4:x LR = blur(x HR , 𝑠 𝑡 ) ⊲ Get LR image 5:m = binarize(ℓ) ⊲ Get binary mask 6:if 𝑝 obj ≤ U (0, 1) then 7: x = m ⊙ x LR + m ⊙ x HR ⊲ Blur objects 8:EQUATION)𝐻 𝑡 = (1 -𝑠 𝑡 ) • (𝐻 -𝐻 0 ) + 𝐻 0 (2)Using a bilinear image resizing operation 𝜓 (x, 𝑤, ℎ), we can then generate the low-resolution (LR) image x LR,𝑡 ∈ R 𝑊 ×𝐻 ×𝐶 for the current timestep 𝑡 by first downsampling x HR to 𝑊 𝑡 , 𝐻 𝑡 and subsequent upsampling to match the image resolution of x HR , thus removing high-frequency details while retaining structural content:EQUATIONWe have two options to perform object-level blurring: blur the foreground objects or the background. To produce the former, we can cut-and-paste the object regions of x LR into x HR using the binary mask m to produce xLR→HR . Similarly, we can generate an alternative image with a blurred background by cut-and-pasting the object regions of x HR into x LR to get xHR→LR .EQUATION)where m denotes the inverted mask, and ⊙ is the element-wise Hadamard product. To control how often we want to use an image with blurred objects as opposed to an image with blurred background, we define the probability 𝑝 obj of blurring the objects as opposed to the background, and randomly choose whether to return xHR→LR or xLR→HR for the current sample.Table 1 : Main results. We report the mean and standard deviation over three runs and mark the best mean in bold. Using our proposed ObjBlur schedule, we achieve better performance across global image FID and object-level SceneFID while often reducing the variance at the same time. Note: [43] uses a slightly different evaluation protocol; thus, the scores are not directly comparable to [10, 29] . In other words, for each image sample, we either blur the objects as defined by the layout ℓ or the background before continuing with model training. The blur strength is progressively reduced, using the blur schedule function s(𝑡) and the current training step 𝑡, while the start resolution used to compute x LR defines the initial blur strength. See Figure 2 for a visualization and Algorithm 1 for an algorithmic overview of our method. We analyze different blur schedules s(𝑡), start resolutions, object probability 𝑝 obj and schedule durations in section 5.COCO Visual Genome Model FID ↓ SceneFID ↓ DS ↑ FID ↓ SceneFID ↓ DS ↑ LostGAN [In contrast to CutBlur [36] , which performs fixed blurring of a random patch, our method consists of a semantic-driven curriculum. Through progressive object-level blurring, we do not induce spatial confusion, unrealistic patterns, or loss of semantic content in the training data. Our proposed ObjBlur focuses on object-level blurring and implicitly guides the model into respecting the input layout by learning the boundaries between blurred and non-blurred areas. Because the blurring schedule converges towards clean images, the data augmentation does not leak into the generations of the final model. 
experiment setup 4.1 datasets: 
 We use COCO-Stuff [3] and Visual Genome [17] and follow standard data filter procedures as in the corresponding works [10, 29, 43] . In COCO-Stuff, the annotations contain 80 thing classes (person, car, etc.) and 91 stuff classes (sky, road, etc.). Boxes that are smaller than 2% of the image area are eliminated, and we use images with 3 to 8 objects. Finally, images that belong to crowd are filtered, resulting in 74,777 train and 3,097 val images. In Visual Genome, we select object and relationship categories occurring at least 2000 and 500 times in the train set, respectively, choose images with 3 to 30 bounding boxes, and ignore all small objects, resulting in 62,565 train, 5,506 val, and 5,088 test images.We choose multiple metrics to evaluate our model and compare it with baselines. To evaluate the quality and diversity of the images, we use FID [11] . To assess the visual quality of individual objects, we choose the SceneFID [30] , which corresponds to the FID applied on cropped objects as defined by the bounding boxes, and the classification accuracy score (CAS) [20] , which measures how well an object classifier trained on generated image crops can perform on real image crops. To evaluate the diversity between images generated from the same layout, we adopt the procedure from LayoutDiffusion [43] and use LPIPS [38] as the diversity score (DS) between two images generated from the same layout (one-to-many mapping). As it is a distance metric, higher values indicate greater image diversity.We choose two adversarial [10, 29] and one recent diffusion [43] models and use the official PyTorch implementations available on GitHub to evaluate our proposed ObjBlur schedule. For [29] and [10] , we increase the batch size to 512 and train on 8 NVIDIA RTXA6000 GPUs to speed up training. Given this change, we also retrain the baseline, improving it compared to the reported performance as a side effect. We train for 200 epochs, which takes about two days. For [43] , we leave everything unchanged and train on 8 NVIDIA A100 GPUs for 300k iterations with a batch size of 64, which takes about two days. The image resolution is set to 128×128. We train every model three times and report mean and standard deviation to evaluate training stability. 
results: 
 We first discuss our main results in Table 1 and then perform an extensive analysis on the effect of different hyperparameters using LostGAN [29] and the COCO [3] dataset as a simple baseline. Finally, we critically examine the importance of performing object-level blurring as opposed to full image blurring, random patch blurring (as in CutBlur [36] ), and random mask blurring.Our main results are summarized in Table 1 and Table 2 . ObjBlur significantly improves the performance of LostGAN [29] and CAL2IM [10] across both FID and SceneFID in terms of mean and standard deviation in three runs without any changes to the model architecture or optimization. We achieve a relative improvement of 19.82% in FID and 16.89% in SceneFID on COCO with [29] . With [10] , our 2 shows that ObjBlur produces much more recognizable objects across all tested models and datasets, improving [29] on COCO by 2.44pps, and [43] on Visual Genome by 3.70pps.To better understand the influence of our method, we analyze the performance during training of [29] with and without ObjBlur (Figure 1 ). In terms of FID, our method leads to a much smoother convergence with better final performance. Compared to the baseline, performance improves steadily throughout training time, and the standard deviation across runs is also much lower, especially at convergence. Therefore, our approach can be seen as an effective stabilization and regularization method. We also compare our method's capability to produce diverse images for the same layout using LPIPS as the diversity score and find that ObjBlur achieves comparable scores. In other words, our proposed schedule does not lead to a loss of sampling diversity.The initial blurring strength (i.e., the resolution used to compute LR image regions) plays an important role and must be balanced to 3a . We find that starting with an LR resolution of 4 and 8 works best with steady degradation when starting higher.We propose to select between the blurring of objects or the background at random defined by 𝑝 obj = 0.5. To better understand the impact of the ratio, we conduct additional experiments that range from always blurring the background 𝑝 obj = 0.0 to always blurring the objects 𝑝 obj = 1.0. Our results in Figure 3b show that many configurations lead to better performance, but 50% works best. Applying a blur schedule focusing on the background is generally more stable. We hypothesize that blurring objects too often negatively affects performance due to overfitting on low-resolution object boundaries.The blurring schedule does not necessarily have to run for the entire duration of training. For example, it is possible to use the CL schedule within the first 𝑛 epochs and then fine-tune on clean images for the remaining iterations. We ablate the impact of using it within 70% to 100% of the training time, see Figure 3c and find that the model benefits from a short fine-tuning stage on clean images in the last 5% of epochs. Interestingly, the standard deviation across multiple runs increases towards longer fine-tuning phases on clean images, indicating problematic convergence behaviour. We hypothesize that this is due to shorter adaptation times during the CL schedule.A comparison of generated images using baseline models and our method is shown in Figure 4 for adversarial approaches LostGAN [29] , CAL2IM [10] , and LayoutDiffusion [43] . When using ObjBlur, we find that the images are generally of similar quality. However, looking closer, one can see that our method often produces better images and more recognizable objects in comparison, especially on the GAN-based models. We provide four more pages of visual results in the appendix.We ablate several other functions to compute the current blur strength, see Figure 5 : linear, step functions with step sizes of 4 and 8, power of two steps with an exponential increase of steps to allow more time for adjustment and exponential functions with different rates. Although most achieve better results than the baseline, sin performs significantly better than others, providing an initial warmup and final fine-tuning along a symmetric transition throughout the schedule. Interestingly, an exponentially schedule function emphasizing a long warm-up performs second best, indicating the potential benefits of a kind of "pre-training" on low-resolution images. We leave further exploration to future work.To test the importance of our proposed object-level schedule, we compare it with a CutBlur [36] version of our CL schedule. We use the official implementation to select random patches and keep all other parameters, such as blur strength and schedule functions, constant. The results can be found in Table 3 . Using CutBlur leads to degraded performance and yields a generator that produces blurred patches after training. Combining CutBlur with our schedule improves the baseline, indicating that a blurring schedule is effective. Our proposed ObjBlur, which applies blurring based on semantic information provided by the object annotations, is the best across both FID and SceneFID by a significant margin.A critical question to investigate is whether the benefits of our proposed blurring schedule are due to semantic differentiation between foreground objects and background or if similar performance can be achieved by choosing the correct amount of blurring on a dataset level. To answer this question, we perform a random mask assignment experiment. We reuse the object-based masks computed with our proposed approach and shuffle the image-to-mask assignment during training. This will effectively keep the amount of blur on a dataset level constant and answer whether the semantic-driven masks are significant. Alternatively, we test whether applying the blur on the entire image is beneficial. Different blur techniques are visualized in Figure 6 . Our results in Results in Table 3 .semantic-driven blurring schedule is critical and that performance suffers whenever objects are inconsistently blurred. 
limitations & future work: 
 While our method can stabilize training and improve performance, there are a few limitations and possibilities for future work. We exclusively studied blurring as an augmentation function. The potential applicability of other augmentations, such as additive noise, remains unexplored. Uniform blurring across all images and object classes ignores object and sample difficulty differences. Investigating a dynamic blurring schedule for individual objects, object classes, or specific samples could benefit future research. Although our findings underscore the necessity of object masks, we also demonstrate that combining CutBlur [36] with our CL schedule surpasses baseline performance. Further studies are needed to determine whether similar performance can be achieved using masks without requiring object annotations. Combining ObjBlur with blurring diffusion models such as in [6, 12] would be interesting. Finally, we ask whether our approach could work on single-object datasets such as ImageNet by blurring important (object-centric) areas and whether it benefits low-data regime scenarios. 
reproducibility & ethics statement: 
 Our method can easily be reproduced as it only requires changes to the data loader of existing layout-to-image models to apply a progressive object-level blur using standard down-and upsampling operations. We show an example implementation in the appendix. Our method is plug-and-play and improves existing layout-to-image generation models. As such, it inherits risks such as being misused to spread fake news, invading privacy, and potential copyright issues due to using real-world datasets. To counter these issues, it's important to develop advanced deepfake detection technologies and enforce ethical guidelines to differentiate synthetic images from real ones, ensuring responsible use of the technology. 
conclusion: 
 In this work, we introduced ObjBlur, an innovative curriculum learning strategy based on object-level blurring that significantly improved layout-to-image generation models. Our approach reaches  
 
 Appendix:  


Decision: This paper receives two borderline accept and one borderline reject. Reviewers acknowledge the novelty of the paper but raises several questions regarding the method design and experiments. Authors addressed the questions during rebuttal. After reading the paper, reviews, and rebuttal, the area chair decides to recommend acceptance and suggest the authors to take the reviewers' comments into consideration for the camera-ready version.

Reviews: review: paper_summary: paper_summary: This paper introduces a novel method called ObjBlur, which leverages curriculum learning through progressive object-level blurring to enhance the quality and stability of layout-to-image generation models. This approach significantly improves image quality, training stability, and achieves state-of-the-art results on complex datasets in COCO and Visual Genome.
strength_weakness: strength_weakness: The use of progressive object-level blurring as a curriculum learning strategy is a novel and effective method to improve training stability and image quality in layout-to-image generation models. ObjBlur demonstrates significant improvements in image generation quality and training stability across various models and datasets, achieving state-of-the-art results on COCO and Visual Genome datasets.
limitations: limitations: Strong initial blurring could lead to overfitting on the boundaries of objects, especially if the transition to clean images is not well-balanced. 
The method assumes that progressively reducing blur uniformly increases task complexity in a manner beneficial for all types of images and objects. However, different objects and scenes may have varying inherent complexities that are not adequately addressed by a uniform blurring schedule. 
The effectiveness of ObjBlur hinges on the availability of accurate object annotations. In real-world scenarios, obtaining precise annotations can be challenging, and any inaccuracies can negatively impact the training process. This dependency limits the method's applicability to datasets where high-quality annotations are not available or feasible to obtain.
While the method shows improved performance on specific datasets, its ability to generalize to unseen data distributions is theoretically uncertain. The training improvements observed might be specific to the characteristics of the datasets used (e.g., COCO and Visual Genome) and may not translate to datasets with different properties.
paper_summary: paper_summary: The paper mainly uses the Curriculum Learning method to alleviate the difficulty of the layout-to-image task and improve the model training effect. It can be applied to existing methods in a plug-and-play manner. The main method is to guide the model to learn layout-to-image generation from easy to difficult by blurring images with different strength, and by randomly selecting the foreground or background of the blur, the model focuses on layout information by learning the boundary between the foreground and background. Experiments show the gain effect on GAN and Diffusion Models after using this method, and extensive ablation experiments are conducted.
strength_weakness: strength_weakness: 1. This method is the first to use Curriculum Learning for layout to image tasks, optimizing the learning process; In addition, to enhance the processing of layout information, a random blur foreground and background method is adopted. Overall, it is simple and effective, with innovation
2. The experimental part applied this method on GAN and Diffusion Models, demonstrating its efficiency in image quality and diversity. Sufficient ablation experiments were conducted to demonstrate the effectiveness of the method in training stability, initial blur intensity selection, blur ratio and schedule design.
limitations: limitations: 1. This paper designs a random blurring method to optimize layout learning. Although they explains that its main mechanism is to enhance the model's attention to boundaries between the foreground and background, there is no in-depth and detailed analysis on how this method is beneficial for the model to process layout information.
2. Although this method performs significantly on GAN models, its gain on Diffusion Models is relatively not significant, while in many metrics (such as SceneFID), DM performs the best as a baseline (possibly related to its training method for denoising learning)
paper_summary: paper_summary: This paper proposes a novel curriculum learning approach designed to enhance layout-to-image generation models by progressively applying object-level blurring during training. The method leverages blurring as an intuitive way to adjust task difficulty, maintaining structural content while progressively introducing high-frequency details to the model. This approach stabilizes training, improves image quality, and smooths convergence by gradually transitioning from strong blurring to clearer images. The effectiveness of the proposed method is demonstrated across various datasets.
strength_weakness: strength_weakness: 1. The paper is well-organized and clearly written.
2. The proposed method is both novel and effective.
3. Extensive experiments validate the method's effectiveness in diverse settings.
4. The visualization results are impressive.
5. The limitations of the method are thoroughly discussed.
limitations: limitations: 1. The authors should incorporate error analyses, such as confusion matrices, to provide a deeper understanding of the proposed method's performance.
2. Results for different hyper-parameters (hyper-parameter sensitivity analyses) should be provided to evaluate the robustness of the method.
3. The proposed method applies a uniform blurring strategy across all images and object classes, which does not account for the varying difficulty levels of different objects and scenes. 
4. The method relies heavily on the availability of accurate object annotations to generate the blurring masks. This dependency could limit the applicability of the approach to datasets where such detailed annotations are not available.
5. The progressive blurring technique adds additional computational steps during the data loading process, which could increase the overall training time.