DAFT-GAN: Dual Affine Transformation Generative Adversarial Network for Text-Guided Image Inpainting

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 09:39:44

Abstract: In recent years, there has been a significant focus on research related to text-guided image inpainting, which holds a pivotal role in the domain of multimedia processing. This has resulted in notable enhancements in the quality and performance of the generated images. However, the task remains challenging due to several constraints, such as ensuring alignment between the generated images and the accompanying text, and maintaining consistency in distribution between corrupted and uncorrupted regions, for achieving natural and fine-grained image generation. To address these challenges, previous studies developed novel architectures, inpainting techniques, or objective functions but they still lack semantic consistency between the text and generated images. In this paper, thus, we propose a dual affine transformation generative adversarial network (DAFT-GAN) to maintain the semantic consistency for text-guided inpainting. DAFT-GAN integrates two affine transformation networks to combine text and image features gradually for each decoding block. The first affine transformation network leverages global features of the text to generate coarse results, while the second affine network utilizes attention mechanisms and spatial of the text to refine the coarse results. By connecting the features generated from these dual paths through residual connections in the subsequent block, the model retains information at each scale while enhancing the quality of the generated image. Moreover, we minimize information leakage of uncorrupted features for fine-grained image generation by encoding corrupted and uncorrupted regions of the masked image separately. Through extensive experiments, we observe that our proposed model outperforms the existing models in both qualitative and quantitative assessments with three benchmark datasets (MS-COCO, CUB, and Oxford) for text-guided image inpainting.

Main paper: 
 introduction: 
 Image inpainting [1] [2] [3] [4] [5] [6] [7] [8] [9] is a process that involves realistically filling in missing or corrupted areas within an image, serving a vital role in practical image processing applications such as image reconstruction, photo editing, and completing obscured regions. The core strategy in image inpainting involves predicting missing pixels by leveraging features or patterns from uncorrupted areas of the image. While these models can produce high-quality results, they face challenges in complex scenarios that demand intricate details, especially when multiple objects are involved or when the corrupted region covers a substantial portion of the image. Relying solely on surrounding pixels for generation in such cases can lead to distorted content or artifacts, prompting further research to address these limitations.The pipeline typically follows a generative adversarial network (GAN) inpainting architecture, initially proposed in [10] and later refined by [11] to incorporate global and local aspects in the discriminator. However, addressing irregular mask holes has remained a challenge, prompting the introduction of a coarse-to-fine network [7] that includes a refinement process to enhance the generation quality. Additionally, previous studies [12, 13] proposed improved convolution techniques for feature encoding, while recent studies [14, 15] have demonstrated the benefits of utilizing fast Fourier convolutional (FFC) blocks to extract global and spatial features separately and modulate them individually for improved appearance quality. Nevertheless, accurately generating corrupted areas in complex images lacking specific patterns remains a significant challenge, as relying solely on surrounding pixels may yield semantically diverse solutions, necessitating time-consuming and resource-intensive processes to select the desired output. In specific advanced inpainting tasks, therefore, incorporating external guidance to inpainting models is necessary to control the solution space and efficiently generate desired outcomes. Since external guidance, such as lines [12] , edges [16] , sketches [17] , or exemplars [18, 19] , may offer weak visual directionality and also lack semantic context, text guidance is considered the most effective form of guidance in inpainting tasks to ensure semantic consistency. Thus, text guidance is widely used in other tasks such as text-to-image synthesis and text-guided manipulation. Text-to-image synthesis aims to generate complete images that accurately reflect the text description, while text-guided image manipulation involves modifying existing image content to align better with the text description. Inpainting tasks involve much stricter constraints compared to tasks such as text-to-image synthesis and image manipulation, as they require generating local regions of images. It is crucial to consider whether the text is accurately reflected in the generated image, if the remaining uncorrupted regions are coherent with the generated pixels, and other relevant factors.The task of text-guided inpainting is advancing towards generating high-quality images even in more complex cases such as filling in large masked regions in an image by utilizing text descriptions as external guidance. To do this, it is crucial to effectively utilize features from the uncorrupted regions of the image. Additionally, combining the features of the image with the text features used as guidance is also vital. The attention mechanism is the most commonly used when combining features from multimodal data. In textto-image synthesis, the attention mechanism was first proposed in AttnGAN [20] , demonstrating the effectiveness of combining text features with image features. Also, a deep attentional multimodal similarity model (DAMSM) [20] was proposed for extraction of sentence and word embeddings, which are crucial features when utilizing the attention mechanism. Similarly, in this study, we utilized the pre-trained DAMSM [20] to extract sentence and word embeddings. A recurrent affine network [21] was utilized when combining sentence embeddings with images, and a network with attention added to the existing affine network was employed for refinement at the word level when combining word embeddings with images. By gradually stacking this two-path decoder block, fine-grained images that effectively reflect text were generated. While most existing studies encode features of corrupted images through a single convolution, we propose separated mask convolution blocks to distinguish between corrupted and uncorrupted regions, minimizing the information leakage of uncorrupted image features. As shown in Fig. 1 , Our proposed model successfully generates corrupted regions on three benchmark datasets.The main contributions of this paper are listed as follows.• We propose a novel text-guided inpainting model (DAFT-GAN) for generating fine-grained images with text descriptions.  
related work 2.1 text-to-image synthesis: 
 Text-to-image synthesis is a task aimed at generating complete images using text descriptions. The most crucial aspects in text-toimage synthesis are the authenticity of the generated images and the semantic consistency between the provided text description and the generated images.The foundation of text-to-image synthesis models, such as Stack-GAN [22] , utilizes a multi-stage approach to generate high-resolution images reflecting the input text. This architecture not only gradually generates high-resolution images through stages but also proposes a stacked structure to address the unstable nature of GAN models. Subsequent research [20] focused on combining text and image features, with attention mechanisms being proven effective through such studies. However, attention mechanisms also have drawbacks when generating high-resolution images since they require attention at every scale, which incurs substantial computational costs. This computational burden can make it challenging to fully utilize text information at certain scales. To address these issues, DF-GAN [23] proposes an efficient fusion network that utilizes affine transformation networks. By combining contextual information without attention in every block, it offers significant cost advantages. While this simple and efficient network can produce images with better quality, some limitations still exist. Some regions of the images may not be recognizable or consistent with the text description at the word level. To enhance this, SSA-GAN [24] introduces weakly supervised mask predictors to guide spatial transformations. RAT-GAN [21] adds recurrent networks when connecting affine transformation blocks to address the long-term dependency problem. Through research that effectively combines text and images, the quality of image generation has been improved.Image inpainting involves the reconstruction of specific uncorrupted regions within an image, serving as a critical component in various image processing applications such as photo manipulation and filling in occluded areas. Despite its importance, image inpainting remains a challenging task, with the level of difficulty being heavily influenced by the extent of the damaged regions in the image. When a significant portion of the image is corrupted, there is a higher probability of entire objects being lost from the visual context. Moreover, the absence of usable features from uncorrupted areas further complicates the restoration process. Recent advancements in deep learning models have significantly enhanced the ability to extract and leverage high-level semantic features, resulting in notable improvements in terms of the quality of generated images. Recent studies investigated different model architectures, including transformer-based [25] [26] [27] [28] , GAN-based [15, [29] [30] [31] [32] , or FFC-based [14, 15] models. In addition to model architecture, performance improvements have been achieved through the application of various inpainting techniques such as convolution [12, 13] and attention mechanisms [20] .The goal of the text-guided image inpainting task is to reconstruct corrupted images using textual and visual information to generate realistic images. MMFL [33] proposes a multimodal fusion learning approach that focuses on text descriptions corresponding to objects of interest in each image through the word demand module. Additionally, they construct it as a two-stage coarse-to-refine process to generate high-quality images. TDA [34] proposes a dual multimodal attention module that aims to achieve deeper integration by applying attention to feature maps of two inverted regions. However, all these methods attempted multimodal fusion at the encoding stage and did not utilize the provided text information during the decoding stage. This leads to an incomplete fusion of spatial details, text, and visual information. Furthermore, previous studies [33] [34] [35] [36] utilize structures with two or multiple stages for image refinement, resulting in significant time and space consumption due to repeated encoder-decoder pairs. Therefore, we designed a network that injects text description information at the decoding stage to generate images. This strategy not only mitigates resource inefficiencies but also improves image quality through the utilization of a one-stage-dual-path architecture. 
methods: 
 Given a masked image 𝐼 𝑀 = 𝐼 ⊙ 𝑀 (where 𝐼 is the original image, 𝑀 is given mask metric), text-guided image inpainting is the task of generating an image that aligns with the text description 𝑡 and maintains semantic consistency with the image. We propose DAFT-GAN, which generates images by appropriately processing text and visual features in the decoder stage.As shown in Fig. 2 , the proposed model consists of an encoder composed of separated mask convolution (SMC) blocks and a decoder composed of dual affine transformation (DAFT) blocks forming the Conv-U-Net structure. More specifically, the process of feature extraction involves seven SMC blocks, while text features are integrated at various resolutions through the incorporation of seven DAFT blocks to produce the final image. A comprehensive breakdown of all elements of the approach is provided as follows.Seperated Mask Convolution. SMC block performs two roles. As shown in Fig. 3 , one is to distinguish the masked and the unmasked regions and conduct convolution and normalization separately on them. Another is to update the mask in a way that minimizes information leakage. SMC block takes as input a pair of the image feature 𝐹 𝑒 𝑖 ∈ R 𝐶 𝑖 ×𝑊 𝑖 ×𝐻 𝑖 extracted from the previous block and a mask metric 𝑀 𝑖 ∈ {0, 1} 𝑊 𝑖 ×𝐻 𝑖 of the same size. The mask metric represents the mask status, where the unmasked (valid) regions are marked as 0, and the masked (invalid) regions are marked as 1. The mask metric is used to differentiate between the two regions during the encoding process. The initial 256 × 256 masked image and mask metric are downscaled by a factor of 2 in 6 out of the 7 blocks, excluding the first block, resulting in a final output size of 4 × 4.First, from the previous block, we passed the input feature 𝐹 𝑒 𝑖 through two different convolution layers to obtain a valid feature𝐹 𝑣𝑎𝑙 𝑖+1 ∈ R 𝐶 𝑖+1 × 𝑊 𝑖 2 × 𝐻 𝑖2 and an invalid feature𝐹 𝑖𝑛𝑣𝑎𝑙 𝑖+1 ∈ R 𝐶 𝑖+1 × 𝑊 𝑖 2 × 𝐻 𝑖2 . If we use only one convolutional layer to extract features, the convolutional weights could be updated by being influenced not only by the valid region but also by the invalid region during training. Therefore, we used two convolution layers.EQUATIONEQUATIONThen, we generated𝐹 𝑒 𝑖+1 ∈ R 𝐶 𝑖+1 × 𝑊 𝑖 2 × 𝐻 𝑖 2 composed of 𝐹 𝑣𝑎𝑙 𝑖+1 and 𝐹 𝑖𝑛𝑣𝑎𝑙 𝑖+1 . At this point, using 𝑀 𝑖+1 ∈ {0, 1} 𝑊 𝑖 2 × 𝐻 𝑖2 obtained by updating 𝑀 𝑖 , we assigned 𝐹 𝑣𝑎𝑙 𝑖+1 to the valid region of 𝐹 𝑖+1 and 𝐹 𝑖𝑛𝑣𝑎𝑙 𝑖+1 to the invalid region. The mask metric update process is explained in detail as follows.EQUATIONIn 𝐹 𝑒 𝑖+1 , the valid and invalid regions have distinctly different distributions. To prevent the loss of information in the valid region, we applied MaskNormalize, the normalization method used in [37] . Mask Update. When performing convolution operations on image features, both the valid and invalid regions are inevitably considered at the boundary of the mask due to the receptive field. In response to this situation, we applied a mask update method that treats as a valid feature if any part of the valid region is included to minimize the loss of valid information. We used a pooling layer with the same kernel size, stride, and padding as the convolution layer to extract the image features. By doing so, when performing convolution operations on a specific region of the image feature, we can also perform pooling operations on the exact corresponding region for the mask metric. If any part of the valid region is included in the receptive field, i.e., the region in the mask metric contains both 0 and 1, we need the pooling result for that region to be 0 in order to consider the convolution result as a valid feature. Therefore, we implemented a min-pooling operation to update the mask as follows.EQUATIONGlobal Fusion Path. As depicted in Fig. 2 , the generator generates masked parts by combining the encoded feature with the U-Net structure [38] . A noise vector 𝑧 is sampled from a standard Gaussian distribution, passed through a fully connected layer, and then added to the encoded feature of the same dimension at the beginning of the generator. For each upsampling block, the encoded feature of the same scale was combined with the feature of the previous block through element-wise addition.𝐹EQUATION𝐹EQUATIONwhere 1 ≤ 𝑖 ≤ 𝐿 (𝐿 is the highest level with the smallest spatial size) Unlike the basic U-Net [38] , this model combined features using residual connections instead of channel-wise concatenation. This was done to encourage the network output to closely resemble the input during the early stages of training, which stabilizes the learning process. Additionally, residual learning generally helps the model preserve previous information at each scale and learn high-frequency contents more effectively. The input 𝐹 𝑔 𝑖𝑛 𝑖 passes through each recurrent affine transformation (RAT) module [21] , performing the transformation for upsampling. It sequentially goes through the initial generator block and 6 upsampling blocks (the first generator block does not upsample), ultimately generating a feature map as a size of 256 × 256.Multimodal Cross Affine Transformation. As shown in Fig. 4 , multimodal cross affine transformation (MCAT) takes the RAT output image feature maps 𝐹 𝑔 𝑜𝑢𝑡 𝑖 and word features 𝑤, hidden features ℎ 𝑡 within the same DAFT block as input, and outputs same scale image feature maps 𝐹 𝑠 𝑜𝑢𝑡. The core of the MCAT module is the CrossAffine layer shown in Fig. 5 . The CrossAffine layer performs cross-attention between image feature maps and word features, and obtains attention feature maps 𝐹 spatial ∈ R (2×𝑑 𝑤 ) ×𝐻 ×𝑊 through channel-wise concatenation with hidden features ℎ 𝑡 , and then extracts modulation parameters 𝛾 𝑐 and 𝛽 𝑐 through channelwise multi-layer perception (MLP) for affine transformation. By applying channel-wise MLP to predict modulation parameters while maintaining spatial structure and identifying where text information needs to be complemented in the current image features, spatially refined image feature maps can be obtained. Specifically, given input image feature maps 𝑥 𝑐ℎ𝑤 ∈ R 𝐶 ×𝐻 ×𝑊 , word features 𝑤 ∈ R 𝐿×𝑑 𝑤 , and hidden feature ℎ 𝑡 ∈ R 𝑑 𝑤 , the image feature maps were transformed into the same semantic space as word features by a new perceptron layer 𝑊 𝑄 ∈ R 𝑑 𝑤 ×𝐶 . The query 𝑄 𝐹 = 𝑊 𝑄 𝑥 𝑐ℎ𝑤 ∈ R 𝑑 𝑤 ×𝐻 ×𝑊 , and key 𝐾 𝐹 , value 𝑉 𝐹 are word features 𝑤. Then, it performs cross-attention to obtain image feature maps 𝐹 spatial for predicting modulation parameters as follows:EQUATIONEQUATIONwhere SpatialReplication expands the dimensions by 𝐻 × 𝑊 . By combining the sentence hidden feature ℎ 𝑡 from the long short-term memory with channel-wise concatenation with 𝐹 attn , necessary global semantic information at current stage was injected, at the same time, global-spatial connection was implemented. Additionally, modulation parameters 𝛾 𝑐 and 𝛽 𝑐 were predicted through a channel-wise MLP, and channel-wise affine transformation and residual connection are applied to input feature maps 𝑥 𝑐ℎ𝑤 to generate final output feature maps x𝑐ℎ𝑤 . This process can be represented by the following equation:EQUATIONwhere 𝛾 𝑐 and 𝛽 𝑐 are obtained through two channel-wise MLP layers.EQUATIONEQUATIONEQUATION)In order to learn a wider range of expressions, as described in Fig. 4 , CrossAffine layer and convolution layer were stacked to form a single MCAT module.One-Stage-Dual-Path. Inspired by recent research on image inpainting [15] , we proposed a one-stage-dual-path approach that utilizes global text features to manipulate images, focusing on a global path for manipulating global structure and a spatial path for concentrating on spatial details. As shown in Fig. 4 , the global path generates a semantic-consistent global structure through the RAT module, while the spatial path leverages more semantic-consistent and fine-grained visual details through word-level and pixel-level control using the MCAT module for each scale of the output from the global path. Additionally, by upsampling the previous spatial path output and connecting it to the current output through a residual connection, the model is guided to transform only the necessary parts gradually at each step, enabling initial learning stabilization and finer control over details.This structure resolves the time and space waste issues derived from the two-stage or multi-stage structure of the existing textguided image inpainting [33, 35, 36] , and has the advantage of generating natural images from a human evaluation perspective through organic integration and detailed role differentiation between the two paths. Adversarial Loss with MA-GP. To ensure semantic consistency between the inferred image and the given text description, matchingaware zero-centered gradient penalty (MA-GP) was used [23] .EQUATION)where 𝑠 is the given text description, ŝ is the mismatched text description, 𝑥 is the actual image corresponding to 𝑠, and x is the generated image. 𝐷 (.) is the output of the discriminator, providing matching information between the image and the sentence. The 𝑘 and 𝑝 are hyperparameters of the MA-GP.Reconstruction Loss. The ℓ 1 loss is typically optimized for an average blurry result. Despite being a non-saturating function, we incorporated perceptual loss to improve the naturalness and quality of the final image. where 𝜙 𝑖 (.) refers to the layer activation of the pre-trained VGG-19 network.EQUATIONAdversarial Loss. Adversarial loss is defined as follows.EQUATIONwhere x is generated images.Text-Guided Attention Loss. To enhance text guidance, we implemented the text-guided attention loss [33] . This method involves multiplying the attention map and the generated image x at the final scale of 256 × 256 with the ground-truth image 𝑥, and minimizing the ℓ 1 loss of the two terms.EQUATIONwhere 𝐴(.) performs attention and 𝑤 is the word features of the text corresponding to 𝑥.DAMSM Loss. For fine-grained image-text matching that considers both sentence-level and word-level information, we adopted the DAMSM loss (𝐿 𝐷𝐴𝑀𝑆𝑀 ). Details are described in [33] .Overall Loss. The total loss of the generator is defined as below.EQUATION 
experiments 4.1 datasets: 
 We used the Caltech-UCSD Birds-200-2011 (CUB-200-2011), Oxford-102 Category Flower (Oxford-102), and MS-COCO datasets to train our proposed model and methods, focusing on multimodal inpainting that incorporates both text and images. While all three datasets were used to assess the performance of text-guided inpainting, the MS-COCO dataset was specifically used as a more challenging dataset for evaluating the performance of image inpainting. This is because the MS-COCO dataset contains multiple objects and relatively complex scenes compared to the other two datasets. When training or testing our proposed model, we used randomly generated masks in irregular shapes. By doing so, we can construct a more robust model for inference and demonstrate the ability to generate realistic images even when large corrupted regions are present in the image. Subsequently, we evaluated the performance of the proposed model through several experiments. We confirmed the effectiveness and competitiveness of our proposed model by conduncting quantitative in Table 1 and qualitative evaluations in Table 2 with the other models proposed in previous studies. After that, to validate the effectiveness and performance improvement of each method in the final model we selected, we sequentially evaluated the application of methods from the baseline model as shown in Table 3 . We utilized a training approach with diverse irregular masks. We conducted the experiment on an NVIDIA RTX3090 Ti GPU. The parameters of the generator in the model were optimized with a learning rate of 10 -4 , while the parameters of the discriminator were optimized with a learning rate of 4 × 10 -4 , both using the Adam optimizer. The sentence embedding and word embedding utilized in this study were extracted using a pre-trained text encoder from the DAMSM proposed in AttnGAN [20] . The final objective function weights were set as 𝜆 𝑟𝑒𝑐 = 0.2 and 𝜆 𝐷𝐴𝑀𝑆𝑀 = 0.01The image inpainting task allows for evaluating the quality of generated images by comparing how well the corrupted parts are reconstructed to the ground truth. To assess the quality of generated images, the Fréchet inception distance (FID) [39] and the kernel inception distance (KID) [40] were used to measure the distribution between ground-truth and generated images. When evaluating reconstruction, the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM) [41] were used to measure the difference between generated images and ground-truth at the pixel level. Lower values for FID and KID, and higher values for PSNR and SSIM indicate better performance. As shown in Table 1 , we observed that our proposed model has a significant impact on integrating text and achieving high scores compared to the existing models on all three datasets, with a particularly significant improvement seen on the MS-COCO dataset. Despite being relatively challenging due to its inclusion of multiple objects and complex scenes, the significant improvement on the MS-COCO dataset indicates effective semantic matching between text and images. Despite the significant improvements in FID and KID scores across all datasets, the improvements in PSNR and SSIM are relatively modest. This phenomenon may be explained by the distinction that PSNR and SSIM metrics evaluate distances at the pixel level, while our model operates on feature-level distances to produce images that exhibit a more realistic appearance. Consequently, the improvement in reconstruction metrics compared to previous models that rely solely on pixel-level distances might be lower. Nonetheless, it is evident that in some cases, other models achieve similar or even higher reconstruction scores.In the context of generative models, qualitative evaluation from a human perspective is also important. In the text-guided image inpainting task, the most commonly used metrics for qualitative evaluation are semantic consistency and naturalness. Semantic consistency measures how well the generated image matches the given text, while naturalness measures how natural the generated image appears. We conducted a human evaluation with 15 volunteers with 100 randomly generated images, ranging from rank 1 (high quality) to rank 5 (low quality), to obtain scores for semantic consistency and naturalness. As scored in Table 2 , we observed that our model not only outperforms the other models [8, 9, 33, 34] in quantitative evaluation but also achieves the best scores in qualitative evaluation.As evident from the metrics in Fig. 6 , while other models generated awkward images with blurriness or artifacts, our model generated natural-looking images without any anomalies compared to the ground truth. To ensure high-quality image generation across all datasets, we conducted experiments with various text descriptions using our proposed model. In Fig. 7 , we confirmed the generation of photorealistic images on the CUB and the Oxford datasets. Particularly in Fig. 8 , despite the diverse object types and more complex scenes, our model effectively incorporated the text descriptions to generate remarkably natural images, even for the challenging MS-COCO dataset.In Table 3 , we conducted an ablation study on the CUB-200-2011 dataset to assess the performance changes resulting from sequentially applying our proposed methods to the baseline model. The baseline model utilizes a basic convolutional block to encode the features of the masked image and employs only one affine transformation network [23] along with sentence features. From this baseline, we measured the performance changes when adding the weakly supervised mask predictor proposed in [24] , and when adding the recurrent affine network proposed in [21] . Then, we evaluated the performance when adding cross refinement, which involves using two affine transformation networks to refine by integrating word features. Additionally, we evaluated the performance by incorporating the DAFT Block, which connects the features of each encoder to the decoder using residual connections. Then, we examined the impact of replacing the basic block with the SMC block in our final model. Finally, to build a robust inpainting model that is independent of the shape of corrupted masks, we trained our model using various irregular masks. By sequentially applying our proposed methods, we confirmed significant improvements in performance based on the evaluation metrics.Through text guidance, we can manipulate specific parts to create desired images or generate diverse images that are semantically consistent with various text descriptions. In other words, it is possible to extend text-guided image manipulation using our trained model. We confirmed it through the following process. First, mask accurately locate and generate modifications based on the provided text description. This experiment shows that it is possible to generate natural images that are semantically consistent with any arbitrary mask, and it indicates the potential for expansion into image manipulation.Our goal is to develop a robust model capable of effectively reconstructing images corrupted by various sizes and shapes encountered in real-world scenarios. Therefore, we evaluated the performance of irregular random masks across different ratio ranges 20%-30%, 30%-40%, 40%-50%, and 20%-50%. As shown in Table 4 , the FID and KID metrics steadily worsen as the mask ratio increases from 20% to 50%. This is an unavoidable result, as the available clues for predicting the corrupted regions diminish as the mask size grows larger. However, TDA and our model demonstrated relatively robust performance and ours achieved the best performance across all mask ratio ranges on the FID and KID metrics. 
conclusions: 
 In this study, we propose a novel model (DAFT-GAN) to address the current challenges for text-guided image inpainting. Our approach involves the integration of two affine transformation networks to progressively incorporate text and image features, thereby enhancing the semantic consistency between the generated images and associated text descriptions. Using global text features, our model initially generates coarse results, which are subsequently refined using spatial details, leading to an overall improvement in the quality of the generated images. Additionally, we propose SMC blocks to reduce information leakage of uncorrupted features by encoding corrupted and uncorrupted regions of the masked image separately. Strikingly, we demonstrate that our proposed method outperforms existing methods in terms of both qualitative and quantitative evaluations. Our contributions represent significant advancements in text-guided image inpainting and pave the way for further research in the field of text-guided image inpainting.  
daft-gan: dual affine transformation generative adversarial: 
 Network for Text-Guided Image InpaintingText (top): "a room filled with furniture and accessories in a room. ", Text (bottom left): "the bird is colorful and has black eyerings a spiky tan crown and gray wings. ", Text (bottom right):"this is an orange flower with green stamen and black stripes near the ovary. " techniques, or objective functions but they still lack semantic consistency between the text and generated images. In this paper, thus, we propose a dual affine transformation generative adversarial network (DAFT-GAN) to maintain the semantic consistency for text-guided inpainting. DAFT-GAN integrates two affine transformation networks to combine text and image features gradually for each decoding block. The first affine transformation network leverages global features of the text to generate coarse results, while the second affine network utilizes attention mechanisms and spatial of the text to refine the coarse results. By connecting the features generated from these dual paths through residual connections in the subsequent block, the model retains information at each scale while enhancing the quality of the generated image. Moreover, we minimize information leakage of uncorrupted features for finegrained image generation by encoding corrupted and uncorrupted regions of the masked image separately. Through extensive experiments, we observe that our proposed model outperforms the existing models in both qualitative and quantitative assessments with three benchmark datasets (MS-COCO, CUB, and Oxford) for text-guided image inpainting. 
 
 Appendix:  


Decision: This paper initially received mixed scores, including one "borderline reject," one "weak reject," and one "borderline accept." After the rebuttal, the reviewers were very satisfied with the additional experimental results and related explanations, and all reviewers raised their scores. Ultimately, this paper received two "borderline accept" and one "weak accept" scores. Based on this, I recommend accepting this paper. I hope the authors can include the additional results from the rebuttal in the final version of the paper, as requested by the reviewers.

Reviews: review: paper_summary: paper_summary: This paper proposes a Dual Affine Transformation Generative Adversarial Network to maintain the semantic consistency for
text-guided inpainting. DAFT-GAN integrates two affine transformation networks to combine text and image features gradually for each decoding block. The experiments on three benchmark datasets show its effectiveness.
strength_weakness: strength_weakness: 1. This paper is well-written and the method is easy to follow.

2. This paper conducts comprehensive experiments by employing multiple metrics to evaluate the model's performance on three benchmarks.
limitations: limitations: The major concern with this paper is its inadequate comparative methodology. The experimental section lacks a comparison with the latest methods that have been proposed in recent years, which hinders the paper's ability to substantiate its claim of "outperforming existing models." There have been numerous advancements in text-guided image inpainting based on diffusion, and while it may be unfair to compare the authors' small model with these pretrained models, it is essential for the authors to introduce and discuss them in the related work section to ensure the paper's timeliness and comprehensiveness. I recommend that the authors focus on highlighting the advantages of their own lightweight and fast-generating models when comparing them to diffusion models, emphasizing the aspects of model efficiency and generation speed. This will provide a clearer understanding of the strengths of their small model when compared to diffusion models.
paper_summary: paper_summary: This paper focuses on text-guided image inpainting, and proposes a dual affine transformation generative adversarial network (DAFT-GAN).   Specifically, DAFT-GAN introduces two affine transformation networks to combine text and image features gradually.  The first one leverages text global features generate coarse results. The second one utilizes attention mechanisms and spatial of the text to refine the coarse results. The output reults of these dual paths are connected by residual connections.
strength_weakness: strength_weakness: The experimental results are sufficient, all the proposed modules are well-ablated. The paper is well-organized and esay to follow.
limitations: limitations: Considering the rapid development of the field of image generation and editing, the baselines compared in the paper are too old. For example, TDA is proposed in  2020, MMFL is proposed in 2020, PDGAN is proposed in 2021, which significantly decrease the value of the paper at current time. I also recommend the authors to compare with Stable Diffusion based inpainting methods.
paper_summary: paper_summary: The paper presents DAFT-GAN, a new model designed to enhance text-guided image inpainting by ensuring semantic consistency between text descriptions and generated images. DAFT-GAN uniquely uses two affine transformation networks for a gradual merger of text and visual features, delivering refined image quality. The model advances existing methods with innovative features: a recurrent affine network for integrating sentence embeddings, an attention-equipped network for word-level refinement, and separate encoding of damaged versus undamaged image regions. The proposed DAFT-GAN model demonstrates superior performance over current models on MS-COCO, CUB-200-2011, and Oxford-102 datasets, which underscores its effectiveness in generating fine-grained, textually consistent images.
strength_weakness: strength_weakness: 1. The innovative incorporation of two affine transformation networks allows for a gradual combination of textual and visual features, significantly improving the semantic consistency between text descriptions and generated images.
2. Demonstrating state-of-the-art performance on three notable benchmark datasets (MS-COCO, CUB-200-2011, and Oxford-102) underscores the model’s effectiveness and superiority over existing image inpainting methods.
3. The model uniquely leverages both global features from the text for initial coarse generation and spatial text features through attention mechanisms for refined output, ensuring the generated images closely align with the textual input.
4. The introduction of separated mask convolution blocks to differentiate between corrupted and uncorrupted areas offers a novel approach to handling image inpainting, setting the research apart from conventional single convolution methods.
limitations: limitations: 1. Image and Citation Placement: he current arrangement of images and their corresponding citations is not optimal, leading to potential confusion and disruption in the reader's engagement with the main text. 
2. Analytical Depth and Insight: More analyses and insight are needed in the experiments. Some metrics reported do not outperform those in previous literature, yet the paper lacks a comprehensive discussion on the possible underlying reasons or the implications of these outcomes. A more in-depth analysis could provide valuable insights, explore potential limitations of the current methodology, and suggest areas for future improvement.
3. The discussion regarding the calculation of the loss function, while technically sound, has been misplaced in the main content of the paper. Considering that this is a widely recognized standard practice, it would be more fitting to include such discussions in the background section of the paper. 
4. Some of the formulas are not clearly explained. Specifically, the explanation surrounding Equation (14) lacks clarity regarding the meaning and application of the expression for /sigma_i. This lack of clarity can lead to confusion among readers.