Multi-Tailed, Multi-Headed, Spatial Dynamic Memory refined Text-to-Image Synthesis

Abstract: Synthesizing high-quality, realistic images from text-descriptions is a challenging task, and current methods synthesize images from text in a multi-stage manner, typically by first generating a rough initial image and then refining image details at subsequent stages. However, existing methods that follow this paradigm suffer from three important limitations. Firstly, they synthesize initial images without attempting to separate image attributes at a word-level. As a result, object attributes of initial images (that provide a basis for subsequent refinement) are inherently entangled and ambiguous in nature. Secondly, by using common text-representations for all regions, current methods prevent us from interpreting text in fundamentally different ways at different parts of an image. Different image regions are therefore only allowed to assimilate the same type of information from text at each refinement stage. Finally, current methods generate refinement features only once at each refinement stage and attempt to address all image aspects in a single shot. This single-shot refinement limits the precision with which each refinement stage can learn to improve the prior image. Our proposed method introduces three novel components to address these shortcomings: (1) An initial generation stage that explicitly generates separate sets of image features for each word n-gram. (2) A spatial dynamic memory module for refinement of images. (3) An iterative multi-headed mechanism to make it easier to improve upon multiple image aspects. Experimental results demonstrate that our Multi-Headed Spatial Dynamic Memory image refinement with our Multi-Tailed Word-level Initial Generation (MSMT-GAN) performs favourably against the previous state of the art on the CUB and COCO datasets. 

Reviews and author responses:

This paper proposed a new text-to-image generation method that would like to tackle the entangled textual inputs by multi-tailed word-level initial generation, create the region-contextualized text representations for region-aware image refinement, and propose an iterative multi-headed mechanism to allow multiple different changes at each stage of refinement. --- Strengths ---
1. Entangled sentence-level representation and region/semantic-insensitive image refinement are two noticeable issues in the text-to-image generation task. I am glad that the authors would like to tackle these problems in a unified framework.
2. Experiments can validate the performance of each proposed component to a certain extent.
2. This paper is generally well-written and easy to follow.
--- Weaknesses ---
1. The disentanglement of word-level attributes at the level of generating a small-size image may not be a necessity, in my opinion. Word-level features can be modulated to the visual features in many ways other than disentangling the sentence-level input. I would like to see more evidence about this assumption. Moreover, the performance of disentanglement was not fully validated, the proposed MTWIG looks like redundant representations of the sentence-level features, rather than ``disentangled'' representations.
2. The spatial dynamic memory is an extension of DM-GAN with more region-contextualization, its design may just be one of the choices to enable region awareness, but may not be the only way to fulfill it. Authors should provide more discussions to tell the motivation, design logic, and/or the advantages of the proposed SDM, also including the rest two proposed modules.
3. Moreover, the proposed three components are loosely related to each other, without a concrete idea that integrates them together. This paper is overall well-written and easy to follow, but the ablation study and discussions may be inadequate to validate that the performances, properties, and novelties of the proposed modules. 

The paper proposed a new method to tackle text-to-image generation challenge. In the paper, authors introduced a potential problem that current methods only use sentence embedding at beginning of the network to generate initial images, where different attributes may be entangled and are hard to be refined during following states. Based on this, authors proposed three components to address this limitation. Strengths:
1. Authors introduced a potential entanglement problem in current methods by using sentence embedding at beginning.
2. A spatial dynamic memory module is proposed to capture the contextual information within image regions.

Weaknesses:
1. The paper introduced a problem that current methods only use the sentence embedding at the beginning of the network to generate initial image, where attributes may be entangled. I am confused about this assumption. I think these methods using sentence embedding at beginning is mainly because the initial image resolution is quite small, and sentence embedding is enough to produce the rough image. Then, these methods utilise word-level information to improve the details. These methods can also be modified to incorporate word-level embedding at very beginning of the network, like image resolution at 4 X 4. It is better if authors can add experiments to support this assumption by verifying the existence of entanglement and the need to use word-level embedding at the beginning.

2. In MTWIG, the generation from different couple of words share same conv and upsample blocks (if I am not right, please correct me), does this operation can do disentanglement as claimed by authors? Also, the following fusion operation that combines these different features into single image features. Can this operation cause entanglement? 

3. The proposed spatial dynamic memory module is based on the memory module introduced in DM-GAN, and the main difference shown by authors is the utilization of region-contextualized representations. For me, the convolution operator somehow can also capture the regional information. Could we achieve similar performance by adjusting the size of convolution operator in pixel-level representation? 

4. I am confused about the FID scores of DF-GAN (33.29 for COCO) reported in the paper, which is different from the value claimed in the DF-GAN paper (21.42 for COCO).

5. How to decide the size of region used in spatial dynamic memory module? Is the 8 × 8 grid map enough to capture the contextual information in image?
 The current version of paper may need experiments to support claims shown in the paper. 

This paper is motivated by that most existing text-to-image methods suffer from three limitations and solutions are proposed to address the different limitations.

Firstly, it introduces multi-tailed word-level initial generation to enhance global sentence representation with distinct n-gram word presentation. 

Second, the spatial dynamic memory module is proposed to create a separate region-contextualized text representation for each image region. 

Finally, it introduces an iterative multi-headed mechanism to make multiple distinct modifications to the prior image features. Strength:
1. MTWIG improves the initial generation by word-level representation, alleviating the entanglement of words in the sentence-level representation.
2. The method about region-specific word representation realizes the alignment between text modal and image modal, and the proposed three levels query in SDM are interesting.
3. The experiment can verify the effectiveness of the proposed modules to an extent.

Weakness:
1.  MTWIG method seems unstable. As shown in Table 4(A.2), the MTWIG(ks=1) is better than MTWIG(ks=2) in terms od FID score in the CUB dataset, but MTWIG(ks=3) is better than MTWIG(ks=1). It may be due to that LSTM and TextCNN-like networks are poor to capture n-gram word information.
2. The training epochs are different, e.g. the experiments shown in Table 1 and Table 3. Thus, we cannot convincingly ensure the effectiveness of different components by fairly comparing the proposed method with the SOTA methods.
3. The COCO dataset has more examples compared to CUB.  When the data set is small, it is easy to produce over-fitting. Authors should consider performing an ablation study on COCO to verify the effectiveness of different modules. This paper has a clear description of the found three problems, but the ablation studies are not sufficient. For details, please see weakness part in the main review section.
 
From the experiment, it can be seen that the stability of the model is poor. And the consideration of the performance reproducibility of training complex models with huge parameters is also very important.

Meta-review: The reviewers' evaluation of this paper are borderline/negative.  The AC considered the reviews, rebuttal, and the paper itself, and concurs with the reviewers.  The AC found that the paper is an extension of previous work DM-GAN (DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis, CVPR 2019, https://arxiv.org/pdf/1904.01310.pdf). This work uses the word features in addition to sentence features at the first stage of generation, while DM-GAN and other previous work don’t use word features in the first stage, but use them in the later stages when the feature resolution is higher. The authors improve the dynamic memory in DM-GAN into spatial dynamic memory, and also change the image refinement process in DM-GAN into an iterative refinement. The proposed multi-tailed word-level initial generation, spatial dynamic memory, and iterative refinement are incremental changes to DM-GAN. Moreover, the proposed structure almost doubles the parameter size of DM-GAN (shown in Table 2), yet the evaluation results on COCO are similar to DM-GAN with only minor improvements. It is not clear whether the performance improvement comes from  the increased number of parameters or the architecture design. Especially on the CUB dataset with limited number of images, the model can easily overfit with a larger number of parameters.  The proposed method shares the similar network structure and dynamic memory blocks as DM-GAN, except for a few changes.  Overall, the AC finds this paper not suitable for acceptance at ICLR in present form.

Decision: Reject