Multi-Tailed, Multi-Headed, Spatial Dynamic Memory refined Text-to-Image Synthesis
Authors: Amrit Diggavi Seshadri
Balaraman Ravindran
Venue: ICLR 2022 Submitted (2022)
Keywords: Text-to-Image Generation
Computer Vision

Abstract: Synthesizing high-quality, realistic images from text-descriptions is a challenging task, and current methods synthesize images from text in a multi-stage manner, typically by first generating a rough initial image and then refining image details at subsequent stages. However, existing methods that follow this paradigm suffer from three important limitations. Firstly, they synthesize initial images without attempting to separate image attributes at a word-level. As a result, object attributes of initial images (that provide a basis for subsequent refinement) are inherently entangled and ambiguous in nature. Secondly, by using common text-representations for all regions, current methods prevent us from interpreting text in fundamentally different ways at different parts of an image. Different image regions are therefore only allowed to assimilate the same type of information from text at each refinement stage. Finally, current methods generate refinement features only once at each refinement stage and attempt to address all image aspects in a single shot. This single-shot refinement limits the precision with which each refinement stage can learn to improve the prior image. Our proposed method introduces three novel components to address these shortcomings: (1) An initial generation stage that explicitly generates separate sets of image features for each word n-gram. (2) A spatial dynamic memory module for refinement of images. (3) An iterative multi-headed mechanism to make it easier to improve upon multiple image aspects. Experimental results demonstrate that our Multi-Headed Spatial Dynamic Memory image refinement with our Multi-Tailed Word-level Initial Generation (MSMT-GAN) performs favourably against the previous state of the art on the CUB and COCO datasets. 

Under review as a conference paper at ICLR 2022
MULTI-TAILED, MULTI-HEADED, SPATIAL DYNAMIC MEMORY REFINED TEXT-TO-IMAGE SYNTHESIS
Anonymous authors Paper under double-blind review
ABSTRACT
Synthesizing high-quality, realistic images from text-descriptions is challenging task, and current methods synthesize images from text in multi-stage manner, typically by first generating rough initial image and then refining image details at subsequent stages. However, existing methods that follow this paradigm suffer from three important limitations. Firstly, they synthesize initial images without attempting to separate image attributes at a word-level. As a result, object attributes of initial images (that provide basis for subsequent refinement) are inherently tangled and ambiguous in nature. Secondly, by using common rextreppresnntations for all regions, current methods prevent from interpreting text in fundamentally different ways at different parts of an image. Different image regions are therefore only allowed to assimilate the same type of information from text at each refine- ment stage. Finally, current methods generate refinement features only once at each refinement stage and attempt address all image aspects single shot. This single-shot refinement limits the precision with which each refinement stage can learn to improve the prior image. Our proposed method introduces three novel components to address these shortcomings: (1) An initial generation stage that explicitly generates separate sets of image features for each word n-gram. (2) A spatial dynamic memory module for refinement of images. (3) An iterative multi- multi-headed mechanism to make it easier to improve upon multiple image aspects. Experimental results demonstrate that our Multi-Headed Spatial Dynamic Memory image refinement with our Multi-Tailed Word-level Initial Generation (MSMT- GAN) performs favourably against the previous state of the art on the CUB and COCO datasets.
INTRODUCTION
Generative Adversarial Networks (GANs) have shown great promise for the generation of photo-realistic synthetic images (Goodfellow et al., 2014, Radford et al.] 2015) Denton et al. 2015; [Salimans et al. 2016). and the highly-compelling nature of images generated by GANs has driven research into conditional image-generation and multimodal learning. In this paper, we focus the task of text-to-image generation, that has emerged as an area of active research in recent years. Although much progress has been made in this area, the synthesis of high-quality, realistic images from text-descriptions remains challenging task. Current state-of-the-e methods (Xu et al. 2018 Li et al., 2019a| Zhu et al.) 2019) employ multiple-stages of image generation typically, an initial image is first generated from global sentence-level vector, and subsequent stages incorporate fine-grained information extracted from word-level vectors to refine image details. However, these methods suffer from three important limitations. The first problem is that by attempting to synthesize image features directly from a sentence-level vector, the initial generation stage fails to cleanly separate image attributes at a word-level. If potentially distinct objects such 'cat" and 'tree" for example, are entangled in the sentence-level representation, then the presence of either word in a sentence could prompt the initial stage to generate the same hybrid image attributes. This is important because the subsequent refinement stage relies upon the initial image features to provide a meaningful basis for word-level refinement. By feeding ambiguous and poorly formed initial features, we limit the scope of refinement. Secondly, current methods do not construct region- region-specific representations of text at refinement stages. This prevents us from interpreting words fundamentally different ways based the content of image regions.

---

Under review as a conference paper at ICLR 2022
Whereas, in complex real-world scenes, the requirement for a regiioncontextualizee interpretation of words is commonplace based on the region under consideration, the same word often dictates fundamentally different types of refinement within single image. The word 'raining' for example dictates a requirement in the sky that is fundamentally different from the requirement that it dictates in the region of the ground. While the sky becomes more cloudy, the ground must become wet. To generate realistic images from natural text descriptions, it is important that we construct refinement architecture that allows different image regions assimilate contton-cottttuulizeed information from text descriptions. Finally, we note that current methods generate refinement features (that modify previous image features) only once at each refinement stage and attempt to address all image aspects within single-shot. This single-shot refinement limits the precision with which each refinement stage can learn to improve the prior image.
In this paper, we propose a Multi-Headed and Spatial Dynamic Memory image refinement mechanism with a Multi-Tailed Word-level Initial Generation stage (MSMT-GAN) to address these three issues. Our contributions are summarized as follows:
We introduce novel "Multi-Tailed" Word-level Initial Generation stage (MTWIG), that generates a separate set of image features for each word n-gram, and iteratively fuses these sets together to obtain initial image features. We demonstrate that it is possible to improve the performance of previous methods by replacing their initial generation stage with ours.
We introduce a novel Spatial Dynamic Memory module (SDM) that fuses word-informatin in custom way with each prior image region, obtain region-contextualized text- representations. At each refinement stage we retrieve features for image improvement from this SDM module.
We introduce novel Iterative Multi-Headed Mechanism (IMHM) of image refinement wherein we explicitly allow each stage of refinement to make multiple distinct modifications to the prior image, under common discriminator feedback.
We evaluate our MSMT-GAN model on the Caltech- UCCD Birds 200 (CUB) dataset (Wah et al. 201T) and the Microsoft Common Objects in Context (COCO) dataset (Lin et al. 2014). Experiment results demonstrate that MSMT-GAN is competitive with current methods on the COCO dataset and significantly outperforms the previous state-of-th art on the CUB dataset, decreasing the lowest reported Fréchet Inception Distance (FID) (Heusel et al. 2017) by 21.58% for CUB.
2 RELATED WORK
Text-to-Image Generators:Ress et al. (0166) first demonstrated that translation model from natural language to image pixels could learnt by conditioning both generator and discriminator networks of GAN on input text-descriptions. There has since been surge of interest in training multi-stage attention based GAN architectures for this task. While the conventional setting (Zhang
et al., 2017: Xu et al.) 2018 Li et al., [019a] Zhu et al.] (219)) assumes only the availability of
(text,image) pairs at training time, recently a second setting has emerged that assumes availability of bounding-box//haape bwwbbbbooo/saaaasssseeess information of objects attributes during training (Li et al. 2019b
Hinz et al. 2019, Cho et al., 2020 Liang et al.) 2020). We highlight that this represents a significantly easier problem setting and that such methods are not feasible where bounding- boxtsg-p-bo///ppeeeeeee information is unavailable (such as the CUB dataset). Our method does not assume the availability of information, and we make comparisons against prior work of the same setting.
Memory Networks: Memory Networks (Weston al. 2014) combine inference components with a long-term memory module that can be dynamically written to and read from. Current methods (Miller et al. 2016) query "key encodings" of memory slots to retrieve set of weights. These weights are used to combine separate "value encodings" of the slots into a single response. A Dynamic Memory Generative Adversarial Network (DM-GAN) (Zhu et al. 2019) that retrieves information for image refinement from a memory module was recently proposed for text-io-image synthesis. In our SDM module, we too employ the memory-writing, key-addressing, value-reading paradigm introduced by (Miller et al. 2016), but our method differs from (Zhu et al.) 2019) in all three memory operations (Section 3.2. Fundamentally, DM-GAN does not create region- contextualized representations of text.
2

---

Under review as a conference paper at ICLR 2022
sma bird with a yellow
Multi-Tailed Word-level Initial Generation Stage
Image Pradiction
Text Eac--aaceerr
grey
crown, grey breest,
yellow belly
and back whith
green grey
wings and
wingbars.
Multi-Headed Spatial Dynamic Memory Based Refinement Stage
Valla Fauchl
(m))
legion wilte Bay Addmeuiing
For Ruad
lonning fugmentation Fully Lauer
of memery
Kry Encoding memore
weights comnssponds
Figure I: Our MSMT-GAN architecture for text-to-image synthesis, showing a Mutli-Tailes Word- level Initial Generation stage, Multi-Headed Spatial Dynamic Memory based refinement stage with three refinement heads, and image prediction.
Multi-Headed Attention: Transformers (Vaswani et al. 2017) utilize a key-value mecha- nism similar to memory networks and introduced the idea of multi-headed attention. They linearly project query, keys and values to separate encodings, called "attention heads", and each head is separately used to extract output vector. These vectors are concatenated together and linearly projected to a single response. Inspired by the success of Transformers, we introduce the IMHM method for image refinement. However, our method differs in few respects. We maintain separate SDM modules for each head and we obtain queries and fuse outputs in an iterative fashion. We also adopt a "redundancy loss" (Section 3.4) encourage each head focus on separate image aspects.
MSMT-GAN
Our MSMT-GAN architecture (Figure comprises of three stages a Generation (MTWIG) stage, and two refinement stages, Each refinement stage is Word-level Initial and each refinement head has separate Spatial Dynamic Memory (SDM) module. Section [.] presents our MTWIG stage, Section 3.2 presents our SDM module for single refinement head, and the details of our Iterative Multi-Heades Mechanism (IMHM) are presented Section
3.1 MULTI-TAILED WORD-LEVEL INITIAL GENERATION (MTWIG)
We highlight that previous multi-stage methods (Zhang al. 2017 2018 Li et al. 2019a Zhu et 2019) all rely on the same type of initial generation stage and focus only on improving the refinement stages making the conventional assumption that the performance of multi-staee generators primar- ily determined by the refinement stages, and that the quality of the "rough initial image" is of little importance. our paper, we break from this tradition and demonstrate for the first time that gains can be achieved in the final stage of image refinement by making an improvement to the initial images.
The conventional approach synthesizes initial images directly from sentence-level vector without attempting to separate image attributes at a word-level. As a result, words that are entangled the sentence-level representation generate initial image attributes that inherently ambiguous in nature. our novel Multi-Taile Word-level Initial Generation (MTWIG) stage, we overcome this shortcoming by explicitly creating separate sets of image attributes for each word n-gram.

---

Under review as a conference paper at ICLR 2022
First, we sample a vector of random noise 2 from a normal distribution and use a pretrained text-encoder extract sentence-level vector and word-level vectors: and W from the input text.
W = [W1, W2, ... w wi E IR
SERN
Zn NO, 1); E RN
(I)
Where L is the number of words in the text-description, and N,. N₂ and N are the dimensions of the noise vector, sentence vector and word vectors respectively. To mitigate over-fitting, the Conditioning Augmentation technique (Zhang et al. (0017) is used to resample the sentence-vector from an independent Gaussian distribution. This resampled sentence vector s' and the noise vector are then concatenated with each word-level vector from the input text sequence, and the sequence of concatenated vectors are passed through ID convolutional operation V of stride I (see Figure I..
F ((coocaal(s' 2, wi) wr E W3)
The length T of the output sequence depends on the kernel size used by and the vectors of the output sequence ft F are each separately passed through a series of upsampling blocks to generate corresponding sets of image features St. These sets of image features or "tails" each correspond to a different word n-gram from the input text sequence. If S, corresponds to a single word. If we use a kernel size of 2, then each tail S, corresponds to a word bi-gram, and so on. We combine our sequence of tails {S²) together in an iterative fashion using the adaptive gating fusion mechanism introduced by Zhu et al. (2019) (discussed in Section A.
kernel size of 1 for V, then each tail
S1 = fuse(S St, PMTWIG MTWIG)
R1 = S1:T
(3)
Where PMTWIG and p MTWIG denote parameter matrix and bias terms, S denotes a combination of the first & tails, and S1.1 denotes the first tail Sj. The combination of all tails gives us the final image features Ri for our initial stage. Notice that by concatenating each word vector wy with s' and before the ID convolution, each tail is created with some common information, so they may learn to fuse together coherently. Each upsampling block consists of nearest neighbor upsampling layer and 3x3 convolution operation. An initial image is predicted from R₁ using a 3x3 convolution.
3.2 SPATIAL DYNAMIC MEMORY (SDM)
In this section, we describe the operation of single refinement head. Unlike previous methods, our novel Spatial Dynamic Memory (SDM) module creates a separate region-contextualized text representations for each image region. This allows us interpret the same text in fundamentally different ways at different parts of an image and assimilate region-contextuallied information from text at each part. To begin with, we have the set of word-level vectors W and image features R from the previous stage of generation.
Ra-1 = [ri,1, T12 .... T Tu, E RN,,
(4)
Where is s| the number of image pixels and N, the dimension of pixel features. We obtain refinement features in three steps: Memory Writing, Key Addressing and Value Reading. Memory Writing: First, we divide the Ene-grained s x 8 inital image into coarse h x h sized grid-map and average the pixel features within each grid-cell to get grid-level image features C.
jep
C..
1
Σ
Tu.e
P x
e= -=(i-1)ep+1
11)pp1
Where p == =-==s/a s/h, so that Ip x pl are the number of pixels represented by each grid cell. Then, we create L x h x h memory slots {m,,i,s} one corresponding to each word for each grid-cell (i.j). These slots are our region-contextualized representations of each word, and each slot uses separate memory writing gate و'', fuse information from each grid-cell (i.j) with each word feature un.
gi... (Ra- wI) = (A wi + Bij * C.,5)
(6)
mi,3,; = Mn (w7) O g.... + Mc(C)a @ (1 (1- (1-9h..)
(7)
The grid-level features C are encoded using 2d convolution operation Mc (with stride I and N.. output filters) and we use a common 1x1 convolution operation Mn,, to encode all word vectors to a Nm dimensional space. A and B are 1 x N., and N, matrices respectively.
4

---

Under review as a conference paper at ICLR 2022
Key Addressing: In this step, compute attention weights {ai,Ij... over region- contextualized text-representations (m,,s))- The dimensions (a,b) index pixels within grid-cell (i,j), that each slot my,ij gets matrix arji : p p of attention weights. Each weight is computed as a similarity probability between a key-encoding of the slot: @x (me)a and a query vector: Ji..a.b. where (.).) is a 2d convolution operation with stride I and (N, + p²) output filters.
explóx (mu)a. p.j.a.b)
al..j.a E exp((x (i.j.a.))
In the case of single headed image refinement, we use the previous image features Rg-1 to obtain the query vectors. A query vector qi.i.,,b is made up of three components, I)A global-level query: golobal grid and 3)A pixel-level query: R.j.a.b To obtain these three components, we
2)A grid-level query: 9i. encode R using three separate 2d convolution operations: (quioonr (.), @quria (.) and @epeiiac (-), each with a stride of and N, output filters.
Qglobal = @Qootabal (R
= @Qaria (R (R
Quizel = @quizal (Rx-1)
(9)
Then, the average of all pixel features of Qatabal becomes the global-level query component gatabal The average of pixel features within the grid cell (i, of Qarid becomes the grid-level query y. grrd and the pixel feature at location (a, b) within grid cell (i. is extracted from Quizel to give us the pixel-level query component F.,ja.
iep
jap
global
Qalobal
Pi
grid
Qgrid
(10)
p
a a= (f- 1)
1)-++1
P.j.a.b
Vuti.a) k(j.))
h(i,a) (i 1) +
(11)
Where (h(i,a), M(j.b) indexes the pixel at location (a, b) within grid-cell (i,5). To obtain the final query Jij.., we concatene these three components together.
Value Reading: In the value reading step, for each pixel (a, b) within a grid-cell (i.j), we compute weighted sum of value-encoded memory slots: dy (mi)i along the word dimension (.
e.j.a,bb
a...j.a.b (m())s
(12)
(.). is a 2d convolution operation with stride and N, output filters. We now have e,,; :p x x N, dimensional matrices each of which corresponds to single grid cell of our coarse h x h grid map. To obtain s x fine-gratd -ne-ggrined refinement features, we apply the mapping:
Oh(i.a) k(..6) Cs.j.a,b
(13)
Where M(,,, .) is the function defined Eq That is, we populate each grid cell with lp x pl vectors of N, dimensionality. Since s/h, we are left with a set of refinement features O = (o.. } that are made up of [s x s| vectors of N, dimensionality, each of which corresponds to a single pixel.
3.3 ITERATIVE MULTI-HEADER MECHANISM (IMHM)
Current methods generate refinement features only once each refinement stage and attempt to address all image aspects in a single-shot. This limits the precision with which each refinement stage can learn to improve the prior image. In order make it easier for each refinement stage to precisely address multiple image aspects, we introduce novel iterative multiple distinct modifications to the prior image features under common discriminator feedback. Each head of our mechanism has separate spatial dynamic memory module formed from RE-1 and W. For the first refinement head, we use the previous image features R to obtain a query matrix and extract set of refinement features O exactly as described in Section 3.2 Then, we fuse O and R using the fusion mechanism introduced by Zhu et al. (2019) (described in Section A. to obtain updated set of image features U1. If we use only single refinement head, then this becomes our response for the refinement stage k. However, we use more than one refinement head, then for the next head, we use U, obtain query matrix. That is, follow the same mechanism
mechanism that makes
5

---

Under review as a conference paper at ICLR 2022
outlined in Section 3.2 but replace Rx-1 with U1 in Eq Doing so, extract second set of refinement features O2. and we fuse O2 and U1 to obtain updated image features U2. We proceed in this iterative fashion until we have used all of our refinement heads. The final updated image features are fused with the original image features Rk-1 in a skip-connection to obtain the final response of the refinement stage k. That is, if we have 7 refinement heads:
U. = ['se(UUUU- Oi, Pt: Pt)
response; = fUse(UU, RE-1, pskip, yakiip)
(14)
Notice, we use separate parameter matrix and and bias terms P and p for each fusion operation. so that we combine refinement features and image features in a custom way for each head. The, responsey passed through several residual blocks (He et al.] 2016) and an upsampling block to obtain higher resolution image features Rg. Each block consists of a nearest neighbor upsampling layer and a 3x3 convolution operation. Finally, a refined image ry is predicted from Rg using a 3x3 convolution operation.
3.4
OBJECTIVE FUNCTION
The objective function for our generator network is defined as:
K
= L ALRRED + Σ (Le, + ABLDAMMM.
(15)
k=
k=1
Where, LCA denotes the conditioning augmentation loss Zhang al. 2017), G2 denotes the generator of the 1tth stage so that LG, denotes the adversarial loss for Gg. TRED, denotes our redundancy loss for the htth stage, and LDAMSM, denotes the DAMSM text-image matching loss (Xu et al. (0018) for the kth stage. \1, À2 and )3 are hyperparameters that combine the various losses. Redundancy Loss: To encourage each head of a refinement stage to focus on separate image aspects, we average region-wist information of each head's output refinement features and penalize similarity between different refinement heads. That is, for T refinement heads:
f(t)
Ou,e ; LREDE Σ
(im()()),
(16)
8 X
j=i+1
Where On E Or in stage k (see Section 3.3 and sim is the cosine similarity between vectors. We call this sum of pairwise similarity LRED, the "redundancy loss" of the Eth refinement stage. Adversarial Loss: The adversarial loss for G& defined
LG - Exspa, log Dx(x) + Exzpa, log (x,
(17)
D& is the discriminator network for the 1tth stage of generation. The first term provides feedback image realism independent of the input text, and the second term provides feedback on the realism of the image in light of the input text. Alternate to adversarial training of Gg, each discriminator D₁. is trained to classify images as real or fake by minimizing the discriminator loss LD.-
LDz.
BerPaasa log De (x) + Bempe, log(1 (x)+))] Bewpdora log (2 a) + EewPe log(1 (z +)]]
ancceditiona Joma
crnditionad Joa
(18)
Where the unconditional component distinguishes synthetic and real images independent of the input text, and the conditional component distinguishes them light of the input text.
EXPERIMENTS
Datasets: We evaluate our method on the Caltech-UCSD Birds 200 (CUB) dataset Wah et al. 2011 and the Microsoft Common Objects in Context (COCO) dataset (Lin et al. 2014). The CUB dataset contains 8,855 training images and 2,933 test images, with 10 corresponding text descriptions for each image. The COCO dataset, contains 82,783 training images and 40.504 test images, with corresponding text descriptions for each image. We preprocess the datasets according to the methods introduced by Zhang al. (2017).

---

Under review as a conference paper at ICLR 2022
Evaluation metrics: To evaluate the realism of images, we rely on the Fréchet Inception Distance (FID) (Heusel et al., 2017). FID computes the distance between synthetic and real-world image distributions based on features extracted from a pre-trained Inception v3 network. A lower FID indicates greater image-realism. To evaluate the relevance of a synthetic image it's generating text-description, we rely on the R-precision introduced by Xu et al. (2018). R-precision is computed as the mean accuracy of using each synthetic image retrieve one ground truth text-description from among 100 candidates. To evaluate a model on a dataset, we generate 30,000 synthetic images conditioned on text descriptions from the unseen test set.
Implementation Details: To obtain a sentence-level vector and word-level vectors for a given text description, we use the pretrained bidirectional LSTT text encoder employed by AttGGAN (Xu et al.) 2018). Our MTWIG stage synthesizes images features with 64x64 resolution. Two refinement stages refine these features to 128x128 and 256x256 resolution respectively. At refinement stages, we use T = refinement heads and we use h = 8 to divide each input image into a coarse 8 x 8 grid map. We use the same discriminator network architecture employed by Zhhu et al. (2019).
Further implementation details are provided in Section A.3
4.1 ABLATIVE EXPERIMENTS
Effectiveness of Multi-Tailed Word-level Initial Generation: In our experiments (Appendix A.
we find that our MTWIG stage is most effective when used with a kernel size of 3, so that we generate a separate tail for each word tri-gram. To evaluate the effectiveness of our MTWIG(ks=3) stage in multi-stage models, we train our MTWIG(ks=3) method with DM-GAN (Zhu et al. 2019) style-refinement stages for 700 epochs on the CUB dataset, and observe that it decreases the FID score achieved by the original DM-GAN model by 7.72% and increases R-precision by 2.76% (Table D]. Figure Figure[]shows showss the improved visual quality of the refined images. We again point out that previous multi-stage methods (Zhang et al. 2017; 2018; Xu et al., 2018 Li et al. 2019a Zhu ct al. (019)) all rely on the same type of initial generation stage, and we expect a similar boost in performance if we replace their initial stage with our MTWIG stage.
Effectiveness of Spatial Dynamic Memory: In order to evaluate the effectiveness of our SDM based-refinement stages, we compare a multi-stage model that uses our MTWIG((s=3) stage and DM-GAN's refinement stages against a multi-stage model that uses our MTWIG(Ks=3) stage and our single-headed SDM-based refinement stages. Both models are trained on the CUB dataset for 700 epochs. We observe that our SDM-based refinement out- perfoorfsmm DM-GAN's refinement, decreasing FID score by 4.64 % and boosting R --precision by an additional 0.83% (Table []. Figure 2 shows that SDM-based refinement generates images of better visual quality than those formed by DM-GAN's refinement stages for the same initial generation architecture.
Effectiveness of the Iterative Multi-Headed Mechanism: To evaluate the effectiveness of our IMHM refinement, we compare the model that uses MTWIG(Ks=3) and single-headed SDM-based refinement stages against our full MSMT-GAN model that uses MTWIG(ks=3) and six SDM-bbased refinement heads for each stage. As before, both models are trained for 700 epochs on the CUB dataset. We find that refinement stages that use our multi-headed mechanism out-perform single-headed refinement stages, decreasing FID score 2.38%, and boosting R R-precision by another 1.03% (Table MT. Visually, besides overall better image quality, we find that images generated by multi-headed refinement stages posess text-irrelevant content that is far more detailed than that observed in images generated by single- refinement stages (Figure
4.2 COMPARISON WITH STATE OF THE ART
To compare our architecture against the previous state-of-ther -ftt-orf-hheart for the task of text-to-image genera- tion, we train MSMT-GAN models for 1000 epochs on the CUB dataset and for 210 epochs on the COCO dataset. As shown in Table our MSMT-GAN decreases the previous lowest reported FID score by 21.58% on the CUB dataset maarinng a significant improvement in image realism, and also boosts the previous best reported CUB R R-precision by 4.24 % marking large improvement in the similarity of synthetic images to their generating text. As shown in Table and Table our model is comparable in size previous methods, and outperforms the next closest contender of similar
7

---

Under review as a conference paper at ICLR 2022
Table 1: FID and R-precision of DM-GAN and ablative versions of our model. (With all of our variants trained for 700 epochs on CUB)
CUB
Method
FID
R-pren (%) t
DM-GAN
11.91
76.58 + 0.53
MTWIG DM-GAN's refinement
10.99
79.37 + 0.73
MTWIG w/ SDM refinement
10.48
80.20 + 0.67
MTWIG SDM and IMHM refinement (MSMT-GAN)
10.23
81.23 = 0.68
Our MTWIG our SDM refinement
CV NN0-GAN
Our MIWIC with
Our MTWI with
SDM (MHM
refinemeet (M
small black The small brown
This bind has vibrant blue the body and yellow tail feathers.
The crown 1s
sea D black
The srall brown with
This bird was blue feathers the body and yellow tail feathers.
The crewm 15 the black h111.
bnrdd witt
black and
and brown #ozc1ing wostly brown and red sped beak.
white brown morti ina mestly brown back El Lipped beak,
Hight hew eyes, and
eyes and
black
Figure 2: Comparison of DM-GAN with ablative versions of our model trained for 700 epochs on the CUB dataset.
size for COCO (DM-GAN) by 4.21% on FID score -making it highly competitive with the current state-of-the-art We also observe a slight improvement of 0.23% COCO R-precision. Qualitatively. we observe that synthetic images generated by our model are typically sharper and more realistic than those generated by prior methods (Figure 3). In particular, we observe that our method generates synthetic images that possess greater detail and that are better matches for the generating text.
Table 2: Number of parameters required at test-time (including text-encoders) for the previous state-of-the-art comparison to our MSMT-GAN (approximate values reported millions).
Method
# Paramers for CUB
Parameters for COCO
AttnGAN
9.16M
22.43M
ControlGAN
30.72M
45.36M
DM-GAN
23.52M
30.96M
DF-GAN
14.32M
20.87M
XMC-GAN
>M
Our MSMT-GAN
48.1
55.16M
5 CONCLUSION
In this work, we have proposed the MSMT-GAN architecture for the task of text-to-image synthesis. First, we introduced a novel Multi-Tailed Word Level Initial Generation stage (MTWIG) that explicitly generates separate sets of image features for each word n-gram. Second, proposed novel Spatial Dynamic Memory (SDM) module to contextualize text representations by image-region. Third, we introduced novel Iterative Multi-Heades Mechanism (IMHM) of image refinement to make easier
8

---

Under review as a conference paper at ICLR 2022
Table FID and R-precision the previous cate-of-the-art and our MSMT-GAN trained for 1000 epochs on CUB and 210 epochs on COCO.
CUB
COCO
Method
FID
R-pren (%) t
FID +
R-pren (%) f
AttnGAN
14.01
67.82 4.43
29.53
85.47 = 3.69
ControlGAN
69.33 # 3,23
82,43 + 2,43 92.23 = 0.37
DM-GAN
11.91
76.58 0.53
24.24
DF-GAN
13.48
33.29
Our MSMT-GAN
9.34
80.82 0.54
23.22
92.46 + 0.28
Att/SAN
INSM SSMTT-DA -DAN
1ights сorner poTe in town.
group shoop <itring next
Standing
man
water bird is variouus 1eed inn The water.
row
11 tiny with blue
snall black bi11 that
A mostly tyy with whi face and a br brown beak.
Fis sur Thoard,
shades of gray 1th a yel fow beak.
and brown
riding big
feathers and white beak.
well black evering.
Figure 3: Comparison of MSMT-GAN with state-of-the art models on the CUB and COCO datasets.
for each refinement stage to precisely address multiple image aspects. Our ablative experiments clearly demonstrate the effectiveness each of these three components, and we have shown that are able to boost the performance of prior methods by replacing their initial stage with our MTWIG stage. Experiment results further demonstrate that our MSMT-GAN model significantly out-performs the previous state of the art on the CUB dataset, decreasing the lowest reported FII) score by 21.58% and boosting the CUB R-precision by 4.24%. On the COCO dataset, we have demonstrated that MSMT-GAN is highly comptitive with current methods based on image realism and model size. In future work, we aim to design a discriminator model that provides more region-specific feedback than existing methods, use in conjunction with MSMT-GAN generator architecture.
REFERENCES
Jun Cheng, Fuxiang Wu, Yanling Tian, Lei Wang, and Dapeng Tao. Rifegan: Rich feature generation for text-to-image synthesis from prior knowledge. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10911-10920, 2020.
Jaemin Cho, Jiasen Lu, Dustin Schwenk, Hannanch Hajishirzi, and Aniruddha Kembhavi. X- Ixmert: Paint, caption and answer questions with multi-modal transformers. arXiv preprint arXiv:2009. 1278, 2020.
Emily L. Denton, Soumith Chintala, Rob Fergus, et al. Deep generative image models using a laplacian pyramid of adversarial networks. Advances in neural information processing systems, pp. 1486-1494, 2015.
'We make our comparisons against the pretrained models released by the authors, and report results using the official implementations of FID score.

---

Under review as a conference paper at ICLR 2022
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems. pp. 2672-2680, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. In Advances in neural information processing systems, pp. 6626-6637, 2017.
Tobias Hinz, Stefan Heinrich, and Stefan Wermter. Semantic object accuracy for generative text-to- image synthesis. arXiv preprint arXiv: 13321, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A methodfor stochastic optimization. In International
Conference onr.nigg Representations (ICLR), 2015.
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip Torr. Controllable text-to-image generation. In Advances Neural Information Processing Systems, pp. 2065-2075, 2019a.
Wenbo Li, Pengchuan Zhang, Lei Zhang, Qiuyuan Huang. Xiaodong He, Siwei Lyu, and Jianfeng Gao. Object-driven text-to-image synthesis via adversarial training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12174-12182, 2019b.
Jiadong Liang, Wenjie Pei, and Feng Lu. Cpgan: Content-parsing generative adversarial networks for text-to-image synthesis. In European Conference Computer Vision, pp. 491-508. Springer, 2020.
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Ja- son Weston. Key-value memory networks for directly reading documents. arXiv preprint arXiv:1606.03126, 2016.
Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. Learn, imagine and create: Text-to- image generation from prior knowledge. Advances in Neural Information Processing Systems, 32: 887-897, 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.0643: 2015.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. arXiv preprint 160iv::605: 1605.0566, 2016.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234-2243 2016.
Hongchen Tan, Xiuping Liu, Xin Li, Yi Zhang, and Baocai Yin. Semantics-enhanced adversarial nets for text-to-image synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10501-10510, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances neural information processing systems, pp. 9998-6008 2017.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongic. The caltech-ucsd birds-200-2011 dataset. 2011.
Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. 110.1:1410.3916, 2014.
arXiv preprint
10

---

Under review as a conference paper at ICLR 2022
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attugan: Fine- aene-grinedd text to image generation with attentional generative adversarial networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1316-1324, 2018.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan: Text photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 5907-5915, 2017.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris Metaxas. Stackgan++: Realistic image synthesis with stacked generative adversarial networks. IEEE transactions on pattern analysis and machine intelligence, 41(8)::947-1962, 2018.
Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross-modal contrastive learning for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pp. 833-842, 2021.
Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 1802-5810, 2019.
A APPENDIX
A.1 FUSING TWO SETS OF IMAGE FEATURES TOGETHER
Given any two sets of image/refinement features A {ou,e} and B {bw,e} such that au,v, b RNN, we use the adaptive gating mechanism introduced by Zhu et al. (2019) to obtain combined/updated set of features {ru,e) where tu, NR.
gu.u = o(P concat(au,o bu,e) + Tu, Qa,y @ gu, + bu. (1 11-96. gu,e)
(19)
(20)
Where (u, v) index pixel locations, N, is the dimension of image pixel features, L is gate for information fusion, the sigmoid function and and p are the parameter matrix and bias term respectively. In our paper, we invoke this adaptive gating mechanism by function "fuse" such that fuse(A, A, B, P. = (Ta,e}-
A.2 ANALYSIS OF MULTI-TAILED WORD-LEVEL. INITIAL GENERATION
Table 4: Quantitative comparison of Fréchet Inception Distance and R-precision between previous Initial Generation (IG) and our Multi-Tailed Word-level Initial Generation (MTWIG) stage for varied kernel sizes (ks) on the CUB and COCO datasets.
CUB
COCO
Method
FID 1
R-pren (%) f
FID |
R-pren (%) t
IG
119.39
83.54 # 0.57
51.22
94.57 + 0.53
MTWIG (ks=1)
118.49
82.78 0.58
41.79
94.47 # 0.49
MTWIG (ks=2)
120.76
82.82 # 0.92
40.23
94.65 # 0.39
MTWIG (ks=3)
115.38
82.94 0.74
39.85
94.80 = 0.29
To analyze the effectiveness of our MTWIG stage, we make comparisons with the inital generation (IG) stage employed by previous methods (Zhang al. 2017 2018 Xu et al. 2018 Li et al. 2019a Zhu et al. 2019). We train IG and MTWIG stages without refinement for 300 epochs on the CUB dataset and for 60 epochs on the COCO dataset. Table Thows shows quantitative comparison between IG and MTWIG architectures that use different kernel sizes. We observe that our MTWIG method achieves best results for ks=3 (that is, by forming separate of word tri-grams), decreasing FII)

---

Under review as a conference paper at ICLR 2022
IG
MTWWIG (Ours)
thts vater Fawl bluck und whi Le very book.
thts long recked dark pray bfind cheel.
hht bird has a white throaz superctl 1iary..
budruan with bonkeso and
people le beach Filking ander
sifce of outcce biusady
in mai is sur whille another cown.
potches and eye.
white head.
matyy in a Eirchenn
Figure 4: Multi-Tailed Word-level Initial Generation (MTWIG; kernel size=3) in comparison to conventional Initial Generation (IG).
scores from the previous IG method by 3.36% on the CUB dataset and by 22.2% on the COCO dataset. The larger improvement observed the COCO dataset highlights that our MTWIG method is most beneficial for complex scene generation, where the presence of a large number of distinct objects demands word-level separation of attributes. Our MTWIG stages also achieves competitive R- R-precision scores on both datasets, demonstrating that images synthesized by our method are well conditioned their input text-descriptions. In Figure we visually compare 64x64 images generated by IG and images generated by our MTWIG(Ks=3) model. We observe that images generated by our method typically posses object attributes that are better separated and more clearly discernible.
A.3 ADDITIONAL IMPLEMENTATION DETAILS
We set No. 256, N, 48 and Nm. 96 to be the dimension of text, image and memory feature
vectors respectively. We set the hyperparameters {A1 = 1, A2 0.5, A = 5 for the CUB dataset and {A1 = 1, A 0.5, Ag 50} for the COCO dataset. All networks are trained using an ADAM optimizer (Kingma& & Ba 2015) with 31 = 0.5 and B₂ 0.999. The learning rate is set to be 0,0002 and we use a batch size of 20 for all networks. We train our models on single Tesla V100 gpu. and observe that similar to previous multi-stage methods, training our MSMT-GAN method is GPU intensive process. We require approximately 27min for one epoch of the CUB dataset, and approximately 4.7 hours for one epoch of the COCO dataset.
A.4 NOTE ON EVALUATION METRICS
We point out that there is inconsistency in the FID implementation used to evaluate prior methods.
While some methods (Li et al., 2019b) report scores using the official Tensorflow version of FID. which uses the weights of a pretrained Tensorflow inception model, other methods (Zhu et al. 2019 Zhang et al. 2021) have reported scores using an unofficial Pytorch implementation of FID that uses the weights of pretrained Pytorch inception model.
Each of these implementations computes different values of FID scores for the same sets of images, and scores computed from the two versions are not directly comparable with each other. We highlight that the correlation between FID score and human judgement is empirical observation, that is dependent on the weights of the pretrained inception model used. The correlation with human judgement has only been shown to hold true for the weights of the pretrained Tensorflow inception model (used by the FID authors Heusel et al. (207)), and has not been verified for the unofficial Pytorch inception model (which has different weights). As such, to ensure that we correlate with human judgement, in our paper we only report scores using the official Tensorflow implementation of FID score computing values for prior work from the pretrained models released by their authors. In Table 5 below, we additionally report Pytorch-implementttion FID scores of SEGAN (Tan et al. 2019) and XMC-GAN (Zhang et al. 2021 models, as reported by their authors. It was not possible for to recompute these scores using the official Tensorflow version of FID as pretrained models for these methods have not been made publicly available.
12

---

Under review as a conference paper at ICLR 2022
We again highlight, that the FID scores computed by the official Tensorflow implementation directly comparable with the scores computed by the unofficial Pytorch implementation.
In Table 5 additionally benchmark the performance of models the Inception Score (IS) (Goodtellow et al., 2014). We note however, that the FID score (which reference-based and compares the distributions of real and synthetic images together) has been observed to be more consistent with human judgement of image realism than IS (which is reference- and does not make comparisons to real images) (Heusel et al.) (0017). We were able recompute other metries for RiFeGAN (Cheng et al. 2020) and LeciaGAN (Qiao el al. 2019) as the pretrained models have been made publicly available.
Table FID and R-precision and of the previous methods and our MSMT-GAN trained for 1000 epochs on CUB and 210 epochs on COCO.
CUB
COCO
Method
FID 4
R-pren (%) f
IS f
FID 4
R-pren (%) f
IS f
AttnGAn
14.01
67.82 4.43
4.36 = 0.03
29.53
85.47 3.69
25.89 + 0.47
ControlGAN
69.33 + 3.23
4.58 + 0.09
82.43 2.43
24.06 + 0.60
SEGAN
18
4.67
32.28
27.86 0.31
RiFeGAN
5.23
LeciaGAN
4.62
DM-GAN
11.91
76.58 + 0.53
4.71
24.24
92.23 0.37
32.43 0.58
DF-GAN
13.48
4.70
33.29
18.64 + 0.64
XMC-GAN
9.33
30.45
Our MSMT-GAN
9.34
80.82 + 0.54
4.55 + 0.06
23.22
92.46 0.28
28.91 + 0.35
In Table represents cases where the data not reported is reported in manner which is non-comparable (besides FID values).
*We make comparisons against the pretrained models released by the authors, and report results using the official implementations of FID score.
We use the FID score reported by the paper, but note that this was computed using an unofficial Pytorch implementation of FID which is not directly comparable with the official FID implementation scores. See Section A.4ioonAA44i for further details.
13


Reviews:
[{"review_id": "c-oqV4Ae1iU", "reviewer": "Reviewer_WhWC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. MTWIG improves the initial generation by word-level representation, alleviating the entanglement of words in the sentence-level representation.\n2. The method about region-specific word representation realizes the alignment between text modal and image modal, and the proposed three levels query in SDM are interesting.\n3. The experiment can verify the effectiveness of the proposed modules to an extent.\n\nWeakness:\n1.  MTWIG method seems unstable. As shown in Table 4(A.2), the MTWIG(ks=1) is better than MTWIG(ks=2) in terms od FID score in the CUB dataset, but MTWIG(ks=3) is better than MTWIG(ks=1). It may be due to that LSTM and TextCNN-like networks are poor to capture n-gram word information.\n2. The training epochs are different, e.g. the experiments shown in Table 1 and Table 3. Thus, we cannot convincingly ensure the effectiveness of different components by fairly comparing the proposed method with the SOTA methods.\n3. The COCO dataset has more examples compared to CUB.  When the data set is small, it is easy to produce over-fitting. Authors should consider performing an ablation study on COCO to verify the effectiveness of different modules.", "summary_of_the_paper": "This paper is motivated by that most existing text-to-image methods suffer from three limitations and solutions are proposed to address the different limitations.\n\nFirstly, it introduces multi-tailed word-level initial generation to enhance global sentence representation with distinct n-gram word presentation. \n\nSecond, the spatial dynamic memory module is proposed to create a separate region-contextualized text representation for each image region. \n\nFinally, it introduces an iterative multi-headed mechanism to make multiple distinct modifications to the prior image features.", "summary_of_the_review": "This paper has a clear description of the found three problems, but the ablation studies are not sufficient. For details, please see weakness part in the main review section.\n \nFrom the experiment, it can be seen that the stability of the model is poor. And the consideration of the performance reproducibility of training complex models with huge parameters is also very important.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Z-Wn8WlEgS2", "reviewer": "Reviewer_6fuX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. Authors introduced a potential entanglement problem in current methods by using sentence embedding at beginning.\n2. A spatial dynamic memory module is proposed to capture the contextual information within image regions.\n\nWeaknesses:\n1. The paper introduced a problem that current methods only use the sentence embedding at the beginning of the network to generate initial image, where attributes may be entangled. I am confused about this assumption. I think these methods using sentence embedding at beginning is mainly because the initial image resolution is quite small, and sentence embedding is enough to produce the rough image. Then, these methods utilise word-level information to improve the details. These methods can also be modified to incorporate word-level embedding at very beginning of the network, like image resolution at 4 X 4. It is better if authors can add experiments to support this assumption by verifying the existence of entanglement and the need to use word-level embedding at the beginning.\n\n2. In MTWIG, the generation from different couple of words share same conv and upsample blocks (if I am not right, please correct me), does this operation can do disentanglement as claimed by authors? Also, the following fusion operation that combines these different features into single image features. Can this operation cause entanglement? \n\n3. The proposed spatial dynamic memory module is based on the memory module introduced in DM-GAN, and the main difference shown by authors is the utilization of region-contextualized representations. For me, the convolution operator somehow can also capture the regional information. Could we achieve similar performance by adjusting the size of convolution operator in pixel-level representation? \n\n4. I am confused about the FID scores of DF-GAN (33.29 for COCO) reported in the paper, which is different from the value claimed in the DF-GAN paper (21.42 for COCO).\n\n5. How to decide the size of region used in spatial dynamic memory module? Is the 8 × 8 grid map enough to capture the contextual information in image?\n", "summary_of_the_paper": "The paper proposed a new method to tackle text-to-image generation challenge. In the paper, authors introduced a potential problem that current methods only use sentence embedding at beginning of the network to generate initial images, where different attributes may be entangled and are hard to be refined during following states. Based on this, authors proposed three components to address this limitation.", "summary_of_the_review": "The current version of paper may need experiments to support claims shown in the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "O_BUMoyS5Gh", "reviewer": "Reviewer_4wnG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "--- Strengths ---\n1. Entangled sentence-level representation and region/semantic-insensitive image refinement are two noticeable issues in the text-to-image generation task. I am glad that the authors would like to tackle these problems in a unified framework.\n2. Experiments can validate the performance of each proposed component to a certain extent.\n2. This paper is generally well-written and easy to follow.\n--- Weaknesses ---\n1. The disentanglement of word-level attributes at the level of generating a small-size image may not be a necessity, in my opinion. Word-level features can be modulated to the visual features in many ways other than disentangling the sentence-level input. I would like to see more evidence about this assumption. Moreover, the performance of disentanglement was not fully validated, the proposed MTWIG looks like redundant representations of the sentence-level features, rather than ``disentangled'' representations.\n2. The spatial dynamic memory is an extension of DM-GAN with more region-contextualization, its design may just be one of the choices to enable region awareness, but may not be the only way to fulfill it. Authors should provide more discussions to tell the motivation, design logic, and/or the advantages of the proposed SDM, also including the rest two proposed modules.\n3. Moreover, the proposed three components are loosely related to each other, without a concrete idea that integrates them together.", "summary_of_the_paper": "This paper proposed a new text-to-image generation method that would like to tackle the entangled textual inputs by multi-tailed word-level initial generation, create the region-contextualized text representations for region-aware image refinement, and propose an iterative multi-headed mechanism to allow multiple different changes at each stage of refinement.", "summary_of_the_review": "This paper is overall well-written and easy to follow, but the ablation study and discussions may be inadequate to validate that the performances, properties, and novelties of the proposed modules. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers' evaluation of this paper are borderline/negative.  The AC considered the reviews, rebuttal, and the paper itself, and concurs with the reviewers.  The AC found that the paper is an extension of previous work DM-GAN (DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis, CVPR 2019, https://arxiv.org/pdf/1904.01310.pdf). This work uses the word features in addition to sentence features at the first stage of generation, while DM-GAN and other previous work don’t use word features in the first stage, but use them in the later stages when the feature resolution is higher. The authors improve the dynamic memory in DM-GAN into spatial dynamic memory, and also change the image refinement process in DM-GAN into an iterative refinement. The proposed multi-tailed word-level initial generation, spatial dynamic memory, and iterative refinement are incremental changes to DM-GAN. Moreover, the proposed structure almost doubles the parameter size of DM-GAN (shown in Table 2), yet the evaluation results on COCO are similar to DM-GAN with only minor improvements. It is not clear whether the performance improvement comes from  the increased number of parameters or the architecture design. Especially on the CUB dataset with limited number of images, the model can easily overfit with a larger number of parameters.  The proposed method shares the similar network structure and dynamic memory blocks as DM-GAN, except for a few changes.  Overall, the AC finds this paper not suitable for acceptance at ICLR in present form.
Decision: Reject