Year: 2021

Abstract:
Synthesizing high-quality, realistic images from text-descriptions is a challenging task, and current methods synthesize images from text in a multi-stage manner, typically by first generating a rough initial image and then refining image details at subsequent stages. However, existing methods that follow this paradigm suffer from three important limitations. Firstly, they synthesize initial images without attempting to separate image attributes at a word-level. As a result, object attributes of initial images (that provide a basis for subsequent refinement) are inherently entangled and ambiguous in nature. Secondly, by using common text-representations for all regions, current methods prevent us from interpreting text in fundamentally different ways at different parts of an image. Different image regions are therefore only allowed to assimilate the same type of information from text at each refinement stage. Finally, current methods generate refinement features only once at each refinement stage and attempt to address all image aspects in a single shot. This single-shot refinement limits the precision with which each refinement stage can learn to improve the prior image. Our proposed method introduces three novel components to address these shortcomings: (1) An initial generation stage that explicitly generates separate sets of image features for each word n-gram. (2) A spatial dynamic memory module for refinement of images. (3) An iterative multi-headed mechanism to make it easier to improve upon multiple image aspects. Experimental results demonstrate that our Multi-Headed Spatial Dynamic Memory image refinement with our Multi-Tailed Word-level Initial Generation (MSMTGAN) performs favourably against the previous state of the art on the CUB and COCO datasets.

heading: 1 INTRODUCTION
text: Generative Adversarial Networks (GANs) have shown great promise for the generation of photo-realistic synthetic images (Goodfellow et al., 2014; Radford et al., 2015; Denton et al., 2015; Salimans et al., 2016), and the highly-compelling nature of images generated by GANs has driven research into conditional image-generation and multimodal learning. In this paper, we focus on the task of text-to-image generation, that has emerged as an area of active research in recent years. Although much progress has been made in this area, the synthesis of high-quality, realistic images from text-descriptions remains a challenging task. Current state-of-the-art methods (Xu et al., 2018; Li et al., 2019a; Zhu et al., 2019) employ multiple-stages of image generation - typically, an initial image is first generated from a global sentence-level vector, and subsequent stages incorporate fine-grained information extracted from word-level vectors to refine image details. However, these methods suffer from three important limitations. The first problem is that by attempting to synthesize image features directly from a sentence-level vector, the initial generation stage fails to cleanly separate image attributes at a word-level. If potentially distinct objects such as ‘cat’ and ‘tree’ for example, are entangled in the sentence-level representation, then the presence of either word in a sentence could prompt the initial stage to generate the same hybrid image attributes. This is important because the subsequent refinement stage relies upon the initial image features to provide a meaningful basis for word-level refinement. By feeding it ambiguous and poorly formed initial features, we limit the scope of refinement. Secondly, current methods do not construct region-specific representations of text at refinement stages. This prevents us from interpreting words in fundamentally different ways based on the content of image regions.
Whereas, in complex real-world scenes, the requirement for a region-contextualized interpretation of words is commonplace - based on the region under consideration, the same word often dictates fundamentally different types of refinement within a single image. The word ‘raining’ for example dictates a requirement in the sky that is fundamentally different from the requirement that it dictates in the region of the ground. While the sky becomes more cloudy, the ground must become wet. To generate realistic images from natural text descriptions, it is important that we construct a refinement architecture that allows different image regions to assimilate region-contextualized information from text descriptions. Finally, we note that current methods generate refinement features (that modify previous image features) only once at each refinement stage and attempt to address all image aspects within a single-shot. This single-shot refinement limits the precision with which each refinement stage can learn to improve the prior image.
In this paper, we propose a Multi-Headed and Spatial Dynamic Memory image refinement mechanism with a Multi-Tailed Word-level Initial Generation stage (MSMT-GAN) to address these three issues. Our contributions are summarized as follows:
• We introduce a novel "Multi-Tailed" Word-level Initial Generation stage (MTWIG), that generates a separate set of image features for each word n-gram, and iteratively fuses these sets together to obtain initial image features. We demonstrate that it is possible to improve the performance of previous methods by replacing their initial generation stage with ours.
• We introduce a novel Spatial Dynamic Memory module (SDM) that fuses word-information in a custom way with each prior image region, to obtain region-contextualized textrepresentations. At each refinement stage we retrieve features for image improvement from this SDM module.
• We introduce a novel Iterative Multi-Headed Mechanism (IMHM) of image refinement - wherein we explicitly allow each stage of refinement to make multiple distinct modifications to the prior image, under common discriminator feedback.
We evaluate our MSMT-GAN model on the Caltech-UCSD Birds 200 (CUB) dataset (Wah et al., 2011) and the Microsoft Common Objects in Context (COCO) dataset (Lin et al., 2014). Experiment results demonstrate that MSMT-GAN is competitive with current methods on the COCO dataset and significantly outperforms the previous state-of-the art on the CUB dataset, decreasing the lowest reported Fréchet Inception Distance (FID) (Heusel et al., 2017) by 21.58% for CUB.

heading: 2 RELATED WORK
text: Text-to-Image Generators:Reed et al. (2016) first demonstrated that a translation model from natural language to image pixels could be learnt by conditioning both generator and discriminator networks of a GAN on input text-descriptions. There has since been a surge of interest in training multi-stage attention based GAN architectures for this task. While the conventional setting (Zhang et al., 2017; Xu et al., 2018; Li et al., 2019a; Zhu et al., 2019) assumes only the availability of (text,image) pairs at training time, recently a second setting has emerged that assumes availability of bounding-box/shape-mask information of objects attributes during training (Li et al., 2019b; Hinz et al., 2019; Cho et al., 2020; Liang et al., 2020). We highlight that this represents a significantly easier problem setting and that such methods are not feasible where bounding-box/shape information is unavailable (such as the CUB dataset). Our method does not assume the availability of bounding-box/shape information, and we make comparisons against prior work of the same setting.
Memory Networks: Memory Networks (Weston et al., 2014) combine inference components with a long-term memory module that can be dynamically written to and read from. Current methods (Miller et al., 2016) query “key encodings" of memory slots to retrieve a set of weights. These weights are used to combine separate “value encodings" of the slots into a single response. A Dynamic Memory Generative Adversarial Network (DM-GAN) (Zhu et al., 2019) that retrieves information for image refinement from a memory module was recently proposed for text-to-image synthesis. In our SDM module, we too employ the memory-writing, key-addressing, value-reading paradigm introduced by (Miller et al., 2016), but our method differs from (Zhu et al., 2019) in all three memory operations (Section 3.2). Fundamentally, DM-GAN does not create region-contextualized representations of text.
Multi-Headed Attention: Transformers (Vaswani et al., 2017) utilize a key-value mechanism similar to memory networks and introduced the idea of multi-headed attention. They linearly project query, keys and values to h separate encodings, called “attention heads", and each head is separately used to extract an output vector. These vectors are concatenated together and linearly projected to a single response. Inspired by the success of Transformers, we introduce the IMHM method for image refinement. However, our method differs in a few respects. We maintain separate SDM modules for each head and we obtain queries and fuse outputs in an iterative fashion. We also adopt a “redundancy loss" (Section 3.4) to encourage each head to focus on separate image aspects.

heading: 3 MSMT-GAN
text: Our MSMT-GAN architecture (Figure 1) comprises of three stages - a Multi-Tailed Word-level Initial Generation (MTWIG) stage, and two refinement stages. Each refinement stage is Multi-Headed, and each refinement head has a separate Spatial Dynamic Memory (SDM) module. Section 3.1 presents our MTWIG stage, Section 3.2 presents our SDM module for a single refinement head, and the details of our Iterative Multi-Headed Mechanism (IMHM) are presented in Section 3.3.

heading: 3.1 MULTI-TAILED WORD-LEVEL INITIAL GENERATION (MTWIG)
text: We highlight that previous multi-stage methods (Zhang et al., 2017; 2018; Li et al., 2019a; Zhu et al., 2019) all rely on the same type of initial generation stage and focus only on improving the refinement stages - making the conventional assumption that the performance of multi-stage generators is primarily determined by the refinement stages, and that the quality of the "rough initial image" is of little importance. In our paper, we break from this tradition and demonstrate for the first time that gains can be achieved in the final stage of image refinement by making an improvement to the initial images.
The conventional approach synthesizes initial images directly from a sentence-level vector without attempting to separate image attributes at a word-level. As a result, words that are entangled at the sentence-level representation generate initial image attributes that are inherently ambiguous in nature. In our novel Multi-Tailed Word-level Initial Generation (MTWIG) stage, we overcome this shortcoming by explicitly creating separate sets of image attributes for each word n-gram.
First, we sample a vector of random noise z from a normal distribution and use a pretrained text-encoder to extract a sentence-level vector and word-level vectors: s and W from the input text.
W = {w1, w2, ..., wL}; wl ∈ RNw ; s ∈ RNs ; zn ∼ N (0, 1); z ∈ RNz (1)
Where L is the number of words in the text-description, and Nz , Ns and Nw are the dimensions of the noise vector, sentence vector and word vectors respectively. To mitigate over-fitting, the Conditioning Augmentation technique (Zhang et al., 2017) is used to resample the sentence-vector from an independent Gaussian distribution. This resampled sentence vector s′ and the noise vector z are then concatenated with each word-level vector wl from the input text sequence, and the sequence of concatenated vectors are passed through a 1D convolutional operation V of stride 1 (see Figure 1).
F = V ({concat(s′, z, wl) | ∀ wl ∈W}) (2)
The length T of the output sequence F depends on the kernel size used by V and the vectors of the output sequence ft ∈ F are each separately passed through a series of upsampling blocks to generate corresponding sets of image features St. These sets of image features or "tails" each correspond to a different word n-gram from the input text sequence. If we use a kernel size of 1 for V , then each tail St corresponds to a single word. If we use a kernel size of 2, then each tail St corresponds to a word bi-gram, and so on. We combine our sequence of tails {St} together in an iterative fashion using the adaptive gating fusion mechanism introduced by Zhu et al. (2019) (discussed in Section A.1).
S1:t = fuse(S1:t−1, St, P MTWIG, ρMTWIG) ; R1 = S1:T (3)
Where PMTWIG and ρMTWIG denote parameter matrix and bias terms, S1:t denotes a combination of the first t tails, and S1:1 denotes the first tail S1. The combination of all T tails gives us the final image features R1 for our initial stage. Notice that by concatenating each word vector wl with s′ and z before the 1D convolution, each tail is created with some common information, so they may learn to fuse together coherently. Each upsampling block consists of a nearest neighbor upsampling layer and a 3×3 convolution operation. An initial image is predicted from R1 using a 3×3 convolution.

heading: 3.2 SPATIAL DYNAMIC MEMORY (SDM)
text: In this section, we describe the operation of a single refinement head. Unlike previous methods, our novel Spatial Dynamic Memory (SDM) module creates a separate region-contextualized text representations for each image region. This allows us to interpret the same text in fundamentally different ways at different parts of an image and assimilate region-contextualized information from text at each part. To begin with, we have the set of word-level vectors W and image features Rk−1 from the previous stage of generation.
Rk−1 = {r1,1 , r1,2 , ..., rs,s} ; ru,v ∈ RNr (4)
Where |s× s| is the number of image pixels and Nr is the dimension of pixel features. We obtain refinement features in three steps: Memory Writing, Key Addressing and Value Reading.
Memory Writing: First, we divide the fine-grained s× s inital image into a coarse h× h sized grid-map and average the pixel features within each grid-cell to get grid-level image features C.
Ci,j = 1
|p× p| i∗p∑ u=(i−1)∗p+1 j∗p∑ v=(j−1)∗p+1 ru,v (5)
Where p = s/h, so that |p × p| are the number of pixels represented by each grid cell. Then, we create L× h× h memory slots {ml,i,j} - one corresponding to each word l for each grid-cell (i, j). These slots are our region-contextualized representations of each word, and each slot uses a separate memory writing gate gwl,i,j to fuse information from each grid-cell (i, j) with each word feature wl.
gwl,i,j(Rk−1, wL) = σ (A ∗ wl +Bi,j ∗ Ci,j) (6) ml,i,j =Mw(wl) gwl,i,j +Mc(C)i,j (1− gwl,i,j) (7)
The grid-level features C are encoded using a 2d convolution operation Mc (with stride 1 and Nm output filters) and we use a common 1x1 convolution operation Mw to encode all word vectors to a Nm dimensional space. A and Bi,j are 1×Nw and 1×Nr matrices respectively.
Key Addressing: In this step, we compute attention weights {αl,i,j,a,b} over our regioncontextualized text-representations {ml,i,j}. The dimensions (a, b) index pixels within grid-cell (i, j), so that each slotml,i,j gets a matrix αl,i,j : p×p of attention weights. Each weight is computed as a similarity probability between a key-encoding of the slot: φK(ml)ij and a query vector: qi,j,a,b, where φK(.) is a 2d convolution operation with stride 1 and (Nr + p2) output filters.
αl,i,j,a,b = exp(φK(ml)i,j ∗ qi,j,a,b)∑L l=1 exp(φK(ml)i,j ∗ qi,j,a,b)
(8)
In the case of single headed image refinement, we use the previous image features Rk−1 to obtain the query vectors. A query vector qi,j,a,b is made up of three components, 1)A global-level query: qglobal, 2)A grid-level query: qgridi,j , and 3)A pixel-level query: q pixel i,j,a,b. To obtain these three components, we encode Rk−1 using three separate 2d convolution operations: φQglobal(.), φQgrid(.) and φQpixel(.), each with a stride of 1 and Nr output filters.
Qglobal = φQglobal(Rk−1) ; Q grid = φQgrid(Rk−1) ; Q pixel = φQpixel(Rk−1) (9)
Then, the average of all pixel features of Qglobal becomes the global-level query component qglobal. The average of pixel features within the grid cell (i, j) of Qgrid becomes the grid-level query qgridi,j , and the pixel feature at location (a, b) within grid cell (i, j) is extracted from Qpixel to give us the pixel-level query component qpixeli,j,a,b.
qglobal = 1
|s× s| s∑ u=1 s∑ v=1 Qglobalu,v ; q grid i,j =
1
|p× p| i∗p∑ u=(i−1)∗p+1 j∗p∑ v=(j−1)∗p+1 Qgridu,v (10)
qpixeli,j,a,b = Q local h(i,a), h(j,b) ; h(i, a) = (i− 1) ∗ p+ a (11)
Where (h(i, a), h(j, b)) indexes the pixel at location (a, b) within grid-cell (i, j). To obtain the final query qi,j,a,b, we concatene these three components together.
Value Reading: In the value reading step, for each pixel (a, b) within a grid-cell (i, j), we compute a weighted sum of value-encoded memory slots: φV (ml)ij along the word dimension l.
ei,j,a,b = L∑ l=1 αl,i,j,a,b · φV (ml)i,j (12)
φV (.) is a 2d convolution operation with stride 1 andNr output filters. We now have ei,j : p×p×Nr dimensional matrices - each of which corresponds to a single grid cell of our coarse h× h grid map. To obtain s× s fine-grained refinement features, we apply the mapping:
oh(i,a) , h(j,b) = ei,j,a,b (13)
Where h(., .) is the function defined in Eq.11. That is, we populate each grid cell with |p× p| vectors of Nr dimensionality. Since p = s/h, we are left with a set of refinement features O = {ou,v} that are made up of |s× s| vectors of Nr dimensionality, each of which corresponds to a single pixel.

heading: 3.3 ITERATIVE MULTI-HEADED MECHANISM (IMHM)
text: Current methods generate refinement features only once at each refinement stage and attempt to address all image aspects in a single-shot. This limits the precision with which each refinement stage can learn to improve the prior image. In order to make it easier for each refinement stage to precisely address multiple image aspects, we introduce a novel iterative multi-headed mechanism that makes multiple distinct modifications to the prior image features under common discriminator feedback. Each head of our mechanism has a separate spatial dynamic memory module formed from Rk−1 and W . For the first refinement head, we use the previous image features Rk−1 to obtain a query matrix and extract a set of refinement features O1 exactly as described in Section 3.2. Then, we fuse O1 and Rk−1 using the fusion mechanism introduced by Zhu et al. (2019) (described in Section A.1) to obtain an updated set of image features U1. If we use only a single refinement head, then this becomes our response for the refinement stage k. However, if we use more than one refinement head, then for the next head, we use U1 to obtain a query matrix. That is, we follow the same mechanism
outlined in Section 3.2, but replace Rk−1 with U1 in Eq.9. Doing so, we extract a second set of refinement features O2, and we fuse O2 and U1 to obtain updated image features U2. We proceed in this iterative fashion until we have used all of our refinement heads. The final updated image features are fused with the original image features Rk−1 in a skip-connection to obtain the final response of the refinement stage k. That is, if we have T refinement heads:
Ut = fuse(Ut−1, Ot, Pt, ρt) ; responsek = fuse(UT , Rk−1, P skip, ρskip) (14)
Notice, we use separate parameter matrix and and bias terms P and ρ for each fusion operation, so that we combine refinement features and image features in a custom way for each head. The, responsek is passed through several residual blocks (He et al., 2016) and an upsampling block to obtain higher resolution image features Rk. Each block consists of a nearest neighbor upsampling layer and a 3×3 convolution operation. Finally, a refined image xk is predicted from Rk using a 3×3 convolution operation.

heading: 3.4 OBJECTIVE FUNCTION
text: The objective function for our generator network is defined as:
L = λ1LCA + K∑ k=2 λ2LREDk + K∑ k=1 (LGk + λ3LDAMSMk) (15)
Where, LCA denotes the conditioning augmentation loss (Zhang et al., 2017), Gk denotes the generator of the kth stage so that LGk denotes the adversarial loss for Gk, LREDk denotes our redundancy loss for the kth stage, and LDAMSMk denotes the DAMSM text-image matching loss (Xu et al., 2018) for the kth stage. λ1, λ2 and λ3 are hyperparameters that combine the various losses.
Redundancy Loss: To encourage each head of a refinement stage to focus on separate image aspects, we average region-wise information of each head’s output refinement features and penalize similarity between different refinement heads. That is, for T refinement heads:
f(t) = 1
|s× s| s∑ u=1 s∑ v=1 ou,v ; LREDk = T∑ i=1 T∑ j=i+1 sim(f(i), f(j)) (16)
Where ou,v ∈ Ot in stage k (see Section 3.3) and sim is the cosine similarity between vectors. We call this sum of pairwise similarity LREDk the “redundancy loss" of the k
th refinement stage. Adversarial Loss: The adversarial loss for Gk is defined as:
LGk = − 1
2 [Ex∼pGk logDk(x) + Ex∼pGk logDk(x, s)] (17)
Dk is the discriminator network for the kth stage of generation. The first term provides feedback on image realism independent of the input text, and the second term provides feedback on the realism of the image in light of the input text. Alternate to adversarial training of Gk, each discriminator Dk is trained to classify images as real or fake by minimizing the discriminator loss LDk .
LDk =−
1 2 [Ex∼pdata logDk(x) + Ex∼pGk
log(1 − Dk(x))]︸ ︷︷ ︸ unconditional loss
+− 1
2 [Ex∼pdata logDk(x, s) + Ex∼pGk log(1 − Dk(x, s))]︸ ︷︷ ︸ conditional loss
(18)
Where the unconditional component distinguishes synthetic and real images independent of the input text, and the conditional component distinguishes them in light of the input text.

heading: 4 EXPERIMENTS
text: Datasets: We evaluate our method on the Caltech-UCSD Birds 200 (CUB) dataset (Wah et al., 2011) and the Microsoft Common Objects in Context (COCO) dataset (Lin et al., 2014). The CUB dataset contains 8,855 training images and 2,933 test images, with 10 corresponding text descriptions for each image. The COCO dataset, contains 82,783 training images and 40,504 test images, with 5 corresponding text descriptions for each image. We preprocess the datasets according to the methods introduced by Zhang et al. (2017).
Evaluation metrics: To evaluate the realism of images, we rely on the Fréchet Inception Distance (FID) (Heusel et al., 2017). FID computes the distance between synthetic and real-world image distributions based on features extracted from a pre-trained Inception v3 network. A lower FID indicates greater image-realism. To evaluate the relevance of a synthetic image to it’s generating text-description, we rely on the R-precision introduced by Xu et al. (2018). R-precision is computed as the mean accuracy of using each synthetic image to retrieve one ground truth text-description from among 100 candidates. To evaluate a model on a dataset, we generate 30,000 synthetic images conditioned on text descriptions from the unseen test set.
Implementation Details: To obtain a sentence-level vector and word-level vectors for a given text description, we use the pretrained bidirectional LSTM text encoder employed by AttnGAN (Xu et al., 2018). Our MTWIG stage synthesizes images features with 64x64 resolution. Two refinement stages refine these features to 128x128 and 256x256 resolution respectively. At refinement stages, we use T = 6 refinement heads and we use h = 8 to divide each input image into a coarse 8× 8 grid map. We use the same discriminator network architecture employed by Zhu et al. (2019). Further implementation details are provided in Section A.3.

heading: 4.1 ABLATIVE EXPERIMENTS
text: Effectiveness of Multi-Tailed Word-level Initial Generation: In our experiments (Appendix A.1), we find that our MTWIG stage is most effective when used with a kernel size of 3, so that we generate a separate tail for each word tri-gram. To evaluate the effectiveness of our MTWIG(ks=3) stage in multi-stage models, we train our MTWIG(ks=3) method with DM-GAN (Zhu et al., 2019) style-refinement stages for 700 epochs on the CUB dataset, and observe that it decreases the FID score achieved by the original DM-GAN model by 7.72% and increases R-precision by 2.76% (Table 1). Figure 2 shows the improved visual quality of the refined images. We again point out that previous multi-stage methods (Zhang et al., 2017; 2018; Xu et al., 2018; Li et al., 2019a; Zhu et al., 2019)) all rely on the same type of initial generation stage, and we expect a similar boost in performance if we replace their initial stage with our MTWIG stage.
Effectiveness of Spatial Dynamic Memory: In order to evaluate the effectiveness of our SDM based-refinement stages, we compare a multi-stage model that uses our MTWIG(ks=3) stage and DM-GAN’s refinement stages against a multi-stage model that uses our MTWIG(ks=3) stage and our single-headed SDM-based refinement stages. Both models are trained on the CUB dataset for 700 epochs. We observe that our SDM-based refinement out-performs DM-GAN’s refinement, decreasing FID score by 4.64 % , and boosting R-precision by an additional 0.83% (Table 1). Figure 2 shows that SDM-based refinement generates images of better visual quality than those formed by DM-GAN’s refinement stages for the same initial generation architecture.
Effectiveness of the Iterative Multi-Headed Mechanism: To evaluate the effectiveness of our IMHM refinement, we compare the model that uses MTWIG(ks=3) and single-headed SDM-based refinement stages against our full MSMT-GAN model - that uses MTWIG(ks=3) and six SDM-based refinement heads for each stage. As before, both models are trained for 700 epochs on the CUB dataset. We find that refinement stages that use our multi-headed mechanism out-perform single-headed refinement stages, decreasing FID score 2.38%, and boosting R-precision by another 1.03% (Table 1). Visually, besides overall better image quality, we find that images generated by multi-headed refinement stages posess text-irrelevant content that is far more detailed than that observed in images generated by single-headed refinement stages (Figure 2).

heading: 4.2 COMPARISON WITH STATE OF THE ART
text: To compare our architecture against the previous state-of-the-art for the task of text-to-image generation, we train MSMT-GAN models for 1000 epochs on the CUB dataset and for 210 epochs on the COCO dataset. As shown in Table 3, our MSMT-GAN decreases the previous lowest reported FID score by 21.58% on the CUB dataset -marking a significant improvement in image realism, and also boosts the previous best reported CUB R-precision by 4.24 % - marking a large improvement in the similarity of synthetic images to their generating text. As shown in Table 2 and Table 3, our model is comparable in size to previous methods, and outperforms the next closest contender of similar
size for COCO (DM-GAN) by 4.21% on FID score -making it highly competitive with the current state-of-the-art. We also observe a slight improvement of 0.23% on COCO R-precision. Qualitatively, we observe that synthetic images generated by our model are typically sharper and more realistic than those generated by prior methods (Figure 3). In particular, we observe that our method generates synthetic images that possess greater detail and that are better matches for the generating text.

heading: 5 CONCLUSION
text: In this work, we have proposed the MSMT-GAN architecture for the task of text-to-image synthesis. First, we introduced a novel Multi-Tailed Word Level Initial Generation stage (MTWIG) that explicitly generates separate sets of image features for each word n-gram. Second, we proposed a novel Spatial Dynamic Memory (SDM) module to contextualize text representations by image-region. Third, we introduced a novel Iterative Multi-Headed Mechanism (IMHM) of image refinement to make it easier
for each refinement stage to precisely address multiple image aspects. Our ablative experiments clearly demonstrate the effectiveness each of these three components, and we have shown that we are able to boost the performance of prior methods by replacing their initial stage with our MTWIG stage. Experiment results further demonstrate that our MSMT-GAN model significantly out-performs the previous state of the art on the CUB dataset, decreasing the lowest reported FID score by 21.58% and boosting the CUB R-precision by 4.24%. On the COCO dataset, we have demonstrated that MSMT-GAN is highly comptitive with current methods based on image realism and model size. In future work, we aim to design a discriminator model that provides more region-specific feedback than existing methods, to use in conjunction with our MSMT-GAN generator architecture.

heading: A APPENDIX
text: A.1 FUSING TWO SETS OF IMAGE FEATURES TOGETHER
Given any two sets of image/refinement features A = {au,v} and B = {bu,v} such that au,v, bu,v ∈ RNr , we can use the adaptive gating mechanism introduced by Zhu et al. (2019) to obtain a combined/updated set of features {ru,v}, where ru,v ∈ RNr .
gru,v = σ(P ∗ concat(au,v, bu,v) + ρ) (19) ru,v = au,v gru,v + bu,v (1− gru,v) (20)
Where (u, v) index pixel locations, Nr is the dimension of image pixel features, gru,v is a gate for information fusion, σ is the sigmoid function and P and ρ are the parameter matrix and bias term respectively. In our paper, we invoke this adaptive gating mechanism by a function “fuse" such that fuse(A,B, P, ρ) = {ru,v}.
A.2 ANALYSIS OF MULTI-TAILED WORD-LEVEL INITIAL GENERATION
To analyze the effectiveness of our MTWIG stage, we make comparisons with the inital generation (IG) stage employed by previous methods (Zhang et al., 2017; 2018; Xu et al., 2018; Li et al., 2019a; Zhu et al., 2019). We train IG and MTWIG stages without refinement for 300 epochs on the CUB dataset and for 60 epochs on the COCO dataset. Table 4 shows a quantitative comparison between IG and MTWIG architectures that use different kernel sizes. We observe that our MTWIG method achieves best results for ks=3 (that is, by forming separate sets of word tri-grams), decreasing FID
scores from the previous IG method by 3.36% on the CUB dataset and by 22.2% on the COCO dataset. The larger improvement observed on the COCO dataset highlights that our MTWIG method is most beneficial for complex scene generation, where the presence of a large number of distinct objects demands word-level separation of attributes. Our MTWIG stages also achieves competitive R-precision scores on both datasets, demonstrating that images synthesized by our method are well conditioned on their input text-descriptions. In Figure 4, we visually compare 64x64 images generated by IG and images generated by our MTWIG(ks=3) model. We observe that images generated by our method typically posses object attributes that are better separated and more clearly discernible.
A.3 ADDITIONAL IMPLEMENTATION DETAILS
We set Nw = 256, Nr = 48 and Nm = 96 to be the dimension of text, image and memory feature vectors respectively. We set the hyperparameters {λ1 = 1, λ2 = 0.5, λ3 = 5} for the CUB dataset and {λ1 = 1, λ2 = 0.5, λ3 = 50} for the COCO dataset. All networks are trained using an ADAM optimizer (Kingma & Ba, 2015) with β1 = 0.5 and β2 = 0.999. The learning rate is set to be 0.0002 and we use a batch size of 20 for all networks. We train our models on a single Tesla V100 gpu, and observe that similar to previous multi-stage methods, training our MSMT-GAN method is a GPU intensive process. We require approximately 27min for one epoch of the CUB dataset, and approximately 4.7 hours for one epoch of the COCO dataset.
A.4 NOTE ON EVALUATION METRICS
We point out that there is inconsistency in the FID implementation used to evaluate prior methods. While some methods (Li et al., 2019b) report scores using the official Tensorflow version of FID - which uses the weights of a pretrained Tensorflow inception model, other methods (Zhu et al., 2019; Zhang et al., 2021) have reported scores using an unofficial Pytorch implementation of FID - that uses the weights of a pretrained Pytorch inception model.
Each of these implementations computes different values of FID scores for the same sets of images, and scores computed from the two versions are not directly comparable with each other. We highlight that the correlation between FID score and human judgement is an empirical observation, that is dependent on the weights of the pretrained inception model used. The correlation with human judgement has only been shown to hold true for the weights of the pretrained Tensorflow inception model (used by the FID authors -Heusel et al. (2017)), and has not been verified for the unofficial Pytorch inception model (which has different weights). As such, to ensure that we correlate with human judgement, in our paper we only report scores using the official Tensorflow implementation of FID score - computing values for prior work from the pretrained models released by their authors. In Table 5 below, we additionally report Pytorch-implementation FID scores of SEGAN (Tan et al., 2019) and XMC-GAN (Zhang et al., 2021) models, as reported by their authors. It was not possible for us to recompute these scores using the official Tensorflow version of FID as pretrained models for these methods have not been made publicly available.
We again highlight, that the FID scores computed by the official Tensorflow implementation are not directly comparable with the scores computed by the unofficial Pytorch implementation.
In Table 5, we additionally benchmark the performance of models on the Inception Score (IS) (Goodfellow et al., 2014). We note however, that the FID score (which is reference-based and compares the distributions of real and synthetic images together) has been observed to be more consistent with human judgement of image realism than IS (which is reference-free and does not make comparisons to real images) (Heusel et al., 2017). We were not able to recompute other metrics for RiFeGAN (Cheng et al., 2020) and LeciaGAN (Qiao et al., 2019) as the pretrained models have not been made publicly available.
In Table 5, “-” represents cases where the data was not reported or is reported in a manner which is non-comparable (besides FID values).
*We make our comparisons against the pretrained models released by the authors, and we report results using the official implementations of FID score.
‡ We use the FID score reported by the paper, but note that this was computed using an unofficial Pytorch implementation of FID - which is not directly comparable with the official FID implementation scores. See Section A.4 for further details.

Review summaries:
The paper proposed a new method to tackle text-to-image generation challenge. In the paper, authors introduced a potential problem that current methods only use sentence embedding at beginning of the network to generate initial images, where different attributes may be entangled and are hard to be refined during following states. Based on this, authors proposed three components to address this limitation.

This paper is motivated by that most existing text-to-image methods suffer from three limitations and solutions are proposed to address the different limitations.  Firstly, it introduces multi-tailed word-level initial generation to enhance global sentence representation with distinct n-gram word presentation.   Second, the spatial dynamic memory module is proposed to create a separate region-contextualized text representation for each image region.   Finally, it introduces an iterative multi-headed mechanism to make multiple distinct modifications to the prior image features.