Multi-Tailed, Multi-Headed, Spatial Dynamic Memory refined Text-to-Image Synthesis

1 INTRODUCTION . Generative Adversarial Networks ( GANs ) have shown great promise for the generation of photo-realistic synthetic images ( Goodfellow et al. , 2014 ; Radford et al. , 2015 ; Denton et al. , 2015 ; Salimans et al. , 2016 ) , and the highly-compelling nature of images generated by GANs has driven research into conditional image-generation and multimodal learning . In this paper , we focus on the task of text-to-image generation , that has emerged as an area of active research in recent years . Although much progress has been made in this area , the synthesis of high-quality , realistic images from text-descriptions remains a challenging task . Current state-of-the-art methods ( Xu et al. , 2018 ; Li et al. , 2019a ; Zhu et al. , 2019 ) employ multiple-stages of image generation - typically , an initial image is first generated from a global sentence-level vector , and subsequent stages incorporate fine-grained information extracted from word-level vectors to refine image details . However , these methods suffer from three important limitations . The first problem is that by attempting to synthesize image features directly from a sentence-level vector , the initial generation stage fails to cleanly separate image attributes at a word-level . If potentially distinct objects such as ‘ cat ’ and ‘ tree ’ for example , are entangled in the sentence-level representation , then the presence of either word in a sentence could prompt the initial stage to generate the same hybrid image attributes . This is important because the subsequent refinement stage relies upon the initial image features to provide a meaningful basis for word-level refinement . By feeding it ambiguous and poorly formed initial features , we limit the scope of refinement . Secondly , current methods do not construct region-specific representations of text at refinement stages . This prevents us from interpreting words in fundamentally different ways based on the content of image regions . Whereas , in complex real-world scenes , the requirement for a region-contextualized interpretation of words is commonplace - based on the region under consideration , the same word often dictates fundamentally different types of refinement within a single image . The word ‘ raining ’ for example dictates a requirement in the sky that is fundamentally different from the requirement that it dictates in the region of the ground . While the sky becomes more cloudy , the ground must become wet . To generate realistic images from natural text descriptions , it is important that we construct a refinement architecture that allows different image regions to assimilate region-contextualized information from text descriptions . Finally , we note that current methods generate refinement features ( that modify previous image features ) only once at each refinement stage and attempt to address all image aspects within a single-shot . This single-shot refinement limits the precision with which each refinement stage can learn to improve the prior image . In this paper , we propose a Multi-Headed and Spatial Dynamic Memory image refinement mechanism with a Multi-Tailed Word-level Initial Generation stage ( MSMT-GAN ) to address these three issues . Our contributions are summarized as follows : • We introduce a novel `` Multi-Tailed '' Word-level Initial Generation stage ( MTWIG ) , that generates a separate set of image features for each word n-gram , and iteratively fuses these sets together to obtain initial image features . We demonstrate that it is possible to improve the performance of previous methods by replacing their initial generation stage with ours . • We introduce a novel Spatial Dynamic Memory module ( SDM ) that fuses word-information in a custom way with each prior image region , to obtain region-contextualized textrepresentations . At each refinement stage we retrieve features for image improvement from this SDM module . • We introduce a novel Iterative Multi-Headed Mechanism ( IMHM ) of image refinement - wherein we explicitly allow each stage of refinement to make multiple distinct modifications to the prior image , under common discriminator feedback . We evaluate our MSMT-GAN model on the Caltech-UCSD Birds 200 ( CUB ) dataset ( Wah et al. , 2011 ) and the Microsoft Common Objects in Context ( COCO ) dataset ( Lin et al. , 2014 ) . Experiment results demonstrate that MSMT-GAN is competitive with current methods on the COCO dataset and significantly outperforms the previous state-of-the art on the CUB dataset , decreasing the lowest reported Fréchet Inception Distance ( FID ) ( Heusel et al. , 2017 ) by 21.58 % for CUB . 2 RELATED WORK . Text-to-Image Generators : Reed et al . ( 2016 ) first demonstrated that a translation model from natural language to image pixels could be learnt by conditioning both generator and discriminator networks of a GAN on input text-descriptions . There has since been a surge of interest in training multi-stage attention based GAN architectures for this task . While the conventional setting ( Zhang et al. , 2017 ; Xu et al. , 2018 ; Li et al. , 2019a ; Zhu et al. , 2019 ) assumes only the availability of ( text , image ) pairs at training time , recently a second setting has emerged that assumes availability of bounding-box/shape-mask information of objects attributes during training ( Li et al. , 2019b ; Hinz et al. , 2019 ; Cho et al. , 2020 ; Liang et al. , 2020 ) . We highlight that this represents a significantly easier problem setting and that such methods are not feasible where bounding-box/shape information is unavailable ( such as the CUB dataset ) . Our method does not assume the availability of bounding-box/shape information , and we make comparisons against prior work of the same setting . Memory Networks : Memory Networks ( Weston et al. , 2014 ) combine inference components with a long-term memory module that can be dynamically written to and read from . Current methods ( Miller et al. , 2016 ) query “ key encodings '' of memory slots to retrieve a set of weights . These weights are used to combine separate “ value encodings '' of the slots into a single response . A Dynamic Memory Generative Adversarial Network ( DM-GAN ) ( Zhu et al. , 2019 ) that retrieves information for image refinement from a memory module was recently proposed for text-to-image synthesis . In our SDM module , we too employ the memory-writing , key-addressing , value-reading paradigm introduced by ( Miller et al. , 2016 ) , but our method differs from ( Zhu et al. , 2019 ) in all three memory operations ( Section 3.2 ) . Fundamentally , DM-GAN does not create region-contextualized representations of text . Multi-Headed Attention : Transformers ( Vaswani et al. , 2017 ) utilize a key-value mechanism similar to memory networks and introduced the idea of multi-headed attention . They linearly project query , keys and values to h separate encodings , called “ attention heads '' , and each head is separately used to extract an output vector . These vectors are concatenated together and linearly projected to a single response . Inspired by the success of Transformers , we introduce the IMHM method for image refinement . However , our method differs in a few respects . We maintain separate SDM modules for each head and we obtain queries and fuse outputs in an iterative fashion . We also adopt a “ redundancy loss '' ( Section 3.4 ) to encourage each head to focus on separate image aspects . 3 MSMT-GAN . Our MSMT-GAN architecture ( Figure 1 ) comprises of three stages - a Multi-Tailed Word-level Initial Generation ( MTWIG ) stage , and two refinement stages . Each refinement stage is Multi-Headed , and each refinement head has a separate Spatial Dynamic Memory ( SDM ) module . Section 3.1 presents our MTWIG stage , Section 3.2 presents our SDM module for a single refinement head , and the details of our Iterative Multi-Headed Mechanism ( IMHM ) are presented in Section 3.3 . 3.1 MULTI-TAILED WORD-LEVEL INITIAL GENERATION ( MTWIG ) . We highlight that previous multi-stage methods ( Zhang et al. , 2017 ; 2018 ; Li et al. , 2019a ; Zhu et al. , 2019 ) all rely on the same type of initial generation stage and focus only on improving the refinement stages - making the conventional assumption that the performance of multi-stage generators is primarily determined by the refinement stages , and that the quality of the `` rough initial image '' is of little importance . In our paper , we break from this tradition and demonstrate for the first time that gains can be achieved in the final stage of image refinement by making an improvement to the initial images . The conventional approach synthesizes initial images directly from a sentence-level vector without attempting to separate image attributes at a word-level . As a result , words that are entangled at the sentence-level representation generate initial image attributes that are inherently ambiguous in nature . In our novel Multi-Tailed Word-level Initial Generation ( MTWIG ) stage , we overcome this shortcoming by explicitly creating separate sets of image attributes for each word n-gram . First , we sample a vector of random noise z from a normal distribution and use a pretrained text-encoder to extract a sentence-level vector and word-level vectors : s and W from the input text . W = { w1 , w2 , ... , wL } ; wl ∈ RNw ; s ∈ RNs ; zn ∼ N ( 0 , 1 ) ; z ∈ RNz ( 1 ) Where L is the number of words in the text-description , and Nz , Ns and Nw are the dimensions of the noise vector , sentence vector and word vectors respectively . To mitigate over-fitting , the Conditioning Augmentation technique ( Zhang et al. , 2017 ) is used to resample the sentence-vector from an independent Gaussian distribution . This resampled sentence vector s′ and the noise vector z are then concatenated with each word-level vector wl from the input text sequence , and the sequence of concatenated vectors are passed through a 1D convolutional operation V of stride 1 ( see Figure 1 ) . F = V ( { concat ( s′ , z , wl ) | ∀ wl ∈W } ) ( 2 ) The length T of the output sequence F depends on the kernel size used by V and the vectors of the output sequence ft ∈ F are each separately passed through a series of upsampling blocks to generate corresponding sets of image features St . These sets of image features or `` tails '' each correspond to a different word n-gram from the input text sequence . If we use a kernel size of 1 for V , then each tail St corresponds to a single word . If we use a kernel size of 2 , then each tail St corresponds to a word bi-gram , and so on . We combine our sequence of tails { St } together in an iterative fashion using the adaptive gating fusion mechanism introduced by Zhu et al . ( 2019 ) ( discussed in Section A.1 ) . S1 : t = fuse ( S1 : t−1 , St , P MTWIG , ρMTWIG ) ; R1 = S1 : T ( 3 ) Where PMTWIG and ρMTWIG denote parameter matrix and bias terms , S1 : t denotes a combination of the first t tails , and S1:1 denotes the first tail S1 . The combination of all T tails gives us the final image features R1 for our initial stage . Notice that by concatenating each word vector wl with s′ and z before the 1D convolution , each tail is created with some common information , so they may learn to fuse together coherently . Each upsampling block consists of a nearest neighbor upsampling layer and a 3×3 convolution operation . An initial image is predicted from R1 using a 3×3 convolution .

Summary: The paper proposed a new method to tackle text-to-image generation challenge. In the paper, authors introduced a potential problem that current methods only use sentence embedding at beginning of the network to generate initial images, where different attributes may be entangled and are hard to be refined during following states. Based on this, authors proposed three components to address this limitation.