Multi-Tailed, Multi-Headed, Spatial Dynamic Memory refined Text-to-Image Synthesis

1 INTRODUCTION . Generative Adversarial Networks ( GANs ) have shown great promise for the generation of photo-realistic synthetic images ( Goodfellow et al. , 2014 ; Radford et al. , 2015 ; Denton et al. , 2015 ; Salimans et al. , 2016 ) , and the highly-compelling nature of images generated by GANs has driven research into conditional image-generation and multimodal learning . In this paper , we focus on the task of text-to-image generation , that has emerged as an area of active research in recent years . Although much progress has been made in this area , the synthesis of high-quality , realistic images from text-descriptions remains a challenging task . Current state-of-the-art methods ( Xu et al. , 2018 ; Li et al. , 2019a ; Zhu et al. , 2019 ) employ multiple-stages of image generation - typically , an initial image is first generated from a global sentence-level vector , and subsequent stages incorporate fine-grained information extracted from word-level vectors to refine image details . However , these methods suffer from three important limitations . The first problem is that by attempting to synthesize image features directly from a sentence-level vector , the initial generation stage fails to cleanly separate image attributes at a word-level . If potentially distinct objects such as ‘ cat ’ and ‘ tree ’ for example , are entangled in the sentence-level representation , then the presence of either word in a sentence could prompt the initial stage to generate the same hybrid image attributes . This is important because the subsequent refinement stage relies upon the initial image features to provide a meaningful basis for word-level refinement . By feeding it ambiguous and poorly formed initial features , we limit the scope of refinement . Secondly , current methods do not construct region-specific representations of text at refinement stages . This prevents us from interpreting words in fundamentally different ways based on the content of image regions . Whereas , in complex real-world scenes , the requirement for a region-contextualized interpretation of words is commonplace - based on the region under consideration , the same word often dictates fundamentally different types of refinement within a single image . The word ‘ raining ’ for example dictates a requirement in the sky that is fundamentally different from the requirement that it dictates in the region of the ground . While the sky becomes more cloudy , the ground must become wet . To generate realistic images from natural text descriptions , it is important that we construct a refinement architecture that allows different image regions to assimilate region-contextualized information from text descriptions . Finally , we note that current methods generate refinement features ( that modify previous image features ) only once at each refinement stage and attempt to address all image aspects within a single-shot . This single-shot refinement limits the precision with which each refinement stage can learn to improve the prior image . In this paper , we propose a Multi-Headed and Spatial Dynamic Memory image refinement mechanism with a Multi-Tailed Word-level Initial Generation stage ( MSMT-GAN ) to address these three issues . Our contributions are summarized as follows : • We introduce a novel `` Multi-Tailed '' Word-level Initial Generation stage ( MTWIG ) , that generates a separate set of image features for each word n-gram , and iteratively fuses these sets together to obtain initial image features . We demonstrate that it is possible to improve the performance of previous methods by replacing their initial generation stage with ours . • We introduce a novel Spatial Dynamic Memory module ( SDM ) that fuses word-information in a custom way with each prior image region , to obtain region-contextualized textrepresentations . At each refinement stage we retrieve features for image improvement from this SDM module . • We introduce a novel Iterative Multi-Headed Mechanism ( IMHM ) of image refinement - wherein we explicitly allow each stage of refinement to make multiple distinct modifications to the prior image , under common discriminator feedback . We evaluate our MSMT-GAN model on the Caltech-UCSD Birds 200 ( CUB ) dataset ( Wah et al. , 2011 ) and the Microsoft Common Objects in Context ( COCO ) dataset ( Lin et al. , 2014 ) . Experiment results demonstrate that MSMT-GAN is competitive with current methods on the COCO dataset and significantly outperforms the previous state-of-the art on the CUB dataset , decreasing the lowest reported Fréchet Inception Distance ( FID ) ( Heusel et al. , 2017 ) by 21.58 % for CUB . 2 RELATED WORK . Text-to-Image Generators : Reed et al . ( 2016 ) first demonstrated that a translation model from natural language to image pixels could be learnt by conditioning both generator and discriminator networks of a GAN on input text-descriptions . There has since been a surge of interest in training multi-stage attention based GAN architectures for this task . While the conventional setting ( Zhang et al. , 2017 ; Xu et al. , 2018 ; Li et al. , 2019a ; Zhu et al. , 2019 ) assumes only the availability of ( text , image ) pairs at training time , recently a second setting has emerged that assumes availability of bounding-box/shape-mask information of objects attributes during training ( Li et al. , 2019b ; Hinz et al. , 2019 ; Cho et al. , 2020 ; Liang et al. , 2020 ) . We highlight that this represents a significantly easier problem setting and that such methods are not feasible where bounding-box/shape information is unavailable ( such as the CUB dataset ) . Our method does not assume the availability of bounding-box/shape information , and we make comparisons against prior work of the same setting . Memory Networks : Memory Networks ( Weston et al. , 2014 ) combine inference components with a long-term memory module that can be dynamically written to and read from . Current methods ( Miller et al. , 2016 ) query “ key encodings '' of memory slots to retrieve a set of weights . These weights are used to combine separate “ value encodings '' of the slots into a single response . A Dynamic Memory Generative Adversarial Network ( DM-GAN ) ( Zhu et al. , 2019 ) that retrieves information for image refinement from a memory module was recently proposed for text-to-image synthesis . In our SDM module , we too employ the memory-writing , key-addressing , value-reading paradigm introduced by ( Miller et al. , 2016 ) , but our method differs from ( Zhu et al. , 2019 ) in all three memory operations ( Section 3.2 ) . Fundamentally , DM-GAN does not create region-contextualized representations of text . Multi-Headed Attention : Transformers ( Vaswani et al. , 2017 ) utilize a key-value mechanism similar to memory networks and introduced the idea of multi-headed attention . They linearly project query , keys and values to h separate encodings , called “ attention heads '' , and each head is separately used to extract an output vector . These vectors are concatenated together and linearly projected to a single response . Inspired by the success of Transformers , we introduce the IMHM method for image refinement . However , our method differs in a few respects . We maintain separate SDM modules for each head and we obtain queries and fuse outputs in an iterative fashion . We also adopt a “ redundancy loss '' ( Section 3.4 ) to encourage each head to focus on separate image aspects . 3 MSMT-GAN . Our MSMT-GAN architecture ( Figure 1 ) comprises of three stages - a Multi-Tailed Word-level Initial Generation ( MTWIG ) stage , and two refinement stages . Each refinement stage is Multi-Headed , and each refinement head has a separate Spatial Dynamic Memory ( SDM ) module . Section 3.1 presents our MTWIG stage , Section 3.2 presents our SDM module for a single refinement head , and the details of our Iterative Multi-Headed Mechanism ( IMHM ) are presented in Section 3.3 . 3.1 MULTI-TAILED WORD-LEVEL INITIAL GENERATION ( MTWIG ) . We highlight that previous multi-stage methods ( Zhang et al. , 2017 ; 2018 ; Li et al. , 2019a ; Zhu et al. , 2019 ) all rely on the same type of initial generation stage and focus only on improving the refinement stages - making the conventional assumption that the performance of multi-stage generators is primarily determined by the refinement stages , and that the quality of the `` rough initial image '' is of little importance . In our paper , we break from this tradition and demonstrate for the first time that gains can be achieved in the final stage of image refinement by making an improvement to the initial images . The conventional approach synthesizes initial images directly from a sentence-level vector without attempting to separate image attributes at a word-level . As a result , words that are entangled at the sentence-level representation generate initial image attributes that are inherently ambiguous in nature . In our novel Multi-Tailed Word-level Initial Generation ( MTWIG ) stage , we overcome this shortcoming by explicitly creating separate sets of image attributes for each word n-gram . First , we sample a vector of random noise z from a normal distribution and use a pretrained text-encoder to extract a sentence-level vector and word-level vectors : s and W from the input text . W = { w1 , w2 , ... , wL } ; wl ∈ RNw ; s ∈ RNs ; zn ∼ N ( 0 , 1 ) ; z ∈ RNz ( 1 ) Where L is the number of words in the text-description , and Nz , Ns and Nw are the dimensions of the noise vector , sentence vector and word vectors respectively . To mitigate over-fitting , the Conditioning Augmentation technique ( Zhang et al. , 2017 ) is used to resample the sentence-vector from an independent Gaussian distribution . This resampled sentence vector s′ and the noise vector z are then concatenated with each word-level vector wl from the input text sequence , and the sequence of concatenated vectors are passed through a 1D convolutional operation V of stride 1 ( see Figure 1 ) . F = V ( { concat ( s′ , z , wl ) | ∀ wl ∈W } ) ( 2 ) The length T of the output sequence F depends on the kernel size used by V and the vectors of the output sequence ft ∈ F are each separately passed through a series of upsampling blocks to generate corresponding sets of image features St . These sets of image features or `` tails '' each correspond to a different word n-gram from the input text sequence . If we use a kernel size of 1 for V , then each tail St corresponds to a single word . If we use a kernel size of 2 , then each tail St corresponds to a word bi-gram , and so on . We combine our sequence of tails { St } together in an iterative fashion using the adaptive gating fusion mechanism introduced by Zhu et al . ( 2019 ) ( discussed in Section A.1 ) . S1 : t = fuse ( S1 : t−1 , St , P MTWIG , ρMTWIG ) ; R1 = S1 : T ( 3 ) Where PMTWIG and ρMTWIG denote parameter matrix and bias terms , S1 : t denotes a combination of the first t tails , and S1:1 denotes the first tail S1 . The combination of all T tails gives us the final image features R1 for our initial stage . Notice that by concatenating each word vector wl with s′ and z before the 1D convolution , each tail is created with some common information , so they may learn to fuse together coherently . Each upsampling block consists of a nearest neighbor upsampling layer and a 3×3 convolution operation . An initial image is predicted from R1 using a 3×3 convolution .

Summary: This paper is motivated by that most existing text-to-image methods suffer from three limitations and solutions are proposed to address the different limitations.  Firstly, it introduces multi-tailed word-level initial generation to enhance global sentence representation with distinct n-gram word presentation.   Second, the spatial dynamic memory module is proposed to create a separate region-contextualized text representation for each image region.   Finally, it introduces an iterative multi-headed mechanism to make multiple distinct modifications to the prior image features.