Variational Hetero-Encoder Randomized GANs for Joint Image-Text Modeling
Authors: Hao Zhang
Bo Chen
Long Tian
Zhengjue Wang
Mingyuan Zhou
Venue:  (2020)
Keywords: Deep topic model
image generation
text generation
raster-scan-GAN
zero-shot learning

Abstract: For bidirectional joint image-text modeling, we develop variational hetero-encoder (VHE) randomized generative adversarial network (GAN), a versatile deep generative model that integrates a probabilistic text decoder, probabilistic image encoder, and GAN into a coherent end-to-end multi-modality learning framework. VHE randomized GAN (VHE-GAN) encodes an image to decode its associated text, and feeds the variational posterior as the source of randomness into the GAN image generator. We plug three off-the-shelf modules, including a deep topic model, a ladder-structured image encoder, and StackGAN++, into VHE-GAN, which already achieves competitive performance. This further motivates the development of VHE-raster-scan-GAN that generates photo-realistic images in not only a multi-scale low-to-high-resolution manner, but also a hierarchical-semantic coarse-to-fine fashion. By capturing and relating hierarchical semantic and visual concepts with end-to-end training, VHE-raster-scan-GAN achieves state-of-the-art performance in a wide variety of image-text multi-modality learning and generation tasks. 

Published as a conference paper at ICLR 2020
VARIATIONAL HETERO-ENCODER RANDOMIZED GANS FOR JOINT MMAGE-TEX" MODELING
Hao Zhang, Bo Chen, Long Tian, Zhengjue Wang National Laboratory of Radar Signal Processing
Xidian University, Xian, China
zhanghac xidian& 63 com
bchen xxl.xxiiii sidian..ee. edu. cn zhengjuewangel 63 com
cianlong.xiddant 63, com
Mingyuan Zhou
McCombs School of Business
The University of Texas Austin, Austin, TX 78712, USA m mingyuan. thou@mccombo ute.as. edu
ABSTRACT
For bidirectional joint image-text modeling, we develop variational hetero-encoder (VHE) randomized generative adversarial network (GAN), a versatile deep genera- decoder, probabilistic image encoder,
tive model that integrates a probabilistic and GAN into coherent end-to-end multi-modality learning framework. VHE randomized GAN (VHE- GAIN encodes an image to decode its associated text, and feeds the variational posterior the source of randomness into the GAN image generator. We plug three off-the-shelf modules, including deep topic model, ladder-structured image encoder, and StackGAN++, into VHE-GAN which already achieves competitive performance. This further motivates the development of VHE-raatee-sscan-CAAA NAAN that generates photo-realistic images in not only multi- coarse-to-fine fashion. By capturing and relating hierarchical semantic and visual concepts with end-to-end training. VHE-raster-scan-Gar achieves state-of-the-art performance in a wide variety of image-text multi-modality learning and generation tasks.
scale tow-to-high-resolutii manner, but also a hierarchical-semen
INTRODUCTION
Images and texts commonly occur together the real world. There exists a wide variety of deep neural network based unidirectional methods that model images (texts) given texts (images) (Gomez
cet al. 2017 Kiros& & Szepesvari 2012. Reed et al. 20066] Xu et al. 2018 Zhang et al. 2017a). There also exist probabilistic graphic model based bidirectional methods (Srivastava & Salakhutdinov
2012b a Wang et al., 2018) that capture the joint distribution of images and texts. These bidirectional methods, however, often make restrictive parametric assumptions that limit their image generation ability. Exploiting recent progress on deep probabilistic models and variational inference (Kingma & Welling 2014 Zhou et al. [2016] ZZang et al. 22018a] Goodfellow et al., 2014) 4; Zhang et al. 201 we propose end-to-end learning framework construct multi-modaln deep generative models that can not only generate vivid image-text pairs, but also achieve oftte-of-thh-are results on various
unidirectional tasks (Srivastava & Salakhutdinov 2012b a Wang et al. 2018 Gomez et al. 2017]
Xu et al. 2018 Zhang et al.) 2017a by Verma et al.) 2018 Zhang et al. 2018b). such as generating photo-realistic images given texts and performing rext-based zero-shot learning.
To extract and relate semantic and visual concepts, we first introduce variational hetero-encoder (VHE) that encodes image decode its textual description (e.g., tags, sentences, binary attributes, and long documents), where the probabilistic encoder and decoder are jointly optimized using variational inference (Blei et al.) 2017 Hoffmann al.) 2013 Kingma & Welling 20144, Rezende et
2014). The latent representation of VHE can be sampled from either the variational posterior provided
"Corresponding author

---

Published as a conference paper at ICLR 2020
by the image encoder given image input, or the posterior of the text decoder via MCMC given text input. VHE by construction has the ability to generate texts given images. To further enhance its text generation performance and allow synthesizing photo-realistic images given an image, text, or random noise, we feed the variational posterior of VHE in lieu of random noise as the source of randomness into the image generator of a generative adversarial network (GAN) (Goodfellow et al. 2014). We refer to this new modeling framework as VHE randomized GAN (VHE-GAN).
Off-the-shelf text decoders, image encoders, and GANs can be directly plugged into the VHE-GAN
framework for end-to-end multi-modality learning. To begin with, as shown in Figs. T and [b) we construct SHE-StackGAN++ by using the Poisson gamma belief network (PGBN) (Zhou et al. 2016)) as the VHE text decoder, using the Weibull
variational encoder (Zhang et al.
2018a) as the VHE image encoder, and feeding the concatenation of the multi-stochasttc-laye latent representation of the VHE as the source of randomness into the image generator of StackGAN+ (Zhang et al. 2017b). While WHE-StackGANN+ already achieves very attractive performance, we find that its performance can be clearly boosted by better exploiting the mulli-stochastic-layee semantically meaningful hierarchical latent structure of the PGBN text decoder. To this end, as shown in Figs. ((a) and [[c]
we develop VHE-raster-scae seaerr--a---ANNN GAN to perform image generation in not only a multi-scale www-o-hhihh-esoluue
manner in each layer, as done by StackGAN++, but also a Seerarchical-semantt coarse-to-fine fashion across layers, unique feature distinguishing it from existing methods. Consequently, not only can VHE-raster-scaan-AAA generate vivid high-resolution images with better details, but also build interpretable hierarchical semantic-visua relationships between the generated images and texts.
Our main contributions include: 1) VHE-GAN that provides a plug-and-play framework to integrate off-the-shelf probabilistic decoders, variational encoders, and GANs for end-to-end bidirectional multi-modalitt learning; the shared latent space can be inferred either by image encoder q(2 2)) if given images, or by Gibbs sampling from the conditional posterior of text decoder p(t z), if given texts; 2) VHEE-aater--sen--
that captures and relates hierarchical semantic and visual concepts to achieve state-of-the-art results in various unidirectional and bidirectional image-text modeling tasks.
VARIATIONAL HETERO-ENCOME
RANDOMIZED GANS
VAEs and GANs are two distinct types of deep generative models. Consisting of generator (decoder) p(ær z). a prior p(z), and an inference network (encoder) g(z x)) that is used to approximate the posterior p(z æ)), VAEs (Kingma & Welling 2014 Rezende et al. 2014) are optimized by maximizing the evidence lower bound (ELBO) as
ELBO = (x C(x) := Ezng(=] ()) [inp( =)] - KL [[)][()))[[(]]]]
(I)
where Pdata (z) ΣN i=1 N 1z, represents the empirical data distribution. Distinct from VAEs that make parametric assumptions on data distribution and perform posterior inference, GANs in general use implicit data distribution and do not provide meaningful latent representations (Goodfellow et al.) 2014]; they learn both generator G and discriminator D by optimizing a mini-max objective as minG m (Expppaas))) [In D(=)] Eg~p(=) [In(1 D(G(=)))
(2)
where p(z) is random noise distribution that acts as the source of randomness for data generation.
2.1 VHE-GAN OBJECTIVE FUNCTION FOR END-TO-END MULTI-MODALITY LEARNING
Below we show how to construct VHE-GAN to jointly model images æ and their associated texts t, capturing and relating hierarchical semantic and visual concepts. First, we modify the usual VAE into VHE, optimizing a lower bound of the text merg--aaainallleeee likelihood (t- [In p(t)) as
ELBO whe = Epunntt. (L (t, (t, Lvhe x)) := Ezwa(= m) [In p(t -L [g(2 x)) [)]lp(z)]] (3)
where p(t| 2) is the text decoder, p(z) is the prior, p(t) = Ex>p(=) pt( =)]. and Lyne xt) < p[t| |
In E ax) into its latent representation 2, is used to approximate the posterior p(= tt p(t | )))(2)/
g(= ( |=)](=||| = In p(t). Second, the image encoder q(> x)), which encodes image x
Third, variational posterior q( x)) in lieu of random noise p(2) is fed as the source of randomness
into the GAN image generator. Combing these three steps, with the parameters of the image encoder
2

---

Published as a conference paper at ICLR 2020
0
X
g
0
gm
X
g
0
6
f(x)
0
(a)
(b)
(d)
Figure I: Illustration (a) VHE, (b) StackGAN++, (c) raster-scan-GAN, (d) vanilla-GAN, and (e) simple-
raster-scan-GAN VHE--aster-scann-GAN consists of (a) and (c). æid is down-sampled from æ with scaling factor d. VHE- Stt--,tck consisting of (a) and (b), VHE-vanilla-Gan
consisting of (a) and (d), and VHE-simple saimppe-raste--saal GANN consisting of (a) and (e), are all used for ablation studies.
(= æt), text decoder p(t z), and GAN generator denoted by E, Gvae. and Ggnn respectively, we express the objective function of VHE-GAN for joint image-text end-to-end learning as
min E,G.a.
max (t) [C(t,
c(t,z) : D(x) + KL [9(2 x)) x) [[())]] + Ez~a(= m) In(1 D(G (z))) lnp x)]- (4)
Note the objective function in (4) implies a data-triple-reuse training strategy, which uses the same data mini-batch in each stochastic gradient update iteration to jointly train the VHE, GAN discriminator,
and GAN generator; see related objective function, shown in (10) of Appendix A. that is resulted from naively combining the VHE and GAN training objectives. In VHE-GAN, the optimization of the encoder parameter E is related to not only the VHE's ELBO, but also the GAN mini-mat objective function, forcing the variational posterior q( 2) to serve as a bridge between VHE and GAN, allowing them to help each other. Although there are some models (Mescheder et al., 2017
Makhzani et al., 2015 Tolstikhin et al.) 2018: Dumoulin et al.. 2017 Donahue et al.] 2017] (0017|Che Che et al. 2017:SSrivatavv et al. 2017] Grover et al.) 2018| Larsen et al. 2016 Huang et al. (2018) combining VAES and GANs in various ways, they focus
two different modalities. In Appendix A, we analyze the properties of the VHE-GAN objective function and discuss related works. Below we develop two different VHE-GANs, one integrates off-the-shelf modules, while the other introduces new interpretable hierarchical latent structure.
single-modality tasks while the VHE-GAN
2.2 VHE-STACKGANH+ WITH OFF-THR-SHELF MODULES
As shown in Figs. I(a) and I(b) we first construct VHE- VHE-StackGAN++ by plugging into VHE-GAN three off-the-shelf modules, including a deep topic model (Zhou et al. 2016), ladder-structured encoder (Zhang et al. 2018a and StackGAN++ (Zhang et al. 2017b). For text analysis, both sequence models and topic models are widely used. Sequence models (Bengio et al., 2003) often represent each document as a sequence of word embedding vectors, capturing local dependency structures with some type of recurrent neural networks (RNNs), such as long short-term memory (LSTM) (Hochreiter & Schmidhuber 1997p. Topic models such as latent Dirichlet allocation (LDA) (Blei et al., (2003) often represent each document cooccurrence patterns into latent topics. Suitable for capturing local dependency structure, existing sequence models often have difficulty in capturing long-range word dependencies and hence macro- level information, such as global word cooccurrence patterns (i.e., topics), especially for long documents. By contrast, while topic models ignore word order, they are very effective in capturing latent topics, which are often directly related to macro-level visual information (Gomez et al. 2017 Dieng al., 2017, Lau al. 2017). Moreover, topic models can be applied not only sequential texts, such as few sentences Wang et al. [009 Jin et al.) 2015) and long documents (Zhou et al. 2016), but also non-sequential ones, such as textual tags (Srivastava Salakhutdinov 2012a| 2014 Wang et al. 2018) and binary attributes (Elhoseiny et al. [2017b] Zhu et al. (0018). For this reason, for the VHE text decoder, we choose PGBN (Zhou et al. (0166), sfate-of-the-an topic model that can also be represented
bag of words (BoW), capturing global word
deep generalization of LDA (Cong et al.) 2017). We -dwwnward variational encoder (Zhang
by choosing the Weibull et al. aa) the VHE image encoder, and feeding the concatenation of all the hidden layers of PGBN as the source of randomness to the image generator of StackGAN+ (Zhang et al. (0176).
complete
As in Fig. D we VHE that encodes an image into deterministic-un sww.hadststthhht---osoooooe ladder-structured latent representation, which is used to decode the corresponding text. Specifically, represent each document as a BoW high-dimensional sparse count vector t, Z,Ko, where
3

---

Published as a conference paper at ICLR 2020
Z= (0,0,1.... 1........ and Ko is the vocabulary size. For the VHE text decoder, we choose use PGBN to extract hierarchical latent representation from t. PGBN consists of multiple gamma distributed stochastic hidden layers, generalizing the "shallow" Poisson factor analysis (Zhou et al.] 2012 Zhou & Carin 2015) into a deep setting. PGBN with L hidden layers, from top to bottom, is expressed as
Gam
(r, (r,1/s/(t))
(aamm1/// 1/80):
n
0 Gam
s 11++00 1/s5/11)
1,... .2, 2,,,, t. Pois (8"00..)))
(5)
where the hidden units t E Ki of layer are factorized under the gamma likelihood into the product of topics (()) E B - and hidden units of the next layer, R4, {x, 2 0). 8n 0, and K2 is the number of topics of layer I. If the texts are represented binary attribute vectors b,. we can add Bernoulli- Poisssn link layer as b,. = (t,, 2 1) (Zoou 2015 Zhou et al. 2016). We place a Dirichlet prior each column of 4b()) The topics can be organized into directed acyclic ( the
graph (DAG), whose node k at layer can be visualized with the top words of [ + (*) Φ%
topics tend to be very general in the top layer and become increasingly more specific when moving downwards. This semantically meaningful latent hierarchy provides unique opportunities to build better image generator by coupling the semantic hierarchical structures with visual ones.
Let us denote +- = (T(1) ( T (L) r) as the set of global parameters of PGBN shown in (5) Given T we adopt the inference in Zhang et al. (2018a) to build an Weibull apward-downwend variational image encoder as IT,,
((()) Z
o!(+)), where o (L++)) := r, 0(++1) == 0, and
xx, 4 (1+1), (((+1)) (ei + 4 (1+1) (1+1)
(6)
The Weibull distribution used to approximate the gamma distributed conditional posterior, and its parameters (1) A!! E RK are deterministically transformed from the convolutional neural network (CNN) image features f(x,) (Szegedy et al., 20166), as shown in Fig. [[a)and and described in Appendix D.1. We denote Ω as the set of encoder parameters. We refer to Zhang et al. (0188)) for more details about this deterministic-upwee www.wrd-ttchaatttieeee
aarder-structured inference network, which is distinct from a usual inference network that has a pure bottom-up structure and only interacts with the generative model via the ELBO (Kingma & Welling 2014 Gulrajani et al. 2017).
The multi-stochastic-lyye latent representation = = (0 |/2 1 is the bridge between two modalities. As shown in Fig. [(b]] AtE-SttckGAAN+++ simply randomizes the image generator of StackGAN++
0 We provide the overall objective function ın (15) of Appendix D.2. Note that existing neural newwork- models (Gomez al.) 2017 Xu al.) 2018 Zhang et al., 2017a b Verma et al.] 2018 Zhang et al.) (018b) are often able to perform unidirectional but not bidirectional transforms between images and texts t. However, bidirectional transforms are straightforward for the proposed model, as can be either drawn from the image encoder q(z x)) in (6). or drawn with an -oward-dowwward Gibbs sampler (Zhou et al. 2016) from the conditional posteriors p(> (t of the PGBN text decoder p(t 2) (5).
(Zhang et al. 2017b) with the concatenated vector 0 (0
2.3
VHE-RASTER-SCAN- IMAGE GENERATOR
WITH A NIERARCIICCL-SSMAATTI MULTI-RESOLUTIN
While we find that VHE-StackGAN++ has already achieved impressive results, its simple concate- nation of g does not fully exploit the gemantically-manninge of the PGBN-based text decoder. For three DAG subnets inferred from three different datasets, as shown in Figs. 21 23 of Appendix C.7, the higher-layer PGBN topics match general visual concepts, such as those on shapes, colors, and backgrounds, while the lower-layer ones provide finer details. This motivates us to develop an image generator to exploit the semantic structure, which matches coarse-to- to-fine visual concepts, to gradually refine its generation. To this end, as shown in Fig. [(c] we develop "raster-scan GAN that performs generation not only in multi-scale low-to- www-o-hiigh-ree high-resoluuion manner in each layer, but also hierarchical-ssmannie coarse-to-fine fashion across layers.
hierarchical latent representation
Suppose we are building a three-layer raster-scan GAN to generate an image of size 256 We randomly select an image I, and then sample {o()) 13 Ji=i from the variational posterior IT's ((((() Fn, (1+11) , g((+1)). First, the top-layer latent variable g often capturing general
4

---

Published as a conference paper at ICLR 2020
semantic information, is transformed to hidden features h: (3)
for the jth branch:
h = F (B(3) 1 = F((3) (h (3) i =
(7)
where P.") is CNN. Second, having obtained {
( J= generators {G Ji=1 synthesize low- (3) and 83 are of 16?,
to-high-resolution image samples {s. c (h. ) where $1 82
322, and 642, respectively. Third, x is down-sampled to 83 of size 32 and combined with the information from ((2) to provide the hidden features at layer two:
h (2) C(F(2) (0(2), { n
F! (M 0 i 2,3,
(2)
(8)
where C' denotes concatenation along the channel. Fourth, the generators synthesize image samples {s? (2) G. (h, (2) = where 81 $22 and (12) are of 322, 642, and 128 respectively. The same
process is then replicated at layer one to generate (s!") = (1 ) > where 8/ $2 and s are of size 64?, 128? and 256?, respectively, and $3
(1) becomes a desired high-resolution synthesized described in Fig. 2f Appendix D.3. PyTorch code is provided to aid the understanding and help reproduce the results.
image with fine details. The detailed structure of raster-scan-GAN
Different from many existing methods (Gomez et al. 2017 Reed et al. 2016 Xu al. 2018 Zhang et al. 201 whose textual feature extraction is separated from the end task, VHE-raster-scam GAN -AAN performs joint optimization. As detailedly described in the Algorithm in Appendix E, at each mini- batch based iteration, after updating 4 by the topie-layer-adaptie stochastic gradient Riemannian (TLASGR) MCMC of Cong et al. (2017). a Weibull distribution based reparameterization gradient (Zhang al. 20188) is used to end-to-end optimize the following objective:
min (o Ja. a max, (D(").,, E t.
Th ((0 ma. mn (143) _ 004
log pt., $ (1) (00
+ 1= (L
Dn. $ (1+1) ((+1) p( (1)
(1+1) 0
n
n
3 1= Ex [log D. (x n "h)) log(1 (G
10 (0! 00))
where (0/ 3,3 (=
denote different resolutions of In corresponding ( 1=1,/=1
(!) 13,3
2.4 RELATED WORK ON JOINT MMAE-TTXTT LEARNING
Gomez et al. (2017) develop CNN to learn transformation from images textual features pre- extracted by LDA. GANs have been exploited to generate images given pre-learned textual features extracted by RNNs (Denton al., 2015 Reed et al.) 2016 Zhang al. 2017a: Xu et al. 2018 Zhang et al. 20186 Li et al. 2019). All these works need a pre-trained linguistic model based on large-scale extra text data and the transformations between images and texts are only unidirectional. The recently proposed Obj-GAN (Li et al. 2019) needs even more side information such as the locations and labels of objects inside images, which could be difficult and costly to acquire in practice. On the other hand, probabilistic graphical model based methods (Srivastava & Salakhutdinov 20126 a Wang al., 2018) are proposed to learn joint latent space for images and texts to realize bidirectional transformations, but their image generators are often limited to generating low-level image features. By contrast, VHE-raster-scan scan-GAN GAN performs bidirectional end-to-end learning to capture and relate hierarchical visual and semantic concepts across multiple stochastic layers, capable of wide variety of joint image-text learning and generation tasks, as described below.
EXPERIMENTAL RESULTS
For joint image-text learning, following previous work, we evaluate the proposed VHE-StackGANN+ and VHE-raster-scan-GGr GAN on three datasets: CUB (Wah et al. (011). Flower (Nilsback & Zisserman, 2008), and COCO (Lin et al., 2014), as described in Appendix F. Besides the usual text-to-image generation task, due to the distinct bidirectional inference capability of the proposed models, we can perform a rich set of additional tasks such as image-to-texx pair generations. Due to space constraint, present below some representative results, and defer additional ones to the Appendix. We provide the details of our experimental settings in Appendix F. Py Porchh code is provided at https//ggihbb..omo/occchennnnnnn
image-to-image and noise-to toass-tt--magettrn
5

---

Published as a conference paper at ICLR 2020
Table 1: Inception score (IS, larger is hetter) and Frechet inception distance (FID, smaller is better) of
StackGAN++ (Zhang et al. 2017b), HDGAN Zhang et al. (018b), AttGAN (Xu et al. (018). Ohj-GAN (Li the values labeled with are calculated by the provided
ct al. 2019,, and the proposed VHE-raster-ccan-GAA
well-trained models and the others are quoted from the original publications: see Tab. i Appendix C.I for the
bars of IS. Note that while the FID of Obj-GAN the lowest, does not necessarily imply it produces high-quality images, as shown in Figs. 13 and 27 this is because FID only measures similarity on the image feature space, but ignores the shapes of objects and diversity of generated images. More discussions can be found in Section 3.1 and Appendix G.
Method
StackGAN++
HDGAN
AttiAAN
Obj-GAN
VHE-raster-scan-CC scan-G--
Criterion
IS
FID
IS
FID
IS
FID
IS
FID
IS
FID
Flower
3.26
48.68
3,45
40.12
3.72
35.13
CUB
3.84
15.30
4.15
13.48
4.36
13.02
4.41
12.02
COCO
8.30
81.59
11.86
78.16'
25.89
77.01
26.58
36.98
27.16
75.88
Table 2: Ablation study for image-to-text learning, where the structures of different variations of raster-scan- GAN are illustrated in Figs. [[b)
[(d] and [(e]
Method
PGBN-StackGAN
VHE-vanila-GAAr
VHE-StacKGAN++
VHE-siipie-rattcr-ecnnncc
Criterion
IS
FID
1$
FID
IS
FID
IS
FID
Flower
3.29
41.04
3.01
$2.15
3.56
38.66
3.62
36.18
CUB
3.92
13.79
3.52
21.24
4.20
12.93
4.31
12.35
COCO
10.63
79.65
6.36
97.15
12.63
78.02
20.13
77.18
with
Standette
N/A
StaciA
mater
Figure 2: Comparison on image generation given texts from CUB, Flower, and COCO. Shown in the top row are the textual descriptions and their associated real images; see Appendix C.2 for hegheo-resolution images. Note AttaGAN did not perform experiments Flower and hence its results Flower are shown, and since Obj-GAN only performed experiment on COCO, we defer visual results Appendix C.3.
3.1 TEXT-TO-IMAGE LEARNING
Although the proposed VHE-GANs do not have a text encoder to directly project a document to the shared latent space, given document and set of topics inferred during training, we use the upward- downward Gibbs sampler of Zhou al. (2016) draw (0() ) J1=1 from its conditional posterior under
PGBN, which are then fed into the GAN image generator to synthesize random images.
Text-to-image generation: In Tab. with inception score (IS) (Salimans al. 2016) and Frechet inception distance (FID) (Heusel et al. (017), we compare our models with three state-of-the-ar GANs in text-to-image generation. For visualization, we show in the top row of Fig. 2 different test textual descriptions and the real images associated with them, and in the other rows random images generated conditioning on these textual descriptions by different algorithms. Higher-resolution images are shown in Appendix C.2. We also provide example results on COCO, much more challenging dataset, in Fig. 13 Appendix C.3.
It is clear from Fig. 2 although both StackGAN+ (Zhang et al. 2017b) and HDGAN ZZang ct al., 2018b) generate photo-realistic images nicely matched to the given texts, they often misrepresent or ignore some key textual information, such as "black crown" for the 2nd test text, "yellow pistil" for 5th, "yellow stamen" for 6th, and "computer" for 7th. These observations also apply to AttiGAN (Xu al. 2018). By contrast, both the proposed THE-StackGAN++ and VHE-raster-scan-GAr do a better job in capturing and faithfully representing these key textual information into their generated images. Fig. 1.3 for COCO further shows the advantages of VHE-raster- Scan-Gr- in better

---

Published as a conference paper at ICLR 2020
Class Rhinoceros Auklet
body
calorful body
It seatend, neating in seabirl colortes, Large orangetbrwn kill. Plumage is dark on top and paler below, in offshore and inshore Someimes
grey brown dark lung Lange
red pink
red beed
biue head green back
large petal
black wings
stamen light
long
soli
the and sometimes stand on the strong
block belly
yellow belly
red heown. gnyy dark black
red pink
stamen petal long pellow
orange black
red yellow petzi buown thin
known dark yellow head
yellow blarle dark load body blue
yellow long red petal greea.
yellow
yellow long
black body long colored
green stamen
petal dack
(a)
(b)
(c)
Figure 3: Example results of VHE-raster-scan-GANN on three different tasks: (a) image generation given five textual attributes; (b) image generation given long class specifci document (showing three representative sentences for brevity) from CUB; and (c) latent space interpolation for joint image-text generation on CUB (left column) and Flower (right column), where the texts in the first and last row are given.
representing the given textual information in its generated images. Note Obj-GAN, which learns bounding box generator that restricts object locations, obtains the lowest FID on COCO. However, it appears that this type of restriction significantly improves FID at the expense of sacrificing the diversity of generated images given text, as shown in Fig. 27 of Appendix G. From the results in Fig. 1 it also appears that Obj-GAN overly emphasizes correctly arranging the spatial locations of different visual features, which is important to achieve low FID, but does not do well in generating correct object shapes, which is important to visual effect. Besides, the training of Obj-GAN requires more side information including the locations and labels of objects in the images, which are often not provided in practice (e.g., neither CUB nor Flower comes with this type of side information). While the proposed VHE-GAN models do not need these additional side information, they could be further improved by following Obj-GAN to take them into consideration.
As discussed in Section 2.2 compared with sequence models, topic models can be applied to more diverse textual descriptions, including textual attributes and long documents. For illustration, we show in Figs. 3(a) and 3(b) example images generated conditioning on set of textual attributes and an encyclopedia document, respectively. These synthesized images are photo-realistic and their visual contents well match the semantics of the given texts. Trained on CelebA (Liu et al., 2015). we provide in Fig. 9 examples of facial image generation given attributes; see Appendix B for details.
Ablation studies: We also consider several ablation studies for text-to-image generation, as shown in Tab. 2 First, we modify StackGAN++ (Zhang et al., 2017b), using the text features extracted by PGBN to replace the original ones by RNN, referred to as PGBN+StuckGAN++ It is clear that PGBN+StackGANt+ outperforms the original StackGAN++, but underperforms
St++,ttckKAA++ which can be explained by that 1) the PGBN deep topic model is more effective in extracting macro-level textual information, such as key words, than RNNs; and 2) jointly end-to-end training the textual feature extractor and image encoder, discriminator, and generator helps better capture and relate the visual and semantical concepts. Second, note that VHE-StackGAN++ has the same structured image generator as both StackGAN++ and HDGAN do, but performs better than them. We attribute its performance gain to 1) its PGBN deep topic model helps better capture key semantic information from the textual descriptions; and 2) performs end-to-end joint image-text learning via the VHE-GAN framework, rather than separating the extraction of textual features from text-to- image generation. Third, VHE-vanilla--G.. underperforms WHFESttckGAAN++ suggesting that the stacking structure is helpful for generating high resolution images, previously verified in ZZhang GAN outperforms VHE-StackGAND
ctt al. (2017a).
but underperforms THEEraster-scan GANN, confirming the benefits of combining the stacking and raster-scan structures. More visual results for ablation studies can be found in Appendix C.2. Below we focus on illustrating the outstanding performance of AHEE-ratter-scan-
Latent space interpolation: In order to understand the jointly learned image and text manifolds, given texts t and f2, we draw 0, and f and use the interpolated variables between them to generate
7

---

Published as a conference paper at ICLR 2020
both images via the GAN's image generator and texts via the PGBN text decoder. As in Fig. the first row shows the true texts t and images generated with 01, the last row shows t2 and images generated with 02. and the second to fourth rows show the generated texts and images with the interpolations from 0 to 02. The strong correspondences hetween the generated images and texts, with smooth changes in colors, object positions, and backgrounds hetween adjacent rows, suggest that is both visually and semantically meaningful. Additional
3(c)
the latent space of VHE-raster-scan-CC more fine-gride gridded latent space interpolation results are shown in Figs. [5| of Appendix C.4.
Visualization of captured semantic and visual concepts: Zhou et al. (016) show that the semantic concepts extracted by PGBN and their hierarchical relationships can be represented as a DAG, only a subnet of which will be activated given specific text input. In each subplot of Fig. E we visualize example topic nodes of the DAG subnet activated by the given text input, and show the corresponding images generated at different hidden layers. There is good match at each layer between the visual contents of the generated images and semantics of the top activated topics, which are mainly about general shapes, colors, or backgrounds at the top layer, and become more and more fine-grained when moving downward. In Fig. 5 for the DAG learned on COCO, we show a representative subnet that is rooted at top-layer node about "rooms and objects at home," and provide both semantic and visual representations for each node. Being able to capture and relate hierarchical semantic and visual concepts helps explain the state-of-the-an performance of VHEE-aatte--san-GGAA
3.2 IMAGE-TO-TENT LEARNING
VHE-raster-scan-GAA GAN can perform a wide variety of extra tasks, such as image-to-text generation, text-based sero-shot learning (ZSL), and image retrieval given text query. In particular, given image Rn we draw L,, as é,, On p( $ 02), 0n xx, q52 ( T ..,,)) and use it for downstream tasks.
Image-to-text generation: Given an image, we may generate some key words, as shown in Fig. [[a] where the true and generated ones are displayed on the left and right of the input image, respectively. It is clear that WHE-raster-scan-GAl successfully captures the object colors, shapes, locations, and backgrounds to predict relevant key words.
Text-based ZSL: Text-based ZSL is specific task that learns a relationship between images and texts on the seen classes and transfer settings on CUB and Flower as existing text- bext-based22 ZSL methods summarized in Tab. 3 There are two default splits for CUB the hard (CUB-H) and easy one (CUB-E) and one split setting for Flower, as described in Appendix F. Note that except for our models that infer shared semantically meaningful latent space between two modalities, none of the other methods have generative models for both modalities, regardless of whether they learn classifier for ZSL. Tab. shows that VHE-raster-scant Senter-scan--AAe GAN clearly outperforms the state of the art in terms of the Top-1 accuracy both the CUB-H and Flower, and is comparable to the second best on CUB-E (it is the best among all methods that have reported their Top-5 accuracies on CUB-E). Note for CUB-E, every unseen class has some corresponding seen classes under the same super-category. which makes the classification of surface or distance metric learned on the seen classes easier to generalize the unseen ones. We also note that both GAZSL and ZSLPP rely on visual part detection to extract image features, making their performance sensitive to the quality of the visual part detector that often has be elaborately tuned for different classes and hence limiting their generalization ability, for example, the visual part detector for birds is not suitable for flowers. Tab. also includes the results of ZSL using VHE, which show that given the same structured text decoder and image encoder, VHE consistently underperforms both THE-StackGAN++ and VHE-raster-scan-GAAN This suggests 1) the advantage of a joint generation of two modalities, and 2) the ability of GAN in helping VHE achieve better data representation. The results in Tab. 3 also show that the ZSL performance of VHE-raster-scan-GGa GAN has a clear trend of improvement as PGBN becomes deeper, suggesting the advantage of having also collect the ZSL results of the last 1000 mini-batch based stochastic gradient update iterations to calculate the error bars. For existing methods, since there are no error bars provided in published paper, we only provide the text error bars of the methods that have publicly accessible code.
the unseen ones (Fu et al. 2018). We follow the the same
distance metric in a latent space
aetochatiic-hiddeer layer deep topic model for text generation. We
3.3 IMAGE/TEXT RETRIEVAL
As discussed in Section 2.4 the proposed models are able to infer the shared latent space given either an image or text. We test both VHIE-StccKANN+ and -HEE-aster-scan--Ga GAN on the same image/text
8

---

Published as a conference paper at ICLR 2020
Topic
Tepk
Topi:
Topic
Topic
Topic
Topic
Topk:
Toplc
green
bird
gre
bird
village
view
Layer
flower
leaves
green
Layer
standing
darkt
body
Layer 3
countr
sky
villaage
color
group
coloral
body
large
white
attractive
cloudy
wide
zuftled
stateen yellow
red
bright
large
blue
village
Layer
lange
green
Layer
grey
body
standing body
Layer 2
sxy
low
say
wary
pink
centee
brnd
light
clinayy
many
beautiful
nuffiid
red
clustened
rod
gey
black
Nue
house
road
Layer
peral
petul
cark
Layer
bright
small
munded
Layer 1
sky
clustered
ground
beight
stamen
body
sunshüne
room
grey
Real
This bright colored red flowrr on the Real
This beight red colored lind with dark Real
The picture shows a view of villaage
Image
green Jeatres has pelals that suncund Image the ovary ruffled wary mamner. Text
rounded grey wing and brows beak are standing.
Image
having blue fise sky, low house, grey road green trees.
Text
Text
(a)
(b)
(c)
Figure 4: Visualization of example semantic and visual concepts captured by a three- stoochastic-hidden layer
VHE-raster-scan GANN from (a) Flower, (b) Bird, and (c) COCO. In each subplot, given the real text tu shown the bottom, we draw {0_) y via Gibbs sampling; we show the three most active topics in ( (ranked by the weights of ) layer I = 3, 2, 1, where each topic is visualized by its top three words; and we feed (0()) y
into raster- scan- scan-GaN GAN to generate three random images (one per layer, coarse to fine from layers 3 1).
Table 3: Accuracy (%) of ZSL. CUB and Flower. Note that some of them are attribute-based methods but applicable in our setting by replacing attribute vectors with text features (labeled by "), as discussed Elhoseiiy ct al. 2017b).
Text-7SI. dataset
CUB-H
CUB-E
Flower top-1
Accuracy criterion
top-1
top-1
top-5
MerC-Kennel Ellhoseiy et al. 2017a
7.7
0.28
33.5 0.22
64.3 1 0.20
:: 2.77
ZSLNS Qiao el al. 2016
7.3 0.36
29.1 0.28
61.8 0.22
8.7 1 2.46
FS7SL. (Romeraparedes & forr
2015
7.4 0.31
28.5 0.26
59.9 + 0.20
8.6 + 2.53
SynC (Changpinyo et al. 2016
8.6
28.0
61.3
8.2
7SLPP (Elhoseiny et al. 2017b
9.7
37.2
GAZSL (Zhu et al. 2018)
10.3 0.26
43.7 + 0.28
67.61 + 0.24
VHE--L3
14.0 +
34.6 + 0.25
64.6 0.20
8.9 + 1,57
VHEESSacKGAN++LL
16.1
38,5
68.2
10.6
VHEE-ratter-scan-iiANN
11.7 0.31
32.1 0.32
62.6 0.33
9.4 + 1.88
VHE-raster-scan GAN-L2
14.9
0.26
37.1 0.24
64.6 1 0.25
11.0 1.54
VHEFraster-scan-GAae
16.7 + 0.24
39.6 + 0.20
70.3 0.18
12.1 1.47
Table 4: Comparison of the image-to-text retrieval performance, measured hy Top-1 accuracy, and text-to-image
retrieval performance, measured by AP@50, between different methods on CUB-E,
Mothod
CNN-LSTM
AttnGAN
TA-GAN
VHE-StackGANn+
VHE-raster-scann
Li 2017)
Xu et al. 2018
Nam 2018
Topl-ACC(%)
35.1
60.2
61.7
AP
57.6
51.0
62.8
61.3
62.6
2018), where we use the cosine distance between the inferred latent space given images (q(O) x) image encoder) and these given texts (p(0 t). Gibbs sampling) to compute the similarity scores. Similar with TA-GAN, the top-1 image-to-text retrieval accuracy (Top-1 Acc) and the percentage of matching images in top-50 text-to-image retrieval results (AP@50) on CUB-E dataset are used to measure the performance. As shown in Table 4 VHE-raster- scan-GAN clearly outperforms AttnGAN (Xu et al. 2018) and is comparable with TA-GAN. Note TA-GAN needs to extract its text features based on the fastText model (Bojanowski et al.) 2017) pre-trained on large corpus, while VHE-raster-scan-GAat learns everything directly from the current dataset in an end-to-end manner. Also, VHEEraster-scan-GGA outperforms VHEEStackGAN++ which further confirms the benefits of combining both the stacking and raster scan structures.
retrieval tasks as in TA-GAN Nam et al.
3.4 GENERATION OF RANDOM TEXT-IMAGE PAIRS
Below we show how to generate data samples that contain both modalities. After training a three- hewaastic-hiddde-llyye VHEaraster-saan-GANN following the data generation process of the PGBN text decoder, given { (1)) 13
Ji=1 and r, we first generate 0 Gam (r. 1/s(4) and then downward propagate it through the PGBN as in (5) to calculate the Poisson rates for all words using (1) (()) Given random draw, (0()) )3_ is fed into the raster-scan-GAr image generator generate

---

Published as a conference paper at ICLR 2020
living table furniture stove kitchen television
bed room table window chairs tv
refrigerator kitchen stove cabinets table
room
tv screen flat room wall
table chairs kitchen dining wooden
refrigerator fridge door food kitchen
living pillow window
Figure 5: An example topic hierarchy learned on COCO and its visual representation. We sample ((1:3)
q( (1:3) 2 æn) for all n; for topic node & of layer I, we show both its top words and the top two images ranked by their activations (())
nk
p1ck
20lord
bulideg cit
toke kag skinsy
53000
rd cove trowe
Abroct
kang 100g
lagg
1e1
Iihht
sty
X8220
Icly
210
locg
vion
106g
l00g
txt
datk
pople
wote
30503
1743
lallayy
CEN
gran
oxterfl
dapod
noad
ocksc
pdloos
load
people
midde
light
daspd
(b)
Figure 6: Example results using VHF--uster-ccan-GGNN (a) image-to-tecxtual-ing generation, where generated tags highlighted red are included the original ones; (b) image-text-pair generations (columns from left to right are based on Flower. CUB, and COCO, respectively).
corresponding image. Shown in Fig. 6(b) six random draws, for each of which show its top seven words and generated image. whose relationships are clearly interpretable, suggesting that VHE-raster-scan-AAN
able recode the key information of both modalities and the relationships between them. In addition to the tasks shown above, THE-raster-scan-GA. can also be used to perform image retrieval given a text query, and image regeneration; see Appendices C.5 and C.6 for example results on these additional tasks.
CONCLUSION
We develop variational hetero-encoder randomized generative adversarial network (VHE-GAN) to provide a plug-and-play joint image-text modeling framework. VHE-GAN versatile deep generative model that integrates off- the-shelf image encoders, text decoders, and GAN image discrim- inators and generators into a coherent end-to-end learning objective. It couples its VHE and GAN components by feeding the VHE variational posterior lieu of noise the source of randomness of the GAN generator. We show VHE-StackGANN+ that combines the Poisson gamma belief network, a deep topic model, and StackGAN+ achieves competitive performance, and VHE-raster-scan-Gi which further improves VHE- THE-StackGAN++ by exploiting the semantically cal structure of the deep topic model, generates photo-realistic images not only in a multi-scale low-to-high-ree siihhrresouution manner, but also in hierarchical hierarchicalssemantie semantic coarse-to-fin foneee fashion, achieving outstanding results in many challenging smage-to-texx rext-to-image, and joint text-image learning and generation tasks.
hierarchi-
10

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGEMENTS
B. Chen acknowledges the support of the Program for Young Thousand Talent by Chinese Central Government, the 111 Project (No. B18039), NSFC (61771361), NSFC for Distinguished Young Scholars (61525105), Shaanxi Innovation Team Project, and the Innovation Fund of Xidian University, M. Zhou acknowledges the support of the U.S. National Science Foundation under Grant UIS-1812699)
REFERENCES
Zeynep Akata, Scott Reed, Daniel Walter, Honglak Lee, and Bernt Schiele. Evaluation of output embeddings for fine-grained image classification. In CVPR. pp. 2927-2936, 2015.
Yoshua Bengio, Rejean Ducharme, Pascal Vincent, and Christian Janvin. neural probabilistic language model. Journal of Machine Learning Research, 3(6):1 137-1155, 2003.
David M Blei, Andrew Y Ng. and Michael Jordan. Latent Dirichlet allocation. Journal of Machine Learning Research, 3:993-1022, 2003.
David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, 1 112(518):859- 877, 2017.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146, 2017.
Soravit Changpinyo, Wei Chao, Boqing Gong, and Fei Sha. Synthesized classifiers for zero-shot learning. CVPR, pp. 5327-5336, 2016.
Tong Che, Yanran Li, Athul Paul Jacob, Yoshua Bengio, and Wenjie Li. Mode regularized generative adversarial networks. In ICLR, 2017.
Yulai Cong, Bo Chen, Hongwei Liu, and Mingyuan Zhou. Deep latent Dirichlet allocation with topic-ayyer-adaptive stochastic gradient Riemannian MCMC. ICMI., 2017.
Emily I. Denton, Soumith Chintala, Arthur Szlam, and Rob Fergus. Deep generative image models using a Laplacian pyramid of adversarial networks. In NIPS, pp. 1486-1494, 2015.
Adji B Dieng, Chong Wang, Jianfeng Gao, and John Paisley. TopicRNN: A recurrent neural network with long-range semantic dependency. ICIR, 2017.
Jeff Donahue, Philipp Krahenbuhl, and Trevor Darrell. Adversarial feature learning. In ICLR, 2017.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Alex Lamb, Martin Arjovsky, Olivier Mastropietro. and Aaron C Courville. Adversarially learned inference. In ICLR, 2017.
Mohamed Elhoseiny, Ahmed M Elgammal, and Babak Saleh. Write a classifier: Predicting visual classifiers from unstructured text. IEEE Transactions on Patlern Analysis and Machine Intelligence, (9<12))253922553 2017a
Mohamed Elhoseiny, Yizhe Zhu, Han Zhang, and Ahmed M Elgammal. Link the head to the "beak": Zero shot learning from noisy text description part precision. In CVPR, pp. 6288-6297, 2017b.
Yanwei Fu, Tao Xiang, Yu Gang Jiang, Xiangyang Xue, Leonid Sigal, and Shaogang Gong. Recent advances in zero-shot recognition. IEEE Signal Processing Magazine, 35, 2018.
Lluis Gomez, Yash Patel, Marcal Rusinol, Dimosthenis Karatzas, and V Jawahar. Self-supervised learning of visual features through embedding images into text topic spaces. In CVPR, pp. 2017- 2026, 2017.
Ian J Goodfellow, Jean Pougetabadie, Mehdi Mirza, Bing Xu, David Wardefarley, Sherjil Ozair, Aaron C Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, pp. 2680, 2014.
11

---

Published as a conference paper at ICLR 2020
Aditya Grover, Manik Dhar, and Stefano Ermon. Flow-GAN: Combining maximum likelihood and adversarial learning in generative models. In AAAI, pp. 0066-0076,220 2018.
Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vazquez, and Aaron C Courville. PixeIVAE: A latent variable model for natural images. In ICLR, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge to local nash equilibrium. In NIPS, pp. 6626-6637, 2017.
Sepp Hochreiter and Jurgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):
1735-1780, 1997.
Matthew D Hoffman and Matthew Johnson. ELBO surgery: Yet another way to carve up the variational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, NIPS, 2016.
Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational inference. The Journal of Machine Learning Research, 14(1):1303 1348, 2013.
Huaibo Huang, Zhihang Li, Ran He, Zhenan Sun, and Tieniu Tan. IntroVAE: Introspective variational autoencoders for photographic image synthesis. In NeurIPS, 2018.
Junqi Jin, Kun Fu, Runpeng Cui, Fei Sha, and Changshui Zhang. Aligning where see and what to tell: Image caption with region-based attention and scene factorization. In CVPR, 2015.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint orXiv:1412.690 2014.
Diederik P Kingma and Max Welling. Stochastic gradient VB and the variational auto-encoder. In ICLR, 2014.
Ryan Kiros and Csaba Szepesvari. Deep representations and codes for image auto-annotation. pp. 908-916, 2012.
Anders Boesen Lindbo Larsen, Soren Kaae Sonderby, Hugo Larochelle, and Ole Winther. Autoen- coding beyond pixels using learned similarity metric. In ICML, pp. 1558-1566, 2016.
Jey Han Lau, Timothy Baldwin, and Trevor Cohn. Topically driven neural language model. In ACL, pp. 355-365, 2017.
Shuang Li, Tong Xiao, Hongsheng Li, Wei Yang, and Xiaogang Wang. Identity-aware textual- visual matching with latent co-attention. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1890-1899, 2017.
Wenbo Li, Pengchuan Zhang. Lei Zhang, Qiuyuan Huang, Xiaodong He, Siwei Lyu, and Jianfeng Gao. Object-driven text-to-image synthesis via adversarial training. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12174-12182, 2019.
Tsungyi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, pp. 740-755, 2014.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint arXiv:1511.0556 2015.
Lars Mescheder, S Nowozin, and Andreas Geiger. Adversarial variational Bayes: Unifying variational autoencoders and generative adversarial networks. In ICML, pp. 2391-2400. PMLR, 2017.
Seonghyeon Nam, Yunji Kim, and Seon Joo Kim. Text-adaptive generative adversarial networks: Manipulating images with natural language. In Advances in Neural Information Processing Systems, pp. 42-51, 2018.
12

---

Published as a conference paper at ICLR 2020
Maria-Elen Nilsback and Andrew Zisserman. Automated flower classification a large number of classes. In Computer Vision. Graphics & Image Processing, 2008. ICVGIP'OR '8. Sixth Indian Conference on, pp. 722-729. IEEE, 2008.
Ruizhi Qiao, Lingqiao Liu, Chunhua Shen, and Anton Van Den Hengel. Less is more: Zero-shot learning from online textual documents with noise suppression. In CVPR, pp. 1249-2257, 2016.
Scott E Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In ICML, pp. 1060-1069, 2016.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In ICML, pp. 1278-1286, 2014.
Bernardino Romeraparedes and Philip H Torr. An embarrassingly simple approach to zero-shot learning. In ICML, pp. 2152-2161, 2015.
Tim Salimans, Ian J Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In NIPS, pp. 2234-2242, 2016.
Akash Srivastava, Lazar Valkoz, Chris Russell, Michael U Gutmann, and Charles A Sutton. VEEGAN: Reducing mode collapse in GANs using implicit variational learning. NIPS, pp. 3308-3318 2017.
Nitish Srivastava and Ruslan Salakhutdinov. Multimodal learning with deep Boltzmann machines. In NIPS, pp.
2012a.
Nitish Srivastava and Ruslan Salakhutdinov. Learning representations for multimodal data with deep belief nets. In NIPS workshop, pp. 2222-2230, 2012b.
Nitish Srivastava and Ruslan Salakhutdinov. Multimodal learning with deep Boltzmann machines. Journal of Machine Learning Research, 15(1):2949 2014.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, pp. 2818-2826, 2016.
lyy Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-encoders. In ICLR, 2018.
Vinay Kumar Verma, Gundeep Arora, Ashish Kumar Mishra, and Piyush Rai. Generalized zero-shot learning via synthesized examples. In CVPR, pp. 4281-4289, 2018
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech- UCSD Dataset. reporrt 2011.
Chaojie Wang, Bo Chen, and Mingyuan Zhou. Multimodal Poisson gamma belief network. In AAAI, 2018.
Dingding Wang, Shenghuo Zhu, Tao Li, and Yihong Gong. Multi-document summarization using sentence-based topic models. In ACI., pp. 297-300, 2009.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. AttnGAN: Fine- ainengrained text to image generation with attentional generative adversarial networks. In CVPR, pp. 1316-1324, 2018.
Han Zhang, Tao Xu, and Hongsheng Li. StackGAN: Text to photo-realistic image synthesis with stacked generative adversarial networks. In CVPR, 2017a.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris Metaxas. StackGAN++: Realistic image synthesis with stacked generative adversarial networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, PP(99):1-1, 2017b.
Hao Zhang, Bo Chen, Dandan Guo, and Mingyuan Zhou. WHAI: Weibull hybrid autoencoding inference for deep topic modeling. In ICLR, 2018a.
13

---

Published as a conference paper at ICLR 2020
Zizhao Zhang, Yuanpu Xie, and Lin Yang. Photographic text-to-image synthesis with a hierarchically- nested adversarial network. In CVPR, 2018b.
Mingyuan Zhou. Infinite edge partition models for overlapping community detection and link prediction. In AISTATS, pp. 1135-1143, 2015.
Mingyuan Zhou and Lawrence Carin. Negative binomial process count and mixture modeling. IEEE Trans. Pattern Anal. Mach. Intell., 37(2):307-320, 2015.
Mingyuan Zhou, Lauren Hannah, David Dunson, and Lawrence Carin. Beta-negative binomial process and Poisson factor analysis. In AISTATS, pp. 1462-1471, 2012.
Mingyuan Zhou, Yulai Cong, and Bo Chen. Augmentable gamma belief networks. Journal of Machine Learning Research, 17(163):1-44 2016.
Yizhe Zhu, Mohamed Elhoseiny, Bingchen Liu, Xi Peng, and Ahmed M Elgammal. A generative adversarial approach for zero-shot leamnng from noisy texts. In CVPR, 2018.
14

---

Published as a conference paper at ICLR 2020
A MODEL PROPERTY OF WHE-GAN AND RELATED WORK
Let us denote g(2) ) Pep g(( x:) N Ex= ((z x.)) as the aggregated posterior (Hoff-
man & Johnson 2016 Makhzani al. 2015). Removing the triple-data-reuse training strategy, we can re-express the VHE-GAN objective in 44 as
min Gyn
max Ehe + Lagals Lagan Pempe In ((x) + Rg In(1 ( gnn(2))))
D
(10)
which corresponds to naive combination of the VHE and GAN training objectives, where the data samples used to train the VHE, GAN generator, and GAN discriminator in each gradient update
iteration are not imposed be the same. While the naive objective function in (TO) differs from the true one in (4) that is used to train VHE-GAN, simplifies the analysis of its theoretical property, as described below.
Let us denote t) =- q(> P) (2, i) as the joint distribution of (æ,, () and z under the mutual in- formation between @ (2) and z ((z), and JDS(pillp2) == (L + 22] to (LLppll(P + p2)/2] as the Jensen-Shannon divergence between distributions p1 and p2. Similar to the analysis in Hoffman & Johnson (016), the VHE's ELBO can be rewritten as ELBO = Byz,,..t) [log p(t z) - 1g 2) [p)))) where the mutual information term can also be expressed as (2, =) = R Ber-pana (w) KI |002=>))]])) [=)]] Thus maximizing the ELBO encourages the mutual information term 12(x, z) to be minimized, which means while the data reconstruction term E (x)x,,,.) [log p(t =)] needs to be maximized, part the VHE optimization objective penalizes a from carrying the information of the æ that it is encoded from. This mechanism helps provide necessary regularization to prevent overfitting. As in Goodfellow et al. (2014), with an optimal discriminator D for generator G, we have min LGAN G) In + 2ISD(paata (3) (m))l (æ)), where PG=(#t (a) denotes the distribution of the generated data G(z) that use z ~ ((z) as the random source fed into the GAN generator. The ISD term minimized when Pc, (x) Pdata (z).
the VHE variational posterior q(= 2x), (2, 2) By(x..0) In g(z))aota
((x,0))
With these analyses, given an optimal GAN discriminator, the naive VHE-GAN objective function in (0)) reduces
min G,G Gree
E, p(t 2)] + KL.(g(=))lp())) + ta 2) 21SD 2ISD(pdata (m)) (11)
From the VHES' point of view, examining shows that it alleviates the inherent conflict in VHE of maximizing the ELBO and maximizing the mutual information (x z). This is because while the VHE part of VHE-GAN still relies on minimizing (2 z) to regularize the learning, the GAN part tries to transform q(z) through the GAN generator match the true data distribution Pattaa (æ:). In other words, while its VHE part penalizes z from carrying the information about the æ that it encoded from, its GAN part encourages a to carry information about the true data distribution Pdota (x). but not necessarily the observed that it is encoded from.
From the GANs' point of view, examining II] shows that it provides GAN with a meaningful latent space, necessary for performing inference and data reconstruction (with the aid of the data data-triple-use training strategy). More specifically, this latent representation is also used by the VHE to maximize the data logelio,l training procedure that tries cover all modes of the empirical data distribution rather than dropping modes. For VHE-GAN { the source distribution is g(> x)), not only allowing GANs to participate in posterior inference and data reconstruction, but also helping GANS resist mode collapse. In the following, we discuss some related works on combining VAEs and GANs.
A.1 RELATED WORK ON COMBINING VAES AND GANs
Examples in improving VAEs with adversarial learning include Mescheder et al. (2017), which allows the VAEs to take implicit encoder distribution, and adversarial and Wasserstein auto-encoder (Tolstikhin et al. 2018)), which drop the mutual information term from the ELBO and use adversarial learning to match the aggregated posterior and prior. Examples in allowing GANs perform inference include Dumoulin et al. (2017) and Donahue et al. (2017).
(Makhzani et al. 2015)
15

---

Published as a conference paper at ICLR 2020
which use GANs to match the joint distribution g(2 x))paat (x) defined by the encoder and the one p(a| x) p(()) defined by the generator. However, they often do not provide good data reconstruction. Examples in using VAEs maximum likelihood to help GANs resist mode collapse include Che et al.
(2017); : Srivastava et al. (2017); Grover et al. (018). Another example is VAEGAN (Larsen et al.
2016) that combines unit- wise likelihood hidden layer and adversarial loss original space, but its
update of the encoder is separated from the GAN mini-max objective. On the contrary. IntroVAE (Huang et al. 2018) retains the pixel-wise likelihood with an adversarial regularization on the latent space. Sharing network between the VAE decoder and GAN generator in VAEGAN and Intro VAE, however, limit them to model a single modality.
B MORE DISCUSSION ON SEQUENCE MODELS AND TOPIC MODELS IN TEXT ANALYSIS.
In Section 3.1, we have discussed two models to represent the text: sequence models and topic models. Considering the versatility of topic models (Wang al. 2009 Jin et al. 2015 Zhou et al.
2016; Srivastava & Salakhutdinov] 2012a: 2014 Wang et al., 2018 Elhoseiny et al. 2017b Zhu al.
2018) in dealing with different types of textual information, and its effectiveness in capturing latent topics that are often directly related to macro-level visual information (ooezz et al., 2017
Dieng et al. 2017 Lau et al., 2017). we choose state-of of-the-art deep topic model, PGBN, model
the textual descriptions in VHE. Due to space constraint, we only provide simple illustrations in this section, more insights and discussions are provided.
Figs. (3a) and 3(b)
Red body
Blue breast
Orange belly Blue head Buff back
Long leg Brown crown
Dark wings
Colorful body
Black belly
White wings
White body
Blue head
Long bill
Striped back
Spotted back
Yellow bill
Green back
Black wings
White bill
Long tail
Striped tail
Round belly
Solid tail
Red head
Short leg
Black throat
Yellow belly
Grey back
Yellow belly
Figure 7: Generated random images by VHE-raster-scan- GGNN conditioning on five binary attributes.
As discussed before, topic models are able to model non-sequential texts such binary attributes. The CUB dataset provides 312 binary attributes (Wah et al. 2011) for each images, such as whether "crown color is blue" and whether "tail shape solid" to dehne the color or shape of different body parts of bird. We first transform these binary attributes for the nihh image to a 311-dimennional binary vector t.p, whose ith element is or depending on whether the bird this image owns the ith attribute not. The binary attribute vectors t, are used together with the corresponding bird images Vn to train VHE-raster-scan-GGr CANN As shown in Fig. 7 we generate images given five binary attributes, which are formed into 212-eimeniional binary vector t (with five non-zero elements at these five attributes) that becomes the input to the PGBN text decoder. Clearly, these generated images photo-realistic and faithfully represent the five provided attributes.
The proposed VHE-GAM can also well model long documents. In text-based ZSL discussed in Section 3.2, each class (not each image) is represented long encyclopedia document, whose global
16

---

Published as a conference paper at ICLR 2020
semantic structure is hard to captured by existing sequence models. Besides good ZSL performance achieved by VHE-raster- scan GAN, illustrating its advantages text generation given images, we show Fig. 8 example results of image generation conditioning long encyclopedia documents the Akata et al. 2015) and Flower (Elhoseiny et al. 2017a).
unseen classes of CUB-E QQao et al. 2016
Class name: Rhinoceros Auklet
Class name: Yellow Bellied Flycatcher
It is seabird, nesting in seabird colonies, with large wrange/brown bill. Plumage is dark on top and paler below, in offshore and inshore water. Sometimes swim the water and sometimes it stand on the strong.
Brownish-olive upperparts, darker on the wings and tail, yellowish underparts.
Have small bill short tail, on perch low or in the middle of a tree.
Its eyes are dark round with radiating vigor, like looking for food or insects.
(a)
Class name: Ball Moss
Class name: Barberton Daisy
It tends form a spheroid shape ranging in size from golf ball soccer ball. It may hinder tree growth.
bear large capitulum with striking, two-lipped ray floret yellow orange. Colors include white, yellow, and pink. Its petals medium, and each of them round the number about six.
Its petals stripe-like yellow ones and stamen also round dark brown or yellow.
(b)
Figure 8: Image generation conditioning on long encyclopedia documents using THE-raster-see GAN trained on (a) CUB-E and (b) Flower. Shown in the top part of each subplot are representative sentences taken from the long document that describes an unseen class; for the three rows of images shown in the bottom part, the first row includes three real images from the corresponding unseen class, and the other two rows include a total six randomly generated images conditioning on the long encyclopedia document of the corresponding unseen class.
Analogous to how the Bird images are generated in Fig, / we also perform facial image generation given set of textual attributes. On CelebA dataset, given attributes, train VHE-stackGANN+ and
17

---

Published as a conference paper at ICLR 2020
eyeglasses receding hairline mouth slightly open male smiling
eyeglasses straight hair young rosy cheeks beard
mustache black hair male high cheekbones sideburss
smiling blond hair wavy hair attractive high cheekbones
bald male narrow eyes no beard old
VHE-StackGAN++
VHE-raster-scan-GAN
Figure 9: Example results of facial image generation conditioning on five textual attributes, by VHEStackGAN++ and VHE-raster-scan-GANN trained on the CelebA dataset. Both models trained with 20 epochs, with the output resolution set as 128 x 128. Note our current network architecture, designed mainly for natural images, has not yet been fine-tuned for facial images.
VHE-raster-scan-GAAY GAN to generate the facial images with resolution 128 x 128. As shown in Fig.
after the training of 20 epochs, we generate facial images given five attributes. While the facial images generated by both models nicely match the given attributes, -HE-raster-scan GAN provides higher visual quality and does better job in representing the details.
18

---

Published as a conference paper at ICLR 2020
C MORE EXPERIMENTAL RESULTS ON JOINT IMAGE-TEXT LEARNING
C.I TABLES AND 2
WITH ERROR BARS.
For text-to-image generation tasks, use the official pre-defined training/testing split (illustrated in Appendix F) to train and test all the models. Following the definition of error bar of IS in StackGAN++ (Zhang et al. 2017b), HDGAN (Zhang al., 2018b). and AttnGAN (Xu et al. 2018), we provide the IS results with error bars for various methods in Table where the results of the StackGAN++ HDGAN, and AttnGAN are quoted from the published papers. The FID error bar is not included has not been clearly defined.
Table 5: Inception score (IS) results in Table I with error bars.
Method
StackGAN++
HDGAN
AttnGAN
Obj-GAN
VHE-raster scan- GANN
Flower
3.26 + .01
3.45 .07
3.72 .01
CUB
3.84 + .06
4.15 + .05
4.36 + .03
4.41 = .03
COCO
8.30 + .10
11,86 + .18
25.89 + .477
26.68 + .52
27.16 + .23
Table Inception score (IS) results in Table 2 with error bars,
Method
PGBN+StccGAN+++
VHE-StackGANrr VHE-vanila-GAAN
VHE-siimll-raatee-seenn
Flower
3.29 + .22
3.01 + 06
3,56 + .03
3.62 + .02
CUB
3.92 + .06
3.52 + .08
4.20 + .04
4.31 + .06
COCO
10.63 .10
6.36 + .20
12.63 = .15
20.13 22
19

---

Published as a conference paper at ICLR 2020
C.2 HIGH-QUALITY IMAGES OF FIGURE 2
Due to space constraint, we provide relative small-size images in Fig. corresponding images with larger sizes.
Below we show the
Brown duck playing the lake making poodle.
This hird yollow with grey i lngs and a black crown.
An all black bird with thick, round
black bill.
StackGAN++
HDGAN
AttnGAN
VHE-StackGAN++
SCan--GN
VHE-vanilla-GAN GAN
Figure 10: The images above the blue line are the larger-size replots of CUB Bird images in Figure 2 while the images below the blue line are results for ablation study.
20

---

Published as a conference paper at ICLR 2020
This fower
This long, curling orange petals spots.
perple bell shaped flower, with yellow pistil and long stigma.
contains bundd ed needle like yelllow polals around the brighter yellow stamee..
StackGAN++
HDGAN
VHB-StackGAN++
VHE scan
VITE vanilla GAN
VIB-siaplo-rasseer-caa- GAN
Figure 11: The images above the blue line are the larger-size replots Flower images in Figure while the images below the blue line are results for ablation study.
21

---

Published as a conference paper at ICLR 2020
A wooden desk topped with laptop computer.
A very dark city street with cars and buildings.
StackGAN++
HDGAN
AttnGAN
raE-raster- GAN
VHE-vanilla-GAN
VIE-simmle-rrsteer-ccnn----
Figure 12: The images above the blue line are the larger-size replots of COCO images in Figure 2 while the images below the blue line are results for ablation study.
22

---

Published as a conference paper at ICLR 2020
C.3 MORE TEXT-TO-IMAGE GENERATION RESULTS ON COCO
COCO is a more challenging dataset than CUB and Flower, as contains very diverse objects and scenes. We show in Fig. 13 more samples conditioned different textural descriptions.
Stack
FDGAN
AttnnAN
Obj-GAN GAN
VHC rec--rater-eaa scan-Geh GAN
living room filled with lots of furniture.
child with black clothes is standing of snow covered hill.
beach on cloudy day bunch of people on it.
People playing with kites under the blue sky.
baseball player with white clothes is ready for the gane.
white tall oathedral tovering under the sky.
ger
Figure 13: Example text-to-image generation results on COCO.
23

---

Published as a conference paper at ICLR 2020
C.4 LATENT SPACE INTERPOLATION
In addition to the latent space interpolation results of VHE-raster--can-GGNN in Fig. 3(c) of Section
3.1, below provide more fine-gridded latent space interpolation in Figs. 15 18
Figure 14: Example of latent space interpolation CUB.
Figure 15: Example of latent space interpolation on CUB.
24

---

Published as a conference paper at ICLR 2020
Figure 16: Example of latent space interpolation on CUB.
Figure 17: Example latent space interpolation on Flower.
Figure 18: Example latent space interpolation on Flower.
25

---

Published as a conference paper at ICLR 2020
C.5 IMAGE RETRIEVAL GIVEN TEXT QUERY
For image Un. we draw its BoW textual description in as 0n p(t Φ. On), On En ~ qn $ v..) Given the BoW textual description ages ranked by the cosine distances between and in 's. Shown in Fig. 19 are three example image retrieval results, which suggest that the retrieved images are semantically related to their text queries in colors, shapes, and locations.
text query, retrieve the top five im-
white
pink
purple
yellow
center
blue
long
black
tail
dark
motorcycle
parked
parking
black
ground
Figure 19:
Top-5 retrieved images given text query. Rows 3 are for Flower, CUB, and COCO, respectiell
C.6 IMAGE REGENERATION
We note for VHE-GAN, its image encoder and GAN component together can also be viewed as an "autoencoding" GAN for images. More specifically, given image æ, VHE-GAN can provide random regenerations using G (% [ (,))). We show example image regeneration results by both VHE-StackGANN and HHE-rrater-scaa-GGie regenerated random images by the proposed VHE-GANs more of less resemble the original real image fed into the VHE image encoder.
in Fig. 20 These example results suggest that the
Example results of image regeneration using VHF-StackGAAN+ and -HE-raster-scan-GA. An original image is fed into the VHE image encoder, whose latent representation is then fed into the GAN image generator to generate corresponding random image. The models columns 1-4 are trained Flower, columns 5-8 CUB, and columms 9- on COCO.
Figure 20:
C.7 LEARNED HIERARCHICAL TOPICS IN VHE
The inferred topics different layers and the inferred sparse connection weights between the topics of adjacent layers are found to be highly interpretable. In particular, we can understand the meaning
of each topic by projecting it back to the original data space via T (() l and understand the
relationship between the topics by arranging them into directed acyelic graph (DAG) and choose
26

---

Published as a conference paper at ICLR 2020
22 and 23 example subnets taken from the DAGs its subnets visualize. We show in Figs. 21
inferred by the three-layer VHE-raster-scan GAN of size 256- 128-64 on Flower, CUB, and COCO, respectively. The semantic meaning of each topic and the connection weights between the topics of adjacent layers are highly interpretable. For example, in Figs. 21 the topics describe very specific flower characteristics, such special colors, textures, shapes, and parts, the bottom layer, and become increasingly more general when moving upwards.
60 green ndges long bright pedioel larpe dark sepal
pedical pare wavy long shapad eddees storm
wavy green rounded wide wrinkled pedicals curied piaoi
green pink aegels stamen looking
nat oolor bright gueen siggu pedicel pittil thown
yentrey beight petíl large
brown towers
stalks pizti pedicals
ruffiidd rounded
pedicel popals color
red stamen pink bright
str Itraatione iped
soft Smooth bunch
sepels pedicel
yelloww golden bright
clustered round
Kushia
shown
bunch
(lament
stigmm
shidking
stales
curled
grewn
clotteeed
Right
pixti
round
large
Figure 21: An example topic hierarchy taken from the directed acyclic graph learned by a three-layee VHE- raster-scan- scan-GAN of size 256-128-66 64 on Flower.
36
orange long crown wings head feet large bodt
yellow green grey smill belly gray while beak cowa hoad
22
br own bally breast white beok throat long wings crown light small
white long head eye body pointy culor cings resl
Nack orsage feet white red crowa webbed besk Jong head tarses
yellow grey wings hoad belly body small krown winghers
brown
white belly
larsus black
black
beakt
feet
webbed footed
Black
vellow bolly
beak
light
webbed
large
wings
gray
sharp
snowy
short
head
yellow
breast
body
wateriiin standing
body
winghars greenish
timy
tall
thest
nope abdomen
unique
feather
dars
petite retrices
flat
chorp
head
grest
blends
(wide
rock
secondary
multscolorad
Figure 22: Analogous plot Fig.
on CUB.
26
bear tedily stuffed bears sitting brown white holding animal
living table couch teeevision sitting furniture chairs letchen
24
70 lugsage sitting bag suitcases old bags red airport
2
37
chair desk table sitting laptop room window bad. wooden chaics old
kitchen retrigerator stave sink table room open
window beds hotel sitting large small pillows
B6 television screen flat
room
white black photo
pink bright
keyboard sittng
old sitting vintage antique
window looking sill
table chairs dining
refrigerator fridge door
clock wall
living
furniture
large
fireplace
picture dark
time
large
flower
monitar
wooder
food
mounted
chairs Image
white
style
gold
windows
kitting
backgraphy
cute
desktop
display
open
area
white
ornate
Figure 23: Analogous plot to Fig.
on COCO.
27

---

Published as a conference paper at ICLR 2020
D SPECIFIC MODEL STRUCTURE IN VHE-STACKGAN++ AND VHE-RASTEA SCAN-GAN -AA-AETTT-----GGNN
D.1 MODEL STRUCTURE OF VHE
In Fig. 24 we give the structure of VHE used in VHE-StacKGAN++ and rastrrs
scan-GAN, where f(x) is the image features extracted by Inception v3 network and e(!) ~
K: Uniform(e 00, 1). With the definition of g() [ax), we have
( exp W g ( + b("),
(12)
A (0) exp (1) + b
(13)
g ( 1n/1 + exp( W g (1-1) + bl
(14)
where w E RRix,, w E RkIK, w E RkiKK-11 bl0) b3 E RK
E RK 6 E RK and
3
exp
2.
(3)
(3)
12
e0p
7.
12
(2)
k
(
EXC
(1
2.
(3]
(
f(x)
Inception
Figure 24: The architecture of VHE VHE-StackGAN++ and VHE-raster-scannGGAN
D.2 MODEL OF VHE-STACKGAN++
In Section 2.2, first introduce the VHE-StackGANHt where the multi-layer textual representation
(0 (2) 0 004) is concatenated 0 (1))) .o...._())) (()] and then fed into StackGAN++
(Zhag al., 20176). In Figs. (a) and (b), we provide the model structure of St.-StackGAN+++
We also provide detailed plot of the structure of StackGAN++ used in VHE- THE-StackGAN++ in Fig.
25
where JCU a specific type of discriminator; see Zhang et al. (2017b) for more details.
The same with VHE-raster-scan-GGAN VHE-StackGAN++ is also able to jointly optimize all compo- nents by merging the expectation in VHE and GAN to define its loss function as
(G.)
(D.)
E (e.m e..)
E
ThE ((00 æ 40+1 (0+")et + { logp(tn $ (),)
L Σ KL
In. $ (1+1) @((+1)) (e a (1+1), (1+1),(++))
+ D2 Di 0,) + log(1 (, 0,))]]
(15)
28

---

Published as a conference paper at ICLR 2020
with reshape
Upsampling
Joining
Residual
Conv3x3
JCU D
JCU D,
64x64x3
128x128x3
ICU
G2
00
64x64 x4N,
128x128 x2Ng
256x256
G
X6AN,
xN,
256x256x3
Figure 25: The structure of Stack-GAN GAN++ in Stt-Kttck where JCU is type of discrimina- tor proposed in Zhang et al. (2017b).
29

---

Published as a conference paper at ICLR 2020
D.3 STRUCTURE OF -ASTER-SCAN-GAN
In Fig. 26 we provide a detailed plot of the structure of the proposed raster-scan-GAN.
with reshape
Upsempling
Joining
Residual
Conváx3
Downssmpling
JCU
ICU
JCU 1
G
G
4x4
16x16
32x32
64a64
64x64x3
x16
16x16
64664
I2P
D2P
32x32x3
13
6
G
78x1
xS12
x128
X136
x64
X32
x16
16416
128x128
H64
x32
JCU
ICU
JCU
1
4x4
16x16
16x16
64x64
128x128
256x256
G
x16
256x256x3
Figure 26: The structure of raster-scan-GAN in VHEEraster-scan-GAYY where JCU is type of discriminator proposed in Zhang et al. (2017b).
30

---

Published as a conference paper at ICLR 2020
E JOINT OPTIMIZATION FOR VHE-RASTER-SCAN-GAAN
Based on the loss function of VHE-raster-scan-Ce GAN with TLASGR- (Cong et al. 2017) and WHAI (Zhng al. 2018a). describe in Algorithm how to perform mini- based joint update of all model parameters.
Algorithm Hybrid TLASGR MCMC/VHE inference algorithm for VHE-raster GANN.
Initialize encoder parameters $2, topic parameters of PGBN { (1 Ji, generator G, and discrimi- nator D.
for iter = 2, do
Randomly select a mini- batch containing N image-text pairs d {Ins tn Jn
Draw random noise {
(1) Ex
N.L
from uniform distribution;
Calculate VoL G, ( 1=1,=1
Calculate ( æ);
N.L (
Calculate V by the aid of
En
1=11-1 G m);
Update D as D D + VD
Update as G G VoL x):
Update 52 as 52 Ω Vol:
Sample (0 ][=1 from (6) given Ω and ( M=
and use (t) update topics ( M==1
according to TLASGR-MCMC;
end for
F DATA DESCRIPTION ON CUB, FLOWER, AND COCO WITH TRAINING DETAILS
In image-text multi-mol modality learning, CUB /ah et al. 2011). Flower (Nilsback & Zisserman 2008 and COCO (Lin et al.) 2014) are widely used datasets.
html) CUB contains 200 bird species with 11,788 images. Since 80% of birds in this dataset have object-image size ratios of less than 0.5 (Wah et al.) (011). as preprocessing step, we crop all images to ensure that bounding boxes birds have greater-than-O.. object-image size ratios, which is the same with all related work. For textual description, Wah al. (20IT) provide ten sentences for each image and we collect them together to form BoW vectors. Besides, for each species, Elhoseiny let al. (2017a) provide its encyclopedia document for text-based ZSL, which is also used in text-based ZSL experiments.
CUB
http:/ www vision. caltech. wwtp:aaaaa........iiiieeeeeeeeee wetp://riiippdiaa/u----000000000
For CUB, there bird subspecies belonging to the same super-category should belong to either the training split or test one without overlapping, referred
split settings: the hard and the easy The hard one ensures that the
CUB-hard (CUB-H in our manuscript). A recently used split setting (Qiao et al. 2016 Akata et al.) 2015) is super-category split, where for each super-category, except for one subspecies that left unseen, all the other are used for training, referred as CUB-easy (CUB-E in our manuscript). For CUB-H, there are 150 species containing 9410 samples for training and 50 species containing 2378 samples for testing. For CUB-E, there are 150 species containing 8855 samples for training and 50 species containing 2933 samples testing. We use both of them the for the text-based 7SI,, and only CUB-E for all the other experiments usual.
index html Oxford- 102, commonly referred to as Flower, contains 8,189 images of flowers from 102 different categories. For textual description, Nilsback & Zisserman 20088 provide ten sentences for each image and we collect them together to form BoW vectors. Besides, for each species, Elhoseiny et al. (2017a) provide its encyclopedia document for text-based ZSL, which is also used in text-based ZSL experiments section 4.2.2. There are 82 species containing 7034 samples for training and 20 species containing 1155 samples for testing.
Flower http://www. robots ox ac uu/ data feettafflowerze wwt/f/lowwwsssro
31

---

Published as a conference paper at ICLR 2020
For text-based ZSL, follow the same way in Elhoseiny et al. (2017a) split the data. Specifically, five random splits are performed, in each which 4/5 of the classes are considered as "seen classes" for training and 1/5 of the classes "unseen classes" for testing. For other experiments, follow Zhang et al. (017b) to split the data.
COCO http: /ttt:://cocooo cocodatasst. .wtp///ttttt/eeeeeeeeeeee #download: Compared with Flower and CUB, COCO a more challenging dataset, since it contains images with multiple objects and diverse backgrounds. To show the generalization capability of the proposed VHB-GANs, we also utilize COCO for evaluation. Following the standard experimental setup for COCO (Reed et al., 2016; Zhang et al. 2017b), we directly use the pre-split training and test sets to train and evaluate our proposed models. There are 82081 samples for training and 40137 samples for testing.
Training details:
we train VHE-rater-scan-GAN in four Nvidia GeForce RTX2080 TI GPUs. The experiments are performed with mini-batch size 32 and about 30.2G GPU memory space. We run 600 epochs to train the models on CUB and Flower, taking about 797 seconds for CUB-E and 713 seconds for Flower for each epoch. We run 100 epochs to train the models on COCO, taking about 6315 seconds for each epoch. We use the Adam optimizer (Kingma & Ba 2014) with learning rate 2 x 10 81 0.5, and 32 0.999 to optimize the parameters of the GAN generator and discriminator, and use Adam with learning rate 10 14-4, 31 = 0.9, and 3g = 0.999 to optimize the VHE parameters. The with those in Cong et al. (2017).
to update the topics & with TLASGR-MCMC are the same
32

---

Published as a conference paper at ICLR 2020
G ADDITIONAL DISCUSSION ON OBJ-GAN
Focusing on the COCO dataset, the recently proposed Obj-GAN (Li et al. 2019) exploits more side information, including the bounding boxes and labels of objects existing the images, to perform text-lo-image generation. More specifically, Obj-GAN first trains an attentive sequence to sequence model infer the bounding boxes given a text t:
Bir [B,, B2. Br]] box
(16)
where, are the pre-trained bi-LSTM word vectors of t, B, (4,, by) consists of the class label of the Ith object and its bounding box b = (x, y, w, h) E Rd. Given the bounding boxes B Obj-GAN learn shape generator to predict the shape of each object in its bounding box:
M Gshape (BI- Z1:))
(17)
where 2, ~ NO, 1) is a random noise vector. Having obtained Bı.T and MY.T. Obj-GAN trains an attentive stat image generator to generate the images conditioned on Bı.r, Mi.r, and e.
Although Obj-GAN achieves better FID on COCO, has two major limitations in practice. First, not always possible to obtain accurate bounding boxes and labels of objects in the image; even they can be acquired by manual labeling,
often time and labor consuming. especially on large datasets. Second, each word is associated with one fixed bounding box; in other words, given one sentence, the locations of the objects in the generated images are fixed, which clearly hurts the diversity of the Obj-GAN generated images, shown in Fig. 27
People playing with kites outside under the blue sky.
Obj-GAN
VHE-raster-scan-GAN
Figure 27: The generated random images Obj-GAN given text lack diversity.
33


Reviews:
[{"review_id": "ryeEe7T0Fr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The authors design a new model for bidirectional joint image-text modeling using a variational hetero-encoder\n(VHE) randomized generative adversarial network (GAN) that integrates a probabilistic text decoder, probabilistic image encoder, and GAN into an end-to-end multimodal model. Their proposed VHE-GAN model encodes an image to decode its associated text and feeds the variational posterior as the source of randomness into the GAN image generator. The authors also incorporate a deep topic model, a ladder-structured image encoder, and StackGAN++ into their framework for improved photo-realistic images.\n\nStrengths:\n- The authors have proposed a nice multimodal model that allows inference of latent variables given only text or image, and also allows realistic synthesis of images from images, text, or noise.\n- The paper is quite dense but generally well written. \n\nWeaknesses:\n- The experimental comparison only included old baselines and the authors should compare to some more recent work such as TA-GAN (NIPS18), and Object-GAN (CVPR19).\n- It would help if the paper contained more ablation studies across different modules that the framework uses.\n\n### Post rebuttal ###\nThank you for your detailed answers to my questions. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SkxqhVvaYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a combined architecture for image-text modeling. Though the proposed architecture is extremely detailed, the authors explain clearly the overarching concepts and methods used, within limited space. The experimental results are extremely strong, especially on sub-domains where conditional generative models have historically struggled such as images with angular, global features - often mechanical or human constructed objects. \"Computers\" and \"cars\" images in Figure 2 show this quite clearly. The model also functions for tagging and annotating images - performing well compared to models designed *only* for this task.\n\nThe authors have done a commendable job adding detail, further analysis, and experiments in the appendix of the paper. Combined with the included code release, this paper should be of interest to many. \n\nMy chief criticisms come for the density of the paper - while it is difficult to dilute such a complex model to 8 pages, and the included appendix clarifies many questions in the text body, it would be worth further passes through the main paper with a specific focus on clarity and brevity, to aid in the accessibility of this work. \n\nAs usual, more experiments are always welcome, and given the strengths of GAN based generators for faces a text based facial image generator could have been a great addition. The existing experiments are more than sufficient for proof-of-concept though.\n\nFinally, though this version of the paper includes code directly in a google drive link it would be ideal for the final version to reference a github code link - again to aid access to interested individuals. Being able to read the code online, without downloading and opening locally can be nice, along with other benefits from open source release. However the authors should release the code however they see fit, this is more of a personal preference on the part of this reviewer.\n\nTo improve my score, the primary changes would be more editing and re-writing, focused on clarity and brevity of the text in the core paper.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HylBJaH6FS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed VHE-GAN for the text-to-image generation task. The proposed method utilizes the off-the-shell modules and feeds the VHE variational posterior into the generator. The experiments are conducted on three datasets. \n\nThe motivation for the paper is not clear. Most of the components used, such as text-encoder, image-encoder, generator-discriminator follow previous works. Therefore, the authors should claim how the proposed VHE variational posterior can help the task. However, I did not see the clear motivation for this part. \n\nBesides the basic version VHE-StackGAN++, it proposed another version  VHE-raster-scan-GAN. However, the paper also fails to tell the intuition of the deep topic model and PGBN text decoder. \n\nThe experimental results are not solid. The comparison only included old baselines. However, several recent state-of-the-art approaches are missing: a. attnGAN (CVPR18), b. TA-GAN (NIPS18), c. Object-GAN (CVPR19). Without these comparisons, it is difficult to evaluate how the method works. In addition, the paper does not provide an ablation study to analyze the effect of each component proposed (e.g., Poisson gamma belief network, a deep topic mode).   ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a bidirectional joint image-text model using a variational hetero-encoder (VHE) randomized generative adversarial network (GAN). The proposed VHE-GAN model encodes an image to decode its associated text. Three reviewers have split reviews. Reviewer #3 is overall positive about this work. Reviewer #1 rated weak acceptance, while request more comparison with latest works. Reviewer  #2 rated weak reject raised concerns on the motivation of the approach, the lack of ablation and lack of comparison with the latest work. During the rebuttal, the authors provide additional comparison and ablation, which seem to address the major concerns. Given the overall positive feedback and the quality of rebuttal, the AC recommends acceptance.
Decision: Accept (Poster)