Title: TFGAN: IMPROVING CONDITIONING FOR TEXT-TO-VIDEO SYNTHESIS

Abstract: Developing conditional generative models for text-to-video synthesis is an extremely challenging yet an important topic of research in machine learning. In this work, we address this problem by introducing Text-Filter conditioning Generative Adversarial Network (TFGAN), a GAN model with novel conditioning scheme that aids improving the text-video associations. With a combination of this conditioning scheme and a deep GAN architecture, TFGAN generates photo-realistic videos from text on very challenging real-world video datasets. In addition, we construct a benchmark synthetic dataset of moving shapes to systematically evaluate our conditioning scheme. Extensive experiments demonstrate that TFGAN significantly outperforms the existing approaches, and can also generate videos of novel categories not seen during training.

1. INTRODUCTION:

Generative models have gained much interest in the research community over the last few years as they provide a promise for unsupervised representation learning. Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) have been one of the most successful generative models till date. Following its introduction in 2014, significant progress has been made towards improving the stability, quality and the diversity of the generated images (Salimans et al., 2016; Karras et al., 2017) . While GANs have been successful in the image domain, recent efforts have extended it to other modalities such as texts (Wang et al., 2018a) , graphs (Wang et al., 2018b), etc. In this work, we focus on the less studied domain of videos. Generating videos are much harder than images because the additional temporal dimension makes generated data extremely high dimensional, and the generated sequences must be both photo-realistically diverse and temporally consistent. We tackle the problem of text-conditioned video synthesis where the input is a text description and the goal is to synthesize a video corresponding to the input text. This problem has many potential applications, some of which include producing multimedia special effects, generating synthetic data for model-based Reinforcement Learning systems and domain adaptation, etc.

Two recent works that address the problem of text-conditioned video generation include Li et al. (2018) and Pan et al. (2017) . Both these methods are variants of conditional GAN model applied to the video data. In spite of some successes, they have the following limitations: (1) They employ 3D transposed convolution layers in the generator network, which constrains them to only produce fixed-length videos. (2) Their models are trained on low-resolution videos -results are shown only at a 64×64 resolution. (3) Text conditioning is performed using a simple concatenation of video and text features in the discriminator: Such a conditioning scheme may perform well on certain datasets, but has difficulty in capturing rich video-text variations.

In this work, we aim to address all the concerns above. First, to model videos of varying length, we use a recurrent neural network in the latent space and employ a shared frame generator network similar to (Tulyakov et al., 2018) . Second, we present a model for generating high-resolution videos by using a Resnet-style architecture in the generator and the discriminator network. Third, we propose a new multi-scale text-conditioning scheme based on convolutional filter generation to strengthen the associations between the conditioned text and the generated video. We call our model Text-Filter conditioning GAN (TFGAN). Finally, we construct a benchmark synthetic moving shapes dataset to extensively evaluate the effectiveness of the new conditioning scheme we proposed. to a GRU network to get a trajectory in the latent space. These latent vectors are fed to a shared frame generator to produce the video sequence. The generated videos are then passed to conditional discriminator networks. The box highlighted in red is where the conditioning is performed and is expanded in Fig. 2 In summary, our contributions in this work are as follows: (i) A new conditional GAN with an effective multi-scale text-conditioning scheme based on convolutional filter generation is proposed;

(ii) A benchmark synthetic dataset for studying text conditioning in video generation is presented;

(iii) Photo-realistic video synthesis is achieved using a deeper generator-discriminator architecture.

2. RELATED WORK:

Two popular approaches to generative modeling include GANs (Goodfellow et al., 2014) and Variational Autoencoders(VAEs) (Kingma & Welling, 2014) . GANs are formulated as a 2player minimax game between a generator and a discriminator network, while VAEs are based on variational inference where a variational lower bound of observed data log-likelihood is optimized. Among the two approaches, GANs have generated significant interest as they have been shown to produce images of high sample fidelity and diversity (Karras et al., 2017) .

A variant of GAN models are conditional GANs where the generator network is conditioned on input variables of interest. Such a conditioning input can be labels (Odena et al., 2017) , attributes (Yan et al., 2016 ), text (Zhang et al., 2017; Xu et al., 2018) or even images (Zhu et al., 2017) . We focus on text conditioning since it is relevant to this work. One of the first works to perform textconditioned image synthesis is Reed et al. (2016) . Their method was only shown to synthesize low-resolution images. 

3. METHOD:

We first provide a formal description of the problem being address. We are given access to n data points {(v i , t i )} n i=1 sampled from an underlying joint distribution p(v, t) in the video-sentence space. Here, each v i ∈ R T ×C×W ×H is a video clip and t i is a sentence description. We are interested in learning a model capable of sampling from the unknown conditional distribution p(v|t). Similar to conditional GANs, we formulate the problem as learning a transformation function G(z, t) from a known prior distribution P z (z) and the conditional input variable t to the unknown conditional distribution p(v|t). The function G is optimized using an adversarial training procedure.

3.1. MODEL FRAMEWORK:

The framework of our proposed model is shown in Fig. 1 . The text description t is passed to a text encoder T to get a frame-level representation t f and a video-level representation t v . Here, t f is a representation common to all frames, and contains frame-level information like background, objects, etc. from the text. The video representation t v extracts the temporal information such as actions, object motion, etc. The text representation along with a sequence of noise vectors {z i } l i=1 are passed to a recurrent neural network to produce a trajectory in the latent space. Here, l denotes the number of frames in the video sequence. These sequence of latent vectors are then passed to a shared frame generator model G to produce the video sequence.

The generated video is then fed to two discriminator models -D F and D V . D F is a frame-level discriminator that classifies if the individual frames in the video are real/fake, whereas the video discriminator D F is trained to classify the entire video as real/fake. The discriminator models D F and D V also take the text encoding t f and t v respectively as inputs so as to enforce text-conditioning.

3.2. TEXT-FILTER CONDITIONING:

To build strong conditional models, it becomes important to learn good video-text associations in the GAN model. A standard technique is to sample negative (v, t) pairs (wrong associations) and train it as fake class, while the correct (v, t) pairs are trained as real class in the discriminator network. Since the generator is updated using the gradients from the discriminator network, it becomes important to effectively fuse the video and text representations in the discriminator so as to make the generator condition well on the text. Previous methods (Li et al., 2018; Pan et al., 2017) use a simple concatenation of text and video features as the feature fusion strategy. We found that this simple strategy produces poor conditioned models in datasets where there are rich text-video variations (refer to Section. 4 for more details).

Our proposed model Text-Filter conditioning GAN (TFGAN) focuses on improving text conditioning. In TFGAN, we employ a scheme based on generating convolutional filters from the text features. This scheme, which we call Text-Filter conditioning, is shown in Fig. 2 . Let us first divide the discriminator network D (which can be

D F or D V ) into multiple sub-networks {D (i) } m i=1 so that D(x) = D (m) • D (m-1) • . . . D (1) (x)

. These sub-networks can be as small as a single layer, or can be a cascade of multiple layers. Let d (i) denote the output of the i th sub-network of the discriminator. From the text features, we generate a set of convolution filters {f i } m i=1 . Each filter f i is now convolved with the discriminator response d (i) , and this convolved output is passed through additional convolutions after which they are pooled to get a single video-text representation. We use this pooled feature vector to classify the (v, t) pairs as real or fake. Because the generated convolutional filters {f i } are applied to discriminator sub-network outputs {d (i) } from different layers, the resulting text conditioning effectively imposes semantic constraints extracted from input texts to the generated individual frames and video clips at different feature abstraction levels.

3.3. TRAINING ALGORITHM:

The discriminator model D and the generator model G are trained using an adversarial game as done in the standard conditional GANs. However, since we employ deep Resnet-style architectures for our G -D networks, it was important to stabilize the GAN training. We use the regularizer as proposed in Mescheder et al. (2018) where the norm of the discriminator gradients are penalized. With this regularizer, our optimization objective can be expressed as From the text features, we extract a set of convolution filters {f i } m i=1 and apply the filter f i to the output d (i) of sub-network D (i) . These responses are then passed to additional convolutional layers before they are pooled to get a single video-text representation that is used to classify the input (v, t) pair as real/fake.

L real =E (v,t)∼p data,real [log(D(v, T (t))) + γ 2 ∇D(v, t) 2 ]

(1)

EQUATION

The text encoder T is optimized as follows

max T L T = E (v,t)∼p data,real log(D(v, T (t))) + E (v,t)∼p data,f ake log(1 -D(v, T (t))) (4)

In the above set of equations, p data,real denotes the real data distribution with correct video-text correspondences, whereas p data,f ake refers to the distribution with incorrect video-text correspondences. Note that we have two discriminator networks -D F , D V in our models, and the above equations have to be repeated for both models. Eq.1-4 are optimized by alternating between the minimization and maximization problems as done in standard GAN. Please refer to Appendix A for the detailed training algorithm.

4. EXPERIMENTS:

This section discusses the experimental validation of our TFGAN model. We first describe a benchmark synthetic dataset we created for the task of text-to-video generation, and use it to better analyze our system. Then, we show results on a challenging real-world video dataset -the Kinetics human action video dataset (Kay et al., 2017) . Finally, we show how our method can be extended to the task of text-to-image synthesis and show results on the CUB birds dataset (Welinder et al., 2010) . all parameters results in 360 unique parameter configurations. Some samples from this dataset are shown in Fig. 3a . We call this dataset Shapes-v1 dataset.

The above dataset contains videos with static background (all black). While this is a reasonable assumption to make, it is hardly true in practice as many videos have dynamic backgrounds. So, we create a second dataset called Shapes-v2 dataset which is a version of Moving Shapes dataset with dynamic backgrounds. To generate the background, we choose images from the Kylberg Texture Dataset (Kylberg, 2011) and sample a sequence of patches corresponding to a random trajectory. Each patch in this sequence forms the background of an individual frame in the video. These background textures are blended with the moving object resulting in videos as shown in Fig. 3b . This dataset is much more challenging than the Shapes-v1 dataset as the generative models should learn to ground the text description to the moving object but not to the randomly moving background. Both these datasets were created with videos containing 16 frames at a 64 × 64 frame resolution. 

4.1.2. QUANTITATIVE EVALUATION:

An important advantage of creating the synthetic dataset is that it provides a framework for quantitative evaluation of the text-conditioning. First, we train five attribute classifiers (shape, size, color, motion and direction classifiers) on the real data using the ground truth attributes (we have access to ground-truth attributes as we stored them while creating the dataset). We then use these trained attribute classifiers to verify if the attributes of the generated videos correspond to those described by the input text in the test set. For each text description in the test set, we generate the video using our TFGAN model and measure the attribute prediction accuracy. Higher this accuracy, better conditioned is our GAN model.

We experiment with the following models: (1) FeatCat: a conditional GAN model trained using simple text-video feature concatenation in the discriminator network (2) FeatCat branchingD: conditional GAN model with branching D structure where responses at intermediate layers of D are pooled, and this pooled feature is concatenated with text embedding. This model is essentially TF-GAN but without performing the convolutions from text-filters, and (3) TFGAN with Text-Filter conditioning. The architecture and hyper-parameter details are described in the Appendix C. Some sample generations of our Text-Filter conditioned GAN model is shown in Fig. 3a and 3b Table 1 reports the quantitative evaluation of the three conditional GAN models on Shapes dataset. We observe that TFGAN with text-filter conditioning achieves the best performance among the three models on both the datasets. An important observation to note is that using a branching architecture in the discriminator network alone does not improve the text conditioning. This shows that the effectiveness of our method comes not from the branching architecture, but in how text conditioning is applied (using convolutions) at multiple layers of the discriminator network.

4.1.3. EXPLORATORY EXPERIMENTS:

In this section, we report some exploratory experiments we perform on the Shapes dataset.

Sentence interpolation:

In this experiment, we depict conditional interpolation whereby frames in a video transition corresponding to the interpolation between two sentence descriptions. Let S 1 and S 2 denote the two sentences that are interpolated, and

(t S1 f , t S1 v ) and (t S2 f , t S2 v )

denote their corresponding feature representation obtained by passing through the text encoder T . For each frame to be generated, the corresponding conditioning feature is obtained by a linear interpolation between these two representations: 

(t i f , t i v ) = (1 -α)(t S1 f , t S1 v ) + α(t S2 f , t S2 v )

Instead of using a fixed text representation (t f , t v ) as conditioning argument to all the frames in the Generator network, we use (t i f , t i v ) as input to the frame i. The resulting interpolated videos are shown in Fig. 4 . We observe that we are able to obtain smooth transitions. When interpolating between the blue square and the red square, we obtain some intermediate frames with pink shade. Interestingly, none of the samples in the dataset contain pink color. In the second figure, we observe a smooth decrease in the object size while the object continues to move in the specified trajectory. Of the 360 unique parameter configurations in the Shapes dataset, we randomly hold out n configurations from the training set. After training the model on this training set, we feed the text descriptions from the held-out n configurations and measure the attribute classification accuracy in this set. In this experiment, we choose n = 20. The results are reported in Table 2 . We observe that our model achieves good accuracy and this illustrates the ability of our method to generalize.

Long Sequence Generation One of the benefits of using a RNN-based GAN model is that it allows us to model variable-length video sequences. To demonstrate this, we perform an experiment where we train our TFGAN model on 16-length video sequences and generate 32-length sequences. This can be performed easily as we could potentially generate a latent trajectory of any length using the RNN model in the latent space, and the videos are generated using a shared generated acting on this latent trajectory. Fig. 4 shows the output of one such 32-length sequence generated. We observe that the model is able to clearly perform the zig-zag motion beyond 16 frames.

4.2. KINETICS DATASET:

To demonstrate the practical relevance of our approach, we perform experiments on real-world video datasets. We use the dataset proposed in Li et al. (2018) The results of training our TFGAN model on the Kinetics dataset are shown in Fig. 5 . We observe that our model is able to produce videos of much higher quality than the comparison method (Li et al., 2018) . We are able to generate fine motions like golf swing, while Li et al. (2018) produces a blobby region. Also, we train the model to produce 128 × 128 resolution videos, while the method in Li et al. (2018) was trained only on 64 × 64 videos. As done in Li et al. (2018) , we report a simplified version of inception score whereby a video classification model is trained on the real data, and the accuracy on generated data is reported. We report the performance on the following five categories as done in Li et al. (2018) : kite surfing, playing golf, biking in snow, sailing, swimming and water skiing. As can be seen from Table . 3, our methods achieves significantly higher accuracy than the method in Li et al. (2018) . In-set refers to the performance obtained on the test set of real videos, thus serves as an upper bound. We report additional results, architecture and hyper-parameter details in Appendix C. 2010), a benchmark dataset for text-to-image generation. Some of the samples from the generated images are shown in Fig. 6 . We observe that our model is able to produce photo-realistic images. We also report Inception score as a quantitative metric. As can be seen from Table . 4, our method achieves higher inception scores than the comparison methods.

4.3. TEXT TO IMAGE GENERATION:



5. CONCLUSION:

In this work, we address the problem of generating videos conditioned on text. We propose a novel text-conditioning framework whereby conditioning is performed using convolution operations on image feature maps with filters generated from text. To better understand the text conditioning, we construct a synthetic dataset and show that our conditioning scheme achieves superior performance compared to other techniques. Finally, by using deeper architectures in the discriminator and generator networks, we generate photo-realistic videos on the challenging Kinetics dataset. Update

D V : φ V = φ V + α∇[L V real + L V f ake ] 6: Update G: θ = θ -α∇[L F real + L F f ake + L V real + L V f ake ] 7:

Update T :

φ T = φ T + α∇[L F T + L F T + L V T + L V T ] 8: end for

APPENDIX B DATASET GENERATION -SHAPES DATASET:

Both Shapes-v1 and Shapes-v2 dataset contain videos with objects moving along a specific trajectory. There are 5 control parameters -shape, size and color of object, type and direction of motion. Table 5 lists the possible values each parameter can take. The (shape, color, size) tuple describes the structure of the object, while (motion type, direction) tuple dictates how the object moves in the video. For straight line and zig-zag motion, the object could move in north, south, west and east direction, while for diagonal motion, the possible directions include north-west, north-east, south-west and south-east. The zig-zag motion was generated using a sinusoidal function. 

:



Figures and Tables:

Figure1: Our TFGAN framework. Text representations are extracted from the input text and passed to a GRU network to get a trajectory in the latent space. These latent vectors are fed to a shared frame generator to produce the video sequence. The generated videos are then passed to conditional discriminator networks. The box highlighted in red is where the conditioning is performed and is expanded in Fig.2

Figure2: Illustration of our Text-Filter conditioning strategy. From the text features, we extract a set of convolution filters {f i } m i=1 and apply the filter f i to the output d (i) of sub-network D(i) . These responses are then passed to additional convolutional layers before they are pooled to get a single video-text representation that is used to classify the input (v, t) pair as real/fake.

Figure 3: Samples from the Moving Shapes Dataset: In each set of images, the top row corresponds to the original video and the bottom two rows correspond to the video generated by our TFGAN model (Better viewed in color).

Figure 4: Exploratory experiments on Shapes-v1 dataset. The image on the top shows the long sequence experiment where we generate 32-length sequence from a model trained for 16 frames. The top row of this video are the first 16 frames and the bottom row corresponds to the next 16. The images on the bottom illustrate the interpolation experiments where we generate a video corresponding to a smooth transition between two input sentences.

Figure 5: Sample generations from models trained on Kinetics dataset

Figure 6: Sample generations of Text2img synthesis from our model trained on CUB-birds dataset

the various loss terms we use in our TFGAN model. Equation 1 gives the expressions for the loss terms L real , L f ake and L T . Denoting L F real , L F f ake , L F T as the losses for frame-level discriminator D F and L V real , L V f ake , L V T as the losses for video-level discriminator D V , the training algorithm we use is mentioned in Alg. 1 Training algorithm Require: θ: Parameters of G, φ F : Parameters of D F , φ V : Parameters of D V , φ T : Parameters of T Require: N iter : number of training iterations Require: α: Learning rate, N b : batch size 1: for t in 1 : N iter do 2:Sample N b real samples with correct video-text correspondence{(v r i , t r i )} N b i=1 3: Sample N b real samples with incorrect video-text correspondence {(v f i , t f i )} N b i=1 4: Update D F : φ F = φ F + α∇[L F real + L F f ake ]5:

Figure 7: Additional results on Kinetics dataset

Figure 9: Results on Epic-Kitchens dataset

Attribute classification accuracy (in %) on Shapes dataset

Table content: <table><tr><td>Method</td><td colspan="3">Shape Color Size Motion Direction</td></tr><tr><td/><td>Shapes-v1 dataset</td><td/><td/></tr><tr><td>Feature concat</td><td>70.18 99.23 83.69</td><td>96.83</td><td>99.00</td></tr><tr><td colspan="2">Feature concat multiscale D 65.25 99.91 74.31</td><td>99.12</td><td>99.11</td></tr><tr><td>Text-filter conditioning</td><td>97.66 99.99 98.60</td><td>99.40</td><td>100.00</td></tr><tr><td/><td>Shapes-v2 dataset</td><td/><td/></tr><tr><td>Feature concat</td><td>64.49 99.98 81.46</td><td>97.40</td><td>99.40</td></tr><tr><td colspan="2">Feature concat multiscale D 56.12 99.71 65.15</td><td>97.20</td><td>98.00</td></tr><tr><td>Text-filter conditioning</td><td colspan="2">88.52 99.98 94.46 100.00</td><td>99.80</td></tr></table>

Attribute classification acc. on novel categories

Table content: <table><tr><td colspan="2">Attribute Accuracy (in %)</td></tr><tr><td>Shape</td><td>96.21</td></tr><tr><td>Color</td><td>99.78</td></tr><tr><td>Size</td><td>98.77</td></tr><tr><td>Motion</td><td>96.23</td></tr><tr><td>Direction</td><td>99.42</td></tr></table>

for this purpose. This dataset

Table content: <table><tr><td>A bird with a</td><td>This bird is</td><td>A small bird</td><td>The bird is</td></tr><tr><td>white body</td><td>mostly black</td><td>with a white</td><td>red with the</td></tr><tr><td>and black</td><td>with dark</td><td>ring, black</td><td>exception of</td></tr><tr><td>wings, white</td><td>blue and</td><td>secondaries</td><td>the wings,</td></tr><tr><td>tail, webbed</td><td>purple</td><td>and is grey</td><td>it's beak is</td></tr><tr><td>feet and a</td><td>secondaries,</td><td>and light</td><td>short and</td></tr><tr><td>long black</td><td>and large</td><td>brown in the</td><td>eyes are</td></tr><tr><td>stripe</td><td>and pointy</td><td>rest of its</td><td>small</td></tr><tr><td/><td>beak</td><td>body</td><td/></tr></table>

Classification accuracy

Table content: <table><tr><td>Method</td><td>Acc. (%)</td></tr><tr><td>In-set</td><td>78.1</td></tr><tr><td>T2V( Li et al. (2018))</td><td>42.6</td></tr><tr><td>Ours</td><td>53.2</td></tr></table>

Inception Score on CUB-Birds dataset V and the RNN network in the latent space. We train our GAN model with Text-Filter conditioning on the CUB-Birds dataset Welinder et al. (

Table content: <table><tr><td>Method</td><td>Inception Score</td></tr><tr><td>StackGAN</td><td>3.7 ± 0.04</td></tr><tr><td>StackGAN v-2</td><td>3.82 ± 0.06</td></tr><tr><td>Ours</td><td>4.12 ± 0.18</td></tr></table>

Hongwei Wang, Jia Wang, Jialin Wang, Miao Zhao, Weinan Zhang, Fuzheng Zhang, Xing Xie, and Minyi Guo. Graphgan: Graph representation learning with generative adversarial nets. In AAAI. AAAI Press, 2018b. P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical Report CNS-TR-2010-001, California Institute of Technology, 2010. Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In ICCV, 2017. Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Computer Vision (ICCV), 2017 IEEE International Conference on, 2017.

Table content: <table><tr><td>Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong</td></tr><tr><td>He. Attngan: Fine-grained text to image generation with attentional generative adversarial net-</td></tr><tr><td>works. 2018.</td></tr><tr><td>Xinchen Yan, Jimei Yang, Kihyuk Sohn, and Honglak Lee. Attribute2image: Conditional image</td></tr><tr><td>generation from visual attributes. In Computer Vision -ECCV 2016 -14th European Conference,</td></tr><tr><td>Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, 2016.</td></tr><tr><td>Han Zhang,</td></tr></table>

Peer review (iclr2019):
 This paper presents a GAN-based method for video generation conditioned on text description. A given text description is first encoded, then combined with latent noise, and finally input to the generator for generating video frames. The frame discriminator discriminates between the generated frames and real frames conditioned on the encoded text description. The discriminator ensures that the frame generation is smooth. Positives: - Easy-to-read paper - New conditioning method that generates convolution filters from the encoded text, and uses them for a convolution in the discriminator - New toy dataset for evaluation - High-quality generated videos on Kinetics dataset Negatives and suggestions: - Novelty is limited. While the paper presents interesting empirical results for computer vision, they may not be of broad interest to the ICLR community. The main contribution seems to be in combining the encoded text with visual features by using the convoluted operation instead of the common concatenation. - Clarity can be improved. In section 3, there is no description about the text encoder T (section 3.1). Clarity could be improved by providing a figure of the overall architecture of D, and how to generate a set of convolved f_i from text features (they are described in the appendix, which makes the text hard to read and understand). - Experiments: (a) It not clear how to generate fake pairs (v, t). (By shuffling ground-truth pair or fake video or fake text?) (b) It is not clear how to construct the text description from 5 control parameters. (c) There are no qualitative results for the Shape-v1 dataset and the Shape-v2 datasets with background. One suggestion for Shape-v2 is that the background should include color. Otherwise, it is a very easy setting, because the background is a grayscale image while the objects are colored. (d) The metric for classifier accuracy seems strange, because the classifier is not perfect. We cannot trust the classifier for evaluating the generated results.