SlowGuess/NAACL_2021_OCR huggingface dataset:
Story Encoder. The story encoder  $E(\cdot)$  encodes the entire story,  $S$  into a single representation,  $h_0$ , which functions as the initial memory state of the MART model. The input  $S$  is the concatenation of sentence embeddings  $s_k \in \mathcal{R}^{1 \times d_s}$  from all timesteps. The conditional augmentation technique (Zhang et al., 2017) is used to convert  $S$  into a conditioning vector by using it to construct and sample a conditional Gaussian distribution i.e.,  $h_0 = \mu(S) + \sigma^2(S)^{1/2} \odot \epsilon_S$ , where  $\epsilon_S \sim \mathcal{N}(0,1)$  and  $\odot$  represents element-wise multiplication. This introduces a loss term which is the KL-Divergence between learned distribution and Gaussian distribution i.e.,
$$
\mathcal {L} _ {K L} = K L (\mathcal {N} (\mu (S), \mathrm {d i a g} (\sigma^ {2} (S))) | | \mathcal {N} (0, I))
$$
Discriminators. There are two discriminators in the model, each aimed at capturing a different aspect of the task. The image discriminator focuses on local consistency and is provided with the generated image  $\hat{x}_k$ , the sentence  $s_k$ , and the context information vector from the story encoder  $h_0$ , and must attempt to distinguish between this and a real triplet, containing the same information except for the real image  $x_k$  instead of the fake image  $(\mathcal{L}_{img})$ . Additionally, the image discriminator is also used to classify the characters in the frame, when labels are available. The story discriminator is instead concerned with the global consistency of the generated sequence. The generated image sequence  $\hat{X}$  and story  $S$  are provided to the discriminator, which must distinguish it from an equivalently encoded real pair  $(\mathcal{L}_{story})$
MART Context Encoder The MART encoder, as described in the main paper, is followed by a
layer of GRU cells that take the contextualized embedding as input along with isometric Gaussian noise,  $\epsilon_{k}$ , and produce an output vector  $g_{k}$ . The outputs  $c_{k}$  and  $g_{k}$  are concatenated and transformed into filters, and subjected to convolution with a projection of the sentence embedding  $s_k$  i.e.
$$
g _ {k}, q _ {k} = \operatorname {G R U} \left(s _ {k}, \epsilon_ {k}, q _ {k - 1}\right)
$$
$$
o _ {k} = \operatorname {F i l t e r} \left(\left[ c _ {k}; g _ {k} \right]\right) \circ \tanh  \left(W _ {I} s _ {k}\right)
$$
where  $q_{k}$  is the hidden state of the GRU cells. Filter(. ) transforms the concatenated vector  $[c_k;g_k]$  into a multi-channel filter of size  $C_{out} \times 1 \times 1 \times len(W_{I}s_{k})$ , where  $C_{out}$  is the number of output channels. The convolution operation can be interpreted as the sifting of information from local context  $s_t$  with the use of filters that have access to the global context.
B GAN Training
The training procedure for our GAN architecture is similar to StoryGAN. The objective function for the generative model is:
$$
\min  _ {\theta_ {G}} \max  _ {\theta_ {I}, \theta_ {S}} \mathcal {L} _ {K L} + \mathcal {L} _ {i m g} + \mathcal {L} _ {s t o r y} + \lambda_ {d u a l} \mathcal {L} _ {d u a l}
$$
where  $\theta_{G}$  is the parameters of the generator,  $\theta_{I}$  is the parameters for the image discriminator, and  $\theta_{S}$  is the parameters for the story discriminator. Note that the video captioning dual learning component is pretrained and then frozen while the rest of the model is trained.
Each of the components in the model has a conditional loss, which is concerned with whether the input caption and generated image align. The adversarial loss function for the generator is then as follows:
$$
\begin{array}{l} \mathcal {L} _ {G _ {i}} = - \frac {1}{2} \mathrm {E} _ {\hat {x} _ {i} \sim p _ {\hat {x} _ {i}}} [ l o g (D _ {i m g} (\hat {x} _ {i}, s)) ] \\ - \frac {1}{2} \mathrm {E} _ {\hat {X} _ {i} \sim p _ {\hat {X} _ {i}}} [ l o g (D _ {s t o r y} (\hat {X} _ {i}, S)) ] \\ \end{array}
$$
where  $\hat{x}_i$  is the generated image sampled from the distribution  $p_{\hat{x}_i}$  during the  $i^{th}$  stage of generation. The first term is the conditional loss of the image discriminator, and the second term is the conditional loss for the story discriminator.
The adversarial losses for the discriminators are:
$$
\begin{array}{l} \mathcal {L} _ {D _ {i m g}} = - \frac {1}{2} \mathrm {E} _ {x _ {i} \sim p _ {x _ {i}}} [ l o g (D _ {i m g} (x _ {i}, s)) ] \\ - \frac {1}{2} \mathrm {E} _ {\hat {x} _ {i} \sim p _ {\hat {x} _ {i}}} [ l o g (1 - D _ {i m g} (\hat {x} _ {i}, s)) ] \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {L} _ {D _ {\text {s t o r y}}} = - \frac {1}{2} \mathrm {E} _ {X _ {i} \sim p _ {X _ {i}}} [ \log (D _ {\text {s t o r y}} (X _ {i}, S)) ] \\ - \frac {1}{2} \mathrm {E} _ {\hat {X} _ {i} \sim p _ {\hat {X} _ {i}}} [ l o g (1 - D _ {s t o r y} (\hat {X} _ {i}, S)) ] \\ \end{array}
$$
where  $\hat{x}_i$  is the generated image sampled from the distribution  $p_{\hat{x}_i}$ , and  $x_i$  is the real image sampled from the distribution  $p_{x_i}$ , during the  $i^{th}$  stage of generation.
For additional algorithmic details we refer readers to Li et al. (2019b).
Our model is constructed using PyTorch, building off of the original StoryGAN codebase. All models are trained on the training set, tuned on the development set, and evaluated on the test set. We report results for each of the latter. We select the best checkpoints and manual tune hyperparameters for each model by using the validation character classification F-Score. We use the ADAM optimizer with betas of 0.5 and 0.999. We train the model on a single Nvidia 2080TI GPU. Each epoch takes 30 minutes, with the model being saved every 10 epochs. At 120 epochs of training, the total training time is nearly 60 hours for a batch size of 4. We did 1-5 runs for hyperparameter search using manual tuning. The number of trainable parameters in our proposed DUCo-STORYGANis 101,718,981.
Many of our hyperparameters are shared with the StoryGAN model. The image size that we use is 64-by-64, and the length of the story is 5 images/captions. The learning rate of the generator is 2e-4, while the learning rate of the discriminator is slightly lower at 1e-4. We train the model for 120 epochs and set the learning rate to decay every 20 epochs. For each training update of the discriminators, we perform two updates for the generator network, with different mini-batch sizes for image and story discriminators (Li et al., 2019b). The image discriminator batch size is 20 and the story discriminator batch size is 4. We found in our experiments that all story visualization models are susceptible to mode collapse with small changes in the discriminator learning rate. Additionally, we attempted replacing the attention-based alignment module from Xu et al. (2018) with a cross-attention layer and observed mode collapse in later epochs
for the first generated frame in the story. We also used an update ratio of 3:1 for generator vs. discriminator and did not find it useful.
The MART hyperparameters are as follows. The hidden size of the model is 192. The number of memory cells is 3. The number of hidden layers is 2. The dropout values across the model are 0.1. The layer normalization epsilon is 1e-12. The number of attention heads is 6. The word embedding size is 300, and the embedding is initialized using the 840B glove training checkpoint.
E PororoSV Dataset
We utilize the Pororo-SV dataset from the original StoryGAN paper (Li et al., 2019b). This dataset was originally a video QA dataset (Kim et al., 2017), consisting of one second video clips paired with multiple descriptions. A sequence of these video clips forms a story, which then has QA pairs associated with it. There are 9 characters frequently featured in the dataset; a distribution of them can be seen in the supplementary. Annotations are available for the distribution of characters in each frame. It can be seen that each character is featured in at least  $10\%$  of the frames, making it crucial for the model to be capable of generating each of them. To convert this to a story visualization task, Li et al. (2019b) sample the one second videos, obtaining a single, representative frame. Five sequential frame-description pairs are then considered to make up a single story. We use the training and test splits outlined in Li et al. (2019b) for comparable results. However, since this split is also used for tuning in both papers, we carve an equally-sized held-out split of unseen samples from the training set for fair evaluation of the models.
Character Frequency. The PororoSV dataset contains 9 characters that are frequently featured; a distribution of them can be seen in Figure 6.
Video Captioning Accuracy. Video Captioning models use a sequence of image embeddings from the sequence of frames in a video segment as input and perform decoding on the processed features to produce a caption of single sentence or multiple sentences. However, they assume that there are multiple frames within a single video segment, unlike our story dataset where there is exactly one frame for each sentence in the story caption. There-
Figure 6: Distribution of PororoSV characters in various data splits.
fore, we adapt existing state-of-the-art video captioning models to perform decoding from a single frame for each sentence in the caption.
R-Precision. Several prior works on text-to-image generation report the retrieval-based metric R-Precision (Xu et al., 2018) for quantifying the semantic alignment between the input text and generated image. R-Precision is computed using the similarity between encodings extracted from a pretrained Deep Attention-based Multimodal Similarity Model (DAMSM). Since this model only evaluates a single text-image pair for similarity, it is not suitable for evaluating story visualization. Therefore, we train a new version of DAMSM to encode all text-image pairs in each story and compute the global similarity for consecutive frames from a story and their respective captions in addition to sentence and word similarity. We introduce an additional bidirectional LSTM network for encoding frame captions into a story representation and average pool the image features for individual frames to extract a global visual embedding for the story. The cosine similarity between these two vectors is used to rank the retrieval-based search between the query visualization and candidate story narratives. This improved model, referred to as Hierarchical DAMSM (H-DAMSM), is trained using two additional story-level losses  $\mathcal{L}_{st0}$  and  $\mathcal{L}_{st1}$  with a smoothing coefficient of  $\gamma = 15$ , and the pretrained model is used for evaluation. We refer the reader to Xu et al. (2018) for details on the DAMSM model.
Example of Discriminative Dataset. Figure 7 shows an example from our discriminative dataset that is used in the discriminative evaluation.
Figure 7: Example of the Discriminative Dataset.
Dual Learning. The actual implementation of the encoder-decoder framework in our dual learning approach can vary. For our primary model, we adapt the MART architecture (Lei et al., 2020) to accept a 2D matrix of image features where each column corresponds to a sub-region, instead of a sequence of image features from adjacent frames in a video segment, for each time step (see details in the Evaluation section of the main paper). However, we compare this model with several variations of dual learning networks: (1) Transformer-based Image Captioning, (2) CNN-LSTM-based Video Captioning and (3) CNN-LSTM-based Image Captioning. The Transformer-based image captioning network is essentially a non-recurrent version of the MART-based video captioning. The CNN-LSTM based image captioning model is similar to Qiao et al. (2019). The generated image  $\hat{x}_k$  is fed into a CNN, which produces a feature vector. The feature vector is then fed through an LSTM decoder, which produces the caption  $\hat{s}_k$ . The CNN-LSTM video_captioning model is an extension of this, using 3D convolutions to pool over all frames within a story. We pretrain these models on the Pororo-SV dataset and freeze the parameters before utilizing the weights to get the dual learning loss while training DUco-STORYGAN.
As seen in Table 5, the image captioning approach using CNN-LSTM has a limited impact on performance. Next, we explore Transformer for implementing the captioning model and see larger improvements for character classification and BLEU scores. However, there is limited improvement in performance on R-Precision using image captioning as dual learning. We hypothesize that this is due to the image captioning model's inability to capture information across frames; essentially,
Dual Model	Char. F1	BLEU2/3	R-Precision	Frame Acc.	Top-1 Acc.	Top-2 Acc.
Image Captioning (CNN-LSTM)	47.08	4.29 / 1.83	5.23 ± 0.06	22.29	24.47	46.48
Video Captioning (CNN-LSTM)	46.19	3.98 / 1.73	4.04 ± 0.29	22.12	23.93	46.22
Image Captioning (Transformer)	47.21	4.58 / 1.81	5.37 ± 0.11	22.47	24.47	46.51
Video Captioning (Transformer)	47.23	4.78 /1.87	6.32 ± 0.08	22.30	24.53	47.41
Table 5: Results from variations of Dual Learning on Pororo-SV dataset.
Character	Support	StGAN	TF	DuCoGAN
Pororo	4400	0.59	0.59	0.58
Loopy	2279	0.07	0.08	0.21
Crong	3327	0.50	0.51	0.49
Eddy	3154	0.48	0.50	0.58
Poby	2346	0.25	0.26	0.44
Petty	1564	0.16	0.17	0.49
Tongtong	717	0.15	0.16	0.14
Body	1073	0.21	0.20	0.41
Harry	1503	0.40	0.41	0.42
this method of dual learning is only capable of considering local consistency and not global consistency. Therefore, we use a video captioning model, where all frames are considered simultaneously, allowing it to capture both local consistency and global consistency. The performance of CNN-LSTM based video captioning model on the captioning validation set was low. Hence, using this model for dual learning loss negatively affected performance of our story visualization model. The Transformer-based image_captioning model outperforms video_captioning with CNN-LSTM, suggesting that a sophisticated dual model is as important as global context for story visualization. Consequently, the MART-based video captioning model leverages additional global context and outperforms Transformer-based image captioning across all metrics.
Individual Character Accuracy. As detailed in the Experiments section in the main paper, there are 9 characters which are featured throughout the Pororo-SV dataset. The distribution of characters varies across scenes, with some occurring more frequently than others. Using StoryGAN, Pororo, the most frequently occurring character in the dataset, has the highest F-Score, while the decrease in F-Score for other characters roughly correlates with their frequency in the data. With DUco-STORYGAN, we saw marginal improve
Table 6: Character Classification F-Scores on Pororo-SV validation set (StGAN=StoryGAN, TF=StGAN+Transformer).
Model	Metric	Score
Inceptionv3	Frame Acc.	41.93
Precision	74.66
Recall	64.12
F-Score	68.99
Accuracy	80.68
MART	METEOR	15.06
ROUGE_L	18.13
CIDEr	102.34
H-DAMSM	R-Precision	88.05 ± 0.00
Table 7: Upper Bounds of models used for Metrics on Pororo-SV validation set.
ments for Pororo and up to  $30\%$  absolute improvement in F-Score for less frequent characters like Loopy. See Figure 6 for a detailed breakdown of each character. While this confirms the data intensive nature of story visualization, it also shows that advanced modelling approaches can alleviate the issue of data scarcity to some extent. However, models in this domain will ultimately need to be extended to more diverse datasets with more characters and settings before they can be useful in practical applications (see Introduction).
Evaluation Metric Upper Bounds. Many of the evaluation metrics that we use take advantage of other external model architectures (see Evaluation section in main paper), similar to prior work in this domain (Li et al., 2019b, 2020). Therefore, the quality of the evaluation metrics is contingent upon the accuracy of these models. Table 7 contains the upper bound results for these models on the Pororo-SV dataset. The finetuned Inceptionv3 model achieves high overall accuracy i.e. more than  $85\%$  on validation and test sets. Video captioning model MART achieves high scores on the Pororo-SV validation set for several NLG metrics. The H-DAMSM model achieves  $88.05\%$  R-precision on the validation set.
More Generated Examples. Figure 8 contains additional examples that our DUCO-STORYGAN model generated.
Figure 8: Additional generated examples using our model. On the left is the generated examples and on the right is ground truth.