# A Pipeline for Story Visualization from Natural Language
## 3 The Proposed Pipeline Framework
### Relation Extraction and Vocabulary Mapping  
We considered phrases that described the main animal characters' behavior. We also focused on some of their common and uncommon basic behaviors. Table 1 shows some common sample phrases used in this work as well as their related actions. It is worth noting that animal behavior that is not listed is considered to be non-common animal behavior. From the resultant phrases of the previous step, we obtained all of the triples in the form _<object, relationship, object>_ using a scene graph parser. Due to practical reasons, it was not possible to create images for all of the extracted triples from the story text. Due to this restriction, as in the case of the visual genome dataset [10], the vocabulary mapping used a semantic similarity based on word2 vec to find the nearest tokens from the model vocabulary, as shown in Algorithm 2. ``` Input = vocabulary [], triples [] Output = similar_triples [] Begin
1. model = gensim.models.Word2 Vec (vocabulary, size = 100, min_count = 1, sg = 1)//initialize from Gensim library ([[https://radimnehurek.com/gensim/models/word2](https://radimnehurek.com/gensim/models/word2) vec.html]([https://radimnehurek.com/gensim/models/word2](https://radimnehurek.com/gensim/models/word2) vec.html) (accessed on 12 February 2023))
2. for entity in triples:
3. top_similar = model.wv.most_similar (positive = entity, topn = 1)//get most similar token to entity
4.... similar_triples.append (top_similar)
5. end for return similar_triples ```

**Algorithm 2**Get_mapping extended extracted entities and relations with model vocabulary Thus, the mapping also helped us to map non-common actions such as _\"approach\"_ to the similar common action in the list such as _\"stand\"_ and _\"walk\"_. If we failed to find a match, we checked for a mapping while including the verb's proposition such as _\"close to\"_, _\"next to\"_, etc. For instance, for the tokens of the sentences mentioned earlier in Figure 1, we computed their similarities with the terms in the vocabularies and took the maximum value among them all. As an example, the token _\"elephants\"_ was mapped to the term _\"elephant\"_ with a similarity value of 1.0; however, the token _approached_ was mapped to the term _\"stand\"_ with a similarity value of 0.1, using the word2 vec similarity function. ### Image Generation

We split the image generation step into two main tasks. One task tackled the generation of a single image to visualize sentences in isolation. The second task was directed towards the detailed generation of image sequences, i.e., multiple images that were highly coherent with the whole story. After, obtaining the objects and relationships that composed the scene graph, we used a graph convolution network [11] composed of several graph convolution layers to process the scene graph. **Single image generation**. We generated images from scene graph triples of actions and characters using a pre-trained model for PyTorch [11]. Basically, the architecture consisted of three main modules: a graph convolution network (GCN), a layout prediction network (LN) and a cascade refinement network (CRN). First, the GCN took a scene graph as an input and produced an embedding label vector output for each object. Then, these object embedding vectors were used by LN to compute a scene layout by predicting a segmentation mask and bounding box for each object. Given a scene layout, the CRN was then responsible for generating an image that respected the object relations in the scene layout. Finally, discriminators were used to generate realistic output images by adversarially training the image generation network against a pair of image discriminator networks and an object discriminator network. The generated realistic output images were adversarially trained by the image generation network against a pair of discriminator networks \(\text{D}_{\text{image}}\) and \(\text{D}_{\text{object}}\) to minimize the weighted sum of six losses [11].