# Storytelling with Image Data: A Systematic Review and Comparative Analysis of Methods and Tools
## 4 Task 2. Story Modeling
### Narrative Engineering
The Planner, formulated as an RL problem, invokes the controllers to explore the environment and answer the question in the IQA task. * **Generative Adversarial Networks (GANs). ** [PERSON] et al. [75] present a Correlated Collaborative Model (CCM) that guarantees the coherence of the generated textual explanations and answers. CCM collaborates with the answer and explanation features and employs generative adversarial strategies for training. In addition, the results are robust to noise in images and text (even if the model is not trained for noise-based attacks). * **Combined Methods (RL + GAN). ** To generate more human-like answers to questions, [PERSON] et al. [76] combine RL with GANs and introduce a novel framework. This framework updates the generator by employing the reward of the generator policy at each training step. After sampling data from the dialog history (a sequence of textual data), maximum likelihood estimation (MLE) is used to update the generator. [PERSON] et al. [77] introduce a conditional generative adversarial captioning technique to extend RL-based architectures. Furthermore, CNN- and RNN-based architectures are presented for the discriminator module. The discriminator judges whether a human described the resulting caption or if it is machine-generated. **Feature-Map-Based Methods. ** One of the critical challenges in narrative construction based on an image is extracting visual features which best represent the image content. Therefore, we categorize these methods into three main groups based on various strategies proposed for extracting visual features: CNN-based, attention-based, graph-based, and vision-transformer-based methods, which are discussed in detail in the following paragraphs. * **CNN-Based Methods. ** [PERSON] et al. [71] employ a vision CNN model to extract image features, since CNNs can embed images in fixed-length vectors. They utilize the result feature vector of GoogleNet [141] for the visual feature extraction step. Moreover, [PERSON] et al. [70] used the visual features extracted from AlexNet [142]. Many studies employ the CNN modules to extract features such as [143; 72; 144]. * **Attention-Based Methods. ** Simple CNN-based methods have the main advantage of being compact and straightforward. At the same time, the extreme compactness and lack of granularity are crucial issues that need to be addressed. An attention module handles this issue by computing attention weights and attending to specific parts of the visual features. Many approaches fit into attention-based methods that can be divided into three main categories: grid-based, region-based, and self-attention. In the following items, these three categories are further discussed. * **Grid-Based Attention Methods. ** A significant proportion of image captioning methods use the attention mechanism to make captioning more flexible and provide better granularity. [PERSON] et al. [78] introduced an attention-based image captioning model inspired by a recent study in machine translation [79] and object detection [80; 81]. The authors proposed two attention-based image caption generators: a \"soft\" deterministic attention mechanism and a \"hard\" stochastic attention mechanism. An adaptive encoder-decoder model [82] also automatically determines when to rely on the language model and when to look at the image (the spatial CNN features at each of the k grid locations of the image). * **Region-Based Attention Methods. ** [PERSON] et al. [83] employ a novel bottom-up and top-down attention module (based on Faster R-CNN [49] and task-specificcontext). First, the bottom-up visual attention module extracts salient regions of an image and represents them using convolutional feature vectors. Then, the top-down module estimates the distribution of attention over image regions (specifies weights of the features). As a result, a weighted average of all image features is the final attended feature vector.