## 3 Multiple Image Captioning
### VAE-Based Captioning Model Using CAM  
The VAE structure was used to construct a network that provides diversity in captions. While previous studies provided variation in sentence style, we proposed a method to give variation in relation to image region characteristics. The KWL [8], which is one of the attention-based caption generation networks, is applied to the VAE encoder and decoder. The overall framework is described in Figure 3.  
The sentence generation model with VAE basically uses RNN as the encoder and decoder, and most simply projects the hidden state of the last step of RNN to a latent distribution [11, 12, 16]. The structure of the base model is represented in Figure 3a. The hidden state \(h_{t}\) is equivalent to the embedding of a sentence, which indicates the characteristics of the caption. \(h_{t}\) is transformed into a latent dimension through a linear layer to obtain mean \(\mu\) and variance \(\sigma^{2}\), respectively. While decoding a latent variable, \(z\) (representing the sentence style) is sampled from the latent space. Different samples affect the orange shaded vectors to induce the generation of different sentences. \(z\) is projected to vector \(s\) via a fully connected layer. The size of \(s\) is the same as the word embedding size, and it is inserted into the first step of the LSTM after being multiplied by a weight. [PERSON] et al. [11] used hidden state mapping when solving the question generation problem, by setting the latent variable dimension to 20. The dimension size of \(z\) is borrowed from [11]. The proposed model transforms CAM to a latent representation. In the image captioning task, information about images plays an important role. An image and its ground-truth caption pair gives the style of the image regions that the caption describes. The structure of the proposed model is illustrated in Figure 3b. First, the 49-size CAM is extracted from the VAE encoder using the method described in Section 3.2. Next, this vector is projected to a latent space via linear layers. If we set the size of the latent variable too large or too small, training does not work properly. The dimension of 256 was chosen for \(\mu,\sigma,z\), whose size is reasonable for representational power. In the decoding process, a sampled latent variable \(z\) is projected as a guide map \(G\) after going through the linear layer. The vectors affected by \(z\) are shaded in orange. This 49-size guide map \(G\) is designed to apply different weights for different areas of the image. We extracted the region-guided image feature by inserting  
Figure 3: Illustration of variational autoencoder-based multiple image captioning models. (**a**) base model that projects sentence embedding to a latent distribution; (**b**) proposed model that projects CAM to a latent distribution. into the CNN with the image. Specifically, \(G\) is scaled up to \(G_{up}\), whose size is the image input size, and then concatenated to the fourth channel of the image. The four-dimensional tensor, including this map, is inserted as an input to the image encoder, ResNet-152 [17]. The input of the first convolution layer of ResNet is changed to four-dimensional, and the remaining layers are the same as in previous work [17]. The outputs of the ResNet, image feature vector \(V\) and global image feature vector \(V_{g}\) can be expected to reflect the region weighting of the image. The latent distribution is trained to be a form of the normal distribution, as described in Section 3.1. Using inference, latent variables are sampled from the distribution. Various sampling methods can be applied according to the purpose of each model. To extract several captions with different style or expressions, we should sample points that are far away from each other in the latent space. On the other hand, to improve the accuracy of the sentence, which calculates similarity using the ground truth caption, it is necessary to sample a point near the origin. The latent variable is randomly sampled to evaluate whether our model is well constructed.