## 3 Multiple Image Captioning
### Caption Attention Map  
The CAM is defined as a vector representing the weight value of how much the corresponding part of the image is related to a sentence about that part. The information that can be obtained from the image-caption set is an attention area in the image that the sentence describes. Intermediate results from the model in the paper [8] are used to induce a CAM. We refer to this method as KWL after the title of the paper \"Knowing When to Look\" [8]. The KWL derives the attention area associated with each word. We combined this information to provide attention to the entire sentence. An example diagram of this process is represented in Figure 1.  
The attention module in the model that is attached after each step of the RNN highlights the image area and effects when predicting words. An image is divided into \(k\) areas: \(7\times 7\), for a total of 49 areas. Following [8], \(A=\{a^{1},...,a^{k}\},a^{i}\in R^{2048}\) is a spatial image features extracted from ResNet at each of the \(k\) grid locations. A global image feature can be obtained by: \(a_{g}=\frac{1}{k}\sum_{i=1}^{k}a^{i}\). Single layer perceptron and rectifier were used to transform the image feature vectors \(a^{i}\) and \(a_{g}\) into new vectors \(v_{i}\) and \(V_{g}\) with dimension \(d\). The transformed spatial image feature form \(V=[v_{1},...,v_{k}]\). The \(V\) is inserted to attention modules and the \(V_{g}\) is concatenated with the term \(W_{i}x_{i}\) and inserted into each step of LSTM, where \(x_{i}\) and \(W_{i}\) represent an \(i\)th word of the sentence and a weight matrix, respectively. \(h_{t}\) denotes the hidden state of each time step \(t\) in LSTM. The LSTM memory stores both long and short term visual and linguistic information. For each time step \(t\), a visual sentinel \(s_{t}\) provides a latent representation of what the decoder already knows. \(\alpha\in R^{k}\) is the attention weight over features in \(V\). Sentinel gate \(\beta_{t}\) represents the weight of whether to attend the image or to attend the \(s_{t}\) when deriving the word of the step. The \(\beta_{t}\) is derived from the \(s_{t}\), the \(h_{t}\), and an image feature in an attention module, and its value is between 0 and 1. The smaller this value is, the higher the weight that is placed on the image. Conversely, the larger the value of the visual grounding (\(1-\beta_{t}\)) probability, the more information is obtained from the image. The \(\alpha_{t}\) and \(\beta_{t}\) are extracted within an attention module for each step of LSTM. Then, the weighted sum of \(\alpha_{t}\) and (\(1-\beta_{t}\)) for words in the caption is calculated to obtain CAM. The more a word is associated with the information in an image, the larger the weight of the region corresponding to that word in the image. The \(CAM=[cam_{1},...,cam_{k}]\) for one image-caption set can be expressed as:  
\[cam_{i}=\sum_{t=1}^{N}\alpha_{ti}*(1-\beta_{t}),\quad for\;i\in[1,k], \tag{3}\]  
Figure 1: Illustration of the process generating a caption attention map (CAM). An attention module for each step derives attention weight \(\alpha\) and visual grounding (\(1-\beta\)) [8]. The calculation method of the CAM is described in the blue box, which is the weighted sum of \(\alpha\) and (\(1-\beta\)). where \(t\) and \(N\) denote the step index of LSTM and the total number of steps, respectively. The attention of each area is expressed as a value between 0 and 1. Figure 2 represents examples of a generated CAM derived from each caption for images in the training data. In each image, the part corresponding to the main content described in the sentence is highlighted. In the case of (a), the diversity of the ground truth captions is low (Div-1 = 0.321). Div-1 is the diversity evaluation metric and it is described in Section 4.2). All five captions mainly describe \"A woman takes a picture in front of a mirror\", and the words appearing are similar to \"woman\", \"picture\", \"photo\", and \"mirror\". Consequently, similar parts are highlighted in the resulting CAM.