# Bi-LS-AttM: A Bidirectional LSTM and Attention Mechanism Model for Improving Image Captioning
## 3 Methodology
### Architecture Model  
The general layout of the model is illustrated in Figure 4. It is mainly composed of three modules: Fast RCNN for encoding image input, Bi-LS-AttM for encoding sentence input, and embedding picture and caption into common space and decoding it into image captions and evaluation scores.  
The Bi-LS-AttM generates word vectors by comparing similarity using the context information from the frontend and the backend. More accurate words are selected after passing by attention. In our work, the model calculates the front hidden vector \(\overset{\rightarrow}{h}\) and the back hidden vector \(\overset{\leftarrow}{h}\). The front cell starts at \(t=1\), while the back cell starts at \(t=T\). Our framework works such that for an initial input frame I, the encoding is performed as follows:  
\[I_{t}=F(I;\theta_{m}) \tag{9}\]  
\[\overset{\rightarrow}{h_{1}^{\dagger}}=B\left(\overset{\rightarrow}{M}; \theta_{n}\right) \tag{10}\]  
\[\overset{\leftarrow}{h_{1}^{\dagger}}=B\left(\overset{\leftarrow}{M};\theta_ {n}\right) \tag{11}\]  
Figure 3: Bi-LSTM cell structure.  
where \(F\) and \(B\) represent Fast RCNN and Bi-LSTM, respectively. \(\theta_{m}\) and \(\theta_{n}\) are their corresponding weight coefficients. \(\overset{\rightarrow}{M}\) and \(\overset{\leftarrow}{M}\) are forward and backward vectors learned from the neural network, respectively. Afterward, the obtained vectors \(\overset{\rightarrow}{h}\) and \(\overset{\leftarrow}{h}\) are input into attention. The bilinear scoring procedure is applied to calculate the correlation between the query \(q\) and \(\overset{\rightarrow}{h}\) and \(\overset{\leftarrow}{h}\). Next, a SoftMax is applied to these scores to normalize them and obtain the attention distribution \(a=[a_{1},a_{2},\dots,a_{t}]\). The bilinear scoring function and SoftMax are defined as follows:  
\[s\left(\overset{\rightarrow}{h_{1}^{1},h_{1}^{1},q}\right)=\overset{ \rightarrow}{h_{1}^{1}}\overset{\leftarrow}{h_{t}^{1}}Wq \tag{12}\]  
\[a_{t}=\frac{\exp\left(s\left(\overset{\rightarrow}{h_{t}^{1},h_{1}^{1},q} \right)\right)}{\sum_{j=1}^{n}\exp\left(s\left(\overset{\rightarrow}{h_{j}^{1 },h_{j}^{1},q}\right)\right)} \tag{13}\]  
where \(W\) is a trainable parameter matrix. \(s\) is a bilinear function.  
After training the model, it can predict the word \(x_{t}\) with a given image context \(V\) and forward word context \(x_{1:t-1}\), predicted either in a forward direction using \(P(x_{t}|x_{1:t-1},V)\) or in a backward direction using \(P(x_{t}|x_{t+1:t},V)\). For both forward and backward directions, \(x_{1}=x_{T}=0\) is set at the starting point. Finally, for sentences generated from both directions, the last sentence of the given image \(P(x_{1:T}|V)\) is determined by the sum of all words' probabilities in the caption:  
\[p(x_{1:T}\mid V)=max\left(\sum_{t=1}^{T}\left(\overset{\rightarrow}{p}(x_{t} \mid V)\right),\sum_{t=1}^{T}\left(\overset{\leftarrow}{p}(x_{t}\mid V) \right)\right) \tag{14}\]  
\[\overset{\rightarrow}{p}(x_{t}\mid V)=\prod_{t=1}^{T}p(x_{t}\mid x_{1},x_{2}, \dots,x_{t-1},V) \tag{15}\]  
Figure 4: Proposed model architecture.  
\[\overset{\leftarrow}{p}(x_{t}\mid V)=\prod_{t=1}^{T}p(x_{t}\mid x_{t+1},x_{t+2}, \ldots,x_{T},V) \tag{16}\]  
The Bi-LSTM module and its training parameters are similar to those presented in [PERSON] [8]. The difference is that an attention mechanism is added to it. It can focus more on comparing the forward and backward context information to obtain the attention distribution. When extracting features, the Fast RCNN is more efficient and saves time.