# Bi-LS-AttM: A Bidirectional LSTM and Attention Mechanism Model for Improving Image Captioning
## 1 Introduction  
Image captioning is a hot topic involving several fields such as computer vision (CV) and natural language processing (NLP), known as image semantic description or \"talking about pictures\" [1; 2; 3; 4; 5; 6]. Image captioning technology not only needs to recognize the entity object information in the image and the relationship between objects but also needs to learn how to integrate them into the ability to describe reasonable sentence descriptions. Traditional methods use models based on visual space search, sentence template usage, and the most matching sentence in the dataset to accomplish the tasks of image captioning. The disadvantage of these methods is the low efficiency of generating real and accurate sentences and the poor ability to generate structurally novel sentences. In recent research [7; 8; 9; 10; 11], visual and language information has been embedded into a common space via recurrent neural networks (RNNs) initially. Convolutional neural networks(CNNs) were then embedded within the visual space and combined with long short-term memory (LSTM) to produce more effective results. Most models extract image features by embedding the CNN into visual space. While this method can achieve good results, the extracted image features are not highly accurate and efficient, wasting a lot of time. Many models embed LSTM and Bi-LSTM into language space to generate sentences, but the results are not accurate enough. Therefore, it is challenging for subtitling models to perform novel subtitling tasks with accurate and efficient image-sentence retrieval. To address these issues, we propose a model leveraging a bidirectional LSTM coupled with an attention mechanism (Bi-LS-AttM). This innovative model substitutes the region convolutional neural network (RCNN)--commonly used for feature extraction--with a more efficient fast region convolutional neural network (Fast RCNN). This adjustment enhances the identification and extraction of features within the image's regions of interest (Rols). The optimized model is then applied to refine the LSTM network's performance. By juxtaposing forward and backward outcomes and incorporating the attention boost, the Bi-LS-AttM is able to predict word vectors with greater precision and generate more fitting image captions. Why do we use the model? We employed the model to break through the boundaries of the traditional Bi-LSTM model, which is not focused enough on the comparison of historical and future word results. In the traditional LSTM cells, the prediction of the next word \(x_{t}\) using the visible context \(V\) and historical context \(x_{1:t-1}\) is performed by estimating \(\log P(x_{t}|V,x_{1:t-1})\). However, in the Bi-LS-AttM, the prediction of the word \(x_{t}\) depends on the forward and backward results of separately maximizing \(\log P(x_{t}|V,x_{1:t-1})\) and \(\log P(x_{t}|V,x_{t+1:t})\) at time \(t\). By combining the Bi-LSTM with the attention model, the model focuses increasingly on comparing historical and future word results and using their dependencies to predict and generate appropriate image captions. Figure 1 shows the example image of the Bi-LS-AttM model generating a sentence that supports our hypothesis that the Bi-LS-AttM model can generate more complementary and focused captions. We tested the efficiency of our model on the datasets Flickr30K and MSCOCO and performed a qualitative analysis. The analysis showed that the method performs efficiently, and the proposed Bi-LS-AttM model outperforms other published models. The principal contributions of this paper are threefold:  
* We proposed a trainable model incorporating a bidirectional LSTM and attention mechanism. This model embeds image captions and scores into a region by capitalizing on the long-term forward and backward context. * We upgraded the feature extraction mechanism, replacing the conventional CNN and RCNN with a Fast RCNN. This improvement enhances the model's ability to rapidly detect and extract features from items within an image's regions of interest.