# Middle-Level Attribute-Based Language Retouching for Image Caption Generation
## 1 Introduction  
Image caption generation is attractive research in the field of artificial intelligence and has emerged in recent years. It is an interdisciplinary subject combining computer vision (CV) and natural language processing (NLP). It focuses on generating a readable sentence to depict the visual content of a given image. The representative applications of image caption generation are assisting visually impaired people to perceive the visual content of the surrounding scenes, providing visual intelligence to chatting robots, image retrieval, and semantic visual research. Therefore, image caption generation is a significant part of scene understanding. Benefiting from the rapid development of deep learning, many novel methods have been proposed for image caption generation in recent years. Researchers have used convolution neural networks (CNNs) [1] to extract the visual information of a given image, and to generate the corresponding language description by a decoder, such as a recurrent neural network (RNN) [2]. Briefly, most methods of image caption generation are based on an encoder-decoder framework. CNN-based models are the most common encoders, such as VGG16 [3] and ResNet-101 [4]. Meanwhile, RNN-based models are commonly used encoders, such as Gated Recurrent Unit (GRU) [5], Bidirectional RNNs (Bi-RNNs) [6], Long Short-Term Memory cell (LSTM) [7], etc. However, most methods for image caption generation based on the encoder-decoder framework have an obvious limitation. The image feature extracted by CNNs is a fixed-length feature vector,which may lose significant identification information of objects. For example, the attention-based methods of image captioning, which use an attention mechanism to decide which part of the input information is important, cannot fully exploit the role of the attention mechanism. The contribution of an attention mechanism is that it can fuse several feature vectors by assigning a weight for each input feature vector, and then generating one fused feature vector. Nonetheless, the attention mechanism may weaken the role of some significant information, such as the shape of objects. The attribute-based methods of image caption generation extract high-level semantic attributes from the raw image, such as \"running\", \"standing\", \"birds\", etc., and then directly inject them into the language generation model. The obvious problem with these methods is that they ignore the role of the middle-level attributes of objects contained in the raw image, such as color, texture, etc. Therefore, we propose a new middle-level attribute-based language retouching (MLAR) method for image caption generation to solve the problem discussed earlier. Our MLAR method is based on an encoder-decoder framework. As opposed to other methods, MLAR extracts the middle-level attributes from the object regions and then uses them as supplementary information to retouch the intermediate image caption generated by our language generation model. It is worth noting that the middle-level attributes are not included in the input information of our attention mechanism. We tried to separate the middle-level attributes and use sequential means to generate the final image caption. One advantage of the sequential method is that the role of middle-level attributes will not be weakened by the attention mechanism. After that, the well-described natural language sentence of the raw image can be generated by our method. The method we propose consists of four key parts: (1) image feature extraction; (2) middle-level attribute prediction; (3) language generation model; and (4) language retouching of the intermediate image caption. The first two parts of our method are used to extract valuable information from the raw image. The last two parts of our method mainly focus on generating the intermediate image caption and retouching it, respectively. In the first part of the study, we used the ResNet-101 model to extract the global image feature, since it can extract accurate high-level semantic features compared with VGG16.