# Middle-Level Attribute-Based Language Retouching for Image Caption Generation
## 2 Related Work
### Attention-Based Methods  
An attention mechanism is a powerful algorithm which is inspired by human's visual attention. It is widely used in CV and NLP. Therefore, various attention-based methods have been proposed for image caption generation.  
The \"hard\" and \"soft\" attention methods, proposed by [23], are used to better understand the visual content of the raw image. In [23], the \"hard\" and \"soft\" attention methods were respectively applied to two different image captioning frameworks. \"Hard\" attention is not dependent on all hidden states of the language model, and the gradient in \"hard\" attention needs to be estimated by Monte Carlo-based sampling. On the other hand, \"soft\" attention is used to calculate the weights of all input feature vectors and to generate an encoded feature vector. Therefore, \"soft\" attention is a parameterized method, and it can be embedded in the language model for training directly. Most existing methods use \"soft\" attention as the basic attention mechanism.  
To solve the problem of the visual attention mechanism, i.e., that it is active all the time, [PERSON]. [24] proposed an adaptive encoder-decoder framework to automatically decide whether to use the attention mechanism or not. In [24], some words could be generated by previous words, such as \"sign\" after \"behind a red stop\". In this case, the attention mechanism is unnecessary. On the other hand, the attention mechanism is active when generating other words which have little relationship with the previously generated words, such as \"of\", \"the\", etc. The shortcoming of the method in [24] is that the attention mechanism might be suppressed all the time when generating image captions.  
Furthermore, to address problems related to object missing and object misjudgment, [PERSON]. [25] proposed a global-Icoal attention (GLA) method to solve these issues during the image captioning process. The advantage of the method in [25] is that the image features are split into two parts, one of which is used as the local image features and is integrated by \"soft\" attention. The drawback of GLA is that the model is not fine-tuned in different datasets. In [26], the authors proposed a novel attention-based model for automatic image captioning named \"Areas of attention\". It can be trained without bounding-box supervision. The contribution of the method in [26] is that the corresponding image areas can be marked when generating words at each time step.  
The authors in [27] proposed a sequence-to-sequence RNN for image caption generation. Different from previous methods, [27] made the input image a sequence of detected objects to generate the corresponding captions by using an attention mechanism. The advantage of the method in [27] is that a sequential attention layer is introduced when generating each word. However, the middle-level attributes of the object were not taken into consideration in [27].  
Furthermore, [PERSON]. [28] proposed a bottom-up and top-down method for image caption and visual question answering (VQA), which also uses an attention mechanism to integrate the objects and other salient image regions. The advantage of the method in [28] is that \"soft\" and \"hard\" attention are combined to achieve image captioning.  
As described above, the attention mechanism is usually used to integrate different pieces of image information during the word generation process. The power of the attention mechanism relies on extracting sufficient image information.