# Middle-Level Attribute-Based Language Retouching for Image Caption Generation
## 3 Materials and Methods
### Middle-Level Attributes Prediction  
In this part, we mainly focus on predicting the middle-level attribute labels from object regions, which can be used to retouch the intermediate image description generated by our language generation model. The process can be broadly divided into three steps, including extracting the object regions, predefining the middle-level attributes, and training and applying the middle-level attribute predictors.  
From the above subsection, we know that the object features and object regions of the raw image can be generated simultaneously by the faster-RCNN model. Hence, we used the object regions as the raw data to predict the valuable middle-level attribute labels.  
Since the categories of the objects contained in the raw image can be recognized by faster-RCNN, we only need to predict the middle-level attribute labels of these objects. We roughly used some valuable middle-level attributes of human and non-human objects predefined in previous works [36; 37].  
As described in [36], the images in the PubFig dataset consist of positive and negative examples, all of which have been labelled with \"0\" or \"1\" for each attribute to indicate whether an attribute exists or not. The attributes are in [36], including expression, lighting, scene, etc. Thus, the PubFig dataset can also be used for training binary classifiers to recognize the presence or absence of describable aspects of visual appearance, such as gender, age, etc.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Categories** & **Optional Values** \\ \hline Objects & child, man, woman, desk, polar bear, rock, cat... \\ Object regions & the corresponding bounding-box: (\(P_{x}\), \(P_{y}\), \(P_{w}\), \(P_{h}\)) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Possible objects and object regions contained in raw images.  
Therefore, the attribute labels used to retouch a human's appearance in our research are a subset of those used in [36]. The details of the human attributes are shown in Table 2, including gender, age, and hair color.  
The ImageNet dataset [35] contains a subset [37] which has 9600 images collected from 384 synsets, and each image is paired with 25 object attributes. The 25 semantic appearance attributes in [35] can be divided into four categories, including \"color\", \"pattern\", \"shape\", and \"texture\". Hence, the middle-level attributes of non-human objects used in this research are the semantic appearance attributes, which are similar to those in [37] and are illustrated in Table 3.  
Finally, we combined multiple attribute predictors into an ensemble attribute predictor to predict the valuable middle-level attributes from the object regions. We used the VGG16 [3] model as the basic classifier, which was trained six times separately for different aims, as shown in Table 4.  
We used the PubFig dataset to train our attribute predictors VGG16(GENDER), VGG16(AGE), and VGG16(HAIR COLOR) to predict human attributes. The attribute predictors VGG16(SHAPE), VGG16(COLOR), and VGG16(TEXTURE) were pre-trained on the subset of the ImageNet dataset to recognize the middle-level attributes of non-human objects.  
After that, these pre-trained attribute predictors were combined into an ensemble attribute predictor to predict the middle-level attributes from the object regions generated by faster-RCNN. It is worth noting that only when the probability of a predicted attribute is greater than a predefined threshold is the predicted attribute applied to retouch the intermediate image description. Here, the threshold was set to 70%.