# Controllable Image Captioning with Feature Refinement and Multilayer Fusion
## 1 Introduction  
Image captioning is a complex task of automatically producing natural language sentences to describe the content of a given image. It requires not only understanding the relationship between each object, but also generating sentences to describe the most conspicuous objects. In the early stages, the image captioning task is based on templates [1, 2] or retrieval [3, 4]. In recent years, with the rapid development of deep learning [5, 6], most current image captioning methods have been based on deep learning and adopted an encoder-decoder structure, in which the encoder extracts features to represent the content of the image. The decoder uses the extracted features to generate a description. The latest caption model even achieved better performance than humans in all accuracy-based metrics. However, these established models tend to generate sentences by describing the most conspicuous objects, but fail to describe a desired region or object as human. It is essential for practical applications. For example, when assisting visually impaired people to walk, the generated caption should describe what is on the road or the color of traffic lights. Meanwhile, many studies have indicated that traditional models tend to produce generic descriptions to capture frequent descriptive patterns, but fail to describe particular objects. To endow captioning models with controllability, several models introduce extra control signals to generate captions, called controllable image captions (CIC). The CIC model can easily generate diverse captions for the same image by feeding different control signals. One type of CIC [7, 8, 9] focuses on controlling describing styles,such as factual, sadly, and happy; the other type aims to control the content, such as region [10], object [11; 12], part-of-speech tags [13; 14], and length level [15]. To produce controllable object image captioning, [PERSON] et al. [11] proposed a model to control the content and the order of the image caption explicitly grounded on a sequence of image regions. [PERSON] et al. [16] proposed a control signal that represents a targeted activity as a verb and some entities involved in this activity as semantic roles. Many researchers [12; 14; 15] work in this direction. In current models, the accuracy of the generated sentences depends on the accuracy of the understanding of the object role. However, the object feature obtained by the detection model lacks prior information. For example, given the features of a man and a baseball, it would be difficult to infer their relationship and the concept of a player. Besides, most of the models adopt cross-entropy loss, which leaves a lack of diversity in the generated sentences. Based on the above problems, this paper introduces a framework based on the information-augmented and node-relation estimation network (IANR) to improve the performance of controllable image captioning. This method is an encoder-decoder structure. The information-augmented graph encoder consists of an information-augmented embedding module and a multi-relational weighted graph convolution network (MR-WGCN). The information-augmented embedding module is designed to add prior information for objects and relationships. The MR-WGCN emulates the message passing from one node to others in different ways. In terms of the decoder, this paper designs a model that dynamically pays attention to control signals or features with prior information. To further increase the diversity of descriptions, an additional similarity loss is added to the traditional cross-entropy loss. The main contributions are summarized as follows:  
* The proposed information-augmented embedding module adds prior information for each object and relation node. * A Multi-Relational Weighted Convolution Graph (MR-WGCN) is proposed to aggregate messages from related nodes in different ways.