Abstract: Image description task has been invariably examined in a static manner with qualitative presumptions held to be universally applicable, regardless of the scope or target of the description. In practice, however, different viewers may pay attention to different aspects of the image, and yield different descriptions or interpretations under various contexts. Such diversity in perspectives is difficult to derive with conventional image description techniques. In this paper, we propose a customized image narrative generation task, in which the users are interactively engaged in the generation process by providing answers to the questions. We further attempt to learn the user's interest via repeating such interactive stages, and to automatically reflect the interest in descriptions for new images. Experimental results demonstrate that our model can generate a variety of descriptions from single image that cover a wider range of topics than conventional models, while being customizable to the target user of interaction.

Introduction: Recent advances in visual language field enabled by deep learning techniques have succeeded in bridging the gap between vision and language in a variety of tasks, ranging from describing the image BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 to answering questions about the image BIBREF4 , BIBREF5 . Such achievements were possible under the premise that there exists a set of ground truth references that are universally applicable regardless of the target, scope, or context. In real-world setting, however, image descriptions are prone to an infinitely wide range of variabilities, as different viewers may pay attention to different aspects of the image in different contexts, resulting in a variety of descriptions or interpretations. Due to its subjective nature, such diversity is difficult to obtain with conventional image description techniques. In this paper, we propose a customized image narrative generation task, in which we attempt to actively engage the users in the description generation process by asking questions and directly obtaining their answers, thus learning and reflecting their interest in the description. We use the term image narrative to differentiate our image description from conventional one, in which the objective is fixed as depicting factual aspects of global elements. In contrast, image narratives in our model cover a much wider range of topics, including subjective, local, or inferential elements. We first describe a model for automatic image narrative generation from single image without user interaction. We develop a self Q&A model to take advantage of wide array of contents available in visual question answering (VQA) task, and demonstrate that our model can generate image descriptions that are richer in contents than previous models. We then apply the model to interactive environment by directly obtaining the answers to the questions from the users. Through a wide range of experiments, we demonstrate that such interaction enables us not only to customize the image description by reflecting the user's choice in the current image of interest, but also to automatically apply the learned preference to new images (Figure 1 ).

Question: How they evaluate quality of generated output?
Answer: Through human evaluation where they are asked to evaluate the generated output on a likert scale.