Deep Learning for Automatic Image Captioning in poor Training Conditions
Authors: Caterina Masotti, Danilo Croce, Roberto Basili (Università di Roma Tor Vergata, 2017)
Topic: Language Models

Abstract
Recent advancements in Deep Learning show that the combination of Convolutional Neural Networks and Recurrent Neural Networks enables the definition of very effective methods for the automatic captioning of images. Unfortunately, this straightforward result requires the existence of large-scale corpora and they are not available for many languages. This paper describes a simple methodology to automatically acquire a large-scale corpus of 600 thousand image/sentences pairs in Italian. At the best of our knowledge, this corpus has been used to train one of the first neural systems for the same language. The experimental evaluation over a subset of validated image/captions pairs suggests that results comparable with the English counterpart can be achieved.

Introduction
The image captioning task consists in generating a brief description in natural language of a given image that is able to capture the depicted objects and the relations between them, as discussed in AUTHOR. More precisely, given an image I as input, an image captioner should be able to generate a well-formed sentence {S(I)=(s_1,...,s_m)}, where every s_i is a word from a vocabulary V=\{w_1,...,w_n\} in a given natural language. Some examples of images and corresponding captions are reported in Figure fig:mscoco_images. This task is rather complex as it involves non-trivial subtasks to solve, such as object detection, mapping visual features to text and generating text sequences. Recently, neural methods based on deep neural networks have reached impressive state-of-the-art results in solving this task AUTHOR. One of the most successful architectures implements the so-called encoder-decoder end-to-end structure AUTHOR. Differently by most of the existing encoder-decoder structures, in AUTHOR the encoding of the input image is performed by a convolutional neural network which transform it in a dense feature vector; then, this vector is ``translated'' to a descriptive sentence by a Long short-term memory (LSTM) architecture, which takes the vector as the first input and generates a textual sequence starting from it. This neural model is very effective, but also very expensive to train in terms of time and hardware resources on a dataset of over 580,000 image-caption examples takes about two weeks even with a very performing GPU.}, because there are many parameters to be learned; not to mention that the model is overfitting-prone, thus it needs to be trained on a training set of annotated images that is as large and heterogeneous as possible, in order to achieve a good generalization capability. Hardware and time constraints do not always allow to train a model in an optimal setting, and, for example, cutting down on the dataset size could be necessary: in this case we have poor training conditions. Of course, this reduces the model's ability to generalize on new images at captioning time. Another cause of poor training conditions is the lack of a good quality dataset, for example in terms of annotations: the manual captioning of large collections of images requires a lot of effort and, as of now, human-annotated datasets only exist for a restricted set of languages, such as in English. As a consequence, training such a neural model to produce captions in another language (e.g. in Italian) is an interesting problem to explore, but also challenging due to the lack of data resources. A viable approach is building a resource by automatically translating the annotations from an existing dataset: much less expensive than manually annotating images, but of course it leads to a loss of human-like quality in the language model. This approach has been considered in this work to perform one of the first neural-based image captioning in Italian: more precisely, the annotations of the images from the MSCOCO dataset, one of the largest datasets in English of image/caption pairs, have been automatically translated to Italian in order to obtain a first resource for this language: this has been exploited to train a neural captioner and whose quality can be improved over time (e.g., by manually validating the translations). Then, a subset of this Italian dataset has been used as training data for the neural captioning system defined in AUTHOR, while a subset of the test set has been manually validated for evaluation purposes. In particular, prior to the experimentations in Italian, some early experiments have been performed with the same training data originally annotated in English, to get a reference benchmark about convergence time and evaluation metrics on a dataset of smaller size. These results in English will suggest if the Italian image captioner shows similar performance when trained over a reduced set of examples or the noise induced in the automatic translation process compromises the neural training phase. Moreover, these experiments have also been performed with the introduction of a pre-trained word embedding, (derived using the method presented in AUTHOR), in order to measure how it affects the quality of the language model learned by the captioner, with respect to a randomly initialized word embedding that is learned together with the other model parameters. Overall the contributions of this work are threefold: (i) the investigation of a simple, automatized way to acquire (possibly noisy) large-scale corpora for the training of neural image captioning methods in poor training conditions; (ii) the manual validation of a first set of human-annotated resources in Italian; (iii) the implementation of one of the first automatic neural-based Italian image captioners. In the rest of the paper, the adopted neural architecture is outlined in Section sec:arch. The description of a brand new resource for Italian is presented in Section sec:resource. Section sec:exp reports the results of the early preparatory experimentations for the English language and then the ones for Italian. Finally, Section sec:concl derives the conclusions.

Conclusion
In this paper a simple methodology for the training of neural models for the automatic captioning of images is presented. We generated a large scale of about 600,000 image captions in Italian by using an automatic machine translator. Although the noise introduced in this step, it allows to train one of the first neural-based image captioning systems for Italian. Most importantly, the quality of this system seems comparable with the English counterpart, if trained over a comparable set of data. These results are impressive and confirm the robustness of the adopted Neural Architecture. We believe that the obtained resource paves the way to the definition and evaluation of Neural Models for Image captioning in Italian, and we hope to contribute to the Italian Community, hopefully using the validated dataset in a future Evalita} champaign.