Online Automatic Post-Editing across Domains
Authors: Rajen Chatterjee, Gebremedhen Gebremelak, Matteo Negri, Marco Turchi (Fondazione Bruno Kessler, 2016)
Topic: Language Models

Abstract
%learn correlations between linguistic and visual features. On the %other hand, the same systems have struggled with questions that %require deeper reasoning. In particular, "number questions' have been %associated with low performance, and only been studied %surfacely. In this paper, we focus on linguistic questions over images which may be answered with a quantier (e.g. How many dogs are black? Some/most/all of them, etc.). We show that in order to learn to quantity, a multimodal model has to obtain a genuine understanding of linguistic and visual inputs and of their interaction. We proposed a model that extracts a fuzzy representation of the set of the requested objects (e.g. dogs) and of the requested property in relation to that set (e.g. black with respect to dogs), outputting the appropriate quantier for that relationship. %this challenging task.

Introduction
Computational linguistics and artificial vision systems are going through a particularly favourable time, well represented by the development of multimodal models capable of carrying out tasks that seemed out of reach until a few years ago, and that are now integrated in many applications intended for use by users. The task of 'Visual Question Answering' (VQA), aimed at answering questions from visual inputs, and the automatic generation of captions ('caption generation') are only some of the tasks in which most is witnessed a rapid and unstoppable improvement. In addition to direct applications in the real world, where visual content plays a crucial role, the performance of these language and vision models is also an indicator of the extent to which they capture meaning, providing us with important theoretical insights. Recent improvements in the field of VQA applications are mainly due to the new generation of artificial deep learning networks ('deep learning neural networks'), along with the growing availability of large image datasets. These models have shown that even very simple neural networks (RN) can capture complex interactions between the properties of a dataset, often surpassing in performance models much more complex. For example, AUTHOR has recently shown that a simple bag-of-words model (BoW) can achieve cutting-edge performance in one of the most important VQA AUTHOR datasets. The authors of the study, however, object that the performance of the model is to be attributed more to the excellent ability of the network to store correlations than to a real capacity of reasoning and understanding, and conclude by supporting the need to move to the second objective. Their study also shows how difficult it is to assess whether a model is capable of actually grasping the meaning of images and words. In this work we propose a task that we believe to be useful for the community in order to achieve this objective. In VQA's task, model assessments are based on questions concerning the properties of specific objects, such as location, color, etc., while in the task of generating captions the entire image is taken into account. In both cases, it is obviously possible to learn correlations between the words present in the question, the words present in the answer and the image itself (or parts of it). Presented to an RN model along with the question What is eating?, for example, the image of a person who eats an active cake is definitely the association between the verb eat and properties (both linguistic and visual) connected to the food, producing the correct answer. In order to test a model beyond this simple mechanism, it emerges the need to propose a task in which a particular association between demand and image does not always give the same answer as a result. Questions concerning the "number' have been studied only marginally in literature, and the performance of VQA systems on these questions has been shown to be quite poor AUTHOR. In addition, the previous works focused almost exclusively on the modelling of exact cardinals, thus analyzing only partially the phenomenon of quantification. In this article we investigate a new type of question that requires a certain degree of understanding of the generalized quantifyors (few, most, all, etc.). The reason why we are interested in these questions is that, in order to answer you, it is not enough to identify the area of an image related to the question. The question In what proportion are dogs black?, for example, requires something more than identifying the dog and black properties in the image: the net must be able to reflect on the relationship between these properties, and to generate one of the quantifyors, which are potentially similar to each other. The ability to identify the members of a set and their shared properties requires a certain degree of deeper reasoning which, we argue, cannot be obtained by a simple storage mechanism. In the following section, we consider the task of VQA as a problem of "fill in the blank' (fill an empty space with the correct word), and ask the question In what proportion are dogs black? in the following form: \underline{\hspace{5mm} dogs are black}. The possible answers are selected within a set of linguistic quantifiers, i.e. no, few, some, most and all. To assign the correct quantifier, the model must be able to focus on the relevant objects (the quantifier shrink) and to quantify the objects that, within this restricted domain, possess the required property (the quantifier range). We show that a simple BoW model is not enough to perform this task effectively, and we propose an alternative and linguistically motivated RN model, the performance of which is superior to the AUTHOR model.

Conclusion
The RNQ neural network model achieves 38.9% accuracy compared to 30.8% of the iBOWIMG model. The RNQ model is very good with "no" and "all" and is quite good with "few" and "most." But its accuracy decreases by "some" (18.16%), adversely affecting the overall result. A different analysis instead emerges from the results of the other model, which is quite good in predicting "no" and "all," but it is very bad in other cases. For it, "few", "most" and "some" are equally difficult to learn. Its errors in particular in learning "few" and "most" (see Table 1). These quantifiers require a more precise understanding of the proportions between name and property than "no" and "all" require. We think, therefore, that this demonstrates the limitations of a model that does not learn to abstract and elaborate proportions. The RNQ model makes understandable mistakes, confusing, for example, "few" with "no" but not with "some", "most" and "all." This shows that this model has learned to quantify and not simply keep track of correlations in the dataset. On the contrary, the other model confuses "few" with "no" and "all" in equal number. In general, the RNQ model appears to be rather immune to possible effects related to the frequency with which each of the 24 properties is required in the dataset. One might think that very frequent combinations are better learned than infrequent combinations, but this is only true for a few cases, on which stand out "red" and "rectangular." In most cases, the model behaves in a similar way, in terms of right/wrong responses, in the face of little or very frequent combinations. In the case of some colors (black, blue and violet), we notice a higher accuracy for infrequent combinations than in very frequent cases (see Figure 1). We can say that the performance of the RNQ model does not depend on the frequency effects of the combinations present in the dataset. Again, therefore, it is not a question of learning correlations present in the dataset, but to carry out the task of quantification a deeper understanding of the visual scenario and the demand is necessary.