A Comparison of Representation Models in a Non-Conventional Semantic Similarity Scenario
Authors: Andrea Amelio Ravelli, Oier Lopez de Lacalle, Eneko Agirre (Università di Firenze, University of the Basque Country, 2019)
Topic: Language Models

Abstract
Representation models have shown very promising results in solving semantic similarity problems. Normally, their performances are benchmarked on well-tailored experimental settings, but what happens with unusual data? In this paper, we present a comparison between popular representation models tested in a non-conventional scenario: assessing action reference similarity between sentences from different domains. The action reference problem is not a trivial task, given that verbs are generally ambiguous and complex to treat in NLP. We set four variants of the same tests to check if different pre-processing may improve models performances. We also compared our results with those obtained in a common benchmark dataset for a similar task.

Introduction
Verbs are the standard linguistic tool that humans use to refer to actions, and action verbs are very frequent in spoken language (∼50% of total verb occurrences) (Moneglia and Panunzi, 2007). These verbs are generally ambiguous and complex to treat in NLP tasks, because the relation between verbs and action concepts is not one-to-one: e.g. (a) pushing a button is cognitively separated from (b) pushing a table to the corner; action (a) can also be predicated through press, while move can be used for (b) and not vice-versa (Moneglia, 2014). These represent two different pragmatic actions, despite the verb used to describe it and all the possible objects that can undergo the action. Another example could be the ambiguity behind a sentence like John pushes the bottle: is the agent applying a continuous and controlled force to move the object from position A to position B, or is he carelessly shoving an object away from its location? These are just two of the possible interpretations of this sentence as is, without any other lexical information or pragmatic reference. Given these premises, it is clear that the task of automatically classifying sentences referring to actions in a fine-grained way (e.g. push/move vs. push/press) is not trivial at all, and even humans may need extra information (e.g. images, videos) to precisely identify the exact action. One way could be to consider action reference similarity as a Semantic Textual Similarity (STS) problem (Agirre et al., 2012), assessing that lexical semantic information encodes, at a certain level, the action those words are referring to. The simplest way is to make use of pre-computed word embeddings, which are ready to use for computing similarity between words, sentences, and documents. Various models have been presented in the past years that make use of well-known static word embeddings, like word2vec, GloVe, and FastText (Mikolov et al., 2013; Pennington et al., 2014; Bojanowski et al., 2017). Recently, the best STS models rely on representations obtained from contextual embeddings, such as ELMO, BERT, and XLNet (Peters et al., 2018; Devlin et al., 2018; Yang et al., 2019). In this paper, we are testing the effectiveness of representation models in a non-conventional scenario, in which we do not have labeled data to train STS systems. Normally, STS is performed on sentence pairs that, on one hand, can have very close or distinct meaning, i.e. the assertion of similarity is easy to formulate; on the other hand, all sentences derive from the same domain, thus they share some syntactic regularities and vocabulary. In our scenario, we are computing STS between textual data from two different resources, IMAGACT and LSMDC16 (described respectively in 5.1 and 5.2), in which the language used is highly different: from the first, synthetic and short captions; from the latter, audio descriptions. The objective is to benchmark word embedding models in the task of estimating the action concept expressed by a sentence.

Conclusion
In this paper we presented a comparison of four popular representation models (one-hot encoding, GloVe, BERT, USE) in the task of semantic textual similarity on a non-conventional scenario: action reference similarity between sentences from different domains. In the future, we would like to extend our Gold Standard dataset, not only in terms of dimension (i.e. more LSMDC16 video-caption pairs annotated with acs from IMAGACT), but also in terms of annotators. It would be interesting to observe to what extent the visual stimuli offered by video prototypes can be interpreted clearly by more than one annotator, and thus calculate the inter-annotator agreement. Moreover, we plan to extend the evaluation to other representation models as well as state-of-the-art supervised models, and see if their performances in canonical tests are confirmed on our scenario. We would also try to augment data used for this test, by exploiting dense video captioning models, i.e. videoBERT (Sun et al., 2019).