Linking IMAGACT ontology to BabelNet through action videos
Authors: Lorenzo Gregori, Alessandro Panunzi, Andrea Amelio Ravelli (Università di Firenze, 2016)
Topic: Lexical and Semantic Resources and Analysis

Abstract
Herein we present a study dealing with the linking of two multilingual and multimedia resources, BabelNet and IMAGACT, which seeks to connect videos contained in the IMAGACT Ontology of Actions with related action concepts in BabelNet. The linking is based on a machine learning algorithm that exploits the lexical information of the two resources. The algorithm has been firstly trained and tested on a manually annotated dataset and then it was run on all the data, allowing to connect 773 IMAGACT action videos with 517 BabelNet synsets. This linkage aims to enrich BabelNet verbal entries with a visual representations and to connect the IMAGACT ontology to the huge BabelNet semantic network.

Introduction
Ontologies are widely used to represent language resources on the web, allowing them to be easily accessed and exploited by machines. For this reason, data interconnection between different semantic resources is a crucial task in order to enhance disambiguation and information retrieval capabilities in Artificial Intelligence, as evidenced by the increasing research into mapping and linking techniques among ontologies AUTHOR. Nevertheless, ontology mapping has to face the problem of concept representation mismatch between resources, due to different building criteria and purposes AUTHOR. Instance matching techniques play an important role in this context, allowing to connect entities from heterogeneous data resources which refer to the same real-world object AUTHOR. Aside the general interest for knowledge bases interconnection in a web-based perspective, there is also a growing interest in multimodal resources, which combine textual and visual data. These resources can be exploited by intelligent algorithms integrating vision and natural language processing techinques}. This integrated approach was successfully applied for some challenging tasks involving verbs and their action reference as a video. Regneri et al. \shortcite{TACL502} developed machine learning models for the automatic identification of similarity among actions, by using a corpus of natural language descriptions, derived from the videos of the MPII Cooking Composite Activities dataset, which represents actions involved in basic cooking tasks. Instead, the algorithm developed by Mathe et al. \shortcite{mathe4563042} extracts higher level semantic features in common among a sample set of verbs, using a fine-grained analysis of the represented action concepts, intended as a subsequent stable set of abstract features of the objects involved in the videos. Within this interdisciplinary perspective, a knowledge base which relates verbal lemmas in different languages with video prototypes can help in serveral applications, and be exploited by both humans and machines.

Conclusion
The experiment described in this paper shows that is possible to obtain an extensive linking between IMAGACT and BabelNet through visual entities (see Figure fig for a visual representation of a linking example); this can be advantageous for both the resources. BabelNet can add a clear video representation of the verbal synsets that refer to actions; IMAGACT can import verb translation candidates from many languages by exploiting the BabelNet semantic network; their integration can be exploited as a unified multimedial resource to accomplish complex tasks that combine natural language processing and computer vision. Finally, we feel important to note that this procedure is scalable and the statistical model can be retrained at resource changes. This is a fundamental feature, especially considering the continuous update of IMAGACT languages and lemmas inventory. This research has been supported by the MODELACT Project, funded by the Futuro in Ricerca 2012 programme (Project Code RBFR12C608);