Learning Grasping Possibilities for Artifacts: Dimensions, Weights and Distributional Semantics
Authors: Irene Russo, Irene De Felice (CNR-ILC, Università di Pisa, 2015)
Topic: Distributional Semantics

Abstract
In this paper we want to test how grasping possibilities for concrete objects can be automatically classified. To discriminate between objects that can be manipulated with one hand and the ones that require two hands, we combine conceptual knowledge about the situational properties of the objects, which can be modeled with distributional semantic methodologies, and physical properties of the objects (i.e. their dimensions and their weights), which can be found on the web through crawling.

Introduction
Distributional semantic models of word meanings are based on representations that want to be cognitively plausible and that, as a matter of fact, have been tested to produce results correlated with human judgments when concepts’ similarity and automatic conceptual categorizations are the aim of the experiment (Erk, 2012; Turney and Pantel, 2010). These approaches share the idea that two nominal concepts are similar and can be clustered in the same group if the corresponding lexemes occur in comparable linguistic contexts. Their success is also due to the expectations of the Natural Language Processing (henceforth NLP) community: both for count and predictive models of distributional semantics (Baroni et al. 2014), the core idea is that encyclopedic knowledge packed in a big corpus can improve the performance in tasks such as word sense disambiguation. However, purely textual representations turn out to be incomplete because in language learning and processing human beings are exposed to perceptual stimuli paired with linguistic ones: the old AI dream to ground language in the world requires the mapping between these two sources of knowledge. One of the aim of this paper is to understand how much physical knowledge can be retrieved in language. Can distributional representations of concrete nouns be helpful for the automatic classification of objects, when grasping possibilities are the focus? Could they help to discriminate between objects that can be manipulated with one hand and the ones that require two hands? More generally, how much knowledge about the physical world can be found in language? Inspired by the cognitive psychology literature on the topic, in this paper artifactual categories are theorized as situated conceptualization where physical and situational properties meet (Barsalou 2002). These situational properties describe a physical setting or event in which the target object occurs (as grocery store, fruit basket, slicing, picnic for apple). In an action-based categorization of objects, these kinds of properties function as a complex relational system, which links the physical structure of the object, its use, the background settings, and the design history (Chaigneau et al. 2004). Situational properties can be derived from distributional semantic models, where each co-occurrence vector approximates the encyclopedic knowledge about its referent. A complementary, but more action-oriented idea, is the psychological notion of affordance as the possibilities for actions that every environmental object offers (Gibson 1979). Conceptual information concerning objects’ affordances can be partially acquired through language, considering verb-direct object pairs as the linguistic realizations of the relations between the actions that can be performed by an agent, and the objects involved in those actions. Affordance verbs, intended as verbs that select a distinctive action for a specific object, can be discovered through statistical measures in corpora (Russo et al. 2013). The main assumption of this paper is that the primary affordance for grasping of an artifact largely depends on its physical properties, in particular dimensions and weight. Such features are found in e-commerce websites. Extracting these values for many similar items, for example for all instances of ``plate'', may help to automatically represent average dimensions for that object. However, combining this knowledge with situational properties of objects modeled as distributional semantics vectors can help understanding if they can be combined. This issue is relevant for the implementation of a module that automatically classifies grasping possibilities for objects in embodied robotics. The paper is structured as follow: section 2 reports on the manual annotation of grasping possibilities for a set of 143 artifacts, discussing the definition of the gold standard that will be the dataset for classification experiments in section 3. Section 4 presents conclusions and ideas for future work.

Conclusion
In this paper we test how distributional representations of nouns denoting artifacts can be combined with physical information about their dimensions and weights automatically extracted from an e-commerce website and with co-occurrence information about their affording parts as found in a corpus of do-it-yourself instructions. The starting hypothesis - concerning grasping possibilities as basic manipulative actions for object - was that they are conceptually a combination of situational and physical properties. \\ As a consequence, we expect the best performance from a mixed features models. This hypothesis is not confirmed; for the two-classes dataset (``onehand" VS ``twohand") both physical knowledge and distributional semantics vectors give good resultsbut they don't improve the classifier's performance when combined. \\ These results are in line with the current trend to mix textual and visual features from computer vision algorithms (Bruni et al. 2012) in order to go beyond the limitations of purely textual semantic representations that cannot encode information about colors, dimensions, shapes etc. As future work we plan to integrate the features used for the experiment in this paper with representations of words as bag of visual words derived from the scale-invariant feature transform (SIFT) algorithm (Lowe 1999) that in computer vision helps to detect and describe local features in images.