Exploiting Spatial Relational Knowledge for Visual Cognitive Tasks

Exploiting Spatial Relational Knowledge for Visual Cognitive Tasks Medhat M. Riad (medhat.riad@phil.uu.nl) Utrecht University, Heidelberglaan 6 3584 CS Utrecht, The Netherlands Florian R¨ohrbein (roehrbei@informatik.uni-bremen.de) University of Bremen, Enrique-Schmidt-Str. 5 28359 Bremen, Germany Nils Einecke (nils.einecke@honda-ri.de) Julian Eggert (julian.eggert@honda-ri.de) HONDA Research Institute Europe GmbH, Carl-Legien-Str. 30 63071 Offenbach am Main, Germany Abstract Reasoning about objects in a visual scene can be substantially improved if we have a representation that includes both in- formation about the object properties and information about the relations that hold between them. For this, we have built a semantic-relational network that makes use of textual com- monsense knowledge of both sorts and allows for inference within a Bayesian framework (R¨ohrbein, Eggert, & K¨orner, 2007). In order to enrich this graphical representation, we tar- get at adding complementary information gained, for example, from sets of labeled images. We are especially interested in re- lational knowledge, a demand that rules out most existing im- age databases since they usually contain only one labeled ob- ject per image. LabelMe proved as a promising alternative and we started our efforts by analyzing the statistical dependencies between all objects in fully-labeled images of that database. In the contribution here, we report these results and discuss how the gained information can be used to build contexts as a first important step in reaching the final goal of learning a relational knowledge representation in an autonomous way. Keywords: inter-object relations; spatial context; labeled im- ages Introduction Visual scenes contain a wealth of information which would be very useful for technical systems. The complexity of visual information and the processing and storage costs involved in acquiring any form of this information that is directly usable, makes the use of visual information very challenging. This is further complicated by the problem of deciding what kind of information is useful and what is simply superfluous. How- ever, a lot of research has been carried out over the years to obtain useful information from visual scenes whether it be high-level, global context information (Fei-Fei & Perona, 2005), object recognition (Wersing & K¨orner, 2003), or low- level image processing for feature detection (Lowe, 2004). In this paper, we report on preliminary results of statistical anal- ysis on images at an intermediate-level where object informa- tion is assumed to be readily available showing that there are some statistical regularities in visual scenes at a level below that of global cues (like texture or gist models) and above that of local analysis (such as edge detection or single objects). Working at object-level means that annotated images must be available. Some of the common analysis techniques in- clude object positions, percentage area covered by object, and object complexity (boundary points). However, these tech- niques do not scale very well with number of images and if any regularities are seen then they are more representative of artistic preferences of photographers than actual object prop- erties in real-world scenes (e.g. photographs of airplanes will usually be centered on the plane when obviously an airplane is seldom in the center of the visual field in everyday scenes). For this reason, it is our belief that analyzing object-object relations in images rather than object-scene relations would reduce the effect of photography bias and, as shown in the results, could detect some regularities in spatial relations be- tween these objects. The following section will give a brief introduction to the LabelMe database that we use in our analysis followed by an overview of region connection calculus, of which we used a modified version to describe relations between objects in im- ages. In the results section, we report on some of the interest- ing results we found on different levels of detail. Following this, we dedicate the final section to discussing the results and how to interpret them as well as present the problems faced and how these problems affected the results. There is also a subsection discussing related work and the reported results and the discussion section ends by briefly talking about future plans and incorporating this information into our semantic- relational network. Method Annotated images One of the main requirements for this research is to have a large database of images with multiple annotated objects per image. Even though there are several publicly available an- notated image data-sets, most of them either have too few images for our purpose or contain only one object per image. Such single-object images are very useful as training data for object recognition but are useless for trying to uncover regu- larities in larger, complex, multi-object scenes. For this pur- pose, we use LabelMe (Russel, Torralba, Murphy, & Free- man, 2008) which is an on-line, publicly accessible database which depends on Internet users to upload and annotate im- ages.