Perceptual Operations and Relations between 2D or 3D Visual Entities

AbstractIn this paper, we present a set of perceptual relations, namely, co-colority, co-planarity, collinearityand symmetry that are deﬁned between multi-modal visual features that we call primitives. 1 Introduction According to Marr’s paradigm [29], vision involves extraction of meaningful representations from inputimages, starting at the pixel level and building up its interpretation more or less in the following order: localﬁlters, extraction of important features, the 2 12 -D sketch and the 3-D sketch.There is psychophysical evidence and evidence from the statistical properties of natural images that the hu-man visual system utilizes a set of visual-entity-combining processes, called perceptual organization in theliterature,forformingbigger,sparserandmorecompleteinterpretationsofthescene(see,e.g.,[18,19,35]). Suchprocessesinclude(i)extractionoftheboundaryoftheobjectsintheimagefromthesetofunconnectededge pixels or features [3, 8, 10, 21, 27, 31, 39] utilizing Gestalt laws of grouping, and (ii) interpolation andextrapolation of unconnected sparse 3D entities for forming more complete 3D surfaces (see, e.g., [13]) uti-lizing the relations between the 3D entities. Gestalt principles include collinearity, proximity, common fateand similarity whereas inference of 3D surfaces from a set of 3D entities include relations like coplanarity,collinearity, co-colority etc. These are essentially second order and higher order relations of local features.In [26], we have introduced a speciﬁc form of a local descriptor that we call a ’multi–modal primitive’(see section 2) and which can be seen as a functional abstraction of a hypercolumn (see [24]). We distin-guish between 2D primitives describing local image information and 3D primitives covering local 3D sceneinformation in a condensed symbolic way.These primitives serve as a basis for an early cognitive vision system [23, 26, 33] in which operations andrelations on these primitives realizing perceptual grouping principles are used in different contexts (see [26]for applications). We have utilized these relations for different problems including stereo [34], RBM [32],estimation of initial grasping reﬂexes from stereo [5], estimation of depth at homogeneous image structures[16], and analysis of second-order relations between 3D features [17]. In this paper, we present the set of 2D and 3D relations deﬁned upon the primitives. These relations includecollinearity, cocolority, coplanarity and symmetry. Of these relations, collinearity, cocolority and symmetryare deﬁned for 2D as well as 3D primitives whereas by deﬁnition, coplanarity is meaningful only for 3Dprimitives. Table 1 summarizes the relations and on which dimension they are deﬁned. Relation 2D 3Dco-planarity ×√co-colority√ √collinearity√ √symmetry√ √Table 1: The relations and in which dimension they are deﬁned. This paper does not focus on any speciﬁc application domain but provides a technically detailed deﬁnitionof these relations that are usually not described in such detail in publications making use of them.The paper is organized as follows: In section 2, we brieﬂy introduce our visual features, namely primitives.In section 3, we describe our deﬁnitions of perceptual relations between the visual primitives. In section 5,we conclude the paper.1