Multiple Representations in the Recognition of Faces at Novel Orientations

Bilaterally-symmetric objects such as faces are easily recognized from an orientation that is symmetric with respect to the original view. Several authors have proposed that a simple image-based transformation acting on relatively raw image information can account for this effect. In a series of perceptual and memory experiments, we explore the nature of the abstractions that are used in addition to an image-based representation, and contrast the image-based approach against one based on structural descriptions. We demonstrate that by itself the image-based model is insufficient to account for the recognition of inverted and tilted faces, faces with texture asymmetries, and data from memory experiments where faces are studied in a smooth motion sequence. When used, these additional representations appear more robust against texture asymmetries, stress relative (rather than absolute) location of important features, provide robust generalization to the symmetric orientation view in memory paradigms, and may include local depth information provided by smooth motion. Extensions to other domains beyond symmetric orientation recognition are explored. FACE RECOGNITION AT SYMMETRIC ORIENTATIONS PAGE 3 This article addresses the nature of the representations that allow us to recognize faces at novel orientations. This task is of everyday importance since we can recognize faces from a variety of viewpoints, even under conditions in which we have been exposed to only a single viewpoint of a novel face (e.g., Rock & DiVita, 1987; Tarr & Pinker, 1989; 1990; Bülthoff & Edelman, 1992). When generalizing to novel viewpoints, one particular orientation stands out as having special properties. Several authors, (see Vetter, Poggio & Bülthoff, 1994; Liu, et al., 1995, Troje & Bülthoff, 1996) have pointed out that the symmetric orientation could be used by the observer in conjunction with properties of the human head to enable good generalization and therefore accurate recognition. This explanation takes advantage of the approximate bilateral symmetry of the human head. For example, a head viewed at a 45° angle will appear as an approximate mirror image when viewed at a -45° angle, despite the fact that the other side of the head is now visible (Troje & Bülthoff, 1996). Human observers perform quite well at the symmetric orientation, and in fact perform better than at the frontal view. This strong recognition of this particular novel orientation is known as the symmetric orientation effect. Vetter, Poggio & Bülthoff (1994) and Vetter & Poggio (1994) have pointed out that this process can be accomplished with image-based (also known as view-based) transformations that work in a two-dimensional coordinate system. The salient landmark features are reflected across the vertical meridian and then matched to the face shown at the symmetric orientation. No knowledge of the three-dimensional structure of the head is required, and the technique is specific to bilaterally-symmetric objects. Models that rely on this transformation accurately capture many of the aspects of human recognition, including the fact that the symmetric orientation is recognized better than the frontal view, and that changes in lighting prove devastating to performance, suggesting that humans do not extract enough three-dimensional structure from the head to compensate for lighting changes (Troje & Bülthoff, 1996). FACE RECOGNITION AT SYMMETRIC ORIENTATIONS PAGE 4 This image-based transformation is an extension of a class of theories known as viewbased models, which seek to explain as much of object recognition as possible without resorting to representations other than transformations applied to two-dimensional views. These two-dimensional views are seen as holistic representations that enable matching whole images to images stored in memory. The images in memory are not exact copies of the raw image, since preprocessing allows normalization for location and size on the retina. In addition, a hierarchy of views are possible, such that parts of objects are stored as separate views. However, these representations do not represent relational information except those represented implicitly by the stored image views. Even in the hierarchical view case, no knowledge of the relation between parts in the hierarchy is allowed under the viewbased framework, because the addition of such relational knowledge would change the model into a structural description model (as described below). A vigorous debate has existed between those who support view-based models and those who propose an additional set of processes that extract relational information about the relevant features in an object. This second class of models has been termed structural description models, because the models process relevant features into parts (e.g. geons or other feature primitives) and then represent the relations between these features independently of the features themselves (e.g. Biederman, 1987; Hummel & Stankiewicz, 1996, 1998; Stankiewicz & Hummel, 1996). This represents a critical difference between structural description models and view-based models, because in view-based models the only representation of relational information is implicit in the location of the features in the image. The view based models cannot represent relational information separately from the parts or features of an object. The structural description models are also consistent with the symmetric orientation effect. For example, one instantiation by Hummel & Biederman (1992) proposes that the horizontal relations between relevant features are not coded as 'left-of' or 'right-of', but simply as 'beside'. Thus to the structural description model the original view at 45° would be FACE RECOGNITION AT SYMMETRIC ORIENTATIONS PAGE 5 viewed as nearly identical (barring asymmetries in the head shape) to the -45° view, providing strong symmetric orientation recognition. The structural description model would parse a frontal view with a different set of relations, and thus would have more difficulty matching the frontal view to the original view, thus accurately accounting for the human difficulties with the frontal view. The structural description models might have difficulty accounting for the performance degradation seen with changes in illumination, since the structural description models are designed to be robust against these shading changes. At the very least there is no strong a prior prediction for cases in which illumination changes, while the view-based models clearly predict a decline in performance. In the literature, the extant data has been portrayed as in support of the view-based account, and for several good reasons. First, the view-based models account for the symmetric orientation effect, as well as the disruption in performance brought on by changes in illumination. According to the view-based models, this disruption occurs because observers do not abstract three-dimensional information that would overcome changes in lighting. Second, the view-based approach is parsimonious, in that it can account for the symmetric orientation effects without additional processing of relational information independent of features. Finally, the template-match nature of view-based models is consistent with the general view that faces are processed holistically (Tanaka and Sengco, 1997). Authors of structural description models have remained relatively agnostic about whether structural description models would apply to faces, or remain restricted to object recognition. All of these suggest that view-based models provide a fairly complete account of symmetric orientation effects. To provide a quantitative test of image-based transformation in the view-based approach, Troje (1998) developed an image-based transformation he termed the Symmetry Corrected Image Distance, which computed the distance in pixel difference space between two images for purposes of predicting recognition data. This transformation horizontally flips the original view and performs a pixel-based template match with the test image FACE RECOGNITION AT SYMMETRIC ORIENTATIONS PAGE 6 through a cross-correlation calculation. This image-based transformation predicts the recognition data quite well, accounting for the translation from one orientation to another. The model that includes the correction for symmetry in its calculation of image distance fails only with novel views of studied faces (other than the symmetric orientation) where it tends to reject studied faces at rates much greater rates than humans. These misses suggest that humans generalize to arbitrary novel orientations better than the template match model, and thus the model cannot model all aspects of face recognition. However, the model does capture the essential aspects of the generalization to the symmetric orientation. Troje (1998) notes that additional image-based transformations could be applied, each with an associated cost. For example, scaling, translation, rotation, and compensation for illumination could all be image transformations that could be used to facilitate recognition at the symmetric viewpoint. However, prior work either did not test for these transformations or found that observers did not compensate for changes (as in the case of changes in illumination, see Troje and Bülthoff, 1998), and thus these may not be needed. Despite these successes, conceding victory to the view-based account is perhaps premature. Many of the previous studies on symmetric orientation recognition of faces used testing conditions that may have favored a view-based representation. First, most experiments have used novel faces tested for only brief retention intervals on the order of a few seconds. In the experimental task the observer views two faces sequentially and asked to make a speeded decision about whether the two faces were the same or not. This samedifferent task may have not required anything other