Modeling Visual Classification using Bottom-up and Top-down Fixation Selection

Modeling Visual Classification using Bottom-up and Top-down Fixation Selection Joyca P. W. Lacroix (jlacroix@fsw.leidenuniv.nl) Department of Cognitive Psychology, Leiden University, Wassenaarseweg 52, 2300 RB, Leiden, The Netherlands Eric O. Postma (postma@micc.unimaas.nl) and H. Jaap van den Herik (herik@micc.unimaas.nl) Department of Computer Science, MICC-IKAT, Maastricht University, Tongersestraat 6, 6211 LN, Maastricht, The Netherlands Abstract to N IM -C LASS A by adding an active top-down mechanism to select fixation locations. We then evaluate to what extent the use of the top-down mechanism improves classification by testing N IM -C LASS A on the classification task. Subse- quently, we discuss top-down gaze control models, and the scalability and sensitivity to changes in viewpoint of the N IM - C LASS models. Finally, we present our conclusion. This paper describes two initial steps towards the realization of a plausible model of natural visual classification. As a first step, we extend the recently developed Natural Input Mem- ory (N IM ) model (Lacroix, Murre, Postma, & van den Herik, 2006) to a classification model of natural visual input called N IM -C LASS and evaluate the model in a face-classification ex- periment. Our experimental results show that N IM -C LASS is able to recognize and classify faces after a single encounter. In addition, N IM -C LASS is insensitive to variations in facial ex- pressions, illumination conditions, and occlusions. As a sec- ond step, we extend N IM -C LASS to N IM -C LASS A by adding an active top-down fixation-selection mechanism. We then as- sess to what extent N IM -C LASS A improves the performance on the face-classification task. The results show that the in- corporation of a selection mechanism improves classification performance, particularly when a limited number of fixations are taken during the classification process. Our results lead us to the conclusion that N IM -C LASS A may provide a suitable basis for a model of natural visual classification. Keywords: Perception, memory, classification, gaze control. Extending N IM to N IM -C LASS N IM is a model for recognition of natural images (Lacroix et al., 2006). N IM encompasses the following two stages. 1. A perceptual preprocessing stage during which a natural image is translated into feature vectors. 2. A memory stage comprising two processes: (a) a storage process that stores feature vectors in a straight- forward manner; (b) a recognition process that compares feature vectors of a newly presented image with previously stored feature vectors. Introduction Traditional computational models of cognition (e.g., Shiffrin & Steyvers, 1997) generally operate on an abstract represen- tation space, because they lack a mechanism to derive rep- resentations from the physical features of stimuli, i.e., they are not grounded in the real world. In sharp contrast, natural systems ground representations in physical interaction with the world. Acknowledging the importance of the environ- ment for natural cognition, a recent trend in psychologically motivated cognitive models is to focus on grounding repre- sentations in terms of their real-world referents (e.g., Pecher & Zwaan, 2005). Following this trend, the recently proposed Natural Input Memory model (N IM ; Lacroix et al., 2006) re- alizes a memory model that operates directly on real-world visual input (i.e., natural digitized images). It builds feature- vector representations on the basis of local samples (i.e., eye fixations) from natural images and uses these to make recognition-memory decisions (e.g., Lacroix et al., 2006). We aim at extending N IM (Lacroix et al., 2006) to a model of natural visual classification. This paper provides two initial steps towards achieving this objective. The outline of the re- mainder of this paper is as follows. As a first step, we extend N IM into a classifier of natural images called N IM -C LASS and assess N IM -C LASS ’s performance in a face-classification experiment. As a second step, we aim to approach the in- teractive nature of natural vision by extending N IM -C LASS Inspired by eye fixations in human vision, the perceptual pre- processing stage selects image samples (i.e., fixations) ran- domly along the contours in the image. For each fixation, visual input is translated into a feature vector that resides in a similarity space. The translation is realized using a biolog- ically informed method that involves a multi-scale wavelet decomposition (see, e.g., Rao & Ballard, 1995) followed by a principal component analysis. This method from the domain of visual object recognition models the first stages of process- ing of information in the human visual system (i.e., retina, LGN, V1/V2, V4/LOC; (Palmeri & Gauthier, 2004)). N IM applies the method in a saccadic based manner to build rep- resentations of fixated image parts that together constitute the feature-vector representation of an image. The memory stage stores the feature-vector representation (the storage pro- cess) and makes recognition decisions by matching an incom- ing feature-vector representation with previously stored rep- resentations. For a more detailed description and a schematic overview of N IM we refer to Lacroix et al., 2006. While N IM is a model for recognition of natural images, here we show that it can readily be adapted into a model for classification of natural images which we call N IM -C LASS . The feasibility