Transfer of object shape knowledge across visual and haptic modalities

Transfer of object shape knowledge across visual and haptic modalities Goker Erdogan (gerdogan@bcs.rochester.edu) Ilker Yildirim (iyildirim@bcs.rochester.edu) Robert A. Jacobs (robbie@bcs.rochester.edu) Department of Brain & Cognitive Sciences, University of Rochester, Rochester, NY 14627 USA Abstract We investigate the hypothesis that multisensory representa- tions mediate the crossmodal transfer of shape knowledge across visual and haptic modalities. In our experiment, par- ticipants rated the similarities of pairs of synthetic 3-D objects in visual, haptic, cross-modal, and multisensory settings. Our results offer two contributions. First, we provide evidence for a single multisensory shape representation common to both vi- sual and haptic modalities. Second, our analyses suggest that these representations are part-based, representing objects as compositions of subparts. Keywords: multisensory perception, visual perception, haptic perception, object perception, shape representation Introduction Imagine the following simple scenario. You see an object, and later you are asked to find that object among a set of objects by using only your sense of touch. How might the visual information about the object be transferred to the hap- tic modality to achieve object recognition in this task? One hypothesis is that haptic input is mapped to a visual shape rep- resentation, maybe like a form of visual imagery, and object recognition is achieved in this visual shape space. It is also plausible that an analogous haptic imagery process is at play, and object perception takes place in a haptic shape space. The third alternative and the hypothesis we are arguing for is the Multisensory Hypothesis. This hypothesis states that peo- ple use sensory representations of objects to infer amodal or multisensory representations characterizing objects’ intrinsic properties, and object perception is mediated by these multi- sensory representations. The first question we address is exactly this question of whether people use modality-specific (vision-specific and haptic-specific) object representations or whether they use modality-independent (multisensory) representations. The second question concerns the fine-grained structure of object representations. If multisensory representations underlie our object perception, what can we say about the nature of these representations? An influential hypothesis in the cognitive science literature is that object representations are part-based, meaning that objects are represented in terms of their parts and the spatial relations among these parts (Marr & Nishi- hara, 1978; Biederman, 1987). To address these questions, we collected similarity judg- ments from people about pairs of novel objects when objects are viewed, when objects are grasped, when one object is viewed and the other is grasped, and when both objects are viewed and grasped. We found that participants gave similar ratings in all experimental conditions, providing evidence for the existence of multisensory object representations. More- over, our analyses suggest that our multisensory object repre- sentations are part-based. Related Research Previous studies provide behavioral and neurophysiological evidence for the existence of multisensory representations. Quiroga (2012), for example, reported the existence of “con- cept cells” which are neurons that respond selectively to par- ticular persons or objects regardless of the modality used to sense those persons or objects. One neuron, for instance, re- sponded when a person viewed an image of the television host Oprah Winfrey, viewed her written name, or heard her spoken name (Quiroga, Kraskov, Koch, & Fried, 2009). Additionally, brain imaging studies (Amedi, Jacobson, Hendler, Malach, & Zohary, 2002) show that LOtv, a neural region within the hu- man lateral occipital complex, is activated both by viewing and touching objects. Behavioral results are consistent with neurophysiological findings. Konkle, Wang, Hayward, and Moore (2009) re- ported that motion aftereffects transferred between vision and touch—when adapted to visual motion in a certain direc- tion, people felt tactile motion aftereffects in the opposite direction, and vice versa. Such a finding provides strong evidence for a shared representation underlying visual and tactile motion perception. In another study, Lacey, Pappas, Kreps, Lee, and Sathian (2009) found that subjects initially showed viewpoint-dependent object recognition in both vi- sual and haptic modalities. However, following unimodal training with either visual or haptic stimuli, people’s object recognition performances became viewpoint-independent in both modalities. A set of studies by Wallraven, B¨ulthoff, and colleagues also provide evidence for common object rep- resentations underlying visual and haptic object perception. In these studies (Cooke, J¨akel, Wallraven, & B¨ulthoff, 2007; Gaissert, Wallraven, & B¨ulthoff, 2010; Gaissert, B¨ulthoff, & Wallraven, 2011; Gaissert & Wallraven, 2012), subjects pro- vided similarity judgments for different sets of objects, both artificial and natural, in vision alone, haptic alone, and vision- haptic conditions. It was found that subjects’ similarity rat- ings were similar in all three sensory conditions, thereby sug- gesting that these ratings were based on shared, multisensory representations. The experiment reported here uses a similar procedure, but extends this earlier work by focusing on the part-based nature of these representations.