Verbal and Nonverbal Cues Activate Concepts Differently, at Different Times

Verbal and Nonverbal Cues Activate Concepts Differently, at Different Times Pierce Edmiston (pedmiston@wisc.edu) Department of Psychology, 1202 W. Johnson Street Madison, WI 53706 USA Gary Lupyan (lupyan@wisc.edu) Department of Psychology, 1202 W. Johnson Street Madison, WI 53706 USA Abstract Although the word “dog” and an unambiguous barking sound may point to the same concept DOG , verbal labels and nonverbal cues appear to activate conceptual information in systematically different ways (Lupyan & Thompson-Schill, 2012). Here we investigate these differences in more detail. We replicate the finding that labels activate a more prototypical representation than do sounds, and find that sounds activate exemplars consistent with the source of the sound, such that after hearing a barking sound, people are faster to recognize a dog with an open-mouth than a closed mouth, but critically, only when the sound and picture are presented simultaneously. The results are consistent with perceptual cues indexing their source while labels activating a more decontextualized representation of the target category. Keywords: categorization, concepts, sounds, recognition, cross-modal effects, language Introduction Most concepts are multimodal and can be activated in a variety of ways (Hoffman & Ralph, 2013). For example, the concept DOG can be activated by seeing a wagging tail, hearing a bark, or petting its furry coat. However, the concept DOG can also be activated by hearing the word ‘dog’—without seeing, hearing, or touching an actual dog. This raises the question of how concepts activated by nonverbal sensory cues compare to those activated by verbal category labels. In the experiments reported here we compare how verbal and nonverbal cues activate representations of purportedly the same concepts. In particular, we focus on visual aspects of familiar animals and artifacts as cued by natural sounds: auditory events with a distinct source (e.g., cat meowing, chainsaw revving), and how these same concepts are activated by verbal labels (words like “cat” and “chainsaw”). The mechanisms underlying recognition of nonverbal sounds and of speech appear to be quite similar. Recognition of both words and natural sounds varies as a function of familiarity, frequency, and context (Ballas, 1993; Stuart & Jones, 1995). Perception of both natural sounds and speech is influenced by signal ambiguity and noise in similar ways (Aramaki, Marie, Kronland-Martinet, Ystad, & Besson, 2010; Gygi, Kidd, & Watson, 2004). Both labels and natural sounds elicit similar N400 event-related potentials—a coarse index of semantic processing (Cummings et al., 2006; Van Petten & Rheinfelder, 1995)— even when the identification of the natural sound is incidental to task demands (Orgs, Lange, Dombrowski, & Heil, 2008). Functional imaging during similar sequential processing tasks reveals largely overlapping cortical areas recruited in processing labels and natural sounds (Dick et al., 2007). Lastly, patterns of naming deficits in patients with aphasia suggest the labeling of everyday objects and the visual recognition of natural sound sources rely on similar cognitive resources (Goll et al., 2010; Saygin, Dick, Wilson, Dronkers, & Bates, 2003). The perception of meaningful nonverbal sounds and of words is thus dependent on many of the same properties and activate largely the same semantic networks. Although it may seem that verbal and nonverbal cues are in important respects equivalent, there are critical differences. One such difference is that natural sounds, unlike labels, have a causal relationship with a specific physical source (Ballas, 1993). Recognizing these relationships requires learning, but the relationship between a referent and its natural sound is not arbitrary. We call these relationships “motivated”: that is, they are determined by physics (e.g., thunder) or driven by biology (e.g., large dogs—and agitated dogs—have deeper barks). Auditory perceivers are able to exploit such “motivated” relationships and surmise features of a hidden physical source, such as the size of a barking dog (Taylor, Reby, & McComb, 2008), the shape of resonating plates (Kunkler-Peck & Turvey, 2000), or the hardness of percussion mallets (Freed, 1990). The perception of these auditory sources is surprisingly accurate, reflecting the lawful relationships between signals and sources in the environment (Fowler, 1990). Importantly, sounds covary lawfully within as well as between categories. For example, a barking sound informs us not only that its source is a dog, but can inform us of the approximate size of the dog. In contrast, the relationship between labels and their referents is “unmotivated.” By this term we do not simply mean that words are arbitrary, i.e., that “dog” refers to dogs by convention (cf. Hockett, 1966), but that there exists a word “dog” that denotes the entire category of dogs rather than a particular type or instance (dachshund, German shepherd, dog-on the left, dog-far away, etc.). In short, barks index specific occurrences of dogs. Even though we can interpret natural sounds at a more categorical level, the surface properties of a specific bark still indexes a particular dog. Verbal labels, on the other hand, abstract over these specifics. When we say “dog” we can leave all