Remembering connectives: Visual-auditory consonance influences representations in time

Remembering connectives: Visual-auditory consonance influences representations in time Bradley J. Morris (morrisb@gvsu.edu) Department of Psychology, Grand Valley State University Allendale, MI 49401 USA Rebecca Neiman (niemanre@mail.gvsu.edu) Department of Psychology, Grand Valley State University Allendale, MI 49401 USA Abstract the “gist” of information reduces working memory burden but has the potential to distort inferences derived from these representations. Reasoning errors often arise when information is represented incorrectly. One example is the “conjunction effect”, a tendency to misremember OR statements as AND statements. We investigated the how the time course of consonance between visual and auditory information influenced memory for connectives. We measured memory for statements after manipulating the consonance between language and visual information. The results indicated that subjects correctly recalled more statements given language-visual consonance and were more likely to false alarm (e.g., incorrectly recalling OR statements as AND) given language-visual dissonance. We modeled performance using a Simple Recurrent Network. Our model, in which the training set was structured similarly to natural language input, provided a reasonable analog. Taken together, the results suggest that connective representations are influenced by the concordance between visual and language input and that the bias toward conjunctions arises, in part, from the relatively high frequency of conjunctions in visual and linguistic input. One such a distortion is the tendency to misremember statements as conjunctions, or the conjunction effect (Rader & Sloutsky, 2001). For example, given the statement There is a star or a pencil in the box people often recall this statement as There is a star and a pencil in the box. This small change yields large effects on drawing valid or true conditions because the conditions for truth-falsity differ for AND and OR statements. These results are clearly incompatible with current accounts in which representation is based on syntactic form (e.g., Rips, 1994) because semantic information was recalled at significantly higher levels than form (i.e., connective). Keywords: Representation, logical connectives, neural network, memory Reasoning errors often arise from defective representations of natural language (Johnson-Laird, 2003). One influential theory of language comprehension is that people create semantic-situational models of information during comprehension (Johnson-Laird, Zwaan Radavansky, 1998). Mental models are cognitive simulations of objects and relations that preserve the structure of their input, specifically, meaning and the situations in which information occurs (e.g., spatial relations between objects; Johnson-Laird, 2003). For example, during comprehension, readers often integrate images and texts into a single model (Gernsbacher, 1990) and represent descriptions of space visually (Spivey, Tyler, Eberhard, & Tanenhaus, 2000). Inferences occur by searching and evaluating models within a bounded working memory space. There is considerable evidence that reasoners use models, rather than formal rules (Rips, 1994), for deductive reasoning. For example, subjects forget the form of arguments rapidly during problem solving (Johnson-Laird & Stevenson, 1970) yet maintain the “gist” of the information (i.e., retain meaning but forget connectives; Johnson-Laird, 2003). The tendency to retain Although the effect is clear, there is no consensus on why the conjunction effect occurs. Although formal rules cannot account for this effect, current accounts of mental model theory are also insufficient to account for these effects. One factor that likely plays a role is the complexity of statements, defined as the number of models required to produce a veridical representation. Johnson-Laird (2003) suggests that conjunctions are simpler because they require only one model while other forms require more than one. Although there is evidence to support this suggestion (Feldman, 2000; Morris & Hasson, 2010), it focuses on the complexity of evaluating models rather than the cause of creating defective representations (e.g., models). We examine two possibilities: the influence of visual and auditory information and the time course of model construction. Integrating visual and auditory information in time As discussed above, models are mental simulations of information derived from sensation and language. That is, models are constructed from information derived from multiple modalities (e.g., visual and auditory information). Evidence form previous research demonstrates that visual and auditory information influence the types of comprehension models created (Gernsbacher, 1990). Spivey et al. (2000) demonstrated that language comprehension influenced visual perception of complex scenes.