Narrative conjunction's junction function: A theoretical model of "additive" inference in visual narratives

Narrative conjunction’s junction function: A theoretical model of “additive” inference in visual narratives Neil Cohn (neilcohn@visuallanguagelab.com) Center for Research in Language, University of California, San Diego 9500 Gilman Dr. Dept. 0526, San Diego, La Jolla, CA 92093-0526 Abstract Visual narratives often depict images of individual characters without showing the larger scene, meaning that this whole spatial environment must be inferred from these component parts. However, few theoretical models of narrative or discourse have attempted to explain the generation of such “additive” inference. This paper explores the complex interactions between narrative structure and meaning within these types of discourse phenomena, situated within the model of Visual Narrative Grammar based on Jackendoff’s (2002) Parallel Architecture of linguistic structure. Narrative “Conjunction” repeats a single narrative category within a broader constituent, allowing for expansion of a sequence beyond the canonical narrative arc. These conjoined units then correspond to semantic structures in a variety of ways, allowing an “additive” inference of actions, scenes, characters, and/or semantic associative networks. This simple yet powerful architecture enables us to account for a large variety of phenomena in visual narratives and other discourse contexts, while providing a structure that can be tested in empirical research. Keywords: Narrative; discourse; inference; Kuleshov effect; situation model; comics; film; visual language. Introduction Theories of film editing have long noticed that disparate images of a scene or individual are understood as “adding up” a larger conceptualization. Experiments conducted by filmmaker Lev Kuleshov in the 1920s (Kuleshov, 1974) combined film shots of moving body parts of different women (hands, feet, eyes, heads), yet film viewers interpreted these shots as a single woman in motion. In another experiment, people responded to different shots of a scene as if they formed a coherent spatial environment when each of the shots was actually filmed in different locations. These experiments nicely showed that people inferentially construct a coherent representation, in a way that is different from the linear bridging inferences commonly discussed in research on discourse and visual narrative (Magliano & Zacks, 2011; McCloud, 1993). This issue becomes even more apparent in static visual narratives, such as those in comics. Consider Figure 1. These sequences only differ in that information from the single panel in 1a is dispersed into two panels in 1b. There is no difference in meaning—just how the panels selectively create a “window” on the different characters. Because of this fact, it stands to reason that the two panels in 1b “add up to” the single panel in 1a. In other words, we must infer that these characters belong to a single spatial environment, thereby creating a “virtual” single panel like that in 1a. Furthermore, in 1b knowledge of both characters within this inferred environment must connect to the final panel. Figure 1: Panels providing a “window” on a visual scene. All of this belies a linear focus on inference in the understanding of this sequence. First, the inference is constructed out of the combination of two panels’ content, not simply from what is left out “between” them. Second, the additive value of those panels must progress to the third panel, not simply the linear juxtaposition between the penultimate and final panels. In contrast to linear approaches, Visual Narrative Grammar (VNG) assigns narrative categories to discourse units like panels, and then organizes them into larger constituents, analogous to the way that words are organized into larger phrase structures in syntax (Cohn, 2013b). This constituent structure can directly address the issue of “additive inference” at a single narrative state. VNG is based on Jackendoff’s (2002) Parallel Architecture for language, meaning narrative structure and semantics are kept separate, yet interface in specific and predictable ways. This differs from previous “grammatical” approaches to narrative structure such as story grammars from the late 1970s (e.g., Mandler & Johnson, 1977), and grammars of film (e.g., Carroll, 1980), where the relationship between structure and meaning remain ambiguous. In VNG, narrative categories are independent from semantics, yet they interface in prototypical ways. This is analogous to how grammatical categories like nouns and verbs might have prototypical correspondences to objects and events, though these semantic qualities do not solely define them. The canonical narrative arc in VNG starts with an Establisher, which passively introduces an interaction or situation. An Initial then begins the events of the interaction (as in the boxer’s preparation in Figure 1a), which climaxes