<!-- image -->

## Alison Gibbons*

## Multimodal Literature 'Moves' Us: Dynamic Movement and Embodiment in VAS: An Opera in Flatland

## Abstract

Multimodality is a recent academic development, fuelling a surge of related research (Kress/van  Leeuwen  1996;  2001;  Baldry/Thibault  2006;  Royce/Bowcher  2007). In  parallel  to  this,  the  turn  of  the  millennium  has  seen  an  increase  in  the  inclusion of  typography,  graphics  and  illustration  in  fi  ction  yet,  with  only  a  few  exceptions (Gibbons forthcoming a; forthcoming b), printed literature has often been neglected in multimodal study. Focusing on the 'imagetext novel' VAS: An Opera in Flatland , written by Steve Tomasula and designed by Stephen Farrell (2002), this paper explores multimodal  printed  literature  through  cognitive-poetic  analysis.  The  examination  of visual elements is aided by theories from visual perception and multimodal research. This  cognitive  and  perceptual  methodology  is  strengthened  through  refl  ection  upon recent fi  ndings from neuroscientifi  c work on embodiment. In consequence, this paper presents a fresh approach to multimodality, an approach which not only attends to all modes of meaning-making equally, as well as collaboratively, but one which considers the cognitive and embodied aspects of a multimodal literary experience.

## 1. Introduction

The turn of the millennium has seen an increase in the inclusion of typography, graphics and illustration in fi  ction. Nevertheless, multimodal literature, that is literature that utilises more than one semiotic mode in the expression of its narrative, is not a new literary phenomenon. In fact, the practice is long standing. For instance, one need only think of

* Alison Gibbons University of Sheffi  eld School of English Literature, Language and Linguistics Sir William Empson House Shearwood Road Sheffi  eld, South Yorkshire S10 2TD, England a.gibbons@sheffi  eld.ac.uk

Hermes - Journal of Language and Communication Studies no 41-2008

<!-- image -->

<!-- image -->

107

<!-- image -->

<!-- image -->

<!-- image -->

William Blake's 'illuminated printing' in the 18th century, and even this has many precedents.

The millennial revival of multimodality has resulted not only in major  publishing  houses  releasing  works  with  graphic  elements.  It  also means that one can identify a spectrum of multimodal literature. This ranges from using pictures in a merely illustrative fashion, as can be seen in Alex Garland's The Coma (2004) and Marisha Pessl's Special Topics in Calamity Physics (2006)  for  instance,  to  a  mediating  form in which type face, type setting and images play a role in the progression of the narrative, of which Extremely Loud and Incredibly Close by Jonathan Safran Foer is an example. Toward the extremity of the spectrum are situated texts in which the presence of visual modality forms a more equal relationship between verbal and visual. In other words, the different modes of expression are located on the page not in an autonomous or separate fashion, but in such a way that, while these modes have distinct means of communicating their narrative voice, they constantly interact in the production of textual meaning. As a result, they often emphasise the dynamic and embodied nature of the practice of reading.

In this paper, I will be considering an extreme example of multimodal printed literature, namely VAS: An Opera in Flatland by Steve Tomasula and Stephen Farrell (2002). The paper will explore the readerly encounter with the novel VAS using cognitive poetic analysis, including the consideration of recent neuroscientifi  c evidence of embodied cognition. The examination of visual elements will employ theories from visual perception and multimodal research. Frameworks used in the course of  the  paper  will  be  elucidated  as  they  emerge  in  analysis.  The  theoretical merger of cognitive poetics and multimodal research enables an original approach to multimodal texts. Expanding on existing work in  multimodal  studies  (Kress/van  Leeuwen  1996;  2001;  Burn/Parker 2003; O'Halloran 2004; Ventola et al. 2004; Carlsson et al. 2005; Cranny-Francis 2005, Baldry/Thibault 2006; Royce/Bowcher 2007; Baldry/ Montagna forthcoming; Gibbons forthcoming a; forthcoming b), this approach advances current  applications  by  considering  the  cognitive possibilities produced by and implications of the interaction of modalities in a literary context.

<!-- image -->

<!-- image -->

<!-- image -->

## 1.1.  Multimodal study

Multimodality is a theoretical endeavour of the 21st century as the recent  arrival  and  surge  in  publication  on  the  topic  illustrates  (Kress/ van Leeuwen, 1996; 2001; Ventola et al. 2004; Levine/Scollon 2004; O'Halloran 2004; Carlsson et al. 2005; Baldry/Thibault 2006; Royce/ Bowcher 2007). Presently,  three  books  hold  a  prominent  position  in multimodal studies; Kress/van Leeuwen's (1996) Reading Images: The Grammar of Visual Design ,  Kress/van  Leeuwen's (2001) subsequent book Multimodal Discourse: the Modes and Media of Contemporary Communication , and Baldry/Thibault's (2006) Multimodal Transcription and Text Analysis .

Kress/van Leeuwen's (1996) Reading Images brought multimodality into academic attention, referring to 'composite or multimodal texts' which they defi  ne  loosely  as  'any  text  whose  meanings  are  realized through more than one semiotic mode' (Kress/van Leeuwen 1996: 183). In this seminal work, Kress and van Leeuwen put forward a frame  work to be used in study and analysis, which attends to the formal features of layout and design of both visual and multimodal texts. As such Reading Images was a groundbreaking publication, and retains its importance as a central resource in multimodal study.

In a later co-authored book, Multimodal Discourse: the Modes and Media of Contemporary Communication , Kress and van Leeuwen's approach shifts towards a greater social semiotic concern. In Kress and van Leeuwen's words, they 'focus on practices ', identifying 'four domains of practice in which meanings are dominantly made' (Kress/van Leeuwen 2001: 4). These domains are Discourse, Design, Production, and Distribution. By acknowledging the production processes and technologies of multimodal texts, Kress and van Leeuwen's work presents a useful focus for the study of multimodality in an increasingly digital age.

Baldry/Thibault's (2006) Multimodal Transcription and Text Analysis is the latest textbook on multimodality. In this work, they articulate (2006: xv), 'The term multimodality covers a diversity of perspectives, ways of thinking and possible approaches. It is not a single principle or approach. It is a multipurpose toolkit, not a single tool for a single purpose'. Baldry and Thibault's work builds on Kress and van Leeuwen's by extending the multimodal toolkit as well as attending to addition- al semiotic resources such as sound, for example in fi  lm dialogue and soundtracks.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

This particular article is not the place for a lengthy review of the works briefl  y mentioned above (see Gibbons forthcoming a for more discussion). Nevertheless, between these authors, a defi  nition of multimodality can be agreed. In the narrowest sense, multimodality is the coexistence of more than one semiotic mode within a given context. In the widest, it is the experience of living; we experience everyday life in multimodal terms through sight, sound, movement… Even the simplest conversation entails language, intonation, gesture, and so forth. In today's world, multimodality in academic research seems to have become even more urgent and even more relevant. The present epoch is a technologically multimodal era. As technology progresses so too do the literary, poetic, and artistic forms of expression through that technology.

In the present study, I take a cognitive approach to multimodal texts. While I would be hesitant to engage in multimodal politics here, I can state with confi  dence that existing work in the fi  eld does not explicitly take account of the cognitive processes of readers, viewers, and receivers of multimodal texts. Thus, Baldry and Thibault's acknowledgement that there are many more avenues of analysis for multimodal study are encouraging. In consequence, the cognitive poetic analysis which follows is both original and revealing. Expanding on existing work in multimodal studies, this approach advances current applications by considering the cognitive possibilities produced by and implications of the interaction of modalities in a literary context.

## 1.2.  Cognitive poetics

Cognitive poetics (Stockwell 2002; Gavins/Steen 2003) is a new approach to the study of literature, an approach which emphasises the cerebral procedures underlying the writing of a text in addition to those required by the reader in order to 'read', interpret, and comprehend the text as a literary artefact. In his textbook defi  nition, Stockwell (2002: 1, original italics) briefl  y explicates the linguistic elements of the name: ' cognition is to do with the mental processes involved in reading, and poetics concerns the craft of literature'. Cognitive poetics is thus a discipline in which both the creativity and reception of a text are conceived as important elements in the production of textual meaning. It seeks to look at form, style, and language in literature in context but through the conviction that structures of language and literary devices are expressions and materialisations of patterns of human thought. Thus as Gavins/Steen (2003: 1) put it, literature is 'a specifi  c form of human experience' and as such its study may reveal to us the cognitive practices by which we not only read literature but perceive and understand the world. Cognitive poetics is an approach constantly in the process of development, refi  ning its frameworks alongside advancements in the domains of cognitive science and psychology upon which it draws.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

Since cognitive poetics has an undeniable literary-linguistic inclination, having developed from stylistics among other sources, multimodal literary works may appear to pose a challenge to the discipline. Yet, as a discipline with an underlying cognitive inclination, cognitive poetics ought to be able to account for interpretive practice in its entirety, regard  less of communicative mode. In their introduction to Cognitive Poetics in Practice , Gavins/Steen (2003: 1) make the following assertion:

The appeal of literature has been challenged by new art forms directed at new groups of audiences through new media, and it has become inevitable to consider the resemblance and difference between these art forms and literature in terms of their psychological and social effects. This is precisely what cognitive poetics promises to bring into view, by relating the structures of the work of art, including the literary text, to their presumed or observed psychological effects on the recipient, including the reader.

Gavins and Steen's description of the goal of cognitive poetics shows that  the  discipline  is  conceived  as  applicable  to  all  forms  of  artistic expres  sion and, by implication, all forms of artistic expression therefore exhibit and demonstrate the cognitive capacities of human nature. While this is true, the idea behind it has yet to be fully investigated.

It is my belief that by using cognitive poetics, along with elements of visual perception and multimodal research, new light will be shed upon the reading experience of literary multimodality, including insights into the interaction of word and image on the page and, crucially, the way in which cognitive structures underlie both verbal and visual modes of expression. In addition, recent discoveries from neuroscience relating to embodied experience are utilised in order to plausibly analyse the embodied nature of the reading process. This interdisciplinary approach is by no means an extraordinary leap since at the heart of these disciplines stands  an  essential  desire  to  relate  to  general  human  experience  and to advance understanding of the cognitive practices by which we see, read, and make sense of the personal and social world.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

## 2. VAS: An Opera in Flatland : Contextual information

VAS: An Opera in Flatland (Tomasula/Farrell 2002) is described in the publisher's  blurb  as  an  'imagetext'  novel,  including  colour,  images, and innovative typographical arrangements. Set in an unspecifi  ed future time, the novel's main plot centres upon a character named Square, who agonises over the decision to undergo a vasectomy at the request of his wife Circle. The names of these characters, complemented by the presence of their daughter Oval, are evidently shape nouns. This is signifi  cant as the subtitle of VAS is an intertextual allusion to Edwin A. Abbott's (1884) short novella, Flatland: A Romance of Many Dimensions which conceives of a world populated by two-dimensional shapes. As an English theologian, cleric, and teacher, Abbott was a prolifi  c writer,  but Flatland is  considered his defi  ning work, functioning both as a social satire and a creative science-fi  ction. The latter is underpinned by mathematical laws and abstract conjectures on time and space (the presence of a fourth dimension) which foreshadowed the scientifi  c concerns of its period, particularly in the arena of theoretical physics. VAS: An Opera in Flatland engages with many of the topics of Abbott's novella, refl  ecting upon present-day American society while also extending Abbott's concern over scientifi  c ethics by inspecting contemporary subjects of controversy. As such, the 'Flatland' in which the narrative of VAS is set is a world which is becoming progressively postbiological, with its culture of surgery, cloning, biological patenting, and genetic commerce. The book's structure augments these themes by utilising scientifi  c quotation, facts and fi  gures about the human genome and evolutionary process as well as commercial adverts related to cloning and DNA testing.

What differentiates VAS from many other multimodal printed novels is that it is a collaborative work between author and academic professor Steve Tomasula and graphic designer Stephen Farrell. Both are named on the book's cover and both of their names appear on the book's spine, giving them equal recognition. The working relationship between To- masula and Farrell appears to be a partnership, implicit in the way in which they discuss the creative process of collaboration. In interview, Farrell accentuates the 'writer and designer together' (Farrell qtd. in Burdick 1996: no pagination):

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

I  think that just by allowing the designer to have the piece to work with, the writer is already allowing certain liberties to be taken. And its important that they know and respect this up front: that design isn't just beautifying a piece, that's it's going to be co-authorship, a joint venture. In return the designer must respect the text, respect the craft of writing.

In this interview, Farrell continues to discuss his working relationship with Tomasula, giving the impression that both form and content were matters of debate between them in order to reach mutual decisions. Farrell's potent choice of wording, emphasising 'co-authorship' and 'joint venture', suggests that he perceives the designer's role to play an equivalent part to the writer in the fi  nal meaning-making potential of a text.

Collaborations between writer and designer inevitably raise important issues for multimodal studies, since they instantly destabilise notions of authorship and authorial intention. Farrell's employment of the prefi  x 'co-' and modifi  er 'joint' suggests that the power of a singular authorial fi  gure is undermined. Interestingly, this is in accord with Barthes' (1968) deconstructive attack upon the supremacy and prestige attributed to the author in humanist thinking. Barthes (1968: 147) deems the notion of author to be restrictive, infl  icting a limit upon the text and closing it  off  from  numerous possibilities. Admittedly, Barthes argument was somewhat deterministic since all texts are subject to various readings as is evident in literary criticism's ability to read the same text from different scholarly perspectives. Nevertheless, Farrell (Farrell qtd. in Burdick 1996: no pagination) sees design as opening up a text, 'enriching meaning by creating contradiction, ambiguity, all the things that imagery can add; that presence, that immediacy'.

By displacing the author, Barthes bestows greater import on the active responsibility of the reader, postulating (1968: 148);

a text is made up of multiple writings… but there is one place where this  multiplicity  is  focused  and  that  place  is  the  reader,  not,  as  was hitherto said, the author. The reader is the space on which all the quotations that make up a writing are inscribed without any of them being lost; a text's unity lies not in its origin but in its destination.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

Like Barthes, (multimodal) cognitive poetics does not recognise the author as the sole determiner of textual meaning, but also looks to the reader. However, even with its acute attention to the cognitive reception of texts, cognitive poetics would not agree with Barthes' (1968: 148) now much-quoted words, 'the birth of the reader must be at the cost of the death of the author'. Rather, the production of meaning is a 'joint venture' between author(s) and reader(s).

Multimodal texts, particularly those which are a shared creative vision of two or more authorial minds, intensify the multiplicity of literary writing. Their polyphonic nature, composed of 'writing and design…  interdependent  modes  of  representation'  as  Farrell  (qtd.  in Burdick 1996: no pagination) would term it, exposes the reader to multiple forms working in synchronicity to communicate narrative meaning. Thus such works assign further cognitive demands to the reader than the conventional novel. Since the reader must negotiate and make sense of not only a multiplicity of meanings, but also of modes which in term expand and add to those meanings, it is imperative to analyse VAS: An Opera in Flatland (Tomasula/Farrell 2002) using an approach which considers readerly cognitive and imaginative process as well as the novel's multimodal design.

## 3. Analysis

## 3.1.  Into VAS

VAS: An Opera in Flatland is a novel with an unusual narrative inception (Tomasula/Farrell 2002: 9-10): 'First Pain / Then Knowledge: a paper cut'. At the story level, this refers to a physical act: while writing, the main character Square has given himself a paper cut. Importantly, the opening sets up the theme of embodiment, which grips the novel. Embodiment is a central concept in cognitive science, linguistics, and poetics (Johnson 1987; Varela et al. 1991; Lakoff/Johnson 1999; Glenberg/Kaschak  2002;  Gibbs  2005;  Gibbs  2006,  Zwaan/Taylor  2006), which consider mind and body as a syndicate through which conceptual information is understood. As Gibbs (2005: 66-67) articulates,

Cognition is what happens when the body interacts with the physical/ cultural world. Minds are not internal to the human body, but exist as webs encompassing brains, bodies, and world… 'embodiment' re- fers to the dynamical interactions between the brain, the body, and the physical/cultural environment.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

Thus,  it  is  the  body's  physical,  sensory,  and  perceptual  interactions within the world that infl  uence the way in which the mind structures and conceptualises human experience.

A framework that has been central to cognitive poetics, and more signifi  cantly, which is seen to demonstrate the connections between conceptual and embodied experience is conceptual metaphor theory (Lakoff/Johnson 1980; Lakoff/Turner 1989; Lakoff/Johnson 1999; Turner 2000). Conceptual Metaphor theory has been a substantial research area  since  the  1980s,  suggesting  that  human  conceptual  patterns  are metaphorical by nature. They always consist of a familiar, often bodily, source domain and an abstract target domain. Conceptual elements from the source are transferred through metaphoric connection to the target which is often re-characterised as a result. In this way, conceptual  metaphors  provide  a  means  for  understanding  abstract  concepts (the target) by comparison to a basic-level domain (the source) which is grounded in bodily and/or everyday experience.

One of the most frequently cited examples of a conceptual metaphor is known as the CONTAINMENT metaphor. This metaphor, which is based upon the physical understanding of containment, relies upon a structure of inside, outside, and boundary between (Lakoff/Johnson 1999:  32).  Obviously,  it  is  not  diffi  cult  to  think  of  everyday  objects which exemplify containment; a box, a cup for tea, etc. We even perform many actions on a daily basis which rely upon experiential containment such as getting into bed at night or out of bed in the morning, walking into or out of a room. The fact that these physical manifestations are an essential factor in forms of conceptual understanding can be seen in idioms such as 'getting the most out of life' or in the common notion of being ' in love' with someone. My italics show that the prepositions 'in' and 'out' play a signifi  cant part in the CONTAINMENT metaphor. Moreover, each of the examples given presents a different variation of the CONTAINMENT metaphor, the former being LIFE IS A CONTAINER while the latter is LOVE IS A CONTAINER. In both cases, a concrete domain (CONTAINER) is mapped onto an abstract domain (LIFE and LOVE respectively), demonstrating the pervasive infl  uence our bodies and bodily experience has upon conceptual thought.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

Signifi  cantly, conceptual metaphor theory is particularly suited to the analysis of multimodal texts since as it underlies the way in which human thought is organised, conceptual metaphors are articulated and perceptible within all forms of human representation. Indeed, Crisp (2003: 100) states, '[m]etaphor can be expressed in different modalities because its underlying reality is conceptual and so not confi  ned to any single mode of expression'. Notably, recent work on conceptual metaphor has begun to tackle non-linguistic manifestations (e.g. Forceville 2005; Gibbons forthcoming a; Gibbons forthcoming b).

The opening to the novel VAS: An Opera in Flatland (Tomasula/Farrell  2002) foregrounds embodied meaning. Moreover, the initial sentence, 'First Pain / Then Knowledge: a paper cut' (Tomasula/Farrell 2002: 9-10), spans two pages, and the positioning of these pages within the book requires a page-turn in its midst. As a result, Tomasula/Farrell (2002) ensure that from the moment a reader picks up the novel, s/he is affected immediately by the materiality of the book, thus foregrounding the embodied aspect of the practice of reading literature.

## 3.2.  Around VAS

Tomasula and Farrell explore the theme of embodiment both thematically through the novel's interest in genetic modifi  cation as well as multimodally by designing the visuality of the book to emphasise its status as a material artefact with which the reader interacts. In this way, VAS seeks to involve the reader physically in its narrative. At the end of a highly political discussion of genetic development, ethnic cleansing and U.S. legislation, the nature of the universe of the book as a 'fl  atland' is suddenly referenced (Tomasula/Farrell 2002: 139): '…astronomers concluded, the universe… must be… fl  at'. The last linguistic content of this page appears in a very small elaborate font. More unusually, these fi  nal words are upside-down:

<!-- image -->

'

'.

Figure 1. 'Still, it Moves' (Tomasula/Farrell 2002: 139)

Due to the diminutive type size and ornate script-like font, it is diffi  cult to read this clearly without rotating the book. On doing so, one is able to decipher the text as 'Still, It Moves'. The word 'move' is defi  ned by the OED as to 'go or cause to go in a specifi  ed direction or manner' or 'change or cause to change position'. It is therefore associated with dynamic action. Even when the word 'move' is used metaphorically, such as expressing that we 'feel moved by' something, it is underwritten by a conceptual metaphor (EMOTION IS MOTION) which relies upon our experiences of physical manipulation and activity. This is in line with cognitive linguistic dogma, believing that all abstract thinking, particularly metaphor is grounded in bodily experience (Lakoff/Johnson 1980; 1999; Johnson 1987; Sweetser 1990; Gibbs 2005; 2006).

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

Research in cognitive science and neuroscience demonstrates that embodied experience is fundamental to linguistic processing and comprehension, particularly in the usage of action verbs such as 'to grasp', both when they are used literally ('He grasped the handle') (Glenberg/ Kaschak 2002; de Vega et al. 2004; Hauk et al. 2004; Zwaan et al. 2004; Gallese/Lakoff 2005; Pulvermüller 2005; Pulvermüller et al. 2005; Tettamanti et al. 2005; Zwaan/Taylor 2006; Glenberg 2007) and when they are used metaphorically ('He grasped the idea') (Wilson/Gibbs 2007). The former research evidences that when reading action-related sentences,  some of the same regions in the brain are activated as when performing the action, thus priming the neurological circuitry for the linguistically-referenced  movement.  Although  this  research  looks  at action  verbs  which  are  specifi  cally  related  to  parts  of  the  body  (e.g. 'grasp' is associated with the hands), it nevertheless suggests that 'real bodily action is at the root of meaning conveyed by language' (Glenberg/Kaschak 2002: 653). Thus, the word 'moves' in VAS , like all uses of this word, has an undeniable motoric dimension.

Furthermore, it is signifi  cant that the reader must physically move the book in order to read, 'Still, It Moves'. The priming operation of language for action also works in reverse; language which refers to a said action can be processed with increased speed and ease if a subject  has  previously performed that same action. This can be inferred from Glenberg/Kaschak's (2002) results as well as Pulvermüller et al.'s (2005), who show that 'sensorimotor areas can play a specifi  c functional role in recognizing action words' (Pulvermüller et al. 2005: 795). Wilson and Gibbs (2007) also reach a similar conclusion which Gibbs (2005: 88) summarises; 'performing an action facilitates understanding of a fi  gurative phrase containing that action word, just as it does for lit- eral phrases'. Consequently, the kinesthetic movement of rotating the book primes the concept of 'movement' and assists linguistic comprehension of 'Still, It Moves'.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

It is also worth noting that the visual feature on this page (Tomasula/ Farrell 2002: 139), a diagrammatic swirl of arrows which resemble the shape of a tornado, may also contribute to this process since the direction and curvature of the arrows appear to be acting under a centripetal force thus creating the impression of rotary motion. In this way, the diagram may be a form of pictorial priming for rotational movement. If this is the case, 'Still, It Moves' is a multimodal inscription. In reading it, word, image, and physical kinesthetics are cognitively and neurologically united.

Above the words 'Still, It Moves' is written (Tomasula/Farrell 2002: 139), 'After all, if a person turned Galileo's telescope end-for-end and looked through it, wouldn't an elephant appear to be the size of an ant?' This hypothetical and rhetorical question is concerned with perception - looking at something in a different way or from a different vantage results in a different vision. This is lucidly pertinent to the reader's experience of 'Still, It Moves' since it is only by altering their approach to the text, physically rotating it as with Galileo's telescope, that the words may be comprehended.

The  reference  to  Galileo  becomes  even  more  important  overleaf where on the right hand page 'Still, It Moves' is repeated (Tomasula/ Farrell  2002:  141),  having  been  imprinted  in  exact  replication  of  its previous appearance. Above this, appear a series of vertically aligned circles and foreign words, of which 'Eppur, Si Muove' sits central and in larger type. The lowest line of this archaic inscription seemingly authenticates the material with a date (April and the year in roman numerals)  and  signatory. The presence of a signatory, seemingly named as Galileo, enables the reader to infer that the mysterious words, 'Eppur, Si Muove' belong to the seventeenth century astronomer and physicist Galileo Galilei. 'Eppur, Si Muove' is, in fact, an Italian phrase which Galileo is reported to have muttered at the close of the inquisition of 1633 as he rose from his knees after having recited the abjuration in which he was forced to retract his belief in the Copernican solar system, namely that the earth (and all planets) move in orbit around the sun (Fahie 1903: 324; Næss 2005: 177). While this speech has been discarded as inaccurate by historians and biographers, it nevertheless holds magnitude for a reading of VAS .

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

To begin with, the 'O' in 'Eppur, Si Muove' is not realised in the same typography as the rest of the words. It is actually created using one of the vertical circles. These circles, ten in total, can be said to represent nine planets and the earth's moon in the solar system, an idea confi  rmed by the heading word 'Méridianum'. This version of the word 'meridian' refers in astronomy to an imagined circle passing through the celestial poles and zenith of a given place, enabling astronomers to plot the positions of stars and planets. In effect, the reader holds in their hands a solar system, all be it a two-dimensional fl  atland version. By paying attention to both the linguistic content and visual design in this extract, the semantic inferences triggered by each semiotic mode interact with each other, bringing the interpretation articulated here into focus.

Since 'Still, It Moves' once again sits upside-down at the bottom of the page, the book requires rotation. Those readers who choose to revolve the book in this manner, rather than lazily reading it 'wrong way up', take on the role of the sun spinning the planets, putting them into orbit, so to speak. This physical interaction creates a multimodal conceptual  blend  in  which  the  notion  of  the  sun  is  transposed  upon  the reader. Thus, like the sun for the earth, the reader is a life-giving force for the novel, bringing it 'alive' in cognitive terms. Ultimately, it is the reader's corporeal transaction with VAS that enables meanings such as this to arise.

## 3.3.  Out of VAS

In terms of sensory modalities, the physical body is part of the haptic sense. As cognitive scientists, Lederman/Klatzy (2001: 71) explain, 'People use the haptic system to perceive and interact with the world of concrete and virtual objects'. Laura Marks (2002: 2) also offers a defi  nition, affi  rming the haptic sense comprises 'the combination of tactile, kinesthetic, and proprioceptive functions, the way we experience touch both on the surface and inside our bodies'. Media critic Laura Marks and art theorist Jennifer Fisher (1997) consider haptic perception in relation to cinema, and art and exhibition spaces, respectively. In considering artistic forms and genres that utilise a haptic aesthetic, as Fisher might term it, they share a promising ideal of the affect such works have on the reception process. While their discussions focus upon more explicitly visual mediums, a shared idea is equally relevant to VAS and the genre of multimodal literary print novels in which it is situated. To quote Marks (2002, p.18) (but with VAS in mind), '…by appearing to us as an object with which we interact rather than an illusion into which we enter, [it] calls on [a] sort of embodied intelligence. In the dynamic movement between optical and haptic ways of seeing, it is possible to compare different ways of knowing and interacting…'. Moreover, in his seminal work, The Body in the Mind , Mark Johnson (1987: xix) states:

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

The  centrality  of  human  embodiment  directly  infl  uences  what  and how things can be meaningful for us, the ways in which these meanings can be developed and articulated, the ways we are able to comprehend and reason about our experience, and the actions we take. Our reality is shaped by the patterns of bodily movement, the contours of our spatial and temporal orientation, and the forms of our interaction with objects. It is never merely a matter of abstract conceptualizations and propositional judgements.

Since embodiment is fundamental to human understanding and experience, it seems logical to suggest if the body is more involved in the act of reading, then that particular reading may become more meaningful, or loaded with greater signifi  cance, as a result. In an article on synaesthesia in poetic language, Shen/Cohen (1998: 125) discuss the organisational ranking of modalities along a scale from high to low. Sight  is  the  'highest'  modality,  followed  by  sound,  smell,  and  taste, with the 'lowest' modality being that of touch. Following Johnson/Lakoff's (1980; 1999) ideas on embodiment, they convincingly argue that concepts belonging to lower modalities are more accessible since they 'involve a more direct, less mediated experience of perception' (1998: 128). Therefore, by tapping into the lower modality of (haptic) touch using the immediate contact of the reader with the text through bodily experience, Tomasula and Farrell enhance the accessibility of their narrative world making VAS a more vivid reading experience.

## 4. Conclusion

Since cognitive processes underlie every act of reading and interpreting, I believe a cognitive approach to multimodality, complete with rig- orous attention to textual detail, will yield invaluable insights into both human understanding and multimodal forms. A more traditional multimodal perspective of the extract of VAS analysed above could certainly have produced an interesting account of the extract's style and design features. However, by adding a cognitive dimension to the analysis, I have been able to reveal the experiential impact that literature which explicitly stimulates the senses may have upon on the reader. Cognitive poetics, with its close focus upon stylistic features and reader involvement, reveals multimodal novels as complex and dynamic forms which induce the reader to invest in narrative physically as well as cognitively. Analogous to our bodies, composed of various matter, chains of DNA, and encompassing our genetic histories, VAS: An Opera in Flatland is multimodal in its employment of the visual, verbal, and somatosensory which collaborate in the production of textual meaning. It demands that, in reading, we employ the capacities of our senses and of our bodies. Multimodal printed literature, such as VAS , not only moves us fi  guratively speaking, it literally makes us move.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

## References

- Abbott, Edwin. A. [1884] 1998: Flatland: A Romance of Many Dimensions . New York and London: Penguin Books.
- Baldry, Anthony/Thibault, Paul J. 2006: Multimodal Transcription and Text Analysis: A multimedia toolkit and coursebook . London: Equinox.
- Baldry, Anthony/Montagna, Elena (eds.) forthcoming: Interdisciplinary Perspectives on Multimodality: Theory and Practice. Campobasso: Palladino.
- Barthes, Roland [1968] 1977: The Death of the Author. In Matéla V (1968). Rpt. ImageMusic-Text . Trans. Stephen Heath. London: Fontana Press, 142-148.
- Burdick, Anne 1996: An Interview with Stephen Farrell. In Émigré 37: no pagination, 6 pages.
- Burn, Andrew/Parker,  David  2003: Analysing  Media  Texts .  London  and  New York: Continuum.
- Carlsson, Maj Asplund et al. (eds) 2005: Multimodality: Text, Culture and Use .  Proceedings of the Second International Conference on Multimodality. Kristiansand: Agder University College/Norwegian Academic Press.
- Cranny-Francis, Anne 2005: Multimedia: Texts and Contexts . London: Sage.
- Crisp, Peter 2003: Conceptual metaphor and its expressions. In Gavins, Joanna/Steen, Gerard (eds.), Cognitive Poetics in Practice . London: Routledge, 99-113.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

- de Vega, Manuel et al. 2004: On doing two things at once: Temporal constrains on actions in language comprehension. In Memory & Cognition 32, 7, 1033-1043.
- Fisher, Jennifer 1997: Relational Sense: Towards a Haptic Aesthetic. In Parachute 87, 4-11.
- Fahie, J. J. 1903: Galileo: His Life and Work . London: John Murray.
- Forceville, Charles 1999: Educating the eye? Kress and van Leeuwen's Reading Images:  The  Grammar  of  Visual  Design (1996).  Review Article.  In Language and Literature 8, 2, 163-178.
- Forceville, Charles 2005: Visual representations of the idealized cognitive model of anger in the Asterix album la Zizanie . In Journal of Pragmatics 37, 69-88.
- Garland, Alex. 2004: The Coma . London: Faber and Faber.
- Gallese, Vittorio/Lakoff, George 2005: The Brain's Concepts: The Role of the SensoryMotor System in Conceptual Knowledge. In Cognitive Neuropsychology 22 (3/4), 455-479.
- Gavins,  Joanna/Steen,  Gerard  (eds.)  2003: Cognitive  Poetics  in  Practice .  London: Routledge.
- Gibbons, Alison forthcoming a: Multimodality and Cognition: reading word and image. In Baldry, Anthony/Montagna, Elena (eds.), Interdisciplinary Perspectives on Multimodality: Theory and Practice. Campobasso: Palladino.
- Gibbons, Alison forthcoming b: 'I contain multitudes': Narrative Multimodality and the book that bleeds. In Page, Ruth (ed.), Narrative and Multimodality: New Perspectives and Practices , presently under proposal with Routledge.
- Gibbs, Raymond W. 2005: Embodiment in Metaphorical Imagination. In Pecher, D./ Zwaan,  R. A.  (eds.), Grounding  Cognition:  The  Role  of  Perception  and  Action in  Memory,  Language,  and  Thinking .  Cambridge:  Cambridge  University  Press, 65-92.
- Gibbs,  Raymond.  W.  2006: Embodiment  and  Cognitive  Science .  Cambridge:  Cambridge University Press.
- Glenberg, A. M. 2007: Language and action: creating sensible combinations of ideas. In Gaskell, G. (ed.), Oxford Handbook of Psycholinguistics. Oxford: Oxford University Press, 361-370.
- Glenberg, Arthur M./Kaschak, Michael P. 2002: Grounding language in action. In Psychonomic Bulletin & Review 9, 3, 558-565.
- Hauk, O. et al. 2004: Somatopic Representation of Action Words in Human Motor and Premotor Cortex. In Neuron 41, 301-307.
- Johnson, Mark 1987: The Body in the Mind: The Bodily Basis of Meaning, Imagination and Reason. Chicago: University of Chicago Press.
- Kress, Gunther/van Leeuwen, Theo 1996: Reading Images: The Grammar of Visual Design . London: Routledge.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

- Kress, Gunther/van Leeuwen, Theo 2001: Multimodal Discourse: The Modes and Media of Contemporary Communication . London: Arnold.
- Lakoff, George/Johnson, Mark 1980: Metaphors We Live By .  Chicago  and  London: University of Chicago Press.
- Lakoff, George/Johnson, Mark 1999: Philosophy in the Flesh: The Embodied Mind and its Challenge to Western Thought . New York: Basic Books.
- Lakoff, George/Turner, Mark 1989: More than Cool Reason: A Field Guide to Poetic Metaphor . Chicago and London: University of Chicago Press.
- Lederman, Susan J./Klatzy, Roberta L. 2001: Designing haptic and multimodal interfaces: A cognitive scientist's perspective. In Farber, G./Hoogen, J. (eds.), Proceedings of Collaborative Research Centre 453. Technical University of Munich, 71-80.
- Marks, Laura (2002) Touch: Sensuous Theory and Multisensory Media . Minneapolis/ London: University of Minnesota Press.
- Næss, Atle 2005: Galileo Galileo - When the World Stood Still . New York: Springer.
- O'Halloran, Kay (ed.) 2004: Multimodal Discourse Analysis: Systemic Functional Perspectives . London and New York: Continuum.
- Pessl, Marisha 2006: Special Topics in Calamity Physics . London: Viking.
- Pulvermüller, Friedemann. 2005: Brain mechanisms linking language and action. In Neuroscience 6, 576-582.
- Pulvermüller, Friedemann et al. 2005: Functional links between motor and language systems. In European Journal of Neuroscience 21, 793-797.
- Royce, Terry D./Bowcher, Wendy L. (eds.) 2007: New Directions in the Analysis of Multimodal Discourse . Lawrence Erlbaum.
- Shen, Yeshayahu /Cohen, Michal 1998: How come silence is sweet but sweetness is not silent: a cognitive account of directionality in poetic synaesthesia. In Language and Literature 7, 2, 123-140.
- Stockwell, Peter, 2002: Cognitive Poetics: An Introduction . London: Routledge.
- Sweetser, Eve 1990: From Etymology to pragmatics: The mind-body metaphor in semantic structure and semantic change . New York: Cambridge University Press.
- Tettamanti, Marco et al 2005: Listening to Action-related Sentences Activatea Frontoparietal Motor Circuits. In Journal of Cognitive Neuroscience 17, 2, 273-281.
- Tomasula, Steve/Farrell, Stephen 2002: VAS: An Opera in Flatland . Chicago: University of Chicago Press.
- Turner,  Mark  2000: Death  is  the  Mother  of  Beauty:  Mind,  Metaphor,  Criticism . Christchurch: Cybereditions.
- Varela, Francisco J. et al. 1991: The Embodied Mind, Cognitive Science and Human Experience . Cambridge; London: MIT Press.
- Ventola, Eija et al. (eds.) 2004: Perspectives on Multimodality, First International Congress on Multimodality. Amsterdam and Philadelphia: John Benjamins Publishing Company.

<!-- image -->

<!-- image -->

<!-- image -->

<!-- image -->

- Wilson, Nicole/Gibbs, Raymond 2007: Real and Imagined Body Movement Primes Metaphor Comprehension. In Cognitive Science 3, 4, 721-731.
- Zwaan, Rolf. A. et al. 2004: Moving words: dynamic representations in language comprehension. In Cognitive Science 28, 611-619.
- Zwaan, Rolf. A./Taylor, L. J. 2006: Seeing, Acting, Understanding: Motor Resonance in Language Comprehension. In Journal of Experimental Psychology: General 135, 1, 1-11.

<!-- image -->

<!-- image -->