Visual-Narrative and Virtual Reality

This paper explores the relationship between the image and the narrative in virtual reality (VR) environments. It addresses three main issues; how we engage in images, how the VR technology can enhance this engagement and how narratives can maintain this engagement. We present an initial investigation into the development of a visual-narrative model to increase immersion and engagement in a VR environment and discuss the state of the art VR technology being developed by the Benogo project. Additionally, we propose a three-part study to elicit a set of design requirements to support the development of this visual-narrative model and report the initial stages of this study. Introduction Images and the art of storytelling have a very long and intertwined history. In fact, more than 30,000 years ago, man was creating immersive environments by using images on cave walls to tell stories and transport the viewer into other worlds. These cave images not only tell us more about the lives of these people but also they are examples of ‘virtual environments of the earliest known form of human expression’ (“Through the looking glass”, 2000). Today we are living in a world where many forms of knowledge and entertainment are visually constructed and after centuries of ‘word’ dominance, we are finally realizing that ‘what we see is as important, if not more so than what we hear or read’ (Rose, 2001, p.1). This new post-linguistic and postsemiotic rediscovery of the picture (Mitchell, 1994, p.16) brings with it many new visualizing technologies which are steadily taking the place of written texts as a new exciting means of communication and storytelling. Virtual reality is one of these new and fascinating visual media. In this paper, we are exploring the notion of applying a narrative framework to images in a virtual environment. We claim that the creation of a visual-narrative in a VR environment will help generate a more immersive and engaging experience. We present a theoretical model for the development of a visual-narrative and propose a three-part study using the latest state of the art Benogo VR technology to elicit a set of requirements to support this model. We report the initial stages of this study. How Do We Engage In Images? Using the image to communicate is the nearest we can get to the true nature of reality (Dondis, 1973, p.2). Since prehistoric cave paintings, humans have used the image to deliver their message to the world. And regardless of the fact, that over time, the image has evolved, first into the pictogram and then in turn into the alphabet, (which subsequently has seen the ascendancy of the word in terms of a communicative tool), we are still, today, using images as a primary source of information and communication (Hugo, 1996, p.4). In fact, we are now, more so than any other era, living in a world of images and we are surrounded by different sorts of visual technologies and media. As Nicolas Mirzoeff describes: ‘Visual technology is any form of apparatus designed either to be looked at or to enhance natural vision from oil painting to television and the Internet’ (1998, p.3). It is through this developing technology that we are slowly beginning to realize once again the true values of the image. The new age in communication technology makes evident what has been known and understood about images since they were first created (Lester, 1994-1996). We are beginning to appreciate the whole concept of the ‘spectator’, the action of looking, glancing, gazing and observing, as an alternative way of reading and understanding information. We are realizing that even though a visual rhetoric can inform literature, it can also exist on its own and is not dependent on incorporating a literary component. But how do we engage in images? What does it mean to read an image? All images can report information and ideas and ‘visual literacy allows the viewer to gather information and ideas contained in the image, place them in context and determine whether they are valid’ (Thibault and Walbert, 2003, p.1). But in more detail, when reading an image, we must firstly examine the image and then try to distinguish the composing parts that build the image, for example, the form of the image dot, line, color, shape, light, texture, the subject matter, the medium and the context. After that, it is a case of interpreting, thinking about and understanding what we see and then perhaps connecting a judgment. For effective visual communication, the image must have both ‘strength and clarity’ (Altengarten, 2002, p.1). The term ‘strength’ concerns the ability of the image to attract the viewers attention and the term ‘clarity’ refers to the ability of the image to keep the viewers interest. According to Csikszentmihalyi, for attention to be attracted to the object a whole other set of conditions are needed, in the sense that the object must contain a set of visual challenges that engage the interpretative skills of the beholder and the environment must encourage a centering of attention on the object and a screening out of distractions (1990, p.18-19). For example, when one reads a book for pleasure, the reader experiences an unforced concentration that transports them to another place (Nell, 1988, p.1). The reader is drawn into the world of the book and this in many ways screens out the real world and its distractions. In the visual experience, there is a difference between passive reception and active perceiving (Arnheim, 1969, p.14). As we look around a room, everything we see is there without having to do anything to produce it. We receive this information passively, its just exists but when we start to be attracted to certain objects and to want more information, then we become actively involved. John Berger claims that ‘we never look at just one thing; we are always looking at the relation between things and ourselves’ (1977, p.9). Whenever we try to make sense of information visually, we firstly see the similarities and the differences. These relationships permit us to not only identify objects but also to give them meaning. As soon as we have an understanding of the relationship between elements, we can stitch together the whole story and understand what we are seeing (Wroblewski, 2003, p.1). As Graham Coulter Smith points out ‘visualnarrative does not depend upon art incorporating a literary component, but that the rhetoric of visuality offers a complexity and sophistication that can inform literature’ (2000, p.105). Once we know the basic ways to visually distinguish objects (for example color, line, texture, shape, direction and size), we can concentrate on the bigger picture and use the visual relationships to tell a coherent story. Elements within a visual narrative are organized in a logical ordering, which can be described as a visual hierarchy (Wroblewski, 2003, p.2). To build effective visual hierarchies, we use visual relationships to add or subtract visual weight to image elements and thereby establish a pattern of movement through the whole picture. Visual weight can be measured by the extent to which an image element demands our attention or maintains our interest. Generally, a centre of attention attracts the viewer’s attention and this attention is kept by creating focal points that compel the viewer to look at them. Focal points have a strong attentive value and they help form the parts that glue the story together. However, it is important to point out ‘that a better understanding and mastery of visual communication requires not only this ability to read images but also a thorough understanding of the nature of these new visual technologies and media and their codes and conventions’ (Hugo, 1996, p.4). How Does The VR Technology Enhance Engagement? Virtual Reality is predominantly an image environment, (Lister, Dovey, Giddings, Grant, and Kelly, 2003, p.35). It is a visual technology (Bates, 1991, p.3), which allows for the communication of ideas (Sherman and Craig, 1995, p.2). However, in order for it to join the cinema and television as a successful artistic medium, it needs to have its own language with which to communicate more effectively. The problem is that virtual reality is a relatively new medium and, as such, has not developed its language (Sherman, 1995, p.3). The structure is still largely missing for virtual reality work (Bates, 1991, p.3): ‘Virtual reality has no firmly settled institutional pattern of distribution, exhibition or use’ (Lister et al., 2003, p. 108). To achieve this structure and to understand form in any art, we must be familiar with the medium (Bordwell and Kirsten, 1990, p.126) and of course, any study of one medium helps us to understand all others (McLuhan cited in Sherman et al, 1995, p.3). A virtual reality environment can be defined as ‘a computer generated three-dimensional landscape in which we would experience an expansion of our physical and sensory powers’ (Ryan, 2001, p.1). Through the use of certain virtual reality technologies, one can interact physically with the environment but also recent Image Based Rendering (IBR) virtual reality research (Benogo project) shows potential to visually explore, visually engage and visually immerse in these environments. In fact, ‘virtual reality has a long history of immersive images... the search for illusionary visual space can be traced back to antiquity’ (Grau, 2003). It is about trying to trick the viewer into believing they are in the same space as what they are seeing. When we look to the successful visual medium of cinema, we can get a clearer understanding of the foundations of the virtual reality phenomenon. In fact, like virtual reality, ‘early cinema based itself on its ability to show something, it solicits spectator attention, inciting visual curiosity and supplying pleasure through an exciting spectacle’ (Elsaesser, 1990, p.58). When we look carefully, we can begin to see ‘the clarity and seduction of cinema’s vi