Visual-Narrative and Virtual Reality

This paper explores the relationship between the image and the narrative in virtual reality (VR) environments. It addresses three main issues; how we engage in images, how the VR technology can enhance this engagement and how narratives can maintain this engagement. We present an initial investigation into the development of a visual-narrative model to increase immersion and engagement in a VR environment and discuss the state of the art VR technology being developed by the Benogo project. Additionally, we propose a three-part study to elicit a set of design requirements to support the development of this visual-narrative model and report the initial stages of this study.  Images and the art of storytelling have a very long and intertwined history. In fact, more than 30,000 years ago, man was creating immersive environments by using images on cave walls to tell stories and transport the viewer into other worlds. These cave images not only tell us more about the lives of these people but also they are examples of ‘virtual environments of the earliest known form of human expression’ (“Through the looking glass”, 2000). Today we are living in a world where many forms of knowledge and entertainment are visually constructed and after centuries of ‘word’ dominance, we are finally realizing that ‘what we see is as important, if not more so than what we hear or read’ (Rose, 2001, p.1). This new post-linguistic and postsemiotic rediscovery of the picture (Mitchell, 1994, p.16) brings with it many new visualizing technologies which are steadily taking the place of written texts as a new exciting means of communication and storytelling. Virtual reality is one of these new and fascinating visual media. In this paper, we are exploring the notion of applying a narrative framework to images in a virtual environment. We claim that the creation of a visual-narrative in a VR environment will help generate a more immersive and engaging experience. We present a theoretical model for the development of a visual-narrative and propose a three-part study using the latest state of the art Benogo VR technology to elicit a set of requirements to support this model. We report the initial stages of this study.  We are beginning to appreciate the whole concept of the ‘spectator’, the action of looking, glancing, gazing and observing, as an alternative way of reading and understanding information. We are realizing that even though a visual rhetoric can inform literature, it can also exist on its own and is not dependent on incorporating a literary component. But how do we engage in images? What does it mean to read an image? All images can report information and ideas and ‘visual literacy allows the viewer to gather information and ideas contained in the image, place them in context and determine whether they are valid’ (Thibault and Walbert, 2003, p.1). But in more detail, when reading an image, we must firstly examine the image and then try to distinguish the composing parts that build the image, for example, the form of the image dot, line, color, shape, light, texture, the subject matter, the medium and the context. After that, it is a case of interpreting, thinking about and understanding what we see and then perhaps connecting a judgment. For effective visual communication, the image must have both ‘strength and clarity’ (Altengarten, 2002, p.1). The term ‘strength’ concerns the ability of the image to attract the viewers attention and the term ‘clarity’ refers to the ability of the image to keep the viewers interest. According to Csikszentmihalyi, for attention to be attracted to the object a whole other set of conditions are needed, in the sense that the object must contain a set of visual challenges that engage the interpretative skills of the beholder and the environment must encourage a centering of attention on the object and a screening out of distractions (1990, p.18-19). For example, when one reads a book for pleasure, the reader experiences an unforced concentration that transports them to another place (Nell, 1988, p.1). The reader is drawn into the world of the book and this in many ways screens out the real world and its distractions. In the visual experience, there is a difference between passive reception and active perceiving (Arnheim, 1969, p.14). As we look around a room, everything we see is there without having to do anything to produce it. We receive this information passively, its just exists but when we start to be attracted to certain objects and to want more information, then we become actively involved. John Berger claims that ‘we never look at just one thing; we are always looking at the relation between things and ourselves’ (1977, p.9). Whenever we try to make sense of information visually, we firstly see the similarities and the differences. These relationships permit us to not only identify objects but also to give them meaning. As soon as we have an understanding of the relationship between elements, we can stitch together the whole story and understand what we are seeing (Wroblewski, 2003, p.1). As Graham Coulter Smith points out ‘visualnarrative does not depend upon art incorporating a literary component, but that the rhetoric of visuality offers a complexity and sophistication that can inform literature’ (2000, p.105). Once we know the basic ways to visually distinguish objects (for example color, line, texture, shape, direction and size), we can concentrate on the bigger picture and use the visual relationships to tell a coherent story. Elements within a visual narrative are organized in a logical ordering, which can be described as a visual hierarchy (Wroblewski, 2003, p.2). To build effective visual hierarchies, we use visual relationships to add or subtract visual weight to image elements and thereby establish a pattern of movement through the whole picture. Visual weight can be measured by the extent to which an image element demands our attention or maintains our interest. Generally, a centre of attention attracts the viewer’s attention and this attention is kept by creating focal points that compel the viewer to look at them. Focal points have a strong attentive value and they help form the parts that glue the story together. However, it is important to point out ‘that a better understanding and mastery of visual communication requires not only this ability to read images but also a thorough understanding of the nature of these new visual technologies and media and their codes and conventions’ (Hugo, 1996, p.4).  The structure is still largely missing for virtual reality work (Bates, 1991, p.3): ‘Virtual reality has no firmly settled institutional pattern of distribution, exhibition or use’ (Lister et al., 2003, p. 108). To achieve this structure and to understand form in any art, we must be familiar with the medium (Bordwell and Kirsten, 1990, p.126) and of course, any study of one medium helps us to understand all others (McLuhan cited in Sherman et al, 1995, p.3). A virtual reality environment can be defined as ‘a computer generated three-dimensional landscape in which we would experience an expansion of our physical and sensory powers’ (Ryan, 2001, p.1). Through the use of certain virtual reality technologies, one can interact physically with the environment but also recent Image Based Rendering (IBR) virtual reality research (Benogo project) shows potential to visually explore, visually engage and visually immerse in these environments. In fact, ‘virtual reality has a long history of immersive images... the search for illusionary visual space can be traced back to antiquity’ (Grau, 2003). It is about trying to trick the viewer into believing they are in the same space as what they are seeing. When we look to the successful visual medium of cinema, we can get a clearer understanding of the foundations of the virtual reality phenomenon. In fact, like virtual reality, ‘early cinema based itself on its ability to show something, it solicits spectator attention, inciting visual curiosity and supplying pleasure through an exciting spectacle’ (Elsaesser, 1990, p.58). When we look carefully, we can begin to see ‘the clarity and seduction of cinema’s vi