Mixed Reality and the interactive imagination : Adding the art to the science and technology of Mixed Reality for training , education and entertainment

Technology has advanced to the point where realism in virtual reality is very achievable. However, in our obsession to reproduce the world and human experience in virtual space, we overlook the most important aspects of what makes us who we are—our reality. Yet, it isn’t enough just to trick the eye or fool the body and mind. One must capture the imagination in order to create truly compelling experiences. Mixed Reality (MR)) allows for the merging of our imagination with our physical self and virtual presence. With the additional convergence of entertainment simulation, artistic convention is playing an ever-growing importance in sparking the imagination to achieve compelling and effective simulations. This paper explores the artistic and technical research challenges of mixing these realities with the imagination based on the work at the Media Convergence Laboratory (MCL) presently being conducted at the at the University of Central Florida (UCF). Introduction: Putting the stimulation in simulation The Chief Scientist from the US Army’s Simulation Training and Instrument Command (STRICOM) remarked to the creative producer of Universal Studio’s Spiderman, The (simulation) Ride, “You know, we are in the same business really. We are in the business of making memories that people will never forget.” During the same panel discussion at the 2000 ACM/SIGGRAPH conference, leaders and visionaries of the future of military and entertainment simulation found more in common than they found in their differences. It is not the intent to confuse the importance of training the next generation soldier for life and death scenarios with extravagant amusement park attractions, but to reflect on the importance that simulations need to be more than realistic. They need to be compelling and engaging to make a difference in one’s life and maybe save one’s life. It is the impact on the imagination that can seal a simulated experience into one’s memory for use in preparing for war, learning or just for thrills. In the study of Mixed Reality at the Media Convergence Laboratory, the creative techniques and technology of both experience-based entertainment (theme parks, video games, virtual sports, etc.) and simulation training, are ever more dependent upon each other to extend the art and science of simulation to reach the next generation of simulation experience and obtain its purpose in entertainment, education and training. Background: Evolution of Mixed Reality Not until the late in the last century, have computergenerated virtual simulation been able to engage successfully with live simulation. Both have had their own advantages, but the choice of the user was to use either Virtual or Live simulation. Virtual Reality (VR) was extremely pliable to one’s purpose, but consisted of a disembodied experience. Live Simulation was significant in providing to a soldier the “ground truth” of Combat Reality (CR) in order to properly test the skills under fire. With the introduction of Augmented Reality (AR) as Boeing coined the term in the early nineties, we were able Unmixed Reality: Either Live Or Virtual to apply virtual objects within physical reality. Combining techniques of Sutherland/Sproull’s first optical see-through Head Mounted Display (HMD) from the early 1960’s with complex, real-time, computer-generated wiring diagrams and manuals. Both were registered with each other and manuals were embedded within the actual aircraft for intensely detailed procedures. Yet the challenges in merging the realities in a seamless immersion were truly limited. Optical see-through HMDs produced transparent virtual entities that would be ghosting through reality. Spatial orientation of the user and virtual objects within reality were in conflict due to the lack occlusion of virtual objects with real foreground objects. Items that were seen accurately with 3D perception through stereoscopic HMD would lose their effect when a real object tried to pass in front of the virtual image. In most cases, it was with the user’s own hand or device that would quickly destroy the illusion and flatten the 3D effect without the parallax of foreground object produced by proper occlusion. We depend upon the parallax of the background with foreground objects in order to negotiate spatial environments and to maintain the illusion of the visual hybrid perception. In 1995, Mixed Reality Laboratories (headed by Hideyuki Tamura, PhD) took on an ambitious research initiative to effectively blend the entire spectrum of these realities, which was coined as Mixed Reality (MR) (Milgram 1994). With research in several aspects of technology from optics, graphics generation, video imagery, computer vision, a video see-through Head Mounted Display (HMD) was developed to more accurately represent the mixing of these realities. This was dramatically demonstrated with AquaGauntlet mixed reality game presented at the 2000 ACM/SIGGRAPH conference in New Orleans. This was a significant development in order to consider MR for incorporation into the media research at the Media Convergence Laboratory. The application of augmented reality went beyond the typical ‘text in space” to creating a more seamless interactive MR environment for art and entertainment. There was still significant work to be done in applied research. Mixed Reality Lab, (2000 Canon Inc.) Experiential media research: All media is Mixed Reality Many can easily see how any type of Media can be considered virtual reality, because it takes you somewhere far away from reality into your mind’s eye. Even though the technology is over one hundred years old, feature films are still one of the most immersive media forms of today. That is because over that time, artistic convention has matured the function of the motion picture to transcend the original limitations of merely capturing motion. It is now a powerful expressive art form. So compelling is the media that in the United States, feature films are only a nine million dollar industry, but it impacts a hundred billion dollar market with subsequent product markets. With a hundred years of passive media captivating the imagination and pacifying audiences, it is more difficult to understand how even traditional media is a form of Mixed Reality. Where does the reality meet the virtual story world? All experiences are physical and interact with the real world in some way. Whether you curl up with a romance novel in front of the rustic fireplace or escape into an exhilarating adventure at a theme park. It is usually through the venue that mediates the desires of the physical world with intension of the storyteller through the media reality. Both play a critical role in the audience’s imagination and suspension of disbelief. The major difference is that one venue is designed to pacify the audience physical presence and the other is designed to engage and interact with the audience. It is in the limitation of the technology of traditional media not being able to change that does not allow it to interact with the audience as does live theater. Cinema heightens the fidelity of the limited modalities it can control in order to transport the person within their own imagination. For Augmented Unmixed Reality: Either Live Or Augmented Or Virtual Melgram’s Reality-Virtuality Continuum (1994)