Using Virtual Environments to Evaluate the Operational Benefit of Augmented Reality

At the Norwegian Defence Research Establishment (FFI) we have been experimenting with simulated Augmented Reality (AR) since 2006. We have used virtual environments to evaluate the operational benefit of AR functionality in combination with a Battlefield Management System (BMS) in combat vehicles. Three experiments have been carried out with professional combat-vehicle crews playing through relevant scenarios in virtual environments. The data collected from these experiments, including quantitative measurements, feedback from the participants through questionnaires and after-action review sessions, supplemented by observations of the usage of the system, is being used in the ongoing process of designing a real AR system for combat vehicles. The simulated AR system is designed for use in combat vehicles like infantry fighting vehicles and main battle tanks. It works in conjunction with an experimental BMS, also developed at FFI, and visualizes information like Blue-Force Tracking, observations and waypoints, in the form of graphical symbols directly in the sights and periscopes of the commander, gunner, and driver. This enables the vehicle crew to better exploit the BMS information while keeping their eyes fixed on what is going on in the battlefield. The AR system also makes the BMS information more intuitive. Basic input to the BMS can be given as simple voice commands, to provide a hands-free user interface. Initially our experiments with simulated AR were conducted using an in-house developed combat vehiclesimulator based on the commercial game Unreal Tournament 2004 (UT2004). Later, Virtual Battlespace 2 (VBS2) has taken over as our primary simulation platform for experimentation with simulated AR, since it offers a more complete virtual environment for military virtual simulations. The experiments have shown that the simulated AR system results in faster and more accurate perception of the BMS information, and thus better overall situational awareness. In small test scenarios we observed an average reduction of up to two thirds in target acquisition times. The general idea behind this work has been to test emerging technologies and new concepts in a virtual environment by developing virtual prototypes. Using this method we can evaluate the operational benefit of technology that is not yet available, and assess what properties it should have in order to give the maximum benefit to its users. This approach can also be used to evaluate operational performance and compare different systems in a procurement process. In this paper we describe the general method we have been using, the simulated AR system, and the experiments we have carried out. We also present the most important results from the experiments and the lessons learned from developing and working with a simulated AR system, which has served as a virtual prototype and a technology demonstrator. Using Virtual Environments to Evaluate the Operational Benefit of AR 5 2 STO-MP-MSG-126 1.0 INTRODUCTION At the Norwegian Defence Research Establishment (FFI) we have been experimenting with simulated Augmented Reality (AR) since 2006 [1]. We have been using virtual environments to evaluate the benefit of AR functionality in combination with a Battlefield Management System (BMS) in combat vehicles, by developing virtual prototypes. Three experiments have been carried out with professional combat-vehicle crews playing through relevant scenarios in virtual environments. The data collected from these experiments, including quantitative measurements, feedback from the participants through questionnaires and after-action review sessions, supplemented by observations of the usage of the system, is being used in the ongoing process of designing an operational AR system for combat vehicles. The simulated AR system is designed for use in combat vehicles like infantry fighting vehicles and main battle tanks. It works in conjunction with an experimental BMS, also developed at FFI, and visualizes information like Blue-Force Tracking (BFT), observations and waypoints, in the form of graphical objects displayed directly in the sights and periscopes of the commander, gunner, and driver. This enables the vehicle crew to better exploit the BMS information, even in critical battle situations, since it allows them to keep their eyes fixed on the battlefield. The AR system also makes the BMS information more intuitive, and increases the vehicle crew's overall situational awareness. Basic input to the BMS can be given as simple voice commands to provide a hands-free user interface. Firstly, this paper describes the background for this work, including a short introduction to AR technology. Then, the overall method used in this work is presented, followed by a description of the simulation system, including the latest version of the simulated AR system. After this, the simulation experiments are presented. Finally, this paper summarizes the most important results from the experiments, the lessons learned from this work, and some ideas for future work. 2.0 BACKGROUND Our first work with simulated AR, dating back to 2006, was for an FFI project on Battlefield Management Systems (BMS) for combat vehicles. We conducted a simulator experiment with the purpose of examining the operational benefit of possible future functionality in a BMS. This experiment was carried out by using an in-house developed simulator called NORBASE [2], which was based on the commercial game Unreal Tournament 2004 (UT2004). One of the possible features we examined was the use of AR to display BMS information directly in the commander’s and gunner’s sights. The main results from this experiment were that AR promised to be very useful, and the simple game-based simulator was accepted by all participants as an appropriate tool for experimenting with this kind of new technology [3]. Figure 1 shows images from the simulated AR system in NORBASE. Figure 1: The AR system in NORBASE from the early experiments with simulated AR in 2006. Using Virtual Environments to Evaluate the Operational Benefit of AR STO-MP-MSG-126 5 3 After this first experiment, the next step was to move AR from the laboratory to the field. For this task, collaboration with the Norwegian AR technology company Augmenti (then AR-Lab) was initiated. Augmenti had already developed AR technology for several civilian applications. In October 2008 we jointly conducted a field trial where AR in infantry fighting vehicles was demonstrated. The field trial was a success, and demonstrated not only that this technology was useful for the vehicle crew, but also that it would be possible to develop and implement this technology in the near future. Figure 2 shows the AR system prototype used in the field trial. After the field trial, FFI initiated a follow-on project aimed at developing AR for operational combat vehicles. Within this project further simulator experiments were conducted, the first of these in May 2009. This time VBS2 was used as simulation platform, since it offered a more complete virtual environment for military simulations. In addition, the Norwegian Army had now started to use VBS2 for training, and it was convenient for us to use a simulation platform familiar to the Norwegian Army. The latest, most comprehensive, and most successful, simulation experiment was carried out in November 2011. VBS2 was still used as simulation platform, but this time we had a much more detailed and sophisticated implementation of the AR system. One of the primary goals for this experiment was to get feedback from the participants to support the development of a real AR system. We needed to know how such a system might be used, what the participants considered to be the most important functionalities and aspects, and what they would like an AR interface to look like. The real-world AR system is being developed in parallel to our experiments, and the results from these experiments provide valuable input to this development process. A prototype of the real AR system was also demonstrated under the multinational exercise Bold Quest 2011 in September 2011. The AR system was then used in conjunction with a commercial BMS (Teleplan FACNAV), and the target acquisition and surveillance system Vingtaqs (from Vinghøg). Figure 2: Prototype of a real AR system used in the field trial in October 2008. 2.1 Augmented Reality (AR) AR is a technology for real-time mixing of virtual, computer-generated data with data we perceive from the real world. This gives the user an augmented perception of reality. Mainly, AR means adding virtual objects, in the form of computer graphics, to visual data from the real world. The virtual objects typically provide information in a way that improves the user’s situational awareness, thus helping him or her to perform realworld tasks better. Using Virtual Environments to Evaluate the Operational Benefit of AR STO-MP-MSG-126 5 5 2.3 Military Applications In military aircraft and helicopters pilots have been using Head-Up Displays (HUD) and Helmet-Mounted Sights (HMS) with AR for several years. These systems provide the pilot with navigation and flight information, and for some systems ground or air targets can be marked with graphics. AR systems that display tactical battlefield information are now being developed for ground soldiers and combat vehicles [6]. Typically these systems will be used to increase situational awareness through visualization of Blue-Force Tracking (BFT) data and points of interest like observations and targets. AR is also being used for military training [6][7], but so far mostly for experimentation purposes. Here virtual enemies are visualized in the real world [8][9], and live and simulator-based training are combined, so that real soldiers and vehicles can train together with virtual forces operated from simulators, in a two-way real-time interaction between real and virtual units. 2.4 Simulated AR By using high-end Vi