AN AUGMENTED REALITY UAV-GUIDED GROUND NAVIGATION INTERFACE IMPROVE HUMAN PERFORMANCE IN MULTI-ROBOT TELE-OPERATION

This research proposes a human-multirobot system with semi-autonomous ground robots and UAV view for contaminant localization tasks. A novel Augmented Reality based operator interface has been developed. The interface uses an over-watch camera view of the robotic environment and allows the operator to direct each robot individually or in groups. It uses an A* path planning algorithm to ensure obstacles are avoided and frees the operator for higher-level tasks. It also displays sensor information from each individual robot directly on the robot in the video view. In addition, a combined sensor view can also be displayed which helps the user pin point source information. The sensors on each robot monitor the contaminant levels and a virtual display of the levels is given to the user and allows him to direct the multiple ground robots towards the hidden target. This paper reviews the user interface and describes several initial usability tests that were performed. This research demonstrates the development of a humanmultirobot interface that has the potential to improve cooperative robots for practical applications. INTRODUCTION Multi-robot systems can often deal with tasks that are difficult for single robot. For example, teams of robots may be able to complete tasks such as multipoint surveillance, cooperative transport, and explorations in hazardous environments more efficiently. Additionally, time-critical missions may require the use of multiple robots working simultaneously to efficiently accomplish the tasks. Controlling multiple robots is a challenging humanoperator task. In multi-robot scenarios, one of the main challenges for a human operator in search and detection missions is to remotely control the semi-autonomous robots [1]. Thus, there is a need to research and develop technologies that can enable an operator to control groups of semi-autonomous robots. Most human-robot interfaces for robot control have focused on providing users data collected by the robot and giving status messages about what the robot is doing. The conventional interface consists of several separate display windows to show information from the robot. [2] is an example of a conventional display from the Idaho National Laboratory (INL). The display may require the operator to integrate information, and this may increase the operator’s workload. Another example of a conventional interface for multiple robots control was designed by Humphrey et al. [3]. Operators may have high workload from needing to simultaneously integrate each multiple status bar. An alternative to conventional interface is a 3D virtual environment display based on a robot simulation. In contrast to direct interfaces, a virtual environment provides an external perspective which allows the operator to see the environment and drive the robot from viewpoints generated by the interface. Nguyen et al. [4] describe several Virtual Reality (VR) based interfaces for exploration, one of which Proceedings of the 2011 Ground Vehicle Systems Engineering and Technology Symposium (GVSETS) An Augmented Reality UAV-Guided Ground Navigation Interface Improve Human Performance in Multi-Robot Tele-operation, Lee, et al. Page 2 of 7 is Viz. Viz has shown that VR interfaces can help the user understand and analyze the robot surroundings and improve the operator situational awareness. However, in virtual environments, the operator’s attention is drawn away from the actual environment which can reduce situational awareness and dynamic situations not modeled in the VR world could pose significant problems. In contrast to the virtual environment display, Augmented Reality (AR) is an advanced visualization technology which allows computer generated virtual images to merge with physical objects in real time. Unlike VR, the user enters and interacts with computer-generated 3D environments. AR allows the user to interact with the virtual images using real objects [5]. Researchers in robotics are beginning to use AR techniques in robotics because it provides direct views of the scene combined with the advantages of virtual displays for human-robot collaboration [6-8]. Communicating to robots and human, on the other hand, touch-based input may allow users to perform complex tasks in an intuitive manner [9]. Micire et al. [10] studied the control of a single agent with a multi-touch table. Moreover, a multi-touch (DREAM) controller [11, 12] using a multitouch table is developed for multi-robot command and control [13, 14]. Figure 1: A group of semi-autonomous robots is controlled using the human-multirobot interface. We have developed a system to combine virtual and augmented reality interfaces capabilities with human supervisor’s ability to control the robots [15]. The role of this human-multirobot interface is to allow an operator to control group of heterogeneous robots in real time and in collaborative way. This paper presents results from a user evaluation of the real multiple robot system in which three interface conditions were evaluated (i.e. Joystick, Point-andGo, and Path Planning). Results show that the novel multirobot control (Point-and-Go and Path Planning) reduced their mission completion times compared to the traditional joystick control for target detection performance. SYSTEM DESCRIPTION The system hardware configuration is shown in Figure 1. The human-multirobot interface (see Figure 2) is a top-down view from the stationary camera. We assumed that the topdown view could be taken from a manned or unmanned aerial vehicle. Figure 2: The human-multirobot interface is a top-down view from the stationary camera. We assumed that the topdown view could be taken from a manned or unmanned aerial vehicle. Hardware Four Mindstorms NXT robots were used as the remote robots. The NXT robot includes two NXT motors with encoders used for differential drive and a third passive caster wheel to maintain balance. An infrared sensor with a 240 degree view is attached on the NXT robot (see Figure 3) to search and detect infrared beacons. A marker on top of a NXT robot is used for position reading of the multi-robot system and for viewing robot status using AR software. The NXT robots are controlled through a Bluetooth connection. Proceedings of the 2011 Ground Vehicle Systems Engineering and Technology Symposium (GVSETS) An Augmented Reality UAV-Guided Ground Navigation Interface Improve Human Performance in Multi-Robot Tele-operation, Lee, et al. Page 3 of 7 A HiTechnic infrared electronic ball was used as a contaminant source. The infrared ball was hidden by one of the decoys. The testbed was equipped with a Logitech Webcam Pro 9000 with autofocus to obtain video frames at a resolution of 1280x1024 and at a refresh rate of 10 frames/ sec. The video was displayed on a 17” liquid crystal display (LCD) computer monitors. Figure 3: A marker is attached on the top of a NXT robot. The marker is detected by AR software to measure a robot position and to generate virtual images to merge with a robot in real time. An infrared sensor with a 240 degree view is attached on the NXT robot to search and detect infrared beacons. Software The ARToolKit augmented reality system is used to determine the position and orientation of each robot. A marker is attached to the top of each robot. Client‐server system for data communication has developed in the testbed. The robot server is programmed to communication with NXT robots using Bluetooth. AR client delivers localization data obtained from an overhead camera to ground robots, and it displays the synthesized views. Visual C++ in Visual Studio 2008 is the programming language used for AR and robot communication API software development. Not eXactly C (NXC) in Bricx Command Center is the programming language used for NXT robot to configure the infrared sensor and robot communication. Figure 4: AR client programs share robot state information and display the synthesized view. Robot server programs read robot sensor data and send movement commands. AR INTERFACE FOR MULTI-ROBOT CONTROL This section describes three features of the interface used for user evaluation. First, we describe the Point-and-Go algorithm developed for multi-robot control and the Path Planning implemented for obstacle avoidance. Then, we describe the joystick control of the multiple robots. Finally, we explain the sensor data and robot messages visualization. Point-and-Go Mode A point‐and‐go algorithm is developed for single human operator controlling multiple robots. The operator is able to select any ground robot using a mouse left click, and then designates a goal location in the video feed from an overhead camera (see Figure 5). A navigation algorithm is developed that allows the robot turns toward the desired goal location, drive straightly toward the goal, and then stops at the target. If a robot is stuck with an obstacle, the user is able to reverse the robot using mouse right click (see Figure 5). Figure 5: Point-and-Go is a high level instruction that allows an operator to control multiple semi-autonomous robots simultaneously. Path Planning Mode An A* algorithm path planning algorithm [16] is implemented to the system. The A* algorithm allows the multiple robots to traverse to the target location with obstacle avoidance and shortest path. Proceedings of the 2011 Ground Vehicle Systems Engineering and Technology Symposium (GVSETS) An Augmented Reality UAV-Guided Ground Navigation Interface Improve Human Performance in Multi-Robot Tele-operation, Lee, et al. Page 4 of 7 Figure 6: The path planning algorithm allows the multiple robots to traverse to the target location with obstacle avoidance and shortest path. Joystick Mode A joystick (ExtremeTM 3D Pro; Logitech, California) was used to manipulate the direction in which the robot moved when in joystick mode. The joystick push-pull axe was used to control the forward and back movements for translation of the robot, and the joystick rotati