GESTURE RECOGNITION FOR ROBOTIC CONTROL USING DEEP LEARNING

Can convolutional neural networks (CNNs) recognize gestures from a camera for robotic control? We examine this question using a small set of vehicle control gestures (move forward, grab control, no gesture, release control, stop, turn left, and turn right). Deep learning methods typically require large amounts of training data. For image recognition, the ImageNet data set is a widely used data set that consists of millions of labeled images. We do not expect to be able to collect a similar volume of training data for vehicle control gestures. Our method applies transfer learning to initialize the weights of the convolutional layers of the CNN to values obtained through training on the ImageNet data set. The fully connected layers of our network are then trained on a smaller set of gesture data that we collected and labeled. Our data set consists of about 50,000 images recorded at ten frames per second, collected and labeled in less than 15 man-hours. Images contain multiple people in a variety of indoor and outdoor settings. Approximately 4,000 images are held out for testing and contain a person not present in any of the training images. After training, greater than 99% of the images in the test set are correctly recognized. Additionally, we use the system to control a small unmanned ground vehicle. We also investigate using a Long Short-Term Memory (LSTM) layer for recognizing gestures that require analyzing sequences of images. On this more difficult set of gestures, we achieve a recognition rate of approximately 80% using a smaller data set of approximately 26,000 images. 1 This Project Agreement Holder (PAH) effort was sponsored by the U.S. Government under Other Transaction number W15QKN-08-9-0001 between the Robotics Technology Consortium, Inc, and the Government. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the U.S; Government. Proceedings of the 2017 Ground Vehicle Systems Engineering and Technology Symposium (GVSETS) Gesture Recognition for Robotic Control Using Deep Learning, Chris Kawatsu, et al. Page 2 of 7 INTRODUCTION Can we apply deep learning to recognize vehicle control gestures from a standard camera with high enough accuracy to control an unmanned vehicle? Our goal is to recognize standard gestures defined in Field Manual (FM) 21-60 [1] to allow warfighters to control an unmanned vehicle in the same manner as a vehicle driven by a human. SoarTech has previously investigated intuitive human-robot interfaces that leverage natural modes of interaction such as speech, gesture, and sketch to enable two-way dialogue between operators and robots. Our Smart Interaction Device (SID) [2] [3] has applied a speech and sketch interface on a tablet to control a variety of unmanned ground vehicles. The present paper focuses on adding gestures as an additional modality to SID. Gesture recognition varies considerably across two dimensions: the type of gestures, for example American sign language, and the type of sensor(s) used to recognize the gesture, for example an accelerometer. In the present paper, we limit ourselves to full body gestures used for vehicle control specified in FM 21-60. By using these gestures, warfighters should not need any additional training to use gestures to control an unmanned ground vehicle. Our gesture set consists of the following gestures: Attention, As You Were, Turn Right, Turn Left, Slow Down, Increase Speed, Halt, Move Forward, Move In Reverse. Examples of these gestures taken from FM 21-60 are shown in Figure 1 through Figure 4. In addition to these gestures, we also add two additional categories: No User and No Gesture. No User indicates that there is no person in the image. No Gesture indicates that the operator is standing with arms down and not performing a gesture. Figure 1:As You Were (left) and Attention (right). Figure 2: Turn Right (left) and Slow Down (right). Figure 3:Halt (left) and Increase Speed (right). Proceedings of the 2017 Ground Vehicle Systems Engineering and Technology Symposium (GVSETS) Gesture Recognition for Robotic Control Using Deep Learning, Chris Kawatsu, et al. Page 3 of 7 Figure 4: Move Forward (left) and Move In Reverse (right). There are a wide variety of sensors which can be used to recognize gestures. For arm gestures such as those in our vehicle control gesture set, the Microsoft Kinect provides very high-fidelity data for the position of each joint in the arm. We have previously used the Kinect on similar types of vehicle control gestures, and were able to achieve close to a 100% recognition rate. Unfortunately, the Kinect will not work in outdoor environments because it relies on an IR laser which is overwhelmed by sunlight. For outdoor operation, typically four types of sensors are used: accelerometers, Lidar, stereo cameras, or monocular cameras. Accelerometers and Lidar are both active sensors, while cameras are passive; therefore, assuming other considerations such as recognition rates are equal, cameras are the preferred solution. Lidar sensors that have high point density are prohibitively expensive. Stereo cameras are also expensive compared to monocular cameras and require significant processing power to perform stereo matching. For these reasons, we decided to use a monocular camera for our sensor. In recent years, deep learning approaches based on Convolutional Neural Networks (CNNs) have achieved state of the art results in a variety of image processing tasks such as object recognition and segmentation. Advances in this area have largely been driven by increases in processing power and the availability of large collections of labeled images to use during training. On the processing side, Graphics Processing Units (GPUs) increased enough in computation power and memory size to support running gradient descent on multiple layer neural networks with hundreds of millions of parameters. On the data side, competitions such as ImageNet [4] have released public datasets containing millions of labeled images which are necessary for training large neural networks. In order to take advantage of these improvements in image processing, we decided to use deep learning as the core of our gesture recognition system. METHODOLOGY Our gesture recognition system is subject to several constraints not considered in most deep learning research. First, there is no labeled dataset containing images of our vehicle control gestures. We must create and label training datasets ourselves. Second, we would like to run our gesture recognition system using on board computation power from a small unmanned ground vehicle, an iRobot PackBot. This means that a large GPU such as the Nvidia Titan X is out of the question. Our target is to run on the Nvidia Jetson which has 48GB of RAM and about one tenth the computational power of a Titan X. Our approach is designed to work around these constraints in two different ways. First, we initialize the weights of our CNN to values obtained by training on the ImageNet classification task. We then perform fine tuning of the upper layers of our network using our much smaller vehicle control gesture dataset. Second, we limit ourselves to CNN architectures which we expect to fit in memory of a Jetson and run in real time. Canziani et al. [5] provide an excellent comparison of popular CNN architectures shown in Figure 5. Initially we identified AlexNet [6] as the most promising network to run in real time on a Jetson TX1. With the release of the Jetson TX2 we have also evaluated using ResNet-50 [7] which provides a good tradeoff between performance and required operations. Proceedings of the 2017 Ground Vehicle Systems Engineering and Technology Symposium (GVSETS) Gesture Recognition for Robotic Control Using Deep Learning, Chris Kawatsu, et al. Page 4 of 7 Figure 5: Comparison of popular CNN architectures. The vertical axis shows top 1 accuracy on ImageNet classification. The horizontal axis shows the number of operations needed to classify an image. Circle size is proportional to the number of parameters in the