# Extending Appearance Based Gait Recognition with Depth Data
## 1 Introduction and Motivation  
Gait recognition has been studied extensively in recent years as a biometric identification method that does not require the user to approach the sensor or to interact with the recognition system. Moreover, human locomotion can be analyzed with low resolution cameras and is difficult to hide, making gait recognition a very attractive method in video surveillance and security systems. Depending on how the walking dynamics and body shape data are used in the recognition process, gait recognition techniques may be divided into a model and appearance based techniques. Model based techniques use explicit models to represent and track different parts of the body over time, such as legs and arms, and construct a gait signature that is later used during recognition. These techniques have high computational requirements for model construction and parameter estimation. Appearance based techniques, on the other hand, do not rely on body models. Instead, silhouette representations of persons are extracted from the images. The extraction is not computationally expensive, hence such methods are suitable for real-time usage. In recent years, there has been a growing proliferation of smart 3D sensors that in addition to the usual 2D camera video stream also provide depth information, often combined. Starting with Microsoft Kinect in 2011, several fast mass market 3D sensors became available that provide a real-time depth stream alongside the RGB stream. There are two types of such sensors, structured light sensors that illuminate the object with a light pattern and use a standard 2D camera to measure depth via triangulation, and time-of-flight sensors that measure depth by estimating the time delay from light emission to light detection. In both cases, an active low power IR projector is used to illuminate thetarget and the signal reflected from the target is captured with a receiver. A passive stereo camera setup can also be used to derive the depth information using stereo vision. Stereo vision is computationally expensive task, but it can provide a depth stream in environment conditions that are not suitable for depth sensing using active time-of-flight or structured light technologies, i.e., it can be used outdoors in sunlight and can capture the depth information with sufficient detail at distances larger than 10 m.  
Both active and passive depth sensors are easily available today, are able to provide RGB and depth streams in real-time, and can be used in different sets of environmental conditions. It is therefore of benefit to study how appearance based methods can be extended and modified to harness additional depth information that is now available. For example, the silhouettes can be extracted from depth rather than from RGB images. This is important because depth in many cases provides better foreground/background separation than color based algorithms since it is less affected by changes of lighting conditions, shadows, and foreground and background regions having similar colors. New gait features can be defined and constructed relying on additional volumetric information of the walking subject that is now available, i.e., capturing the movement of body parts in all three dimensions. Finally, a person's size can now easily be estimated using the available information of the subject's distance from the camera. With regard to silhouettes used by appearance based methods, this means that the size of the silhouette can now be used as a feature and taken into account for classification. It is of interest therefore to test the hypothesis that the addition of depth information can extend the applicability and improve the recognition accuracy of gait recognition methods without incurring a considerable performance penalty. In this work, two methods (Height-GEI integration (HGEI-i) and Height-GEI fusion (HGEI-f)) that enhance appearance based gait recognition methods by integrating the height of a person in the recognition process are proposed and studied.