# Extending Appearance Based Gait Recognition with Depth Data
## 3 Image Processing and Feature Extraction
### Input Video Stream Processing  
Depending on the technique used to acquire depth information from the scene, sensors that provide both RGB and depth video streams in real-time, i.e., 3D cameras, can be divided into two categories: active or passive sensors.  
Active sensors use an active source of energy, i.e., a projector, to emit a laser beam or a pattern of light, typically in the infrared spectrum, and a receiver to capture the light reflected from the objects in the scene. Common active sensors are laser range scanners, and, more recently, consumer grade sensors based on structured light and time-of-flight technologies [11]. Sensors based on structured light and time-of-flight technologies, besides being typically much cheaper than laser scanners, have the advantage of capturing the entire scene at the same time, making them suitable for usage in gait recognition and other dynamic scenes applications.  
Active 3D sensors have a limited range due to the projector power being limited for safety reasons. The typical range of consumer grade active sensors used for gait recognition is around 5 m. Furthermore, they are either not usable or perform poorly outdoors when daylight is strong because the sunlight overwhelms the projected pattern, making it undetectable. Moreover, identical sensors, using the same frequency and pattern, should not overlap the view to avoid interference. The raw depth images are typically noisy and contain artifacts and missing pixels, thus the first stage of image processing usually comprises filtering the noise (e.g., median filter) and applying morphological image processing operations such as dilate and emboss operators.  
Passive sensors, on the other hand, do not use an active projector at all. To obtain the image of the scene, the sensor relies on the light naturally present in the scene (either sunlight or from external artificial light sources). The calculation of the depth information is not based on the structure of light source or time measurements. Instead, two or more receivers (e.g., RGB or infrared cameras) are used to recover the range information using stereo vision processing techniques. Depending on the resolution, the baseline distance between cameras, and the lighting conditions in the scene, stereo vision is not intrinsically limited in range, like active sensors, but instead capable of calculating depth information at much larger distances. In addition, they can be used outdoors without any problem. Disadvantages of passive sensors are the additional computational burden for stereo vision processing, and the presence of undesired artifacts in the resulting depth images, due to, for example, featureless parts of the scene, where correspondences between the images of cameras cannot be found, thus no depth information can be extracted. Without an active projector, they are also more sensitive to reflections, shadows, and other features in the scene. This is noticeable in Figure 1.  
The image processing steps applied to raw depth images from passive sensors are similar to those of the active sensors; however, to obtain better results, they should be fine tuned after installation of the cameras and specifically for the particular application and scene requirements.