# Background Subtraction for Moving Object Detection in RGBD Data: A Survey
## 2 RGBD Data and Related Issues for Background Subtraction  
Color cameras are based on sensors like CCD or CMOS, which provide a reliable representation of the scene with high-resolution images. Background subtraction using this kind of sensors often results in a precise separation between foreground and background, even though well-known scene background modeling challenges for moving object detection must be taken into account [PERSON] et al. (2018); [PERSON] et al. (2018):  
* **Bootstrapping**: The challenge is to learn a model of the scene background (to be adopted for background subtraction) even when the usual assumption of having a set of training frames empty of foreground objects fails. * **Color Camouflage**: When videos include foreground objects whose color is very close to that of the background, it is hard to provide a correct segmentation based only on color. * **Illumination Changes**: The challenge is to adapt the color background model to strong or mild illumination changes to achieve an accurate foreground detection. * **Intermittent Motion**: The issue is to detect foreground objects even if they stop moving (abandoned objects) or if they were initially stationary and then start moving (removed objects). * **Moving Background**: The challenge is to model not only the static background but also slight changes in the background that are not interesting for surveillance, such as waving trees in outdoor videos. * **Color Shadows**: The challenge is to discriminate foreground objects by shadows cast on the background by foreground objects that apparently behave as moving objects. Depth sensors provide partial geometrical information about the scene that can help solving some of the above problems. A depth image, storing for each pixel a depth value proportional to the estimated distance from the device to the corresponding point in the real world, can be obtained with different methods [PERSON] et al. (2018):  
1. _Stereo vision_[PERSON] et al. (2018): this is a passive technique where the depth is derived from the disparity between images captured from a camera pair. Stereo vision systems need to be well-calibrated and can fail when the scene is not sufficiently textured. Moreover, algorithms for stereo reconstruction are often computationally expensive. Finally, stereo vision systems cannot work in low light conditions. In this case, infrared (IR) lights can be added to the system, but then, the color information is lost, which generates segmentation and matching difficulties. 2. _Time-of-Flight_ (ToF) [29]: ToF cameras are active sensors that determine the per-pixel depth value by measuring the time taken by IR light to travel to the object and back to the camera. A ToF camera provides more accurate depth images than a stereo vision system, but it is very expensive and limited to low image resolution. The measured depth map can be noisy both spatially and temporally, and noise is content-dependent and hence, difficult to remove by traditional filtering methods. 3. _Structured light_[30]: A structured light sensor consists of an IR emitter and an IR camera. The emitter projects an IR speckle pattern onto the scene; the camera captures the reflected pattern and correlates it against a stored reference pattern on a plane, providing the depth values. Well known examples include the Microsoft Kinect version 1 (in the following simply named Kinect) and the Asus Xtion Pro Live. These sensors can acquire higher resolution images than a ToF camera at a lower price. The drawback is that depth information is not always well estimated at the object boundaries and for areas too far from/too close to the IR projector. Also, the noise in depth measurements increases quadratically with increasing distance from the sensor [31]. Even though depth data solves some of the previously highlighted background maintenance issues, being independent of scene color and illumination conditions, it suffers from several problems, independent of which technology is used for its estimation.