Adaptive pulsed laser line extraction for terrain reconstruction using a dynamic vision sensor

Mobile robots need to know the terrain in which they are moving for path planning and obstacle avoidance. This paper proposes the combination of a bio-inspired, redundancy-suppressing dynamic vision sensor (DVS) with a pulsed line laser to allow fast terrain reconstruction. A stable laser stripe extraction is achieved by exploiting the sensor's ability to capture the temporal dynamics in a scene. An adaptive temporal filter for the sensor output allows a reliable reconstruction of 3D terrain surfaces. Laser stripe extractions up to pulsing frequencies of 500 Hz were achieved using a line laser of 3 mW at a distance of 45 cm using an event-based algorithm that exploits the sparseness of the sensor output. As a proof of concept, unstructured rapid prototype terrain samples have been successfully reconstructed with an accuracy of 2 mm.

section_content: The Kinect sensor by Microsoft Corp. was introduced to the market in November 2010 as an input device for the Xbox 360 gaming console and was a very successful product with more than 10 million devices sold by March 2011 [1]. The Computer Vision society quickly discovered that the depth sensing technology in the Kinect could be used for other purposes than gaming and at a much lower cost than traditional 3D-cameras (such as time-of-flight based cameras). In June 2011, Microsoft released a software development kit (SDK) for the Kinect, allowing it to be used as a tool for non-commercial products and spurring further interest in the product [2]. 

The technology behind the Kinect sensor was originally developed by the company PrimeSense which released their version of an SDK to be used with the Kinect as part of the OpenNI organization [3]. The SDK should be independent of devices and, as of now, the company ASUS also produces a sensor with much of the same capabilities as the Kinect sensor (including depth sensing) that works with the OpenNI SDK. 

A third approach, and actually the first, to using the Kinect sensor came from the "hacker society" which reverse-engineered the USB-stream of data from the Kinect device (before any SDKs were released). This led to the OpenKinect community that is "working on free, open-source software" on multiple platforms [4]. 

There has been a lot of interest in the sensor from the Computer Vision society, however, there are not many extensive scientific investigations of the sensor at the moment. In [6], calibration of the sensor is investigated along with issues about multiple sensors with overlapping field-of-views. In [7] the sensor is used as a motion capture system and [10] evaluate the sensor for 1. INTRODUCTION indoor map building. Careful error analysis of the depth measurements from the Kinect sensor is made in [9]. Our approach is system-oriented and touches on many of the different issues in building real-life Computer Vision systems with the depth sensor. The issues include choice of software framework, depth resolution and precision, spatial resolution and precision, structural noise and multi-cam setups. 

Chapter 2
section_name: Chapter 1 Introduction

section_content: The basic principle behind the Kinect depth sensor is emission of an IR pattern (see figure 2. 1a) and the simultaneous image capture of the IR image with a (traditional) CMOS camera that is fitted with an IR-pass filter (figure 2. 1b). The image processor of the Kinect uses the relative positions of the dots in the pattern to calculate the depth displacement at each pixel position in the image [8], [9]. It should be noted that the actual depth values are distances from the camera-laser plane rather than distances from the sensor itself (figure 2. 3). As such, the depth sensor can be seen as a device that returns (x,y,z)-coordinates of 3D objects. 

The Kinect hardware specifications have been described in many documents [5], [1]. The main nominal specifications are shown in table 2. 1. These are the specifications that we will evaluate and investigate more carefully in the following.
section_name: Kinect depth sensor technology

section_content: Value Angular Field-of-View 57 • horz., 43 The Microsoft SDK is also limited to depths above 0. 8 meters whereas the other two frameworks go down to 0. 5 meters.
section_name: Property

section_content: Figure 3. 1 illustrates the bit values in relation to actual depth values for each of the three software frameworks. The bit values are calculated in the standard way by converting the bit patterns to unsigned integers. It is seen that the OpenKinect framework delivers uncalibrated data without any linearization. This means that every bit value has a distinct depth value. The other two frameworks instead linearize the bit values such that they represent millimeters exactly. This is obviously easier to use in practice, but also implies that not all bit values are possible. Apparently, the linearization simply corresponds to a look-up table conversion.
section_name: KINECT DEPTH SENSOR TECHNOLOGY

section_content: similar results since they use the same hardware and are both linearized. The resolution of the two frameworks translate directly to the smallest possible depth difference the Kinect can meassure at a certain distance. E. g. at 6 m the smallest depth difference meassurable is approximately 0. 1 m. The most significant difference between the two frameworks is the limited range of the Microsoft SDK. The OpenKinect framework, on the other hand, is not linearized and therefore always has a resolution of 1 bit. As the depth value is not linearized this does not mean that the OpenKinect can meassure smaller depth differences. The figures 3. 1 and 3. 2 both come from an initial experiment where a large cardboard box was positioned at different distances from the Kinect sensor and an area of the cardboard box was used to correlate bit values to actual measured depths as well as measuring depth resolutions. The depth resolutions were simply found as the smallest change in depth value in the area on the cardboard box. 

After evaluating all three frameworks, we decided to use only the OpenNI framework for the reasons given in the previous. Therefore, the rest of the
section_name: COMPARISON OF SOFTWARE FRAMEWORKS

section_content: report and experiments only use the OpenNI framework which is examined in more detail.
section_name: COMPARISON OF SOFTWARE FRAMEWORKS

section_content: The purpose of this section is to describe the properties of the Kinect depth sensor. A series of experiments constitutes the basis for the description of the resolution, accuracy and precision properties of depth as well as spatial information from the sensor. 

All experiments in this chapter have been done using the OpenNI software framework.
section_name: Chapter 4 Properties of the sensor

section_content: To investigate the linearity of the sensor, the sensor is pointed perpendicular towards a planar surface at different distances. For each distance the depth estimates are averaged over a small area at the center of the image and compared to the actual distance. The depth estimates are very close to linear within the measuring range of the sensor. This is shown in figure 4. 1, which shows the averaged depth value as a function of distance. The measuring range is approximately 0. 5 m to 9. 7m. If the limits of the measuring range are exceeded, the depth estimate is set to 0. This indicates that the depth sensor is not able to make a depth estimate. The raw measurements provided by the sensor are non-linear, but the data are linearized in the OpenNI software as described earlier. For more information about the non-linearities of the raw data, see e. g. [11].
section_name: Linearity

section_name: LINEARITY

section_content: The experiment for quantifying the resolution is described in the following. In this setup, the depth sensor is pointed perpendicular towards a planar surface and a depth frame is recorded. This is repeated at different distances to the planar surface. For each distance all unique depth values in the recorded frame are extracted and sorted accordingly. Since the real world is continuous, ideally all values between the smallest and largest depth value in a given frame should be present in the sorted depth data. However, since we are dealing with discrete data we are assuming that every possible discrete value between the smallest and largest value is present. The smallest difference between two adjacent values in the sorted list is perceived as the resolution for the given distance. 

The result is shown in figure 4. 2, which clearly shows that the resolution is getting coarser as the distance increases. Therefore it is necessary to consider needs for range and resolution before using the Kinect sensor for a given computer vision application. The given resolution can be adequate for e. g. navigating a robot through large obstacles, but it might be insufficient if you are trying to pick up chocolate buttons off a production line.
section_name: Depth Resolution

section_content: Even if the sensor setup is stationary, if one observes a given depth pixel through time, one will observe that the value of the pixel will fluctuate between a small number of bit levels. To illustrate this phenomenon, we have pointed the Kinect sensor towards a planar surface and recorded the value of a fixed pixel through 1000 frames with a sample rate of 15 frames per second. The actual measured distance from the surface to the Kinect sensor is approximately 2m. Figure 4. 3 shows a normalized histogram of the bit values at a specific position. The value of the pixel is fluctuating between four different levels ranging from 2004mm to 2040mm. That is, the precision is roughly 40mm at a distance of 2m. 

The spacing between the bins in figure 4. 3 suggests that the resolution of the sensor at a distance of 2 meters is approximately 12mm, which is consistent with figure 4. 2. However, it is important to emphasize the fact that not all pixels are fluctuating between exactly four different levels. To illustrate this, we have measured the number of levels for all 640x480 pixels Clearly the resolution is getting coarser as the distance increases. It appears that with a time span of 10,000 samples roughly 2. 5 percent of the pixels have only one level, 20 percent of the pixels have two different levels, 38 percent of the pixel have three different levels and the rest of the pixels have four or more levels. Naturally, the average number of levels per pixel increases with the time span.
section_name: Depth accuracy and precision

section_content: To quantify the precision further, the entropy of individual pixels over a period of 1000 frames are calculated. The entropy for each pixel is given by: H(x, y) =i p(x, y) i log 2 p(x, y) i where p(x, y) is the estimated probability mass function for pixel (x,y). Since
section_name: DEPTH ACCURACY AND PRECISION

section_name: DEPTH ACCURACY AND PRECISION

section_content: entropy can be used as a measure of unpredictability, values as low as possible are most desirable in this situation. 

Again, the setup for this experiment is stationary, where the sensor is pointed perpendicular towards a planar surface. This is done for three different distances; 1. 0 meter, 1. 5 meters and 2. 0 meters. The resulting images are shown in figure 4. 5 and figure 4. 6. 

The thing to notice is that there is structure in the images. All three images have vertical bands of different sizes, where the entropy is significantly higher than in the rest of the image. That is, the number of levels for each pixel is larger than in the rest of the image. Our experience suggests that this may be caused by some adaptive mechanism in the way the sensor operates (see section 4. 8). 

Furthermore, the largest values of entropy for the three images are respectively 2. 1, 2. 6 and 2. 6. Therefore, the distance does not affect the magnitude of the entropy much, but rather the structure. 

Entropy is based on probabilities; consequently it does not provide any information regarding the magnitudes of the fluctuations. Thus, the variance of each individual pixel is calculated as well. Figure 4. 7 shows the variance of each individual pixel, when the sensor is pointed perpendicular towards a planar surface at a distance of 1. 0m. The sample size consists of 1000 frames. In this situation, the variances of all the pixels are similar in magnitude. This is not the case when one or more objects are present in front of the sensor. Figure 4. 8a shows the depth image of a slightly more complex scene and figure 4. 8b displays the corresponding image of variance. The variances of the pixels along the contours of each physical object, e. g. at the top of the chair, are significantly higher than in the rest of the image. It has been observed that the depth values of these pixels are changing between the distance to the object and the distance to the background. Figure 4. 9 shows a plot of the value of a contour pixel as a function of time. As seen in the figure the value of the pixel fluctuates between approximately 1300mm and 2600mm, which correspond to the distance to object and to the background respectively. Notice also that the variances of the pixels near the "shadows" are very large.
section_name: DEPTH ACCURACY AND PRECISION

section_name: Spatial precision

section_content: The depth estimates in the depth image describe the distance from the point to the sensor plane rather than the actual distance from the point to the sensor. This should result in the same depth estimate over the entire image if the sensor is pointed directly at a planar surface. There are, however, small variations over the image. Figure 4. 12 shows an average over 1000 frames from a recording where the sensor was pointed directly at a planar surface. The sensor was located 1. 0m from the surface, but as the figure shows the depth estimate varies by roughly 40 mm. This variation is not entirely random and appears like a circular ripple in the depth image.
section_name: Structural noise

section_content: The same circular ripple continues to appear in the image as the distance from the sensor to the surface is increased. The variation in the depth estimate also increases. This can be seen in figure 4. 13. At 1. 5m the depth varies roughly 80 mm and at 2. 0m this is further increased to roughly 130mm. The increase in variation corresponds to the depth resolution getting coarser at longer distances (see figure 4. 2). That is, the variation in bit levels is roughly the same at all three observed distances. The variation might come from the transformation the sensor performs on the depth data so it represents the distance from point to sensor plane. Another possibility is that the sensors spatial resolution is not sufficient to record the IR pattern correctly or that the lens warps the IR image slightly. These are merely speculations though.
section_name: STRUCTURAL NOISE

section_content: It is possible to cover larger areas with multiple sensors. As the depth image is based on an infrared image of a pattern projected onto the scene by a laser, multiple sensors may interfere with each other. If the pattern projected by one IR laser overlaps with another, the sensor may not be able to make an estimate of the depth. This shows itself in the depth image as blind spots in the overlapping area where the depth estimates are set to 0. Contrary to what would be expected, the sensor is able to make a correct depth estimate in almost all of the overlapping area. Figure 4. 14 shows the percent of the overlapping area which are blind spots as a function of the overlap. 

Figure 4. 14 was created from an experiment with two depth sensor that were pointed directly at a planar surface. One of the sensors was then moved sideways to increase the overlapping area. A sketch of the setup can be seen in figure 4. 15. 

The position and size of the blind spots depend on how the two IR patterns of the sensors overlap. An example of a depth image with blind spots can be seen in figure 4. 16. Except for the blind spots, the depth estimates in the image are not affected by the overlap. That is, the depth values of the non-blind pixels in the overlapping area appear to be completely unaffected by the interference from the other sensor. 

Another problem arises in a situation, where the purpose is to record an object from more than one viewpoint. Figure 4. 17 shows a sketch of such a situation, where two Kinect devices are facing each other. The same problems as previously mentioned occur in areas where the laser patterns of the sensors overlap, but in addition to this there is the possibility that the laser pattern from one sensor may "blind" the camera of the other sensor. With a wellplaced setup of the sensors, the overlapping areas can be minimized and the The percentage is roughly constant as the overlap increases. blinding avoided. As an example, two sensors have been placed such that they face each other to record both sides of a person in a room (see figure 4. 17). Figure 4. 18 shows the RGB and depth images of the two Kinects.
section_name: Multi-camera setup

section_content: Due to the distance between the illuminator and the IR camera in the Kinect device, illuminated objects may cause shadows in the depth image. The shadow in the pattern clearly makes it impossible for the sensor to estimate the depth and therefore the pixels in the shadowed area are set to zero depth. Figure 4. 22 explains why shadowing occurs. 

Figure 4. 23 shows the depth-and RGB-images of a chair. The shadows in the depth image are clearly seen on the leftmost edge of the chair. Shadows will always appear on this side of an object due to the physical positions of the IR camera and the IR laser.
section_name: Shadows in the depth image

section_name: GENERAL CONSIDERATIONS

section_content: During experiments we observed that from the time we power on the device, it has a settling time before the pixel values reach their final state. Figure 4. 24 shows the pixel value for the first 90 seconds after powering on the device. It is clear from the figure, that the device needs approximately 30 seconds to reach its final value. Equivalently, if the sensor is rotated and quickly pointed towards a different scene, a similar adapting sequence can be observed. This might be important in applications, where the sensor itself is moving e. g. for robot vision applications. Chapter 5
section_name: Transients

section_content: In this report, we have investigated many of the topics that are important when the Kinect depth sensor (or a similar device with the same reference design) should be used in practical, real-life Computer Vision systems. We have examined software frameworks, depth and spatial resolution, structural noise and many other effects that occur in this complex device. 

There are also many other parameters that will influence the sensor in a given situation such as the characteristics of the viewed objects. For instance, there are obvious problems with reflective materials which the sensor is simply not able to determine a distance to. This is clearly understood since the sensor emits IR light which will not be reflected back to the camera. Similarly, diffusely reflective material with very low reflectivity will not reflect enough light for the camera to be able to determine distance to the object. This is obvious when the technology behind the Kinect is considered, however, it still limits the applicability of the sensor in different setups. Most alternative distance sensors will suffer from the same defects. 

Another complication is interfering IR light from e. g. the Sun. This clearly limits the usefulness of the sensor in an outdoor setting. From a practical viewpoint, even a strong ray of sunlight may blind the sensor indoor and other sources of household lighting may influence the sensor, although we did not experience many problems in our projects. One solution to the IRinterference problem might be to employ the RGB-images when the depthsensor fails and vice versa. 

Hopefully, this report will give the reader a background to decide whether or not the Kinect depth sensor will be useful for his/her application. We have surely found it useful in many of our projects.
section_name: Conclusion