QUALITY OF THE 3D POINT CLOUD OF A TIME-OF-FLIGHTCAMERA UNDER LUNAR SURFACE ILLUMINATION CONDITIONS:IMPACT AND IMPROVEMENT TECHNIQUES
Kentaro Uno1 , Louis-Jerome Burtz2 , Masafumi Endo1 , Kenji Nagaoka1 ,Kazuya Yoshida1
1
Tohoku University, Aoba 6-6-01, Aramaki, Aoba-ku, Sendai, Japan,
E-mail: {unoken, nagaoka, yoshida}@astro.mech.tohoku.ac.jp, masafumi.endo.s2@dc.tohoku.ac.jp
2
ispace, inc., Azabudai 3-1-6, Iikura katamachi, Minato-ku, Tokyo, Japan, E-mail: l-burtz@ispace-inc.com

ABSTRACTTime-of-Flight (ToF) cameras can deliver dense 3D point
clouds by precisely calculating the round-trip time of infrared (IR) light reflecting oﬀ objects. A Japanese Lunar
exploration rover employs a commercial oﬀ-the-shelf ToF
camera as its main hazard-detection sensor. The lighting
conditions on the Moon reduce the quality of the threedimensional (3D) point cloud. In this paper, we explore
the eﬀects of the Lunar surface lighting conditions on therange, validity, and accuracy of the data delivered by the
ToF camera. To achieve repeatable results, an indoor Lunar lighting testbed was developed. Based on these experiments, we propose a filtering method to fuse the 3D point
cloud frames obtained at diﬀerent integration times.

1 INTRODUCTION
In recent years, as a popular alternative to stereoscopic
cameras, time-of-flight (ToF) cameras have attracted attention for applications in space robot awareness systems.
A recent example is the Astrobee project [2], a robot
scheduled to be launched this year to work alongside theastronauts in the International Space Station (ISS). It monitors its environment and avoid hazards assisted by a ToF
camera. ToF cameras can deliver dense three-dimensional(3D) point clouds by accurately calculating the round-trip
time of the infrared (IR) light reflected oﬀ objects. Considering their particular application on the surface of theMoon, ToF cameras have certain advantages:• They are eﬀective in sensing unstructured, featureless surfaces such as the Lunar surface.
• As they rely on active illumination, they can obtain a
3D point cloud regardless of shadows.
• Compared to rotating LiDARs, they have no moving
parts which allows them to resist the rocket launch
environment.

• Their mass, power consumption, volume, and computing complexity are compatible with the miniaturized architecture of a microrover.

SORATO, the Japanese Lunar exploration rover managed
by ispace, inc. shown in Figure 1, employs a Sentis3D–
M100 commercial oﬀ-the-shelf ToF camera, manufactured by the BECOM Bluetechnix Inc. as its main hazarddetection sensor. We previously experimentally demonstrated that the selected ToF camera hardware is suitable
for space missions to the Lunar surface, through comprehensive qualification tests [3].
This paper is based on the prior work performed tocharacterize noise and errors of the 3D point cloud of the
ToF camera. In particular, they showed the dependencyof the optical characteristics (such as reflectivity) on thesurface of the target object [5]. To correct for the variancein the data, a number of eﬀective filtering methods werereported in [4] to acquire smoothed point clouds. However, most of the related studies are performed in indoor
environments. This paper focuses on the characteristics ofthe extreme lighting conditions of the Lunar surface.
The paper begins with a presentation of the characteristics of the Lunar surface lighting conditions and the
hardware improvements to the ToF camera to achieve amaximum resilience to extreme sunlight conditions. In
Chapter 3, we present an optical testbed, developed toperform repeatable experiments with the camera hardware
under conditions matching those of the Lunar surface. We
demonstrate the eﬀect on the quality of the 3D point cloud
for the respective integration times at distances of 1 m and2 m. Based on the results and analysis, in Chapter 4 we
propose a software filtering method consisting in the fusion of several 3D point clouds obtained at key integration
times. We discuss the impact on the three quality metrics:
achievable range, validity, and accuracy. In the last partof the paper we apply these techniques to an experimentconducted in a Lunar-analog facility in JAXA.

Figure 1 : Flight model of Japanese Lunar exploration mic HAKUTO/ispace. A visible light camera is fitcrorover ⃝ted on the front (top round lens) and it uses a ToF camera
as the hazard detection sensor (middle round lens for thereceiver and golden IR window for the laser emitter).

Figure 3 : Comparison of distances between the cloudy(left) and strong sunlight (right) conditions as well as between the ambient (top) and strong spot light (bottom)
conditions.

2.2

Figure 2 : Intense sunlight on the surface of the Moon generates critical negative eﬀects on both standard camerasc NASA) and ToF cameras.
(Apollo image above ⃝2 IMPACT OF SUNLIGHT ON THE TOFCAMERA SENSOR
2.1 Optical Conditions on the Lunar Surface
On the surface of the Moon, the absence of atmosphere allows the higher intensity sunlight to reach the surface. The
solar spectral irradiance data measured by orbital spacecrafts are freely available, while the “Standard Solar Constant and Zero Air Mass Solar Spectral Irradiance Tables”,
defined by the ASTM International Standard E490-00a
list the spectral irradiance values in the range of 120–
1000 nm [6]. In the range of 840–860 nm, the irradianceis 2.0 mW/cm2 . The reason for the integration range of840–860 nm is the eﬀective sensitivity of the ToF camera
sensor, discussed in the next subsection.

Eﬀect on the ToF Camera Sensor

We observed two main types of negative eﬀects in the 3D
point clouds obtained under strong sunlight illumination
conditions. The images in Figure 3 show the raw data acquired by the ToF camera in two configurations (the distance value is color-coded according to the scale and theunit is mm). On the left pane, the optical environment ismoderate (outdoor but cloudy) while on the right pane, theoptical environment is challenging.
The first type of noise is general ambient noise(shown on the top of Figure 3). We believe that this phenomenon occurs when the external IR flux saturates thesensitive pixels, even while using the patented suppression of background illumination method[1]. The pixel circuit cannot perform correct phase calculations and outputs
unrealistic distance values. We observed that the drop in
signal-to-noise ratio (SNR) aﬀects the entire frame.
The second type of noise is shown on the bottom of
Figure 3. In this case, a well defined circular pattern canbe observed on the upper part of the image. However, nosuch object was physically present in the scene. We interpret this signal as a strong collimated light reflected in
the lens assembly, which selectively saturated the pixels inthe shown circular region. This type of noise is known asghost pixels. We previously presented additional discussion and more details on these specific mechanisms [3].
The aim of this paper is to explore the strategies tosuppress these two types of noises, first through hardware
improvements and then through software methods.

2.3

Hardware Improvements against theSunlight

From a hardware point of view, the spectral sensitivity ofthe sensor of our ToF camera (PMD PhotonICs 19k-S3)

is in the range of 650–1200 nm. Its spectral sensitivity
is particularly high in the range of 800–900 nm. However, the useful signal from the ToF camera emitter is only
∼850 nm. The noise from the external sunlight is strong
all over the visible and IR bands.
To maximize the SNR, a 20 nm band-pass filter was
implemented together with the manufacturer. This filter
is custom made and it has the smallest possible bandwidth, considering the central wavelength accuracy of thevertical-cavity surface-emitting laser (VCSEL) emitters,
the temperature variations between the emitters and thefilter, and the field of view of the sensor (chief ray angle).

Figure 4 : Overview of the developed photodiode measurement system.

3 LUNAR SUNLIGHT
DEMONSTRATION EXPERIMENTThis section presents the 3D point cloud capturing experiment under lighting conditions representative of the Lunar
surface. First, we describe the capabilities of the equipment we developed. Then, we explain the experimental
setup and its components. Third, we present and discuss
the experimental results.

Figure 5 : Calibration curve obtained to determine the irradiance from the voltage of the photodiode circuit.

3.1 Irradiance Measurement Instrumentation
We developed a measurement device that has the same
sensitivity as the ToF camera by using the same bandpass
filter (840–860 nm). An overview of the device is shownin Figure 4. The sensing part consists of the OP950 photodiode, produced by the TT Electronics/Optek Technology,
Inc. The connected CPU board can measure the voltage ofthe photodiode circuit.
In order to determine the irradiance from the measured voltage, we performed a calibration test. For thiswork, we used a Glacier X TE Cooled CCD Spectrometer spectral analyzer, manufactured by B&W Tek, Inc.
For the reference light source, we used the SOLAX series
XC–500EF artificial sunlight simulator, manufactured by
SERIC, Inc. In this test, the voltage of the photodiode andthe spectral irradiance were measured together. Throughout this test, we obtained the curve of the calibrated irradiance as a function of voltage, as shown in Figure 5. In
the experiments of the following sections the irradiancemonitoring was strictly performed with this calibrated irradiance sensor.

3.2 IR Emitter
To recreate the IR illumination conditions on the surfaceof the Moon, we developed an IR illumination simulator,
shown in Figure 6. The system has seven powerful infrared emitter modules. A 850-nm dual junction infrared
LED emitter (LZ4-00R608 manufactured by LED EnginInc.) was chosen. The illumination was concentrated bya narrow field of view (half-angle at 50% from the maximum intensity of 7.4◦ ) lens (LLC69N, manufactured by

GAGGIONE Inc). A microcontroller controls their emission rate and monitors their behavior from a personal computer. The power sources are supplied directly from a
100 V household outlet.
The main features of the developed illumination simulator are:
• The sunlight simulator accurately represents the IR
irradiance of the Sun for the wavelengths around
850 nm.
• It provides repeatable experiments opportunities(which is not possible with natural sunlight in outdoor environments).
• The modular concept of the IR emitter and lens allows for rapid prototyping of various illumination
conditions.
The limitations of the setup as it was used in this studyare:• The illuminated area is small (however, suitable forthe typical size of the hazards we are interested in
imaging with the ToF camera).
• The irradiance received by the illuminated area isinversely proportional to the square of the distanceto the emitters, which makes it highly variable andneeds to be monitored with the developed irradiancelogger.

Figure 6 : IR illumination simulator consisting of 7 IR
emitting modules.

Figure 8 : Experimental setup. The ToF camera takes a
3D point cloud of the checkerboard illuminated by the IR
emitters. The illumination condition is measured by theirradiance detector developed.

Figure 7 : Simulation of the 840–860 nm irradiance output for the 7-emitter configuration based on measurements of a single IR emitter module.
• To ensure reliability and repeatability, the IR emitterscan be turned on for 1 s long flashes only, and then
10 s are needed to cool down. This requires synchronization with the ToF camera for data acquisition.

3.3 Experimental Conditions
Figure 8 shows an overview of the experimental environment. In the experiments, a checkerboard made ofwhite squares (surface of normal printing paper) and blacksquares (printed by black ink), which have diﬀerent IR reflectivities, were set on the wall as target objects. The
checkerboard is fully illuminated by the developed IR
emitters. The illumination condition on the checkerboard
was measured (Figure 9). To stay above the 2.0 mW/cm2
threshold, we defined the region-of-interest (ROI) as thecentral four squares in the checkerboard. The ToF camera
was set in such a way to enable the acquisition of the center position of the target and the real world distance data
was recorded. Moreover, using the irradiance measurement instrumentation, we verified that the ambient room
illumination had no measurable IR component and therefore no eﬀect on the ToF camera.
In this experiment, a 3D point cloud together with thechange of integration time in the range of 100–2500 µs
at every 100 µs were produced by the ToF camera. And

Figure 9 : Verification of the Illumination condition on the
checkerboard. The yellow–red color shows the change in
irradiance. The region-of-interest (ROI) has higher than
Lunar surface irradiance.

this test was performed for the cases where the real-world
distances were 1 m and 2 m. For each pixel of each point
cloud, we set two amplitude-based thresholds to define theunder/over-exposure of the pixels. The lower threshold is
300 and the higher one is 15000 (amplitude values are dimensionless, 2 byte integers). When the amplitude value
is under or over the thresholds, the distance value of the
pixel becomes −1 and 0 respectively.

3.4

Experimental Result and Discussion

Using the controlled environment presented earlier, wequantify the impact of the Lunar lighting conditions on thequality of the 3D point cloud with three metrics: range,
validity and accuracy. At 1m, the ROI corresponds to 289
pixels on the ToF camera sensor. At 2m, the ROI corresponds to 64 pixels.

Figure 10 : Colorized distance maps of the checker board target (with ROI in the center, shown as a red square). Integration time increases from left to right. The top set is 1 m real-world distance, the bottom set is 2 m. In each set, the top
maps are with no external IR illumination, the bottom maps show the eﬀect of external IR illumination. In each distancemap, over exposure pixels are marked white and under exposure pixels are marked dark blue.

3.4.1 Range
Figure 10 arranges the experimental results according to integration time (increasing from left to right), realworld distance (top set is 1 m, bottom set is 2 m) and theimpact of Lunar surface illumination conditions (no external IR for the first line of each set, Lunar surface equivalent external IR for the second line of each set). Figure 10 shows that the range metric is greatly impacted by
the integration time, the external IR illumination and thereflectivity of the target. The range versus integration timetrade-oﬀ can be summarized as follows: high integration
times (> 800 µs) allow the ToF camera to detect low IR reflectivity targets, even at further distances. However, once
Lunar surface illumination conditions are applied, higher
integration times also increase the amount of noise entering the sensor and therefore the amount of noisy pixels.
Those result show that the ToF camera’s ability to obtain data from farther away (which is a key metric for Hazard Detection and Avoidance) is limited if we set only one
integration time. The optimal integration time for low reflectivity (white checkerboard squares) and high reflectivity (black checkerboard squares) targets is diﬀerent. Fur-thermore, the optimal integration time for 1 m and 2 m
distances is also diﬀerent.

3.4.2

Validity

As presented in section 3.3 the validity is defined
based on the amplitude value of each pixel. The top linesof each set of maps of Figure 10 show that even without external IR illumination, some pixels are still valid.
Underexposure pixels (dark blue) are present at low integration times. Overexposure pixels (white) are present
at high integration times. To isolate the negative eﬀect
due only to the external IR illumination, we checked 10
frames obtained without IR illumination. If the pixel wasinvalid more than 7 times, we regarded it as invalid regardless of the eﬀect of Lunar surface lighting conditions.
Figure 11 shows the change of the number of under- andover-exposure pixels caused by the external IR illumination in the ROI according to the increase in the integration
time. The top graph shows the eﬀects that are not due toexternal IR illumination. We define three zones of interest.
In zone (1), there is a significant and decreasing amountof under exposure. We interpret that it is purely becauseof the lack of sensitivity to detect the necessary modulatedIR signal of the ToF camera lasers. In zone (2), there isno under or overexposure. In zone (3), at 1 m distance,
the number of over exposure pixels increases and reaches
a constant value. This is due to all the high reflectivity
pixels saturating due to the laser light of the ToF camera.
Next we consider the specific eﬀects of Lunar surface
lighting conditions. We identify the invalid pixels and subtract those that were already invalid even without the external IR illumination. This process creates the bottom
plot of Figure 11. In zone (1), there are no new invalid
pixels. This shows that the external IR illumination does
not aﬀect short integration times. In zone (2), the percentage of invalid pixels increases linearly with the integration
time. In zone (3) the percentage of invalid pixels continues to increase linearly, except for the case of 1 m distance
where the pixels where already saturated by the laser of
the ToF camera itself (zone (3) in top graph of Figure 11).
We interpret the observation of both under- and overexposure pixels as due to the saturation of the sensitive
pixels. When the imager combines the modulated lightand the external light, it can lead to the saturation of thesensor: overexposure pixels are reported. When the imager is saturated by the external IR flux and cannot detect
the modulated IR signal of the ToF camera itself: under
exposure pixels are reported. In conclusion, the external
IR illumination causes the validity metric of the quality of
the 3D point cloud to deteriorate linearly once the integration time increases above the 500 us threshold.

3.4.3 Accuracy
We define the accuracy of the valid pixels dataset as
the measure of their standard deviation (STD) and median oﬀset. The oﬀset is calculated as the diﬀerence between the real-world distance and the median value fromthe valid pixels. Figure 12 and Figure 13 show the changeof the STD and oﬀset of the distance value in ROI according to the change of the integration time. We can see thatexternal IR made both standard deviation and the maximum oﬀset worse. Still, both are still less than 3% of
the real-world distance. This shows that the thresholding
performed for determining the validity of the pixels in theprevious subsection, is robust to the Lunar surface illumination conditions.

4 TECHNIQUES FOR IMPROVEMENT
OF THE 3D POINT CLOUD
Based on these insights, we propose a filtering method to
optimize the available range, validity and accuracy by fusing several 3D point cloud frames obtained by the variation of the integration time of the ToF camera. In related
works, Oggier et al. developed an algorithm to automatically select the best integration time [7], and Kazmi etal. proposed a method to combine two frames captured

Figure 11 : The change of the percentage of under- andover- exposure pixels in ROI for each distance (1 m and2 m). Top graph shows the pixels that are invalid regardless of the IR flux illumination. Bottom graph shows the
pixels that are invalid only due to the eﬀect external IR
illumination.
at low and high integration times to increase the accuracy
for short range objects [8]. For our application, 2 m range
is required for the hazard detection algorithm of the rover.
As discussed earlier, there is a trade-oﬀ between the longintegration time resulting in an increased range, but also ahigher sensitivity to sunlight noise.

4.1

Techniques and Performance

The following steps show the skeleton of the fusion algorithm:
1. Set the number and values of the integration time tobe fused and capture the raw 3D point clouds.
2. Set the amplitude thresholds to define validity(under- and over- exposure). In each frame, identify the invalid pixels and remove them from further
computation.
3. Compute the fused 3D point cloud: for each pixel,
calculate the median of the amplitudes and distances
based on the values from the frames captured at thediﬀerent integration times. For each pixel, if there isno valid data from the raw point clouds at the various

Figure 12 : Variation of the STD as a function of integration time for a target in the distance of 1 m away (in blue)and 2 m away (in orange).

Figure 13 : Variation of the median oﬀset from the groundtruth (target 1 m away in blue and 2 m away in orange) as
a function of integration time.
integration times selected, mark the pixel as invalid
in the fused 3D point cloud also.
First, we applied this technique to the 3D point clouddataset obtained in the experiment conducted in our laboratory (explained in Chapter 3). To keep the representativity with the Lunar rover implementation that has real-time
computation and memory constraints, we fused only four
frames. The integration times chosen are: 200, 600, 1000,
and 1500 µs. Table 1 compares the quality metrics: validity (overexposure and underexposure), accuracy (standard
deviation and oﬀset) and range (minimum and maximum)
for each frame and for the resulting fused frame. The bold
font is attributed to the best result for each metric.

4.2 Implementation in a Lunar Analog
Environment
Next, we applied our technique in an experiment performed in the advanced facility for space explorationof the Japan Aerospace Exploration Agency/Institute ofSpace and Astronautical Science (JAXA/ISAS) Sagamihara campus. In this test, the ToF camera located on thefront side of the rover testbed detected three types of haz-Figure 14 : Overview of the hazard slope test at the
Sagamihara campus of JAXA.

Figure 15 : 3D point cloud visualization of the raw frames
at the four source integration times (left) and the output
fused frame (right). View from the above (top) and view
from the right hand side (bottom)
ards: boulder, slope and ditches as defined in [3]. Figure 14 shows an overview of the slope hazard test. Artificial sunlight illuminated the entire test environment. The
hazards themselves were additionally illuminated with thedeveloped IR emitters (creating a 15cm diameter disk at
the intensity of the Lunar surface lighting conditions). Using the same four integration times and the same techniqueas presented above, we obtained the 3D point clouds ofFigure 15.
When the integration time is low (200 µs), the available range is narrow and short. Meanwhile, when the integration time is high (1000 µs and 1500 µs), the range
increases and the slope is visible, but a significant areaof invalid (overexposure) pixels can be observed at shortrange. However, regarding the fused frame (right two images in Figure 15), the quality is preserved at both shortand long range. Through the result in Table 1 and Figure 15, we observe that the fusion method is able to combine almost all the best metrics of each individual frame.
For the standard deviation, the best one is the frame at
1500 µs. This shows the necessary trade-oﬀ between thenumber of valid pixels and the standard deviation metric.

Table 1 : Comparisons of quality metrics: number of under/over-exposure pixels, oﬀset, standard deviation, minimum
range and maximum range. Bold font is attributed to the best result for each metric.
integration
time [µs]
200
600
1000
1500
fusionvalid pixels
in ROI [%](1 m) (2 m)
70.9
53.1
96.9
96.9
81.3
95.3
62.6
76.6
100
100

underexposure
in ROI [%](1 m) (2 m)
29.1
46.9
0.7
3.1
2.8
3.1
15.9
12.5
0
0

over exposure
in ROI [%](1 m) (2 m)
0
0
2.4
0
15.9
1.6
21.5
10.9
0
0

5 CONCLUSION

standard
deviation [mm](1 m) (2 m)
15
52
14
30
10
26
9
19
14
27

medianoﬀset [mm](1 m) (2 m)
11
29
18
26
19
27
21
26
14
24

range
[mm]
(min) (max)
0.2
1.2
0.2
1.6
0.4
2.0
0.5
2.2
0.2
2.2

JAXA/ISAS for the opportunity at their Lunar analog facility.

This paper presents three main developments:• We developed a Lunar surface lighting condition system to conduct repeatable experiments and an irradiance measurement tool to detect the energy received
by the ToF camera sensor.
• We quantified the impact of extreme lighting conditions on the Lunar surface and their evolution withincreasing integration times• We implemented a software solution including thefusion of 3D point clouds taken at key integration
times
The study on the impacts showed that several of thequality trade-oﬀs are a function of the integration time.
A longer integration time results in a higher amount ofoverexposure at short range, but also a higher number ofvalid points at long range. A shorter integration time also
provides a higher SNR even under extreme sunlight illumination. The accuracy oﬀset and STD vary with the integration time and there is a trade-oﬀ between a better STD
value and a higher number of valid pixels.
In conclusion, we demonstrated that the ToF camera
is able to capture 3D point cloud data accurately in challenging lighting conditions by the proposed improvement
technique. The additional software processing, whichfuses frames captured at various integration times, increases the quality of the data under high sunlight conditions such as those on the Lunar surface. As a result,
this increases the suitability of the ToF camera to be a key
component of the environment awareness system of an exploration robot.

Acknowledgement
We express our special thanks to BECOM Bluetechnix
GmBH for their expertise and support, to KONICA MINOLTA for the irradiance measurement calibration and toReferences[1] T. Möller, H. Kraft, et al. (2005) “Robust 3D Measurement with PMD Sensors”, In Proceedings of the
1st Range Imaging Research Day at ETH, Zurich,
Switzerland.
[2] T. Smith, J. Bualat, et al. (2016), “ASTROBEE: A
New Platform for Free-Flying Robotics on the International Space Station”, NASA Technical Reports
Server, ARC-E-DAA-TN31584.
[3] K. Uno, L-J. Burtz, et al. (2017), “Qualification of aTime-of-Flight Camera as a Hazard Detection and
Avoidance Sensor for a Moon Exploration Microrover”, Transactions of The Japan Society for Aeronautical and Space Sciences, Aerospace Technology
Japan, in press.
[4] Y. Cui, S. Schuon, et al. (2010), “3D Shape Scanning with a Time-of-Flight Camera”, In Proceedingsof 2010 IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), San Francisco, USA,pp. 1173–1180.
[5] D. Falie, V. Buzuloiu. (2007), “Noise Characteristics of 3D Time-of-Flight Cameras”, In Proceedingsof the International Symposium on Signals, Circuits
and Systems (ISSCS), Iasi, Romania.
[6] ASTM International. (2014), Standard Solar Constant and Zero Air Mass Solar Spectral Irradiance Tables, Active Standard ASTM E490,
ASTM E490-00a, ASTM International, USA, pp. 2–
13.
[7] T. Oggier, B. Bernhard, et al. (2005), “Swissranger
sr3000 and first experiences based on miniaturized
3D-TOF cameras”, In Proceedings of the 1st Range
Imaging Research Day at ETH, Zurich, Switzerland.
[8] W. Kazmi, S. Foix, et al. (2012), “Plant Leaf Imaging using Time of Flight Camera under Sunlight,
Shadow and Room Conditions”, In Proceedings ofthe International Symposium on Robotic and Sensors Environments (ROSE). IEEE, Magdeburg, Germany pp. 192–197.