Merging SLAM and photometric stereo for 3D
reconstruction with a moving camera
ddRemy Maxence
Hyper Vision Research Laboratory
Keio UniversityYokohama, Japan

Saito Hideo
Hyper Vision Research Laboratory
Keio UniversityYokohama, JapanUchiyama Hideaki
Laboratory for Image and Media Understanding

Kyushu University
Fukuoka, Japan

Kawasaki Hiroshi

Nozick Vincent

Computer Vision & Graphics Laboratory

Laboratoire d’Informatique Gaspard Monge

Laboratory for Image and Media Understanding

Kyushu University
Fukuoka, Japan

Paris-Est Marne-la-Vallée University
Marne-la-Vallée, France

Kyushu University
Fukuoka, Japan

Abstract—By using photometric stereo it is possible tocompute the normal of the surfaces of a scene. This paper
applied photometric stereo using a moving camera. This
implementation is meant for dark environment enlightened by
the torchlight of the camera. Since the photometric stereo isapplied while the camera is moving, it requires a camera pose
estimation. This estimation is performed thanks to ORB
features used in a similar way as ORB SLAM algorithm. A
first estimation of the depth map is provided by stereo
matching.
Keywords—photometric stereo, stereo, SLAM, surface,matching, ORB features

I. INTRODUCTION
AR technologies such as ARCore usually focus on anestimation of the plane surfaces, avoiding to fullyreconstruct the scene they are applied in. In addition, they
perform poorly with close-up scenes and with poor light
conditions. Consequently, some AR concepts are still far
from being implementable because they require roundsurfaces or dark environment. For example, implementing
an AR guide to exit a building after a power outage is achallenging implementation with the current technologies.
Our system uses photometric stereo to generate an accurateestimation of the normal vectors and the albedo of thesurfaces. Usually, photometric stereo is fixed and the lightsource is moving. However, it is possible to apply
photometric stereo even if the camera is moving. A simple
way to do so is to move the light and the camera
simultaneously as in [1]. This method suits perfectly withThomas Diego

the usage of the camera and the torchlight of a smartphonewhich moves altogether. The challenge consists inestimating the camera pose (and consequently the torchlightpose) as the user is moving. This can be done thanks toSLAM algorithm. In this research, the camera pose isobtained thanks to a key points-oriented approach similar toORB SLAM [2]. By using stereo matching, it is possible toget a first estimation of the depth of the scene [3]. Finally,
with the camera/light poses and an initial depth estimation,
it is possible to apply photometric stereo to finally get an
accurate estimation of the normal and the albedo of thescene. The full pipeline is summed up in Fig. 1.
Our contribution consists in (1) the creation of analgorithm combining SLAM and photometric stereo, (2) theadaptation of photometric stereo for a moving user and (3) areconstruction technique that can be applied in dark
environment.
A. Previous works
SLAM technologies are usually used for odometry andcan be used with a monocular camera and provide reallyinteresting results as long as some constraint to get a rightacquisition are respected. With images simply taken by amoving camera, it is possible to get the camera pose in realtime. Various SLAM algorithms exist; some are dense(depth-map oriented) as in [4] and others are sparse (keypoint oriented) as in [2]. In our case, we use a sparse
approach inspired by ORB SLAM. The reason is to get afast process that is robust to various camera movement.

Fig. 1. Key steps of our pipeline.

Photometric stereo is an approved technology that uses a
pixel-wise approach to reconstruct the normal and thealbedo of the technology. Based on the variation of the lightintensity, it performs well when the good conditions arerespected (fixed camera, moving light, fixed small object,dark light conditions). Here, we would like to use the mostconstraint-less approach for photometric stereo. We want toapply photometric stereo with a moving camera. To do so,
we need a large variation of camera movement which
explains why we require the robustness of ORB features(see [5]).
Finally, since we require an initial depth estimation, we
use another often-used technology which is stereo matching[3]. By using two images of the same scene slightly
translated, it is possible to compute easily a depth map ofthe scene.
II. PROPOSED METHODA. Image acquisition
The idea is to acquire the data as if we are using asmartphone with the torchlight turned on. The camera ismoving around the objects with a large variation ofmovement to have a large panel of light orientation. We
need a large number of frames to get an accurate result withthe photometric stereo. In our experiments, we have thecontrol of the colorimetric and compressing settings. The
idea is to get images that can directly feed our ORB SLAM
based algorithm without requiring additional preprocessingof the images.
B. ORB SLAM based-algorithm
The algorithm extract ORB features in all images tofind a set of correspondences between all the images as
explain in [2]. Among all the images, a series of key frames
based on the number of correspondence is found. To
compute the camera pose, the algorithm uses thehomographies between the images and the fundamentalmatrixes. If the scene is rather planar, the homography isused. On the other hand, if the scene is non-planar and has alow-parallax, the fundamental matrix is used. The process isable to work in real-time. A set of Key Frames is obtained
and for each of them, we are able to get the camera pose
between world and camera coordinate system (,)
and vice-versa (,). We also have a sparsereconstruction of the scene in 3D and a mapping of thosepoints with the pixels of each Key Frame. Note that ourreconstruction is unscaled.
C. Stereo Matching
With this camera pose, we are able to get the rotationand the translation
between an image 1 and an
image 2 :(1)
(2)
This is then used to apply stereo matching. After
rectification of the images, we can get a disparity map andconsequently a depth map estimation. We decide to apply
stereo matching for each Key Frame as a left image and a
non-Key Frame as a right image. After rectification of theimages, we apply a semi-dense block matching technique toget a disparity map. From this disparity map, we are able toget a depth map and to project our image in 3D. The depthmap is used as a first estimation for the photometric stereo.
The 3D projection is used for correspondences. We get aseries of 3D point clouds for each Key Frames. By
projecting everything in the world coordinate system, we areable to get a dense point cloud with overlapping points. For
each point we store the following data: the image index
where it was extracted from and the intensity of the pixel
used for this point. By using overlapping points, we get aseries of pixel intensities for each Key Points. As we will
explain in II. D, if we get at least 4 overlapping, we canapply photometric stereo. Overlapping is estimated based on
the Euclidian norm. To refine the result, we also use the 2D
projections to check if the overlapping is preserved.
Ultimately, we only keep the key points that corresponds tofair correspondences. We use the epipolar constraint tocheck if the points are corresponding. The fundamental
Matrix is computed using the intrinsic and extrinsic
parameters. We ensured a rank two to this matrix by using aSVD decomposition to set the last Eigen Values to 0.
D. Photometric Stereo
To represent this 3D reconstruction, we create a list of
3D points such as each member can be represented as X, Y,Z, V where X, Y, Z are the coordinates of the 3D points andV a vector containing a series of tuples. Each tuple contains
the information of different images that will be used forphotometric stereo. The representation of a tuple is (i, p)
where i is the index of the image used to compute this 3D
point and p the corresponding pixel intensity. If the lengthof V is superior to 4, we can apply photometric stereo. The
classic equation used for photometric stereo with lambertiansurfaces is:(3)
where is the intensity of pixel i, the albedo of i, thedirection of the light vector of the torchlight toward i, thenormal vector in the world coordinate system of i and theambient light as received by i. Since the camera is moving,
we need to compute for each pixel and each camera positiona different light vector as followed:

(4)
where
is a first estimation of the normal of thecorresponding pixel i (world coordinate). x is projected fromthe world coordinate system to the camera coordinate
system where it was extracted. Since the camera is in theorigin of the camera coordinate system, it comes that is thevector that is directed from the pixel on the surface to thecamera. For near photometric stereo, the equation (1) needs
to be corrected into (5) by a quadratic or cubic inverse as
explained in [6] where f = 2 or 3 depending on the selectedmodel. If the model is not near photometric stereo, we set

ACAABFig. 2. Bottom: example of 3 input images (bottom) with a textured cube. Upleft: normal vectors for the face A with 1000 input images. Up right: normalvectors for the two faces A and B with 200 input images.

where we can control the light and the camera. We tried
several experiments using different conditions.

(5)

A. Influence of the number of frame

f = 0 which corresponds to the classic case of photometricstereo [7].
We consider the notation (6) where k means it is theindex of the image the value was extracted from. (6) is avector whose size is 3.
(6)With at least 4 different pixels corresponding to thesame point in space, we get our photometric system that wecan solve (7). The light vectors also need to be independent.
Note that if we consider that there is no ambient light, wecan solve the system with only 3 correspondences.

(7)

For better result, we solve this system using theleast-square method with as many equations as possible
instead of just 4. By solving this system, we get. The
norm of the results, provide
and the normalized result
provides. We finally get normal vectors. We do thiscomputation for all the pixels that have enoughcorrespondences. To remove the outliers, we use aRANSAC approach.
III. RESULTS
We applied photometric stereo for sparse points
which provides fair result. To get the full control of thecamera movement, we generated a 3D scene using BlenderFirst, we investigate the influence of the number offrames and correspondences for flat surfaces. The object weused was a cube and we made the camera turned around theobject. We chose a texture colorful but with light colors tofacilitate the key points tracking and the photometric stereo.
The light is virtually put far away of the scene but it still
follows the camera movement. In this case, f=0. We tried touse different number of frames to see how the normalvectors evolve. For each experiment, we were oscillatingaround the cube to be sure that we have a wide variation of
the light vectors for each axis. Without this large variation,
the normal cannot be computed properly since the rank of
the light matrix will not be 3. You can the result of one
experiment on Fig. 2.
As you can see on figure 3, to recover the correct
normal vectors, in practice, we need way more than 3correspondences to get result. The results start to be fairwhen you have at least 8 correspondences. This method
cannot work with just a few images.

Nb Key frame /nb frame

Nb of averagecorrespondences
after RANSAC

Nb good vectors(at most 1% shiftfrom the expecteddirection) / Nb

normal estimated
40/1000
11247 /300
30/4008100/202
24/200632/108
18/100
49/60
9/50
32/50
Fig. 3. Evolution of the percentage of good normal
depending on the number of frames used as input. The
number of correspondences corresponds to the size of all
the systems we solved. As a reminder, 3 is enough withoutambient light.

B. Comparison with stereo matching based normals,
For another experiment, we used a bunny as theobject whose texture is again light and colorful. The camera
trajectory is again moving around the object with a large
variation on each axis. The Fig.4 shows the camera
trajectory and some input images.
We noticed that the number of images has a huge
influence on thegrey part corresponds to the 3D reconstruction of the scene
with stereo matching based on only two frames.
IV. CONCLUSION AND NEXT STEPSSLAM allows an accurate tracking that gives more
freedom to apply photometric stereo by removing the classic
fixed camera constraint. Thanks to photometric stereo, it ispossible to refine the flatness that the stereo matchingusually estimates. This technique is more suitable when
there is no ambient light which makes it good in dark
environment.
As in classic photometric stereo problem, werequire a lot of camera movement and many images asinput. The minimum number of correspondences (3 without,4 with ambient light) is not sufficient this is one it isimportant to get a large number of correspondences.

Fig. 4. On the left: Trajectory of the camera (green lines),
Key frames positions (blue rectangles) and key points (reddots). On the right: examples of 3 images used as input outof 1000.
We compared the rendered normal vectors of ourtechnique with the rendered normal vectors based on thecomputation of the point cloud estimated by the stereo
matching (computed from a normal map). As you can see onFig. 5, our technique is better to reshape the original shape.
With stereo matching solely, the surfaces are estimated way
more flat than they are.

Next step for us is to densify our vectors. Becauseof the poor efficiency of the correspondences found bystereo matching, only a few correspondences can be used forphotometric stereo. Photometric stereo while the camera ismoving is really sensitive to the correspondences and small
shifts of few pixels can make the results wrong. We are now
investigating to find a way to get more reliablecorrespondences. It will in the same time increase thequality of our normal vectors and densify or results.
We also target a pipeline fully applicable with adirect smartphone. Some calculation optimization needs tobe addressed to make it possible.
REFERENCES
[1]

[2]

[3]

[4]
[5]

[6]

[7]

[8]

Fig. 5. Up: the normal estimated thanks to stereo matching.
Down: the normal vectors estimated with 1000 images. The

T. Higo, Y. Matsushita, N. Joshi and K. Ikeuchi, A hand-heldphotometric stereo camera for 3-D modeling, 2009 IEEE 12th
International Conference on Computer Vision, Kyoto, 2009, pp.
1234-1241.
Raul Mur-Artal, J. M. M. Montiel and Juan D. Tardos, ORB-SLAM:a Versatile and Accurate Monocular SLAM System, IEEETransactions on Robotics, 2015.
Geiger A., Roser M., Urtasun R. (2011) Efficient Large-Scale StereoMatching. Computer Vision – ACCV 2010, Berlin Lecture Notes in
Computer Science, 2010, vol 6492.
J. Engel, T. Schöps and D. Cremers, LSD-SLAM: Large-Scale Direct
Monocular SLAM, European Conference on Computer Vision, 2014.
J. Engel, V. Usenko and D. Cremers, A Photometrically Calibrated
Benchmark For Monocular Visual Odometry, CoRR abs Conference
on Computer Vision, 2016, 1607.02555..
Y. Quéau, T. Wu and D. Cremer. Semi-calibrated Near-Light
Photometric Stereo, Scale Space and Variational Methods in
Computer Vision: 6th International Conference, SSVM 2017, 2017.
pp. 656-668
K. Mu Lee,C.-C. Jay Kuo. Shape Reconstruction from
Photometric Stereo, Proceedings 1992 IEEE Computer Society
Conference on Computer Vision and Pattern Recognition., 1992.
H. Hayakawa. Photometric Stereo under a light source with arbitrary
motion, J. Opt. Soc. Am., 1994. A 11 3079-3089