Telecommunications
Digital archive, Virtual reality, Metaverse

Deep photometric stereo for high-fidelity 3D reconstruction
Multimedia Engineering, Graduate School of Information Science and Technology
Assistant Professor Hiroaki Santohttps://researchmap.jp/h_santoProfessor Yasuyuki Matsushita

https://researchmap.jp/yasumat

Telecommunications

Abstract
3D reconstruction from images is one of the fundamental problems in computer vision. In particular, recovering a high-fidelity 3D
shape is desired in various applications, such as a digital archive,
virtual reality (VR), and robotics. We develop a high-fidelity 3Dreconstruction method based on photometric stereo. Photometricstereo estimates a scene shape from multiple images captured bya fixed camera with varying lighting conditions. Since photometric stereo observes shading changes in a per-pixel manner, it issuitable for recovering per-pixel shape, resulting in high-resolution
shape recovery.
Conventional photometric stereo requires knowledge about thematerial of a scene surface in advance to model reflectances on
the surface. Since the reflectance model depends on the surface
material, there has not been a unified method that can be applied
to diverse materials in the real world. To deal with this problem,
we introduce a deep learning-based approach, which learns thereflectance model from training data. As a result, it can handle
diverse materials in the real world, which allows using photometricstereo in a wide range of practical applications.

result, it can handle diverse materials in the real world. Our proposed data-driven approach achieved state-of-the-art accuracy ina benchmark comparison and has become mainstream in recent
photometric stereo studies.

▍ Significance of the research and Future perspective
We have proposed a novel data-driven approach for photometricstereo, which is the first work in the world to introduce the use ofa deep neural network in photometric stereo and achieve highly
accurate estimation for scenes with diverse materials in the realworld.
We are now interested in developing a more simplified capturing
setup and more accurate estimation algorithm so that anyone canquickly obtain 3D data of an object in the real world.

▍ Background & Results
Computer vision is a research field that aims to realize functions
like human vision by computers. In particular, understanding 3D
shape information from 2D images is one of the fundamental problems.
This work focuses on estimating a high-fidelity 3D shape by thephotometric stereo. Photometric stereo estimates a scene shapefrom multiple images captured by a fixed camera with varying lighting conditions. Since the shading information can be obtained in a
per-pixel manner, photometric stereo can recover high-resolution
shape recovery.
Recovering a high-fidelity 3D shape is desired in various applications. For example, a digital archive is a project to digitally preserve the 3D shape and appearance of valuable assets, such as
cultural heritage and arts.
Even though photometric stereo has the advantage of high-fidelity 3D shape reconstruction, it has not been widely used in practice.
The primary reason is that conventional photometric stereo requires
knowledge about the material of a scene surface in advance tomodel reflectances on the surface. Since the reflectance model depends on the surface material, there has not been a unified methodthat can be applied to diverse materials.
To deal with this problem, we introduce a deep learning-based
approach, which we call a deep photometric stereo network. Instead of assuming a specific parametric model, our method learns
the reflectances from data by using a deep neural network. As a

Fig1: Overview of photometric stereo

Fig2: Overview of the proposed deep photometric stereo

P a t e n t
Santo, Hiroaki; Samejima, Masaki; Sugano, Yusuke et al. Deep Photometric Stereo Network, IEEE International Conference on Computer Vision Workshop,
2017, doi: 10.1109/ICCVW.2017.66
Santo, Hiroaki; Samejima, Masaki; Sugano, Yusuke et al. Deep Photometric Stereo Networks for Determining Surface Normal and Reflectances, IEEETransactions on Pattern Analysis and Machine Intelligence (TPAMI), 2020, Vol. 44, No. 1, pp. 114-128, doi: 10.1109/TPAMI.2020.3005219
Tr e a t i s e
Santo, Hiroaki; Michael, Waechter; Matsushita, Yasuyuki. Deep near-light photometric stereo for spatially varying reflectances, European Conference onComputer Vision (ECCV), 2020, doi: 10.1007/978-3-030-58598-3_9
Minami, Kazuma; Santo, Hiroaki; Okura, Fumio et al. Symmetric-light Photometric Stereo, IEEE/CVF Winter Conference on Applications of Computer Vision(WACV), 2022, doi: 10.1109/WACV51458.2022.00039
http://cvl.ist.osaka-u.ac.jp/en/
UR
L https://github.com/hiroaki-santo/deep-photometric-stereo-networkhttps://github.com/hiroaki-santo/deep-near-light-photometric-stereo
K e y w o r d computer vision, physics-based vision, computational photography, 3D reconstruction, photometric stereo