Telecommunications
3D measurement, E-Heritage, Visual inspection

High-fidelity 3D reconstruction via photometric stereo
Department of Multimedia Engineering, Graduate School of Information Scienceand Technology
Professor Yasuyuki Matsushita
https://researchmap.jp/yasumat?lang=en

Telecommunications

Abstract
In this research, we are working on establishing a techniquefor restoring three-dimensional shapes and textures of real-world
objects using the photometric stereo method for the purpose ofhigh-fidelity three-dimensional digitization. In this study, we aredeveloping a data-driven approach to photometric stereo and aone-shot photometric stereo method using a multi-wavelength lightsource and a multi-wavelength sensor. These methods can be applied to the quantification of real-world 3D objects and to the digital
reproduction and preservation of shape and texture.

▍ Background & Results

When we “see” an object in the real world, what our eyes actually perceive is the light that is projected toward our eyes from theobject. The light is observed through interactions with the object,
being reflected or transmitted on the object surface. Therefore, bycorrectly digitizing the three-dimensional shape, surface reflectance,and color of a real-world object, a digital 3D replica can be createdthat has the same appearance as the actual object under any lightsource. The digital replicas can be used for digital preservation ofcultural assets and as contents in virtual spaces. The digital replication technology developed in this research corresponds to thequantification of real-world objects and is expected to be applied toindustrial applications such as product appearance inspections.

Figure 1: Reflectance dataset captured from real-world objects▍ Significance of the research and Future perspective

There is a technique called “photometric stereo” which is a method of 3D shape estimation using shading patterns in images. Photometric stereo can estimate the surface orientation at each pointfrom brightness variations observed in multiple images taken under
different illumination conditions. The surface orientation indicatesthe direction to which the surface on the object is looking at and isexpressed as a surface normal vector that represents the directionin three-dimensional space. Since this can be computed for eachpixel in the image, it is collectively called a surface normal map.
Integrating the normal map yields a depth map, and conversely, thegradient of the depth map corresponds to the normal map. Thus, thenormal map contains a lot of information about the three-dimensional shape. Moreover, because photometric stereo can estimate theshape of each pixel, it can provide high-fidelity 3D shape estimation.
While conventional photometric stereo methods have been applied to diffuse surfaces that are not shiny or glossy, objects in thereal world have a variety of textures represented by different reflectance distributions (Figure 1) that were challenging for the previous
methods. This research has revealed that a data-driven approachthat utilizes prior knowledge of reflectances can be used to accurately estimate 3D shape. As shown in Figure 2, it is now possible
to recover high-fidelity 3D shapes and textures from input image
sequences taken under different light sources. We are also working
on a photometric stereo method that can recover 3D shape in asingle shot by multiplexing measurements at different wavelengths
using a multi-wavelength light source and a multi-wavelength sensor, and are developing a 3D shape recovery method for dynamicobjects (Figure 3).

Figure 2: 3D digitization from 2D photographs

Figure 3: 3D reconstruction of a dynamic scene

P a t e n t
Enomoto, Kenji; Waechter, Michael; Okura, Fumio et al. Discrete search photometric stereo for fast and accurate shape estimation. IEEE Transactions on
Pattern Analysis and Machine Intelligence. 2023, 45, 4355-4367. doi: 10.1109/TPAMI.2022.3198729
Guo, Heng; Okura, Fumio; Shi, Boxin et al. Multispectral photometric stereo for spatially-varying spectral reflectances. International Journal of Computer
Vision. 2022, 130, 2166-2183. doi: 10.1007/s11263-022-01634-4
UR
L http://cvl.ist.osaka-u.ac.jp/en/
K e y w o r d computer vision, 3D reconstruction, photometric stereo
Tr e a t i s e