Social innovation
E-heritage, Metaverse

Shape and albedo recovery by your phone using
stereoscopic flash and no-flash photography
Multimedia Engineering, Graduate School of Information Science and Technology
Specially Appointed Assistant Professor Xu Cao

https://researchmap.jp/caoxu

Professor Yasuyuki Matsushita

https://researchmap.jp/yasumat

Abstract
Recovering shape and albedo for the immense number of existing cultural heritage artifacts is challenging. Accurate 3D reconstruction systems are typically expensive and thus inaccessible
to many and cheaper off-the-shelf 3D sensors often generate
results of unsatisfactory quality. We present a high-fidelity shapeand albedo recovery method that only requires a stereo camera
and a flashlight, a typical camera setup equipped in many off-theshelf smartphones. The stereo camera allows us to infer roughshape from a pair of no-flash images, and a flash image is further
captured for shape refinement based on our flash/no-flash image
formation model. We verify the effective- ness of our method on
real-world artifacts in indoor and outdoor conditions using smartphones with different camera/flashlight configurations. Comparison
results demonstrate that our stereoscopic flash and no-flash photography benefits the high-fidelity shape and albedo recovery on asmartphone.

Goal of the proposal

Social innovation▍ Background & Results
Recording 3D shape and surface reflectance are both invaluable for digitally archiving and analyzing cultural heritage artifacts.
While the importance of digitally archiving artifacts is generally recognized, it is still not widely spread in many museums and libraries,
mostly due to the complexity of the digitization process that comeswith expensive specialized setups. To enable everybody to participate in digital archiving, a method that is simple to operate andonly requires a commodity device is very much wanted.
Our method only requires one stereo camera and a flashlight.
Many commodity smartphones today are equipped with this imaging setup, and we demonstrate that our method is naturally
applicable to such smartphones. With this setup, recording can be
conducted outside a darkroom (e.g., in an office room) and completed in a moment as it only takes two shots without any camera
movement. These properties make the digitization process easy.
Quantitative evaluation using synthetic images justifies our high-fidelity shape and albedo recovery pipeline. Qualitative results using
images captured by a smartphone demonstrate our method’s effectiveness in real scenarios.

▍ Significance of the research and Future perspective

Overview of the proposal

Using our method, people can immediately turn their phones into
high-fidelity 3D scanners, facilitating the digitization of cultural heritage artifacts. We believe that our method is useful in a scenario of
crowd-sourced digital archiving, which accelerates the digitizationof the world’s cultural heritages.

P a t e n t
Tr e a t i s eXu Cao; Michael Waechter; Boxin Shi et al. Shape and Albedo Recovery by Your Phone using Stereoscopic Flash and No-Flash Photography. International
Journal of Computer Vision. 2022, Volume 130, p. 1403-1415, doi: 10.1007/s11263-022-01597-6

URLK e y w o r d geometry reconstruction, stereo camera, flash photography