Efficient Hybrid Zoom using Camera Fusion on Mobile Phones
Authors: name: Xiaotong Wu; name: Wei-Sheng Lai; name: YiChang Shih; name: Charles Herrmann; name: Michael Krainin; name: Deqing Sun; name: Chia-Kai Liang
Published: 2024-01-02 23:28:20

Abstract: DSLR cameras can achieve multiple zoom levels via shifting lens distances or
swapping lens types. However, these techniques are not possible on smartphone
devices due to space constraints. Most smartphone manufacturers adopt a hybrid
zoom system: commonly a Wide (W) camera at a low zoom level and a Telephoto (T)
camera at a high zoom level. To simulate zoom levels between W and T, these
systems crop and digitally upsample images from W, leading to significant
detail loss. In this paper, we propose an efficient system for hybrid zoom
super-resolution on mobile devices, which captures a synchronous pair of W and
T shots and leverages machine learning models to align and transfer details
from T to W. We further develop an adaptive blending method that accounts for
depth-of-field mismatches, scene occlusion, flow uncertainty, and alignment
errors. To minimize the domain gap, we design a dual-phone camera rig to
capture real-world inputs and ground-truths for supervised training. Our method
generates a 12-megapixel image in 500ms on a mobile platform and compares
favorably against state-of-the-art methods under extensive evaluation on
real-world scenarios.