Drag View: Generalizable Novel View Synthesis with Unposed Imagery
Authors: name: Zhiwen Fan; name: Panwang Pan; name: Peihao Wang; name: Yifan Jiang; name: Hanwen Jiang; name: Dejia Xu; name: Zehao Zhu; name: Dilin Wang; name: Zhangyang Wang
Published: 2023-10-05 17:24:36

Abstract: We introduce DragView, a novel and interactive framework for generating novel
views of unseen scenes. DragView initializes the new view from a single source
image, and the rendering is supported by a sparse set of unposed multi-view
images, all seamlessly executed within a single feed-forward pass. Our approach
begins with users dragging a source view through a local relative coordinate
system. Pixel-aligned features are obtained by projecting the sampled 3D points
along the target ray onto the source view. We then incorporate a view-dependent
modulation layer to effectively handle occlusion during the projection.
Additionally, we broaden the epipolar attention mechanism to encompass all
source pixels, facilitating the aggregation of initialized coordinate-aligned
point features from other unposed views. Finally, we employ another transformer
to decode ray features into final pixel intensities. Crucially, our framework
does not rely on either 2D prior models or the explicit estimation of camera
poses. During testing, DragView showcases the capability to generalize to new
scenes unseen during training, also utilizing only unposed support images,
enabling the generation of photo-realistic new views characterized by flexible
camera trajectories. In our experiments, we conduct a comprehensive comparison
of the performance of DragView with recent scene representation networks
operating under pose-free conditions, as well as with generalizable NeRFs
subject to noisy test camera poses. DragView consistently demonstrates its
superior performance in view synthesis quality, while also being more
user-friendly. Project page: https://zhiwenfan.github.io/DragView/.