Cross-view image-based localization and synthesis
Creators: name: Shi, Yujiao
Publisher: name: The Australian National University
Year: 2023

description: We live, work, and entertain in a visual world. Many of our daily activities require estimating correspondences between different views. For example, when lost in a foreign city, we need to localize where we are by matching what we can see in the surroundings to a map of the city in an overhead view. Consider another scenario where suppose we have captured some pictures of a place of interest and want to re-experience it virtually and immersively at home. These tasks require understanding scene geometry from the captured photos and rendering new images from different views. Motivated by these scenarios, this thesis aims to learn cross-view correspondences and leverage them for real-world applications, including cross-view image-based localization and novel view synthesis. Cross-view visual localization aims to localize a query image captured at the ground level by matching it against a large geo-tagged satellite map covering the same region. The task is challenging due to the drastic viewpoint difference between the query and the reference views. This thesis starts by localizing orientation-aligned panoramas. We first design an optimal feature transport module to bridge the cross-view domain gaps and retain the spatial layout information of the scenes for cross-view matching. Then, to further reduce the learning difficulty for neural networks, we introduce a polar transform, which bridges the cross-view domain differences in a parameter-free manner. To localize images with unknown orientation and limited FoV, we introduce a projective transform, which is complementary to the polar transform for bridging the domain gap, and a dynamic similarity matching module to estimate the relative orientation between a ground and a satellite image. This handles the orientation ambiguity and helps to achieve 3-DoF camera pose estimation via ground-to-satellite image matching. Most existing works formulate the ground-to-satellite localization as an image retrieval problem. It is efficient and can be conducted at scale. However, its localization accuracy is limited. To solve this probelm, we first introduce a handcrafted method to accurately pinpoint which pixel in the satellite image corresponds to the query camera location. This method involves the projective transform and an exhaustive search. Then, to improve the robustness and computation speed, we invent a deep pose optimization mechanism for the highly-accurate camera localization problem. The new method adopts a differentiable LM algorithm to optimize the camera pose iteratively within a network feed-forward pass. We also explore the problem of satellite-to-street-view panorama synthesis because it can potentially facilitate the performance of cross-view image-based localization. We aim to synthesize a street-view panorama from a satellite image as if it was captured at a location corresponding to the satellite image center. We introduce a multi-plane image-based method to model the scene geometry and implicitly estimate the height distribution of the satellite image. The synthesized street-view panoramas by our method have more consistent geometry configurations with respect to the input satellite images. In what follows, we consider a more generic setting: synthesizing a novel view image from a few unstructured images. In this problem, we propose a method that estimates the target-view depth and source-view visibility in an end-to-end manner. The underlying idea is: when the projected pixels of a 3D point in different views are similar to each other, a surface likely exists at the 3D point; when a 3D point is visible in a source view, the projected pixel in this source view should be similar to the projected pixels in other source views. Compared to NeRF-based methods, our approach estimates the scene geometry from the given views during inference time and thus has a stronger generalization ability.