# View Synthesis with Scene Recognition for Cross-View Image Localization  
[PERSON]  
1 School of Cybersecurity, Old Dominion University, Norfolk, VA 23529, USA 1, [PERSON]  
1 School of Cybersecurity, Old Dominion University, Norfolk, VA 23529, USA 12 Department of ECE, Old Dominion University, Norfolk, VA 23529, USA 2  
[PERSON]  
3 Department of ECE, University of Arizona, Tucson, AZ 85721, USA 3  
[PERSON]  
1 School of Cybersecurity, Old Dominion University, Norfolk, VA 23529, USA 1, [PERSON]
user tries to match a cell phone image taken at this location with a Google Street View captured at a completely different time or season, the matching performance would be degraded. Experiments in [12] have evaluated the performance degradation caused by the misalignment of matching pairs.  
To alleviate the differences between the matching pairs caused by various lighting, scene structure, and weather conditions, in this paper we propose a framework to convert outdated reference images, such as Google Street View, with a patch image that provides desired weather/light information. The patch images are collected from Flickr, an image-sharing platform, where we can obtain a set of images contributed by people who live in the area of interest. The reconstructed output images synthesize critical features from the original reference images, as well as the weather/light embedding from the patch images. To accomplish the goal of view synthetization, the proposed framework contains three parts: a patch image generator, an autoencoder-based style exchanger [13], and an auxiliary deep learning model, Places365 [14], to provide the semantic understanding of both reference and patch images.  
Although several works have already discussed latent space manipulation [13; 15; 16; 17], i.e., transferring styles between images, the standard approach to determine the quality of transferred images does not apply to our scenario as we need to consider whether the information is equally carried over from patch and reference images. In most cases, users do not have full control of the style transfer, as the parameters and algorithms of the deep neural network determine the transfer process. This can make it challenging to achieve the desired result, especially when transferring styles between images with significant differences in content and style. Therefore, in most cases transfer results are evaluated in the form of similarity instead of fairness. To address this issue and produce a more generative framework for the view systems task, we analyzed the different attributes extracted from patch and reference images before and after the synthesis. A fairness score was calculated from the attributes extracted from the synthesized outputs compared with the individual attributes obtained from the reference image and the patch image.  
In summary, the contributions of this work are as follows:  
1. We proposed a deep generative network-based view synthesis framework to address the challenges in the existing cross-view matching tasks. It was accomplished by updating the reference images, which are usually updated in an infrequent manner, with a weather/time patch. The synthesized reference view contains features from reference views, current time, and weather conditions.  
Figure 1: (**a**) Google Street View taken in Queens, NYC, in November 2016; (**b**) Google Street View taken at the same location in July 2017.  
We propose a novel evaluation metric to measure the quality of the view synthesis instead of a subjective judgment. With the assistance of an auxiliary attribute extraction network, we can effectively select the best synthesis results by comparing the attributes before and after the view synthesizing under various testing conditions.  
The rest of the paper is organized as follows. Section 2 discusses the related works. Section 3 presents the main design of the proposed framework. Section 4 discusses the ways to evaluate the performance, and Section 5 concludes the paper.