# Cross-View Outdoor Localization in Augmented Reality by Fusing Map and Satellite Data
## 3 Methods
### Levenberg-Marquardt Optimization  
To optimize the positional variables, we used the optimizer implemented by [PERSON] et al. [13], which aimed to minimize the difference between the top-level and ground-level. After  
Figure 3: **Overview of AlignNet.** (1) The top-down input consists of satellite and OpenStreetMap data, which are then cropped and rotated around the ground-truth point with the true heading facing to the right and with randomly added translation and rotation noise. The 4 modalities are then mapped through a learned mapping and concatenated. (2) The top-down and street-level inputs are put through a two-branch U-Net feature extractor, resulting in encoded features at different levels called \(S^{1},\ldots,S^{3}\) and \(G^{1},\ldots,G^{3}\), respectively. (3) Next, starting at position \(\mathbf{X_{0}},\mathbf{Y_{0}},\mathbf{R_{0}}\), the model maps the features \(S^{l}\) to the ground-level domain. It tries to minimize the difference between the mapping and \(G^{l}\) by optimizing the pose variables using LM optimization, going first through each level, which is then repeated \(N\) times.  
calculating the deep features from the input images, we denoted the difference of the projected top-level image and ground-level image as  
\[\mathbf{e^{l}}=\mathbf{P^{l}}-\mathbf{G^{l}}, \tag{5}\]  
where \(\mathbf{P^{l}}\) is the projected top-level features at level \(l\) and \(\mathbf{G^{l}}\) is the ground-level features at level \(l\). We minimize this difference by optimizing the positional variables \(\xi=\{\mathbf{R},\mathbf{t}\}\). Our objective becomes  
\[\xi=\operatorname*{argmin}_{\xi}||\mathbf{e^{l}}||_{2}, \tag{6}\]  
where \(||.||_{2}\) denotes the L2 norm and \(\xi\) are the optimal positional variables. Given the small number of variables to be optimized, we can solve this using the Levenberg-Marquardt (LM) optimization algorithm [20].  
For each level \(l\), we calculated the Jacobian matrix and Hessian matrix with regards to the positional variables \(\xi\),  
\[\mathbf{J}=\frac{\partial\mathbf{P^{l}}}{\partial\xi}=\frac{\partial\mathbf{ P^{l}}}{\partial\mathbf{p_{s}}}\frac{\partial\mathbf{p_{s}}}{\partial\xi}, \quad\text{ and }\quad\mathbf{H}=\mathbf{J^{T}}\mathbf{J}, \tag{7}\]  
where \(\mathbf{p_{s}}\) is the satellite's feature map coordinates. We followed the original implementation and used [PERSON]'s damping formula to interpolate between gradient descent (at \(\lambda=\infty\)) and Gaussian-Newton (at \(\lambda=0\)), as formulated by  
\[\tilde{\mathbf{H}}=\mathbf{H}+\lambda\mathbf{I}. \tag{8}\]  
We used the same value of \(\lambda=0.1\) as in prior work [13]. After calculating \(\tilde{\mathbf{H}}\), we can update the pose by  
\[\xi_{t+1}=\xi_{t}+\tilde{\mathbf{H}}^{-1}\mathbf{J^{T}}\mathbf{e}, \tag{9}\]  
where \(t\) denotes the current step.  
The LM optimization was first applied to the coarsest feature level before going to the finer levels, for a total of 3 levels. This repeats 5 times before the model is finished, and the final position is compared to the ground-truth position. This scheme was chosen with the idea of the coarser levels being able to escape local minima while the finer levels can be used to align the final position at a finer scale. We found that theory did not hold up well in practice, although different levels were more well-suited to updating certain positional variables.  
Because this model is end-to-end differentiable, the positional loss can be backpropagated to the feature encoders. Furthermore, the loss between the encoded ground-level features and encoded and top-down features projected at the ground-truth position can be calculated using an L2 loss directly and can be optimized directly to stabilize the training process. This must be carried out carefully as the model could collapse an project all input pixels to the same value, which would result in a zero-valued L2 loss.