An Analytical Solution to Gauss-Newton Loss for Direct Image Alignment
Authors: Sergei Solonets
Daniil Sinitsyn
Lukas Von Stumberg
Nikita Araslanov
Daniel Cremers
Venue: ICLR 2024 oral (2024)
Keywords: featuremetric image alignment
Primary area: optimization

Abstract: Direct image alignment is a widely used technique for relative 6DoF pose estimation between two images, but its accuracy strongly depends on pose initialization.
Therefore, recent end-to-end frameworks increase the convergence basin of the learned feature descriptors with special training objectives, such as the Gauss-Newton loss.
However, the training data may exhibit bias toward a specific type of motion and pose initialization,
thus limiting the generalization of these methods.
In this work, we derive a closed-form solution to the expected optimum of the Gauss-Newton loss. 
The solution is agnostic to the underlying feature representation and allows us to dynamically adjust the basin of convergence according to our assumptions about the uncertainty in the current estimates. These properties allow for effective control over the convergence in the alignment process.
Despite using self-supervised feature embeddings, our solution achieves compelling accuracy w.r.t. the state-of-the-art direct image alignment methods trained end-to-end with pose supervision, and demonstrates improved robustness to pose initialization.
Our analytical solution exposes some inherent limitations of end-to-end learning with the Gauss-Newton loss, and establishes an intriguing connection between direct image alignment and feature-matching approaches.

Published as a conference paper at ICLR 2024
AN ANALYTICAL SOLUTION TO GAUSS-NEWTON Loss FOR DIRECT IMAGE ALIGNMENT
Sergei Solonets" 1,2,, Daniil Sinitsyn' 12,,+
Lukas von Stumberg
Nikita Araslanoy!.2
Daniel Cremers 1.2
Technical University of Munich
2 Munich Center for Machine Learning
3 Valve Software
Figure 1: Direct image alignment is technique for aligning scenes based image intensities. Recent learning- branning-based methods seek to improve its success rate by increasing the convergence basin. We derive analytical solution the core idea of such methods, the Gauss-Newton loss, enabling fine-grained control over the basin of convergence. As result, we can successfully align two scenes despite highly imprecise initialization. From left right, the example above illustrates the convergence of the reprojected keypoints (red) to the ground truth (green) by optimizing the SE(3) camera pose with our analytical solution. The blue color is the re-projection in the first iteration.
ABSTRACT
Direct image alignment is widely used technique for relative 6DoF pose estima- tion between two images, but its accuracy strongly depends on pose initialization. Therefore, recent end-to-end frameworks increase the convergence basin of the learned feature descriptors with special training objectives, such as the Gauss- Newton loss. However, the training data may exhibit bias toward specific type of motion and pose initialization, thus limiting the generalization of these methods. In this work, we derive a closed-form solution to the expected optimum the loss. The solution is agnostic to the underlying feature representa- tion and allows us to dynamically adjust the basin of convergence according to our assumptions about the uncertainty in the current estimates. These properties allow for effective control over the convergence self-supervised feature embeddings, our solution achieves compelling accuracy w: x. f. the state-of state-of-theeart direct image alignment methods trained end-to-end with pose supervision, and demonstrates improved robustness pose initialization. Our analytical solution exposes some inherent limitations of end-to-end learning with the Gauss-Newtar loss, and establishes intriguing connection between direct image alignment and feature-matching approaches]
alignment process. Despite using
INTRODUCTION
Visual localization refers to estimating the camera pose of query image r. f. reference image where the underlying 3D structure (e. g. a point cloud) is available. Traditionally, solutions to visual localization primarily relied estimating correspondences between 2D features in the query image
and 3D features in the reference point cloud (Liu et al. 2017
Sarlin et al. 2019 Sattler et al. 2017
Equual contribution. Correspondence: (s. www.neess,,,,.ooooooooooo d. sinitsyn @tum. de.
Work done while TU Munich. Project code: https://gttuh..co//mmm--
losssaaaltial

---

Published as a conference paper at ICLR 2024
Svarm et al. 2017 Toft et al. 2018). Challenging this approach, recent deep learning frameworks implement direct image alignment by re- projecting the 3D points onto the feature map (Sarlin et al. 2021 von Stumberg et al. 2020a b). Using end-to-end training, deep networks learn dense feature maps suitable for regressing the pose between an image and a point cloud. In analogy to photometric image alignment (Delaunoy and Pollefeys| 2014 Engel al. 2016), this family of methods is referred to as featuremetric image alignment (von Stumberg et al. 2020b).
The feature representation learned by featuremetric image alignment is not only discriminative, but is also spatially smooth for improved convergence (von Stumberg et al. 2020a). A training process enforces the smoothness either explicitly through a specific loss, such the Eauss-Newton loss (von Stumberg et al., 2020a), or implicitly. by backpropagating through an optimization algorithm, such as Levenberg-M Marquuurdd (Sarlin al., (02T). both cases, the resulting feature map embeds the bias of the initial poses in the training data. This may lead to poor generalization, since the training poses can differ substantially from the test scenario. We here take a less bias-prone approach: we use generic feature descriptors (e. g. obtained with self-supervision) and instead control the smoothness of the feature map dynamically at test time.
This work investigates the connection between feature descriptor networks and featuremetric image alignment. The main contribution is the analytical (closed-fore (ooosed-form) solution to the Gauss-Newton loss. On the one hand, this leads to novel technique, which can utilize any feature descriptor generate dense feature map suitable for direct image alignment. Importantly, it allows us to dynamically adjust the smoothness of the feature map, which effectively controls the trade-off between the basin of convergence and the alignment accuracy. We empirically verify our derivation using self-supervised feature descriptors, such as SuperPoint (DeTone al. 2018), and demonstrate on-par even superior alignment accuracy compared to supervised frameworks. On the other hand, the analysis of our closed-form solution reveals inherent limitations of feature learning with backpropagation through Gauss-Newton optimization: featuremetric alignment merely learns form of interpolation between feature descriptors in the points of interest. Although we demonstrate this in the context of direct image alignment, a similar argument extends to other methods, even beyond computer vision, which backpropagate through Gauss-Newton or Sevenberg-Marquaar optimization.
PRELIMINARIES
Image Alignment. Given two images (reference 4, and query 14) with known camera models and an overlapping field of view, and 3D point cloud (p(0) } in the coordinate system of I,, the problem of image alignment is to estimate the relative camera pose T E SEE(3).
Gauss-Newton Optimization. Given a set of functions (r(")(),,... (x=) called residuals, Gauss-Newton (GN) optimization finds the parameters minimizing the sum of squared residuals:
f(x) = Σ || (9) (x))1
Each residual 1.(() (zz) could either be a vector or a single-valued function. In both scenarios, the total residual r(x:) is a stacked vector of residuals. The Gauss-Newtor method seeks minimize [() by iteratively updating the parameter estimate æ. Each iteration linearizes the residuals around the current estimate and computes an update step (r
= (JTJ)- JTr(E), J == dæ
(2)
where J is the Jacobian matrix. The estimate evolves until convergence as き t ã B Ax)(r())).
(3)
Here, the operator B denotes a specific update procedure, which depends on the nature of the optimization space. For linear spaces, E simplifies to the standard subtraction operation; in the context of rigid-body transformation SE(3). operator 3 applies tangential update.
Photometric and featuremetric image alignment. Photometric image alignment estimates the relative 6DoF pose by minimizing the difference between pixel intensities of points in and the
2

---

Published as a conference paper at ICLR 2024
corresponding points in I.. Recalling p() 3D point in the coordinate frame of the reference image, we seek a transformation T E (3) minimizing the following residuals:
(T) = 1. L4 ((Tp)))
Here, (+) is 2D projection operator of 3D point onto the image plane. The projection implicitly uses the corresponding camera intrinsic parameters, assumed to be available for both I, and Ig. We omit them in the notation for clarity. Comprising rotation E (O(3) and translation E R the target transformation T is typically found by minimizing Eq. 4 with non-linear
optimization, such as Gauss-Newer or Syvenberg-Marquaae (LM) methods (Levenberg 1944
Marquardt 1963 Nocedal and Wright 1999).
The success of photometric image alignment critically depends on a favorable initialization of the pose, especially in the conditions of varying illumination and occlusion. As partial cemedy.featurent image alignment (Tang and Tan 2019) uses feature maps instead of pixel intensities. Obtained with deep learning, such feature maps have increased convergence basin compared to that derived from image intensities, which improves robustness to pose initialization.
3 RELATED WORK
Direct and indirect image alignment. Estimating the relative camera pose from two images is fundamental problem in computer vision, with applications in structure from motion (SfM) (Schönberger and Frahm 2016). SLAM and relocalization. To address this problem, indirect (feature- based) approaches detect and match interest points (Bay et al. 2006 Lowe 2004) in both images, and then estimate the pose using PnP (Persson and Nordberg 2018) or by minimizing the reprojection errors (Triggs et al. 1999). In contrast, direct methods sidestep the matching process and minimize the photometric error instead (Horn and Jr. 1988, Irani and Anandan 1999). This means that they can leverage the entire image (Kerl et 2013 Newcombe et al. (011). or focus on pixels with sufficient gradient (Engel et al., [0144] 2016)). The foundation behind these approaches is Lucas-Kanade tracking (Baker and Matthews | Lucas and Kanade 198T). However, direct methods, which are central to this work, optimize for a 6DoF pose instead of individual pixel displacements (i. e. optical flow).
Pose estimation with deep learning. The advent of deep learning revitalized interest in improving pose estimation with deep networks. While some approaches are holistic (Jatavallabhlla et al. 2020), there are broadly three categories of learning-base methods. i) Fully end-to-end pose estimation methods (Kendall al. 2016 Ummenhofer et al. 2017 Zhou et al. (017) directly regress pose estimates with deep neural networks, instead of time-time&oo optimization. ii) Learning-based
indirect methods (DeTone et al., 2018: Dusmanu et al. 2019 Revaud et al., 2019; Yi et al. (016)
replace handcrafted detectors and descriptors with deep representations in indirect pipeline. Some approaches further extend the traditional way of obtaining correspondences. SuperGlue (Sarlin let al. 2020) learns feature matching with graph neural network. LoFTR (Sun et al. 202T) directly regresses correspondences instead of relying on separate feature detection and matching. Similar to indirect methods, dii learning-bused direct image alignment enhances classical direct methods with deep features. This category is the most similar to our work and we discuss it in more detail next.
Learned features for direct image alignment. Previous work differs in their approach to model training and in the final task. For example, Czarnowski et al. (2017) leverage off-the-shelf CNN features to improve optical flow tracking. A number of methods (Han al. 2018 Lv al., 2019
Sarlin et al. 2021 Tang and Tan 2019: Xu et al. 202T) train feature descriptors end-to-end with ground truth poses and backpropagate the gradient through a non-linear optimization process. At test time, these methods employ feature pyramid and refine the initial camera pose in a coarse-to-fine fashion using GN or LM optimization. In addition to the feature pyramid, some methods predict additional properties for image alignment, such uncertainty (Xu et al.) 2021) (Sarlin et al. (021). Jacobians (Han et al., 2018), or optimization parameters (e. g. damping factors (Lv et al. [2019] Sarlin et al. 2021)). In the same spirit, our formulation leads to a continuous image pyramid, where each level of the pyramid can be generated on-the-fly -ly based on the input level of uncertainty.
Another line of work (von Stumberg al.) 2200a b) trains deep network directly on the ground- truth pixel correspondences. GN-Net (von Stumberg et al.] 2020a) minimizes two loss functions. The first is contrastive loss (Schmidt et al.] 2017) facilitating discriminative properties of the
3

---

Published as a conference paper at ICLR 2024
feature representation. The second loss function accounts for likely displacement in the initial correspondence, implemented by adding random 2D offsets to the ground -tunt correspondences ar training time. By learning to minimize the feature discrepancy after a Gauss-Newton step. the model learns to cope with the initially noisy estimates. We revisit this work in detail in the Sec.
The learning- -ased approaches (e. g. (Germain et al. 2021 Sarlin et al. 2021 von Stumberg et al. 2020al) achieve impressive accuracy of pose estimation. Nevertheless, they require pose superviston for training and struggle in scenarios of large-baseline localization. Indeed, GN-Net (von Stumberg et al. 2020a) and PixLoc (Sarlin et al. 202T] tend to exhibit a strong bias toward the noise assumptions of the training process, which cannot be easily reversed. For example. PixLoc trained on the CMU dataset exhibits a strong bias toward horizontal movement as illustrated in Appendix A
Our work takes a different approach. We rely on existing feature descriptors obtained with self- supervision, which contribute no explicit motion bias to the alignment process. We next derive closed-form solution to the Gauss-Newton loss (von Stumberg al.] 2020a) as functional of a probability density governing the noise assumptions of the current pose estimate. This allows to adjust the noise assumptions the alignment process, thus effectively controlling the convergence basin at test time, much akin to the coarse-to-fine strategy, exemplified by Fig. 2
4 THE
LOSS
In this section, we recap the GN-Net (yon Stumberg et al, 2020a) and introduce the notation. GN-Net is convolutional neural network : 0) trained
sparse set of ground-truth truth correspondences. Given coordinates on the image plane, let f IR x IR + Rd continuously map those coordinates to a descriptor from feature grid ddx produced by F 0), e. g. using bilinear interpolation. We define fr ;= E(;;; 0) and fg == Elgy 0) to denote feature representations of reference and query images. GN-Net learns parameters 0 to minimize the expected value of the following loss function, fa) LoN-Ne(LF, L4) Ccontrastive ffr, + CGN (fr. fq),
(5) which comprises a contrastive loss, (caontastive(:::)) and the Gauss-Newton loss, Lax(s.). The contrastive loss minimizes the distance between the features of two corresponding points while maximizing the distance between non-corresponding pairs (Schmidt et al., (017). The contrastive loss facilitates spatially discriminative features, and its particular instantiation has little significance for the following discussion (e. g. GN-Net uses the triplet loss).
The Gauss-Newton loss LuN ensures that the feature map is sufficiently smooth for direct image alignment, thus enlarging the convergence basin. GN-Net implements this by adding random offset to the ground-truth correspondences and encouraging a single Gauss-Newton step to recover the original location. Let us formalize this process.
Given a ground- truth correspondence (x( y ()) between images I, and 1g- the assumption behind the Newson loss is that the initial estimate ( at test time falls in the vicinity of the ground truth x (+), i. e. ã = (i) +e, e, where follows some predefined distribution p(e), such as Gaussian with zero mean. At training time, the Gauss-Newton loss aims at recovering the ground-truth location
(i) from noisy initial location (60) by minimizing the residual w. 2::
م (x(0) ((((()) (z)
(6)
The original Gauss-Newton loss for one point (1), as introduced by von Stumberg et al. (2020a), is
GN (f. :- ||] Dax (2 + D logdet JJJ.
From a probabilistic standpoint (von Stumberg et al. 2020a), the loss balances between the accuracy of the Gauss-Newton step and the direction uncertainty. Here, JTJ represents the inverse covariance matrix propagated through the photometric residuals. The loss function corresponds to the negative ligeliikl of residuals distributed as N (0, (JTJ)-1). In this work, we consider a variant of the Newton loss, in which the covariance matrix is assumed to be identity. Thus, Eq. becomes
C Sy, E) ||e (x) + +
(8)
4

---

Published as a conference paper at ICLR 2024
Although this simplified version does not account for the trade-off between uncertainty and accuracy of the prediction, it admits a closed-fome form minimizer of the expected value, as we show Sec. [5] We also find that the simplified version does not differ from the original one empirically in sigmicaat way (see Appendix C Henceforth, we will refer to the simplified version as the Gauss-Newton loss.
The training process calculates the Gauss-Newton loss stochastically by sampling from p(c). Therefore, it minimizes a Monte-Carlo approximation to the expected value of cl summed over all ground-truth correspondences,
Cas(JrsSaiP)) Bemp!
Hereafter, we use the notation (Jr(Jr: fqi emphasize the dependence of the Gauss-Newton loss on the noise distribution p. and omit this parametarization otherwise to avoid clutter.
Note that the GN-Net's training stage addresses the problem of optical flow, not pose estimation. The underlying assumption is that accurate optical flow facilitates pose estimation, each pixel will contribute to the final pose estimate. At test time, the pose is determined by solving the featuremetric image alignment problem (equivalent to Eq. using Gauss-Newton optimization SE(3).
5 CLOSED-FORM GAUSS-NEWTOR STEP
Decoupling the contrastive and Gauss-Newton losses. The contrastive loss provides sparse constraint on the feature embeddings, since we can only use sparse ground-truth correspondences, the interest points, for supervision. By contrast, the Gauss-Newton loss enforces a pre-defined basin of convergence around each interest point (modeled by p(E)) with little effect on the feature descriptors in the interest points. This is because the residual in Eq. (6) between the corresponding interest points will be negligible, if the contrastive loss for those points Is minimized. It follows that the contrastive and the Mauss-Newton loss essentially optimize over disjoint set of feature locations. Therefore, can decouple the Gauss-Newton loss from the joint optimization objective in Eq. (5) Let us formalize this reasoning. We aim to solve:
f = arg min (C fn) + Cax(frsSa))
(10)
We denote the values of fr. fy the interest points as Fr, Fg: F.!) (z) F The contrastive loss only depends on F, and Fg. while the Gauss-Newtor loss depends on all the query feature values fa and the interest points in the reference map F,.. By eliminating unused parts, we introduce equivalent problem:
())
F fg) arg min [Ceontrastive Pr, Fg) + LGN(Fr, fq)] Eg-f,
(11)
Next, approximate F, with Fg in the second term. This is permissible as the contrastive loss acts as soft constraint, ensuring that F, and F, are close at the optimum of the joint loss function. This allows us decouple the minimization problem follows:
(Fr, F arg min [oomtrative (Fr, Fg) min Lax Sa) F. S arg min (F,(FF fo
fq
(12)
We denote
G(Fgip) = arg min Lox (F Sgip),
(13)
(F = min Lox Fg. JaiP) = LoN G(
fo
5

---

Published as a conference paper at ICLR 2024
Image
10
30
80
200
Figure 2: Controlling the basin of convergence. Image samples and the corresponding feature maps fg (PCA) for different values of a and isotropic Gaussian distribution p.
G(Fgip) is the optimal reconstruction of feature map under sparse feature descriptors Fg and noise density p(c). (Fg(Fg:)) is the corresponding value of the Gauss-Newton loss under Fg. We will refer to G(F fg denote optimal reconstruction under arbitrary sparse input Fg. Note the difference to f7, which an optimal reconstruction under the joint loss, i. e. G(F;; Our main contribution, detailed shortly, is closed-form solution for both ( p) and LGN (F: reveals that the original problem in Eq. (TO) actually depends only the feature values in the interest points:
F arg min [Ccontrastive (F,, + (F,(F
(14)
This means that in training a feature extractor only the features of interest points matter, while the representation of other pixels approximates analytical solution. As the empirical validation, demonstrate compelling results in our experiments by using feature descriptors for the interest points from self-supervised method, while employing our analytical solution for the remaining pixels.
An analytical solution to the Gauss-Newton loss. We propose to calculate the expectation of the Gauss-Newton loss in Eq. (9) functional of fq. and analytically find its closed-for minimizer.
Consider the query image I, as set ((x(1) E(()) containing locations of interest points (+) and the corresponding feature descriptors r2()) (x) The locations x+ can be extracted by an off-the-shed feature detector (e. SuperPoint (DeTone et al. 2018)), while a network trained with the contrastive loss in Eq. 5 can produce the corresponding descriptors F(())
Let us first re-write the expectation in Eq. (9) using Eq. ( and Eq. (2) in the decoupled formulation of the Gauss-Newton loss (ef. Eq. 12)):
(g:t (J J (fg(z(0) (c) - (D(FF)))) p(e)de.
{
(15)
Substituting c= x (1)
we obtain
(Fn = Σ 12
(JJJ)- JI (F((²) - F(1) 2 plē x())dā
(16)
We first consider all à E Ω independently and relax the problem by eliminating the constraint J 0f. This allows derive analytical solutions for minimizers fĄ(+) and J.(.) as functions

---

Published as a conference paper at ICLR 2024
Query image l
Interest points & descriptors
Our analytical solution G(-p)
festure extraction
Reference image I,
Point cloud (p
==CG(p)
low
basin of convergence
high
p())
Scheduler
fr
Featuremetric alignment Algorithm 1
Figure 3: Image alignment with continuous probabilistic feature pyramids. Given reference image with 3D point cloud and a query image paired with an initial coarse pose, start by estimating interest points and their features on the query image, along with features from the projections of the 3D point cloud onto the reference image. Next, perform featuremetric alignment between the 3D point features and derived analytical continuous image pyramid.
of σ. We then observe that 3 (.) coincides with J.(.) for uniform distribution, which confirms that in this case, our derived solution is the solution to the original problem. As a side note, in the context of direct image alignment, it has been suggested that decoupling and predicting the Jacobian independently from the function value may yield superior convergence and results (Han et al. 2018 Appendix provides the full derivation. Below, we summarize the closed-form minimizer to Eq. 16):
fĄ()) J,(#)(E X, (8)) + ym (z),
(17)
J,(#) (Covxy (z) (ā)
y 11)
(18)
(z) Σ F p(ă -z(5))
Σ p(a -x(1),,
(19)
Xm Xm(i) (i) Σ 1o110((3 2011// Σp(E p(o xl1),
(20)
(z)= = Σ (z())= xm (i) (F(() vm (i)) p(ã x()))
Covxy (2)
(21)
Covy($)= = Σ F(J) ym pm())) (F(O) vm ( p(ă z(0),
(22)
where (+) denotes the Moore Penrose pseudo-inverss operator. The optimal point is G(Fg:)) : fa and the loss value at this point (ef. Eq. (13) is
/ Tr [Covx())= Covxy (o) (orrx)) dē.
CGN(FF)
(23)
Our derivation of the optimal embeddings C(FG:P) using fixed interest points Fg provides inter- esting insight into end-to-end learning frameworks. Jointly training both losses requires inverting a high-dimensional matrix Covy($) in Eq. (23 which poses high numerical instability and disconti- nuities. Since previous work (von Stumberg et al. 2020a) can be seen as stochastic approximations solution, this may explain the reported training instability and divergence in those works. Additionally, the form of f, in Eq. (17) suggests that end-to-end pipelines may not necessarily yield any sophisticated representation, as They merely interpolate between features in the interest points.
Featuremetric Image Alignment. As a case study, we employ the analytical form of f, and J, for featuremetric image alignment. Algorithm [ and Fig. provide an overview. Fig. iilustratess stages of direct alignment using our probabilistically reconstructed feature maps. We first extract the interest points (a:(1) and the corresponding feature descriptors {F(5) from I to reconstruct f, and
7

---

Published as a conference paper at ICLR 2024
Require: 7, Jn. {p(0), T(O)
1: {x01) f InterestPoints
60
Ours (mean)
2: {Fg
ElIy; 0) o fo {x
Ours (median)
3: (o() + E(I;; 0) o ( (p(1))}
PixLoc (mean)
4: T (T)
Final [deg]
40
LixLoc (median)
5: for from 0 to Mmxx do p+- --Cceduuler(n)
6:
+ G(( Jip)
7:
8:
(f }
d+AAN - (o(1))
9:
10:
T+0
end for
12: return
20
100
()) denotes 2D projection.
Initial error [deg]
Figure 4: Robustness to initialization. Our closed- form solution exhibits significantly greater robustness compared to PixLoc.
Algorithm I: Image alignment.
Jy using Eqs. and 18 The sparse of 3D points (p()) } projected onto the reference image to obtain reference descriptors. We obtain feature descriptors for these points using the same feature embedding network F² ; 0). Starting with the initial camera pose ( E SE(3), at every iteration we perform Gauss-Newton steps minimizing the residuals y(0), w. r. the camera transform T:
10 (T) = ( - (T)p
(24)
p(c) explicitly controls the basin of convergence. In practice, we any distribution with high initial variance based on the inaccuracy assumed in the initial point projections. As our estimates gradually decrease the variance. Notably, the choice of
improve over the course of optimization,
p(e) lets us define values for points outside of the image boundaries. Fig. iillustrates examples of the feature maps fg with following Gaussian distribution of increasing variance.
Comparison to previous work. Our derivation agnostic to the underlying feature extractor and can operate with off-the-shelf feature descriptors. An interesting property of the derived fy that it enables continuous coarse-to- to-nee alignment. By adjusting the noise prior p in the scheduler (cf. Algorithm [ we can control the basin convergence dynamically at runtime. We can adjust the distribution p either by choosing different parameter set of a pre-defined distribution, or by changing the distribution itself to another family. In practice, we start the optimization with a uniform distribution and then switch to the Gaussian distribution with slowly decreasing variance.
uniform distribution with a wide support offers a large basin of convergence, which sacrifices accuracy for robustness. The ensuing Gaussian distribution with decreasing variance refines the pose and leads to a more accurate solution. Appendix provides implementation details of the scheduler.
Connection between feature matching and featuremetric image alignment. Note that the computation of fy using Eq. (17) only depends the feature descriptors of the interest points in the query image. This observation suggests an interesting interpretation of minimizing the residual in Eq. 24 using fa with fully determined by the neighboring points of interest. Consequently, it implies that the effectiveness of such optimization-bast methods may be limited in comparison to alternative optimization-inspine approaches, which learn the optimization step (Teed and Deng 2020).
optimization as feature matching. The Gauss-Newton step is
EXPERIMENTS
Datasets. We evaluate approach on two most popular datasets for large-scale image localization,
namely the Aachen Diy-Night dataset (Sattler et al. 2018). extended CMU seasons (Toft et all, (022) and 7Scenes dataset (Shotton al. 2013). Aachen Day-Night consists of 98 night and 824 day query
8

---

Published as a conference paper at ICLR 2024
Table 1: Camera localization on Aachen Day-Night and CMU Seasons. Our evaluation shows an overall improved accuracy in diverse scenarios despite using self-super SuperPoint descriptors. We compare to the state-of-the-art methods supervised with pose: GN-Net (von Stumberg al.;
2020a). LM-Reloc (von Stumberg et al. 2020b) and PixLoc (Sarlin et al.
2021b.
Aachen
CMU Seasons
Method
Day
Night
Urban
Suburban
Park
GN-Net
62.4/69.4 76.9
49.0/
/6.33
66.3
75.4/79.9/92.0
64.7/67.0/ 81.4 46.7/48.3//653
LM-Reloc
600.4/6.0//
76.3
37.8/ 46.9/ 59.2
76.6/82.5/93.4
22.3/72.0/22
49.1/53.4/ 66.9
PixLoc
64.3/69.35 77.4
51.0 55.1
/ 67.3
88.3 / /9999944/93..
79.6/81.1/
11/8811/852.
61.0/62
69.4 69.4
Ours (Super(Pour)
66.3 / 72.5 78.8
53.9/50.0/55 56.1
86.0/ 90.6// /95.2
79.8/ 85.0 92.4 63.4 67.9 77.5
images. Extended CMU seasons consists of 14 slices, 5 for urban environment, 5 for suburban and for park. Each slice contains between 3000 and 5000 query images. The 7Scenes dataset comprises seven distinct scenes, each containing multiple sequences of 500 to 1000 frames. For our ablation
study in Appendix D we used Cambridge Landmarks dataset (Kendall et al. 2016).
Results. Table presents the results for camera localization on Aachen Day-Night (Sattler et al. 2018) and the extended CMU Seasons (Toft et al.] (022). We use SuperPoint (DeTTone et al. 2018) as the feature descriptor in these experiments. In each setting (Day/Night for Aachen; Ur- ban/Suburban/Park for CMU Seasons), we report three numbers, which indicate the percentage of the query images that were successfully localized within the specified translation and rotation thresholds We adopt the standard threshold values defined by the benchmarks (translation, rotation): (0.25m, 2 / (0.5m, 59) (5m, 10)). We adopted the implementation from PixLoc (Sarlin et al. 2021] to run these benchmarks A comparison with PixLoc on the 7Scenes dataset is presented in Appendix F
We observe that our solution, despite using supervuue descriptors from SuperPoint (DeTone eet al. 2018), achieves an overall strong localization accuracy across diverse settings in comparison to supervised frameworks. Aachen Day-Night has significant occlusions, hence many outliers. Since the Gauss-Newton loss does not incorporate any outlier filtering, we do not expect high accuracy for our approach on this dataset. Nevertheless, we were surprised to find that on Aachen Day our approach even slightly surpassed previous state of the art. On Aachen Night, our accuracy is inferior to previous work, however. This is somewhat expected, since SuperPoint feature descriptors were not directly trained on day-night correspondences. By contrast, PixLoc was trained with supervision on day-night image pairs, which provides an obvious advantage. On CMU Seasons, our approach demonstrates a clear improvement over previous featuremetric alignment methods.
The proposed approach demonstrates notable robustness to initialization noise. We compared the proposed scheme and PixLoc on the Cambridge Landmarks dataset, varying the levels of random noise applied to the ground-truth pose as illustrated in Fig. and Appendix G
Overall, these results empirically confirm the validity of our closed-form derivation. Furthermore, the comparison suggests a limited benefit of end-to-end learning with the Gauss-Newton loss. By contrast, a dynamic convergence basin offered by our analytical solution provides versatility w. r. t. the underlying feature descriptors and improves robustness pose initialization, result.
7 CONCLUSION
We derived a closed-form solution to the which offers two main advantages. First, it allows for dynamic control of the convergence basin, which improves robustness of the alignment to pose initialization. Furthermore, despite using self- supervised descriptors, such control leads to compelling accuracy of pose estimates in comparison to supervised pipelines established benchmarks. Second, our derivation exposes intrinsic limitations of employing the Gauss-Newter loss in deep learning, as it only leads to a form of interpolation between the feature descriptors in the interest points. This insight offers an interesting connection between direct image alignment and feature matching, and leads to a novel perspective on learning robust features end-to-end, which we will investigate future work.
loss in the context of direct image alignment,
hutps///gthnh.comkegggiittt

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by the ERC Advanced Grant SIMULACRON and by TUM Georg Nemetschek Institute under the project A14TWINNING, We thank Linus Härenstam-Niclsen for helpful discussions and proofreading.
REPRODUCIBILITY STATEMENT
The main contribution of this work the derivation of the closed-form solution to the Gauss-Newton loss. We provide details of this derivation Appendix B To reproduce our experimental results, we elaborate on the implementation details Appendix To facilitate reproducibility in future research, we also publicly release our code.
REFERENCES
Baker and I. A. Matthews. Lucas-Kanade 20 years on: A unifying framework. Int. J. Comput. Vis., 56(3): 221-255, 2004.
H. Bay, T. Tuytelaars, and I.. Van Gool. SURF: Speoded up robust fcatures. In ECCV, pages 404-417, 2006.
Czarnowski, Leutenegger, and A. Davison. Semantic texture for robust dense tracking. ICCV Workshops, pages 851-859, 2017.
A. Delaunoy and M. Pollefoys. Photometric bundle adjustment for dense multi-view 3D modeling. CVPR, pages 1486-1493, 2014.
D. DeTone, Malisiewicz, and A. Rabinovich. SuperPoint: Self-supervised interest point detection and description. In CVPR Workshops, pages 224-236, 2018.
M. Dusmanu, I. Rocco, Pajdla, M. Pollefeys, J. Sivic, A. Torii, and T. Sattler. D2-Net: trainable CNN for joint description and detection of local features. In CVPR, pages (092-8101) 2019.
J. Engel. T. Schöps, and D. Cremers. ASD-SLAM: large-scale direct monocular SLAM. In D. J. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, editors, ECCV, volume 8690, pages 834-849, 2014.
J. Engel, Koltun, and D. Cremers. Direct sparse odometry. [cs.CV] arXiv: 2016.
Germain, Lepetit, and G. Bourmaud, Neural reprojection enor: Merging feature learning and camera pose estimation, 2021.
Gleize, W. Wang, and M. Feiszli. SiL.K Simple Learned Keypoints. [cs.CV] arXiv:2304.061 2023.
Han, M. Ji, Fang, and M. Nießner. RegNet: Learning the optimization of direct image-to- to-e pose registration. [es.CV] arXiv: 1812. 10212, 2018.
B. K. Horn and F. J. W. Direct methods for recovering motion. Int. J. Comput. Vis., 2(1):51-76, 1988.
M. Irani and P. Anandan. A unified approach to moving object detection in 2D and 3D scenes. In ICCV Workshops, volume 1883, pages 267-277. Springer, 1999.
K. M. Jatavallabhula, G. Iyer, and L. Paull. VSLAM: Dense SLAM meets automatic diffenentiation. In ICRA, pages 2130-2137. 2020.
A. Kendall, M. Grimes, and R. Cipolla. PoseNet: A convolutional network for real-time 6-DoF camera relocalization. /cs. CVJ arXiv:1505.07427 2016.
C. Kerl, J. Sturm, and Cremers. Dense visual SLAM for RGB-D cameras. In IROS, pages 2100-2106. 2013.
K. Levenberg. A method for the solution of certain non-linear problems in least squares. Quarterly of Applied Mathematics, 162-166. 1944.
L. Liu, H. Li, and Y. Dai. Efficient global 2D-3D matching for camera localization in large-scale 3D map. In ICCV, pages 2391-2400. 2017.
D. G. Lowe. Distinctive image features from scale-invariant keypoints. Inf. J. Comput. Vision, 60(2):91-110. 2004.
10

---

Published as a conference paper at ICLR 2024
B. Lucas and T. Kanade. An iterative image registration technique with application vision. In IJCAI, volume 81, 1981.
Z. Lv, F. Dellaert, J. M. Rehg, and A. Geiger. Taking deeper look the inverse compositional algorithm. In CVPR. pages 4581-4590, 2019.
D. W. Marquardt. An algorithm for lqaaresquares estimation of nonlinear parameters. Journal the Society for Industrial and Applied Mathematics, 11(2):431 -411, 1963.
R. Newcombe, S. Lovegrove, and A. Davison. DTAM: Dense tracking and mapping in real-time. In ICCV, pages 2320-2327, 2011.
J. Nocedal J. Wright. Numerical Oprimization. Springer, 1999.
A. Paszke, S. Gross, F. Massa, A. Lerer, al. PyTorch: An imperative style, high-performance deep learning library. In NeurlPS, pages 0024-8033 2019.
R. Pautrat, J. Lin, V. Larsson, M. R. Oswald, and M. Pollefeys. SOLD2: Self-supervised acclusion-aware line description and detection. In CVPR, pages 1.368-11378 2021.
M. Persson and K. Nordberg. Lambda twist: An accurate fast robust perspective three point (P3P) solver. In ECCV, volume 11208. pages 334-349, 2018.
J. Revaud, P. Weinzaepfel, C. R. de Souza, and M. Humenberger. R2D2: Repeatable and reliable detector and descriptor. In NeurlPS, 2019.
Sarlin, C. Cadena, R. Siegwart, and M. Dymezyk. From coarse to fine: Robust hicrarchical localization at large scale. In CVPR, pages 12716 12725, 2019.
Sarlin, D. DeTone, Malisiewicz, and Rabinovich. SuperGlue: Learning feature matching with graph neural networks. In CVPR, pages 4937-4946, 2020.
Sarlin. A. Unagar, Larsson.
Back to feature: Learning robust camera localization from pixels to pose. In CVPR, pages 3247-3257, 2021.
Sattler, B. Leibe, and L. Kobbelt. Efficient & effective prioritized matching for large-scale image-based localization, IEEE T. Pattern Anal. Mach. Intell., 39(9):1744 1756, 2017.
T. Sattler, W. Maddern, C. Toft, et al. Benchmarking 6DOF outdoor visual localization in changing conditions. In CVPR, pages 8601-8610, 2018.
T. Schmidt, R. A. Newcombe, and D. Fox. Self-supervised visual descriptor learning for dense correspondence. In IEEE Robotics Autom. Lett., volume 2, pages 420 420-427, 2017.
J. L. Schönherger and J. Frahm. Stracture-roommmation revisited. In CVPR, pages 4104-4113, 2016.
Shotton, B. Glocker, C. Zach, Izadi, A. Criminisi, A. Fitzgibbon. Scene coordinate regression forests for camera relocalization RGB-D images. In CVPR, pages 9930-2937, 2013.
J. Sun, Z. Shen, Y. Wang, H. Bao, and X. Zhou. LoFTR: Detector-free local feature matching with transformers. CVPR, pages 8922-8931, 2021.
L. Svarm, O. Enqvist, F. Kahl, and M. Oskarsson. City-scale localization for cameras with known vertical direction. IEEE Pattern Anal. Mach. Intell., 39(7):1455 1461, 2017.
C. Tang and P. Tan. BA-Net: Dense bundle adjustment networks. In ICLR, 2019.
Z. Teed and J. Deng. RAFT: Recurrent all-pairs field transforms for optical flow. In ECCV, volume 12347, pages 402-419, 2020.
C. Toft, E. Stenhorg, I.. Hammarstrand, I.. Brynte, Pollefeys, T. Sattler, and F. Kahl. Semantic match consistency for long-term visual localization. In ECCV, volume 11206, pages 391-408, 2018.
C. Toft, Maddern, Torii, Long-term visual localization revisited. IEEE T. Pattrnn Anal, Mach. Intell., 44(4):2074-208 2022.
B. Triggs, P. F. McLauchlan. R. I. Hartley, and A. W. Fitzgibbon. Bundle adjustment A modern synthesis. In ICCV Workshops, volume 1883, pages 298-372, 1999.
B. Ummenhofer. H. Zhou, J. Uhrig, N. Mayer, F. 1lg. A. Dosovitskiy, and T. Brox. DeMon: Depth and motion network for learning monocular steren. In CVPR, pages 5038-5047, 2017.

---

Published as a conference paper at ICLR 2024
L. Stumberg, P. Wenzel, Khan, and D. Cremers. GN-Net: The Gauss-Newton loss for melti-weather relocalization. IEEE Robotics Autom. Lett., 5(2):800-877 897, 2020a.
L. von Stumberg, P. Wenzel, N. Yang. and D. Cremers. LM-Reloc: Ayvenberg-Marquarte based direct visual relocalization. In 3DV, pages 968-977, 2020b.
B. Xu, A. Davison, and Leutenegger. Deep probabilistic feature-metric tracking. IEEE Robotics Autom. Lett, 6(1):223-230, 2021.
Trulls, V. Lepetit, and Fua. LIFT: Learned invariant feature transform. In ECCV, volume 9910, pages 467-483, 2016.
T. Zhou, M. Brown, N. Snavely, and D. G. Lowe. Unsupervised learning of depth and ego-motion from video. In CVPR, pages 6612-6619, 2017.
A OPTICAL FLOW BIAS
We demonstrate that PixLoc exhibits bias inher- ited from the training dataset. Fig. illustrates the average translation error after optimization when evaluated on the Cambridge dataset, high- lighting the impact of different initial displace- ments along the image axes. To solely evaluate the bias learned by the feature maps, we dis- abled -ptaset- learned movement priors denoted in PixLoc (Sarlin et al. 202I). Our results demonstrate that when trained on the CMU dataset, the network predicts optical flow along the X axis more accurately. This increased accuracy is directly linked to the structure of the CMU dataset. Specifically, the dataset com- prises image pairs captured by a camera inside moving car, pointed towards the side of the road. Consequently, this positioning results in the net- work predominantly learning horizontal optical flow patterns, reflecting the lateral movement observed in the images of the training dataset.
Oy (MegaDepth)
Ox (MegaDepth)
Oy (CMU)
Ox (CMU)
Final
0
Initial ervor [m]
Figure 5: The final error of the PixLoc solution along X/Y- axes w. E. the initial error PixLoc remains accurate only along X X-axis on CMU, re- flecting the dominant motion in the training set.
B DERIVATION OF THE ANALYTICAL SOLUTION
In this section, provide solution to minimizing Eq. (66) from the main text. Let Ω be the coordinate domain of an image plane [0, 1] x [0, 1]. and let E Rd define the feature space. We aim to find f E:
== arg min Σ ->())))))))) J(Ø) - F(5) x001 p(ē x ( (25)
We start by relaxing the constraint V fz(x) (xx). This makes every value of [(x) and J(x) locally independent and, therefore, the minimum is achieved by minimizing Eq. (25) independently over every point in Ω:
((()) = arg min f.j
Σ σ (jTj)² jo (f-F()))
F(31 z(1) p(ø z0)).
(26)
Note that the only significant part of ff is part which is spanned columns of J. This can be easily by observing that ji f is non-normalized projection onto J. More formally, let find ff as
12

---

Published as a conference paper at ICLR 2024
Ja + J b. where J is basis for an orthogonal complement for span(J). By substituting this into Eq. (26). we obtain:
Σ i w6) - a (JTJ) jT 2
((()) - arg min a. f()) E {J(a)((z) + ((t) b E 2
p(ē -
(27)
Observe that this problem is linear least squares in a and (Ji)) -1 jr and recall that p(-) 2 (by
definition), hence the problem is convex. Minimality conditions for Eq. (27 for a and (jtj) jr
are:
da Σ à 1--at v()
(It) F1F(S)) p(Φ x0))= =0,
(28)
d a
Σ ④ 10)) a + (jt) -1 f1F(D) FOOTp( x161) 0. (29)
(JtJ) -1 jr
:
From Eq. 28 have:
a a= Xm (m) + (JrJ) Jt (+))
(30)
where
(ā) = Σ pf5ppt 2111)// > Ep(ār x0())),
(31)
X (3) = E xl zb1i// Σ p( xz1i).
Substituting a into Eq. 29 we obtain
Σ
Xm xm) (x) x01) + (jtJ) jr (F(0) (110)- -xm(3)) p(UTp(0 x0))) 0,
(32)
-E (x(1) (((0))
xm FUTTpE x(1))
(jiJ) jr Σ (F()) - vm( FUTTp(ē - x())) = 0,
(33)
(σ) + (jtj) -1 jiCovy(ā) = O,
(34)
where
Covxy (x) := > (z) xm (i)) (m())) vm) 7 p(ă 111))
(35)
Covy(#))= == Σ (F() (F(()) )
(( - ym p( xl)).
(36)
Solving Eq. 34) for J and substituting the solution into Eq.
27 results in
j(@) (Covxy( (o) -))
(37)
13

---

Published as a conference paper at ICLR 2024
100
80
Recall [%]
Recall [%)
60
40
20
simplified probabilistic
simplified
0
probabilistic
0.1
0.2
0.3
0.4
0.5
0
2
3
Rotation [degrees]
Translation
Figure 6: Comparison of rotation and translation recalls for our simplified version of GN-loss and probabilistic one
f(x) = J(K)a J( xm)) + J(t) (Ja)) -1 M())Tym(*)
(38)
J()) (J(8)T((3)) T ym (i) can be simplified. Note that in the Gauss-Newton step we
projecting f()) onto J(t). J()) (J()) J(z) T is projection operator, projecting two times. Therefore, there is another equivalent solution which we will further use for the sake of simplicity:
f(i) - J(z)
Xm (3)) + (z))
(39)
Observe that substituting j from Eq. (39) and Eq. 38 into Eq. (66 yields the same loss value.
Notably, V f()) = =())) for a uniform distribution p(-).
C PROBABILISTIC vs. SIMPLIFIED GAUSS-NEWTOR LOSS
In this section, we compare our simplified Gauss-Newton loss formulation,
Σ| (JT)) jr (f r(1)) 20611 plā 21)))
arg min
(40)
f.j
the original loss, which was derived through the maximum likelihood estimation (MLE),
(1 a ( fr -F-F()) x 2 )(= logdett((tt) ) p(a - 2.))).
min Σ f.j
(41)
note that the solution to the probabilistic loss is much more complicated and computationally expensive. Derivatives for this formulation are high-order polynomials in elements of f and J.
It could be solved by general methods of algebraic geometry like Gröbner basis or Homotopy continuation, but the existence of a closed-form solution is not guaranteed, and we were not able to find one.
Nevertheless, in order to compare the two formulations, we approached the problem numerically. Fig. 6 presents the results. We used the Aachen dataset with hloc poses plot the recall of query
14

---

Published as a conference paper at ICLR 2024
images function of rotation and translation errors. We observe that the accuracy difference between these two formulations is negligible in practice. However, the simplified version admits a closed-form solution and can be computed several orders of magnitudes more efficiently.
D A STUDY OF FEATURE DESCRIPTORS
show that our approach is agnostic to the choice the underlying feature de- scriptor. To analyze the accuracy of algorithm in terms of the localization error, we use Cambridge Landmarks (Kendall et al. 2016), which provides ground-trus poses. We experiment with four popular feature de- scriptors: SIFT Lowe 2004), SOLD2 (Pau- trat al. 2021). SuperPoint (DeTone et al. 2018) and the recent SiLK (Gleize et al. 2023). We plug them as the embeddings for inter- est points detected by SuperPoint. Fig. plots recall (the percentage of successfully localized queries) as function of the tolerated translation error.
Recall
ours (SIFT)
We observe that SuperPoint outperforms all other feature descriptors. SOLD2 (Pautrat et al. 202T) is substantially more inferior in terms of accuracy the pose estimates. Since de- signed to be line descriptor, its representation of non-linear structures is not well-defined, thus such outcome is somewhat expected. Interest- ingly, SuperPoint and SIFT both surpass the more recent SiLK on this benchmark. In com- parison to SuperPoint, SiLK has 128-D descrip- tor, which suggests that may be less expressive than the 256-D SuperPoint descriptors.
ours (SiLK)
ours (SuperPuint)
ours (SOLD2)
0
5 10-2
0.1
0.15
0.2
0.25
Translation error [m]
Figure 7: We study our approach with different feature descriptors on Cambridge Landmarks.
E IMPLEMENTATION DETAILS.
We implement our approach in PyTorch (Paszke et al. 2019)). The primary focus of our implementa- tion is the experimental validation of our derived results. Therefore, did not optimize the runtime performance, which heavily depends on the number of interest points and 3D points involved in the computation. On average, the code takes approximately seconds and 10 seconds per alignment on CMU and Aachen, respectively, the distribution p(-) as follows. We initialize p(-) with a truncated uniform distribution radius around all interest points. The radius decreases from 50% of the image diagonal to 5% in the first 30 iterations. Afterward, the scheduler switches to the normal distribution around each interest point with standard deviation Initially, we define such that 99% of the distribution covers 10% of the image around the point, and we decrease the coverage ratio to 1%.
single NVIDIA A4000. The scheduler in Algorithm (ntorithm[aaaapll
fixed
Since formulation does not have any outlier removal, we adopt the cut-off from the coarse tracker of DSO (Engel et al. (2166) The idea is to define threshold that discards but 20% the residuals with the lowest norm. In our experiments with SuperPoint, we set the threshold value 0.4.
F EVALUATION ON 7SCENES
Here, we complement our evaluation with the 7Scenes dataset. This dataset features substantial blur and distortions in some scenes. it was captured with rolling-shutter Kinect camera. Although SuperPoint features were not trained with such distortions, direct alignment with our closed-fore form solution remains competitive with PixLoc, both in terms of the median emrr and the recall Table
15

---

Published as a conference paper at ICLR 2024
Table 2: 7Scenes (Shotton et al., 2013) evaluation and comparison PixLoc. We report the median rotation and translation errors, well as recall values at the specified thresholds of translation and rotation.
Recall
Scene
Method
Median error
(1cm,19)
(5cm,5")
(25cm,29)
(50cm,59)
Ours
0.013m, 1.006*
24.80%
93.0%
86.30%
92.0%
Heads
Pixloc
0.013m, 0.863
36.40%
85.60 %
84.00%
85.90%
Ours
0.028m, 0.941
6.12 %
80.25%
92.55%
99.00%
Office
Pixloc
0.026m, 0.792
7.95%
80.70 %
93.12 %
96.85%
Ours
0.037m, 1.444
1.56 %
64.42%
71.88%
90.16% %
Redkitchen
Pixloc
0.034m, 1.217
3.74%
67.78 %
76.56 %
89.48%
Ours
0.049m, 1.555
1.60 %
51.65%
62.65%
85.35 %
Pumpkin
Pixloc
0.041m, 1.173
2.80%
59.75 %
71.00
84.40%
Ours
0.154m, 3.685
1.90 %
19.90 %
25.60 %
60.50 %
Stairs
Pixloc
0.048m, 1.268
2.60%
51.10
59.10
74.70 %
Ours
0.026m, 0.904
5.50 %
91.95%
95.45%
98.45 %
Chess
Pixloc
0.024m. 0.812
7.75%
90.75 %
94.95%
96.15%
Ours
0.021m, 0.978
10.40 %
90.65%
94.05%
98.45
Fire
Pixloc
0.019m. 0.781
15.85%
87.50 %
87.20%
90.10%
G QUALITATIVE EXAMPLES
Fig. 8 visualizes convergence examples of scene with different pose initializations. The examples
consistently demonstrate successful alignment despite suboptimal initialization of varying degree.
16

---

Published as a conference paper at ICLR 2024
Iteration 0
Iteration
Iteration 54
Iteration 89
Figure 8: Examples of convergence from random initial poses. The green points are projections of 3D points using the current pose estimate: the red points projections of the 3D points with the initial pose, and the yellow points denote the locations of interest points. It can be seen that despite these highly inaccurate initial poses, approach converges the correct solution.
17


Reviews:
[{"review_id": "UBLigz03qp", "reviewer": "Reviewer_hDb9", "summary": "This paper addresses the task of Direct Image Alignment, which is used to estimate the relative 6DoF pose between two images. The task is strongly affected by pose initialization, which has been addressed by prior art by switching to optimization methods that increase the convergence basin, such as the Gauss-Newton loss. The authors claim that these prior methods induce bias towards the training data which limits their generalization. \nThe papers main contribution addresses this problem. The authors introduce an analytical close from solution to the Gauss-Newton loss. This solution is independent of the feature representation and enables adjustment of the convergence basin based on the uncertainty in current estimates, giving control over the algorithm’s convergence properties. This property is used during the experimental evaluation, where optimization is first performed on a uniform distribution with a wider range, but then is switched out to a Gaussian with an increasingly narrowing distribution. \nTheir secondary contributions are insights that the analytical solution provides. Specifically, they show that under their simplified conditions, the Gauss-Newton step is determined by the neighboring points of interest. The author conclude that this is inherently limiting in comparison to other optimization methods. \nExperimental results demonstrate superior performance in almost all results over supervised state-of-the-art methods using self-supervised descriptors. \nThe appendix provides further insights on the derivations, as well as more interesting experimental results.", "questions": "Questions:\n- Eq. 23 leads to numerical instability. Is there a way to avoid this for stochastic optimization?\n- Fig. 4) The median is stable for all tested errors. For what ranges does this hold? I.e How far can the initial error be?\n- Fig. 4) Is there a similar plot for translation?\n\nComments:\n- Sec 2, Image Alignment: Add the variable T to make the text more consistent with the rest -> \"(...) estimate the relative 6DoF camera pose T.\"", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1) Well-written paper. It was a joy to read. It explains the context of the problem well, as well as establishing the necessary preliminary knowledge before delving into its actual contribution. There is a minor exception to this for the derivations (see Weaknesses)\n2) Under the simplifying assumption that eps follows an isotropic Gaussian, the authors derive a close-form solution to the minimizer of the Gauss-Netwon loss in expectation. Under the assumption that the authors claim about poor generalization due to training-data-biased feature maps holds (see Weaknesses), the proposed solutions has the main advantage that it provides unbiased feature map. In addition, it provides the ability to control the basin of convergence, which in turn makes the proposed method more robust to bad initialization (cf. Fig 3). Lastly, the assumed simplification which was necessary to derive the closed-form solution has been shown to lead to negligible differences (cf. Fig 5)\n3) Using self-supervised features, the proposed method is capable of outperforming supervised related work on almost all metrics. This is a strong statement, as the method can be used in conjunction with large and powerful foundation models, enabling bigger generalization due to the superior dataset sizes of such models. Therefore it is complimentary to these works.\n4) The authors provide an interesting insight when using Gauss-Newton as feature matching and indicate that it may be inherently limited. This is important for informing future work in optimization-based methods. Further analysis also shows that joint training of both losses for L_GN may lead to numerical instability and may shed light on reported training divergence of prior work.", "weaknesses": "1) My biggest gripe with the paper is that their claim that motivates the approach is not empirically validated and there is no mention of such validation elsewhere. The authors claim both in the abstract as well as in the appendix that prior art use feature maps that may embed the inductive bias of the training data. While I can comprehend the underlying reasoning, such a claim needs to be empirically shown. \nFor example, an experiment on out-of-distribution test sets demonstrating the superiority of the closed-form solution would back the authors claims and in turn strengthen the paper.\n2) On the same topic of bias, I argue that the authors should explicitly state that their method still exhibits bias, but that the source of this is the underlying feature representation (result of this can be seen in Tbl. 1, Aachen Night dataset). Otherwise it may read that the authors claim their method is not biased. This is stated at the end of Section 3, but I think it should be stated clearly in either Abstract, Introduction and Conclusion section. This is a minor point however and only serves to improve clarity.\n3) A little contradictory to my point in the Strengths section, I believe the math heavy section 4 and 5 could be made a little more clearer when derivations skip multiple steps. Otherwise the sections read as if one equation immediately follows from the other, which is not always the case. (e.g Eq 6. -> Eq. 7). This would enhance the readability of the paper.\n4) On the Aachen-Night dataset, the proposed method clearly suffers. The authors claim that this is due to the underlying feature representation used, which was not trained day-night correspondences. While I find the reasoning sound, it would help the authors claim to have used a feature representation that have used such correspondences during training. This in turn would again strengthen the papers contribution and indicate that it can work in different settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HTdd8YnT6G", "reviewer": "Reviewer_TZP4", "summary": "The authors propose a closed-form solution to the Gauss-Newton loss in the field of direct image alignment. This method allows for dynamic control of the convergence basin to improve the robustness of the alignment to pose initialization. Moreover, the proposed method shows the intrinsic limitations of employing Gauss-Newton loss in deep learning, which offers an insight between direct image alignment and feature matching. The simulation experiments have shown its superior performance.", "questions": "See the weakness part", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper provides an analytical solution to the Gauss-Newton loss, which is a novel technology for generating a dense feature map.\n2.\tThe paper shows the inherent limitations of feature learning with backpropagation via the Gauss-Netwon optimization.\n3.\tThe paper is well-organized and shows the explicit introduction to notion of the Gauss-Newton.", "weaknesses": "1.\tThe paper is required to give more comparisons with state-of-the-art in terms of accuracy of SE3\n2.\tCan the authors provide more training details of the proposed method, for example, the feature embedding network E, the learning rate, the batch size.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7TtbnTi743", "reviewer": "Reviewer_1Yhm", "summary": "This paper builds on the Gauss-Newton loss and establishes a closed-form solution for the expected optimum of this loss; it doesn't depend on the specific feature representation being used, and it enables the adjustment of the convergence basin based on assumptions about the uncertainty in the current estimates. This provides a means to effectively control the convergence properties of the algorithm. Notably, even when employing self-supervised feature embeddings, this approach attains impressive accuracy compared to the SOTA direct image alignment methods that are trained end-to-end with pose supervision. Furthermore, it demonstrates enhanced robustness in terms of pose initialization.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "To the best of my knowledge, the closed-form derivative of the Gauss-Newton loss is innovative, and its effectiveness has been confirmed through empirical evaluation within the domain of direct image alignment, specifically with self-supervised feature descriptors - SuperPoint. What's particularly noteworthy is that this derivative can be applied to other areas to encompass methods employing backpropagation through Gauss-Newton or Levenberg-Marquardt optimization, among others.", "weaknesses": "No major weakness. \n1. It would be interesting to see more discussions on the insight to the end-to-end learning framework's limitation, and a solution to that.\n2. It would be interesting to see this approach handles outliers inherently.\n3. It would be interesting to see this approach is applied to other areas.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)