Robust Model Reasoning and Fitting via Dual Sparsity Pursuit
Authors: Xingyu Jiang
Jiayi Ma
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Model reasoning; Model fitting; Outliers; Sparse subspace learning; Feature matching
TL;DR: Dual Sparsity Pursuit

Abstract: In this paper, we contribute to solving a threefold problem: outlier rejection, true model reasoning and parameter estimation with a unified optimization modeling. To this end, we first pose this task as a sparse subspace recovering problem, to search a maximum of independent bases under an over-embedded data space. Then we convert the objective into a continuous optimization paradigm that estimates sparse solutions for both bases and errors. Wherein a fast and robust solver is proposed to accurately estimate the sparse subspace parameters and error entries, which is implemented by a proximal approximation method under the alternating optimization framework with the ``optimal'' sub-gradient descent. Extensive experiments regarding known and unknown model fitting on synthetic and challenging real datasets have demonstrated the superiority of our method against the state-of-the-art. We also apply our method to multi-class multi-model fitting and loop closure detection, and achieve promising results both in accuracy and efficiency. Code is released at: https://github.com/StaRainJ/DSP.

Robust Model Reasoning and Fitting via Dual Sparsity Pursuit
Xingyu Jiang 1.2
Jiayi M
Huazhong University of Science and Technology. "Wuhan University (jiangxy998, jtt.mmm00000mmam....mmmm
Abstract
In this paper, contribute solving threefold problem: outlier rejection, true model reasoning and parameter estimation with unified optimization modeling. To this end, we first pose this task a sparse subspace recovering problem, search maximum of independent bases under an over-embedded data space. Then we convert the objective into continuous optimization paradigm that estimates sparse solutions for both bases and errors. Wherein a fast and robust solver is proposed to accurately estimate the sparse subspace parameters and error entries, which is implemented by a proximal approximation method under the alternating optimization framework with the "optimal" sub-gradee gradient descent. Extensive experi- ments regarding known and unknown model fitting on synthetic and challenging real datasets have demonstrated the superiority our method against the state- of-the-art. We also apply our method to multi-class multi-mode fitting and loop closure detection, and achieve promising results both in accuracy and efficiency. Code is released at: https: // /gi giittub.c.. cwwccoo///aa/ee S/DaRain//DSP
Introduction
Geometric model estimation is fundamental problem in computer vision, serving as the core of many high-level tasks including cruuttur-frrmmmottee [28, 39], SLAM [33, 9], and data align- ment [24, 27J. The models to specific data can be explained as linc/ellipse/cirele for 2D point sets, plane/cylindder/phere for 3D point sets, or fundament//hoooggghhyiifff correspondences extracted from two-view images, etc. Estimating the parameters of a predefined model from inliers only is well studied [21]. But real-world inputs unavoidably contain severe noises and outliers, posing great challenges for accurate model estimation. Besides, another key problem rarely considered how to recovery the model parameters without knowing the model type.
model for point
Recent researches mainly focus on proposing robust estimators to tackle the impact of noise and outliers, such as regarding it Consensus Maximization (CM) problem. This problem can be well solved with Sample Consensus (SAC) methods, such as RANSAC [19] and its variants[1] I, 38, 3, 22]. They commonly sample a smallest inlier set, to best fit given geometry model following a hypothesize-amp anddviiify strategy [28]. Inside this loop, the model is actually estimated by Direct Linear Transform (DLT) method with a Least-Square (LS) solution. SAC methods can provide probabilistic guarantee of hitting an all-inlier subset. However, this scheme succeeds only if given a predefined model type with sufficient time budget [52]. Another popular strategy is pose it in Global Optimization framework, which formulates the fitting task as subspace learning problem [45, 37, 13, 17, 18, 14J. Specifically, the representative method, Dual Principal Component Pursuit (DPCP) [45], tries to minimize an & co-sparse objective on the sphere. This is further applied to estimate specific models [12, 18], which typically embeds the model as a single hyperplane, or the intersection of multiple hyperplanes, and optimizes globally. This type of methods are admitted with
"Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023), New Orleans, USA.

---

efficient implementations and strong theoretical guarantees, thus arising great research interest in recent years. Deep Learning has also stimulated numerous methods for geometric model learning. which extract deep geometric cues from sparse points using multilayer perceptrons [49, 43, 50], convolutional [51] or graph networks [40]. They can fast output potential inliers once trained, but still require robust estimator such as RANSAC as postprocess for accurate model estimation.
The above mentioned methods can only work on fact that: one is certain about the true geometric models, then uses them to guide the formulation for parameter estimation. But in fact, we can always fit a model less constrained to obtain higher inlier count [31, 37]. Specifically, for homography data, estimating a fundamental matrix may return more consensus points, but many are outliers. While homography estimation can only find the largest plane structure in 3D scene, missing considraabe inliers for full motion data. Fitting an unknown model for heavily contaminated data is much challenging and in general NP hard. Existing methods solve it with model selection criteria [1, 42, 44], which first fit all possible models, then select the best one with a geometric verification, such as widely used GRIC metric [44, 32]. This is what is done in SfM or SLAM pipeline for fundamental matrix and homography identification [39]. However, such strategy is limited by the greedy selection strategy thus requiring huge computational burden. In addition, the used insufficient information would easily cause wrong selection of the true model for constrained motions of camera [37, 31, 32].
In this paper. we will simultaneously solve i) outlier rejection, ti) true model reasoning and iti) parameter estimation in unified optimization modeling. To this end, we start with introducing a common paradigm for exact model fitting, then derive our sparse subspace learning theory, which makes it possible to estimate the true model without knowing the model type, and robust to outliers. On this basis, we convert the objective into continuous optimization paradigm that estimates sparse solutions for both bases and outlier entries. Wherein a fast and robust solution is proposed, that is based on superiorities of projected sub-gradient method (PSGM) and alternating optimization strategy. Finally, the true model and the inliers are directly extracted from our solutions. Contributions: i) We are the first to propose a continuous optimization modeling for geometric model fitting with unknown model type and dominant outliers. ii) We propose sparse subspace recovery theory, which is a novel formulation for model reasoning and fitting. iii) We integrate the proximal approximation strategy and sub- au-grraiient descent method into the alternating optimization paradigm, which solves our dual sparsity problem with a convergence rate of (1/k²). iv) Extensive experiments on known/unknown model fitting and two visual applications are designed to validate the superiority of our method.
2 Methodology
This paper aims to explore a valid solution for geometric model reasoning and robust fitting. Before this, we first give a brief review of the widely-used solution for exact model estimation, which helps to derive our concerned unknown model fitting problem.
2.1 Geometry Preliminaries and New Insights
In the community of model fitting, the objective with geometric error is extremely hard to optimize due to the highly non-linear nature. In contrast, if the data are properly normalized, using linearized error (i.e., algebraic error) to construct objective would show great efficiency to find the optimal solution [18]. For algebraic objective, DLT known as an efficient method. To be specific, suppose we are given a set of feature correspondences S = {s; = (P:, ) where p: (w;, Us 1) and p¹p = (ut, v/. 1)T are column vectors denoting the homogeneous coordinates of feature points extracted from two-view images. Our goal is recover the underlying geometric structure including Fundamental matrix, Homography and Affine. This two-view problem is essential in 3D vision applications, and is the main focus of this paper.
Fundamental Matrix F E g3x3 describes the entire epipolar geometry P. Fp; = 0. is suggested to be represented by single equation living in the polynomial space
(p. pT) "Fc(F)= = 0,
(I)
where
(p TT) (uluu, ulur, ,e u/, ouu,, d/u2, 1/. Ui 02 1).
(2)
the embedding of correspondence (p., p¹) under epipolar constraint, and vee E R the vector form of matrix F in row-major order. N 2 8 correspondences can uniquely determine F up to scale.

---

Homography H E R3* describes the pure rotation or plane projection, which claims that P and Hp, are co-linear, implying xp']x O, where [p'Ix is skew-symmetric matrix. In DLT solution, it usually converts to the following constraint:
(Pi, p() (ec(H)) 0,
(3)
where vee(H) E R2, similarly, And PH (Ps, p/) denotes the homography embedding:
(p p!)
1
0 -uu. -u -
1 - -
-1/u -u
(4)
This constraint suggests that, H can be estimated from given at least 4 correspondences.
Affine Matrix A R3x3 is the degraded case of H, with the last row of A being [0, 0, 1]. This model implies a linear transformation P = Ap:. The affine constraint is also represented as: A (P.. P\) vvc(A) 0,
(5)
where vee(A)= = [a11, 012, 413, 021, 022: 023. 1]T and
4 (P.. P()
Us
0
0
0
(6)
84
the affine embedding. Using Eq. (5), the solution can be given by at least 3 correspondences.
Given sufficient inliers, DLT method provides an efficient solution for geometric model fitting, which converts to solving the following minimal least- problem:
min M 0||
s.t. ||0||2 1,
where M is the data embedding matrix for specific model M i.e., m, = DM Pt) with M (F. H, A), as defined in Eqs. (2), (4) and (6). 0 rec(M) E TR. indicates the parameter vector restricts
of M, and ||0]]2
sphere thus avoiding the trivial solution. The optima solution 0 is exactly the right singular vector corresponding to the smallest singular value of M. However, DLT only works on the outlier-free case, due to the use of le norm. Thus, it is usually applied in SAC paradigm by sampling clean subset to fit a given model. Another practical formulation is using latent variable inside (7) to indicate the inlier, or using truncated or & loss [18]. In contrast, the l objective is easier optimize due to the convex nature by solving
min M 0l11. s.t. ||0||2 = 1.
(8)
This is actually a subspace learning problem as introduced in DPCP series [45, 52, 13], which can be efficiently solved by a projected sub- gradient method (PSGM) [13]. Besides, [18] gives comprehensive comparisons of above mentioned losses under different model cases. These works also demonstrate the robust property of (8), which roughly states that, the estimation task can even tolerate (mm²) outliers (m is the inlier number).
New Insights. Above formulations almost work on knowing the exact information of model type, such that the embedding matrix M is correct, i.e, choosing M from {F, H. Aj to generate & &M-- Practically, predefining a correct model is much difficult. First, the camera type and motion, or the scene in shooting is not always known. Besides, we cannot manually define the correct model for each image pair in large-scale vision tasks. In this regard, this paper tends to simultancously seek the true model and robustly estimate the model parameters, by proposing general continuous optimization framework with dual sparsity constraints.
2.2 Dual Sparsity Formulation
In this paper, we aim to estimate the model parameters without knowing the model type. That means we should find common data embedding ((()) thus avoiding exact & (-) for each model. Instead it converts to constructing model embedding v (M), to generate a common constraint as:
@ T O, ME (F,
Eq. (9) can provide great advantages for our model reasoning and fitting task. First, it avoids the requirement for exact model type to construct (-x(-) before estimation. Moreover, the solution would have the form of (MM, which directly guides the identification of different models. Eq. (9) essentially relies on the following Sparse Subspace Recovery (SSR) theory.
3

---

m. (x, 1,2))
Intiers Embeded inliers
1:y 1, x²
Outliers
Embeded outliers
y==1
m, (x,, 1)
Unit sphere projected points
heddinng
Embedding
T::
0
P: (x))
P:
(x))
(a)
Point model: x-1 =0
(b) 2D embedd. & solution (a)
(c) 3D embedd. & sparse solution of (a)
Figure I: Illustration of fitting a point model (æ = t) (a) original points, (b) embedding under DLT
solution, (c) embedding under our SSR theory. x' or -x" is the optimal solution.
Theorem (SSR) Geometric model fitting can be seen subspace recovery problem represented by the intersection of multiple sparse hyperplanes under an over embedded data space.
To better interpret the proposed Theorem 1, we first give example on 2D point set fitting. Given
a set of clean 2D points S = {s; = (Iz. M:) that are sampled from a structure of line (L), parabola (P) ellipse (E), i.e., .M {L, P. EJ. The model parameters
be estimated by exact data embedding (M. Us) with DLT method (7). For ellipse estimation, & Ui)
[1, Te, Vs, x², v/]T,, while for a line, 1r (1,, vs)
T4, [1] But in our sparse theory, with a higher-order embedding, such as fixing ((x;, Wi) = ΦF y:), we can obtain a sparse solution la, b, c, 0, O] for a line model, and = [a, b, c. d, 0J* or (a, b, c, O, e|" for a parabola model. See Fig. 2, refer Append. and Fig. 6 for details. Thus, the SSR task can be formulated as:
min
|x
s.t. M x =x 0,
||x|| = 1,
(10)
where M is the over embedding for input data with m = (xx v1, vi) = 4 (Zi, yi), and we denote x sparse basis that indicates hyperplane under SSR theory. The E, norm in the objective suggests to use less parameters to fit the given data. Note that, m, can be composed by more polynomials such as ways, uq. y or higher-order forms, but it is not necessary for the model pool P,, E). Problem (10) can be illustrated by a point model fitting x = / as in Fig. 1. The middle plot shows that, the solution x* or -x* is ideally vertical the embedding vector m, of inliers. With higher-order embedding [zv, 1, x TTT the solution would be any vector located in the plane P that is vertical to m, and through the origin. However, with sparsity constraint, the solution reduces to be the intersection of plane P and one of coordinate planes { (Tzoy: Tzoz, you ). Since x; causes higher complexity, the final solution is exactly P Tzoy: i.e., x" +1 [1, -f, O]
VI++
Denoting GM (D, r, as the geometry relationship for model M, where D is the ambient dimension of embedded data, d is the subspace dimension, r denotes the number of basis to be esti- mated, while means the minimum sparsity in each basis (i.e., the least number of zero parameters), the relations for 2D models are G, = (5. 4, 1, Gp = (5, 4, 1), and G. = (5, 4, 1, 0). Next, we emphatically introduce the new formulation for two-view models, which is outlined in Append.D.
Extending to two-view geometry, our SSR task can directly benefit from (8), which a special case in subspace recovery [46, 52, 14], with d D 1. In subspace learning theory. the learning problem is significantly easier when the relative dimension d/D is small [17, 18]. For such purpose, and together with our sparse theory, can obtain the following new formulations. First, for F estimation, since its geometric constraint derives only one single hyperplane living a 9-dimensional space, thus Eq. (9) has the same form with Eq. (1). And the geometric relation is clearly GF (9,8, 1, 0).
Proposition 1 Given n 2 4 clean correspondences conforming to a homography transformation H. the model estimation can be converted to recovering subspace with dimension more than 6, which consists of 3 sparse hyperplanes with sparsity s > 3 under the 9-D embedding T- F p().
Proof. Homography model actually derives r = 3 equations, and each of them indicates a sparse hyperplane with (p, p²) (H) = O, where (H)) E 1xx3 is the embedding of model H, which consists of three sparse intersected hyperplanes (i.e., orthometric bases) as:
01x3, h - h2
V (H) =
- h 0 13x h
(1I)
h h 01x3

---

with - hj : 1. Since (H) is of rank 3, the dimension of subspace d I 6. And the solution for each basis would have least s 3 zeros, thus the geometric relation is Gh (,,6,3, 6,3,3) 3). O
Proposition 2 Given n 2 3 clean correspondences conforming to an affine transformation A, the model estimation can be converted recovering a subspace with dimension no more than which consists 2 sparse hyperplanes with sparsity $ 2 5 under the 9-D embedding T F P\).
Proof. Eq. (5) derives r = 2 linear equations, and each of them indicates sparse hyperplane with ((p,, (() (A) O, where (A) E R is the embedding of model A, which consists of two sparse intersected hyperplanes as:
0. 0, 411 0, 0, 012, -1, 0, A13
v (A) A/xx2
(12)
0, 0, 221 0, 0, 022. 0, 1, A23
Because & (A) of rank 2, the dimension of subspace no more than And the solution for each basis would have at least s = 5 zeros, thus the geometric relation is G = (9, 7,2,
The above two propositions have well interpreted Theorem 1. To be specific, H and A estimation
can be performed as a hyperplane fitring problem with Eq. (3) and Eq. (5). Alternatively, they can be respectively solved recovering 6-dimensional subspace and 7-dimensional subspace in common 9-dimensional data embedding space. For the purpose of our unknown model fitting, the advantages of this conversion have been explained in Eq. (9). Besides, the low-relative dimension d/D significantly makes the subspace learning task easier [17, 18J. In the following, we will give a general formulation for unknown model fitting, then explore its efficient solution.
Formulation for Unknown Model Fitting, With no outliers and noise, unknown model fitting can convert to finding all independent sparse bases X [x,,X,, ,x,] E R Px* by solving
min XER
||X lo:
S.I.
M x 0, rank(X)= =
(13)
where M is the over embedding matrix of data with m (s:)). Rank constraint of X asks for independent bases. But r is generally unknown in advance, thus its estimation is also necessary.
Given sufficient inliers with no noise, the conformed model derives r independent bases, then the solution of SSR satisfies max(rank((X)) =.
Proposition 3
Proof.
Denoting X = X2,, ... x E Rxx the optimal solution, any subset of X* is also
a feasible solution with rank no more than r. For V X,441 with ||x ||| = 1, (x*) Xr+1 = 0, we have Mxx+11
x,+1 # O. Hence, x is not a valid basis, and we can conclude max(rank( (X) = r. D
Proposition 3 suggests that the target of unknown model fitting can be converted to finding maximum number of orthometric bases with sparsity constraint. Considering the outliers and noise, the ideal constraints can be written as M X G E O, where G and E are noise and outlier entries, respectively. Following common perception, G assumed to be Gaussian, which suggests to be constricted by Frobenius norm G F. While outlier matrix E is sparse in row, and any |E,,.la + 0 indicates that the i-th sample can be considered as an outlier, thus we use f to constrain this property. Then our problem is transformed into a multi-objective optimization task, including minimizing the fitting error, the parameter number, the outlier number, and maximizing the basis number. Hence, the general formulation for robust model reasoning and fitting be written unified form:
min X,E, 2
G 1 + X X|o 1o0 + Y E| 2.0 Trank(X),
(14)
S.L M - G E = 0, x
where is an identity matrix of size r x r with r = rank It restricts that the bases are all orthometric. 7, 0 are hyper-parameters
balance different loss items. Problem (14) ultimately outputs an exact sparse subspace X E Dxr, helping to identify the true model M with X (MM) In the following, we will explore an efficient solution for this multi-objective problem.
2.3 Solution
Problem (14) involves Eo minimization, which is the holy grail of sparse approximation. However, to optimization is NP-hard, and the non-linear objective makes it even more difficult. A practical way is
5

---

to use ( norm as the best convex approximation instead. Thus, Problem (14) can be relaxed into:
nin X.E. 2
x - E + X X|X|| + 7 2,1 Trank(X); M
(15)
S.L. |x =1, x, 0, A Vi = 1,,,.... i i
Another troublesome issue is to maximize rank(X). Consider that Problem (15) actually aims to obtain r orthometric bases X1, X2, ... xx] and a sparse outlier matrix E = (ē1, 62 ... 6r] satisfying C(M, X,, é,) = 2 |M & - e - + & 11 + e/ < T, Vi E {1, 2, ... .r.. Thus we decompose (15) into progressively estimating new sparse bases orthometric all given bases up to [(M, x; o₂) < r not holds. For any given bases {B = (x,) )i = 1,2, .......................... - 1] (those have been estimated), new sparse basis (i 1) can be estimated by solving
1 M x e12 + A/x + >//ell1 s.t. ||||| = 1, x Vy E B. (16)
min ((M. x,e
Since there exist two variables (x, e) be estimated, is impractical to optimize them directly with gradient descent method as used in [52, 13, 17J. A valid way is to alternatively optimize x and e in an iterative pipeline, which encourages us to compute the closed-form solution for (x, e) in each iteration. For such purpose, we first calculate the derivatives of objective w.r.f. variables, then make them equal to zero. In the k-th iteration and for given x* k-1, we can easily obtain
et ==
M x ysgn(e") T, x*-1),
(17)
where sgn(-) is sign function. ca is in fact solved by standard threshold shrinkage operation T introduced in the Lemma of [47]. (Refer to Append. B for details)
Given e² Update x Due to the coexistence of quadratic and sparse constraints, it is hard to directly optimize. Fortunately, [7] provides efficient solution for such linear inverse problem with theoretical and practical verification. We exploit its basic idea into our dual sparsity problem, that is to build at each iteration regularization of the linearized differentiable function, then obtain
L (x* 1
1 (VF((x=1))1 + 11
x arg min
(18)
where the smallest Lipschitz constraint of the gradient fx(x) can be calculated by L(f) Amax (MM ), and )max() (-) indicates the maximum eigenvalue of matrix, and we denote
qi(x*
x
(19)
L
where fx(x*-1) M(M
T x )² In this case, Problem (18) is essentially similar to solving e, and we can obtain the update formula of with x T (qz (x)x))) (Proof is in Append. C)
'))
The above solution for x actually reduces to subgradient method [26]. which converges at a rate no worse than (1/k). However, was proven in [34, 35J that there exists a gradient method with an (11/k²) complexity which is an 'optimal' first order method for smooth problems [7]. The core strategy is that the iterative shrinkage operation performs at the point & which uses specific
linear combination of previous two points (xx xx= as x² k-1 = -x+114 + +1++(("*~0 (4--1- (xx (x--1-x
x
with & = (1 + V + 4t 1 Then updating x with convergence rate O(1 (11/22) is (See Fig. 2)
x* TA (x
(20)
Next, we further consider constraints of Problem (16). It can be easily solved by a projection operation [17, 14] in each iteration, including a sphere projection <- x ___ and orthogonal
projection x <- (I - BBT )x, because the projector of orthogonal complement is I BB
Eqs. (17) and (20) provide iterative optimization for searching sparse solution of and e, thus we term the whole algorithm as Dual Sparsity Pursuit (DSP). Repeatedly using DSP, we can successfully address the general unknown model fitting task, and the whole process concluded in Append. D. Note that, in tio- problem, each model has at least = basis (e.g. F) and at most r = 3 bases (e.g. H), thus we denote R as the max number of bases be estimated, with R 3.

---

Model Reasoning Analysis. A critical problem how to reason out the model type and estimate the parameters under a general case. Several meth- ods [44, 41] have been early proposed to use selec- tion strategy [33, 31, 32], but it is not always clear due their ambiguities [37]. This would be well mit- igated in our DSP. To be specific, for 2D point set fitting, we can directly distinguish the correct model from { [L, P. }) through the sparsity of estimated as shown in Fig. 2. As for two-view models, we can also distinguish them via the sparsity and rank of estimated X, referring to the model embedding (MM or their geometric relations G GM((D, (D, d, r.r s). The following remark may deliver good understanding of this property, which is illustrated in Fig. 4.
013R]
Poramoter vallee
2.3
Accelentioon: 00A
600
400
Iteration
Ieeraion
Figure 2: The optimization procedure of our DSP for 2D line fitting. X = [a, b, c, d, e] and x' = [a', ,, c, d',e'T denote the solu- tions w/o and with acceleration, respectively.
Remark If the true model is F, the solution for (x2, would result in a larger fitting error than (x1, e1), i.e., ||M xille & |M x2ll2. In other words, the number of inliers detected by (x2, e2) would be much smaller than (x1, 01). )- Since the second basis is essentially find a homography or affine structure. On the contrary. if the true model is H or A, the detected inliers between these two solutions would have small difference, as the orthometric and sparse bases indeed exist.
3 Experiment
Implementation Details: (1) Parameter setting. In DSP, À and 7 are two hyper-parameters. Based on [44], we set À = 0.005 log( (N) x as default (for 2D model, we set À = log(2NN) x 0.1,10.1,0.1,,,, 1, ). In addition, we set 7 0.06 at the beginning, then update with 0.98y for each twenty iterations, and constrain Ymin = 0.02. Moreover, we set the max iteration as 2k, and stop it e = ||x - x I 6.-6. As for T, it controls the number of estimated basis, i.e., r. We set [(M, x; e2) /2)//M,,xx, x; c;-1), and describe its distribution on all real data as in Fig. 3. Based on the best E. we set = 1. 1.2C(M. -1 e during the estimation of x, (2) Synthesized Data. We synthesize 300 image pairs for each model, which consist of different outlier ratio (OR = 20%, 50%, 80%) and noise level (NI. = 0.2, 0.5, 0.8, 1.0, 1.5 in pixel). (3) Real Image Data. public datasets [8, 2] are used, and we divide them into two groups including Fund: kusvod2, CPC, TUM, KITTI. T&T; and Homo: homogr, EVD, Hpatch. (4) Comparisons. 6 methods are used for evaluation, including three SAC methods RNASAC [19], USAC [38], and MAGSAC+ [6]; one globally optimized method EAS [18]; and two deep methods OANet [50], SuperGlue [40]. (5) Metries. All methods applied the normalized 4- 4-point and 8-point algorithms for H and F estimation (which are applied as post-process [18] in EAS, DSP and two deep methods with 100 iterations). To
Horoograpty
[1,1, 0.5,
Opttnal ç-
0.05
Figure 3: Distribution of
& on H data and F data.
measure the accuracy, we use Geometrical Error (GE, in pixel), which is defined as the re- re-proeection error and Sampson distance [4, 5] for H and F estimation, respectively. We defined the failed case as that the GE is larger than 5 pixels, or the model is wrongly identified. To avoid the failed cases for a dataset, we use the medium value Fmmd of GE measurement. (More details are in Append. E)).
Model Reasoning and Robust Fitting Test:
Table I: Model reasoning results on syn. data w.r.r different OR Each value x indi- cates that ø% models are correctly selected.
(1) Test on Synthetic Data. In the simulation test, an advanced robust estimator EAS [18] is used to esti- mate all possible models first. Then, three widely used model selection criteria, i.e., AIC [1], BIC [42] and GRIC [25], are equipped. The accuracy (%) of model selection are reported in Tab. 1. AIC can achieve good performance for homography and affine models, but not for epipolar cases, particularly for high OR. BIC shows poor performance in fundamental model selec- tion, since it prefers under-fitting in theory. GRIC fully considers the properties of vision problem performing well in most cases. As for our DSP, it can identify the geometric models with the best accuracy even in case of high outlier ratio.
OR
Data Model
AIC
BIC
GRIC
DSP (ours)
F:100 A,100
100
100
20%
100
100
96
100
100
R 100
100
0
100
100
50%
H,100
100
95
100
100
A,100
99
97
98
100
F:100
100
80%
99
A.100
98
all
97.2
70.7
98.4
99.7
7

---

Table 2: Results of model reasoning and fitting on real image pairs. All datasets are divided into Fund.
and Homo. based on the model type. Method with means first estimating all possible models, then using GRIC to select the "best" one. Bold and underline indicate the best and second, respectively.
Methoss
RANSAC
USAC*
MAGSAC+
EAS*
OANer*
SuperGlue
DSP (ours)
2_med
0.8478
0.6545
0.6260
0.6900
0.7579
0.8926
0.6017
Fund.
Time(s)
0.5666
0.0239
0.3918
0.2037
0.0152
0.0721
0.0551
FR
0.2925
0.1992
0.2077
0.2142
0.2632
0.2709
0.1136
E med
1.0428
0.8744
0.8978
0.8472
0.8480
0.9392
0.8227
Homo
Time(s)
2.010
0.0550
1.3419
0.4463
0.0342
0.0716
0.2794
FR 0.3021
0.2604
0.1424
0.0972
0.0903
0.1181
0.066
E. med
0.8794
0.6711
0.6383
0.7091
0.7670
0,9026
0.6249
All
Time(s) 0.6638
0.0260
0,4558
0.22
0.0130
0.0721
0.0703
FR 0.2932
0.2043
0.2033
0.2064
0,2515
0,2606
0.1123
(2) Test on Real Image Data. In this part, those datasets are divided into Fund and Homo based on their model type. In fact, during real applica- tions, it is more urgent make sure whether the true model is or H [41, 33], thus our used real image data are sufficient verify the performance for real cases. addition, con- sidering the competitive performance in Tab. I, we only use GRIC as comparison in model se- lection stage. Six representative estimators are used for comparison. That is, first using those robust estimators to obtain all possible models, then using GRIC to select the best one. Quali- tative results on two representative image pairs are shown in Fig. 4, which obviously reveal the mechanism of our DSP as we assumed in Remark 1. In addition, our algebra results are somewhat coarse, but they can be well refined by a post- pess-processing [18]. Quantitative results are evaluated in Tab. including Emed, Run Time (Time) and failure rate (FR). From this ta- ble, we find that the other estimators typically achieve poor model selection even if using the best criterion GRIC. Three SAC methods get poor model identification, particularly for Homo data, since many image pairs are from extreme view change with high outlier ratios. OANet and SuperGlue similarly achieve low model reasoning accuracy, since some datasets are from complicated scenarios, such as EVD, T&T and CPC, which are not seen by these two deep methods in their training. Their model fitting performances directly affect the model reasoning accuracy. Overall speaking and comparing with GRIC, our DSP reasons out the true model directly from its solutions with much higher accuracy. As for rumning time. OANet and USAC almost achieve the best efficiency. Because OANet is a deep model with GPU acceleration (SuperGlue is slower since it takes descriptors as input to generate matches), and USAC integrates the local optimization and fast model verification strategy in its universal framework, but sacrifices the accuracy to some extent.
Figure 4: Detected inliers of DSP for real data F(left) and H(right). Top to bottom: input data, the first (x1, e1), second basis (X2; e2), and refined results. For visibility, at most 200 random selected matches are shown (blue true positive, green false negative, red = false positive).
Known model estimation is the main focus of current researches, thus we also test on those real image pairs by giving the priori of model type, and provide extensive comparisons. The statistic results are shown in Fig. 5, which only contains four challenging datasets, while others are in Append. E.3. Statistic results show that the performance of SAC-based methods would heavily degrade with more time consuming and less accuracy on these challenging datasets. This is because they intrinsically have to sample an outlier-free free subset to best fit the given model, this is much difficult for these datasets. OANet and SuperGlue achieve poor fitting accuracy due to the same reason particularly for EVD datasets. As for EAS and our DSP, they commonly optimize from global formulation, thus showing similar top accuracy. However, our DSP performs better comparing with EAS, due to the specifical modeling to restrain noise and outliers. And it is much faster, because the use of an acceleration strategy that makes the convergence rate closed to (11k:²).
Known Model Estimation Test:
8

---

Gramgit Emorr (in plesi)
GromANiC Emnor (In p0000
Gaomaite Erree On phm))
USAC, 134
UBAS 0.85
USAG, 631
UBAG, G3T
BAGGSAC-,,
MAGGAC
EAS, 1:55
SAS 0.83
E&S,
OAhee
OANia, 0.53
0.55 Dura,
OANI 0.22
Cors, 1.03
Oarr, 0:00
Ounss 0.75
Ed1
0.6
EVD Cumiiative Datribulion
Hpatihh Cumulaatin Distribution
TST: Cumulative Distribution
CPC: Carrurative Distribution
RAMSAC 100e003
RANSAC T.
RAMEAC 3.56-00
RANEAC 1.
Run Time (m))
M6G0BD-. & 1Deuco
Run Teme (mma)
MADG & Meuul
MMAG6CC- a. 2.00e400
EAS, 1.bee022
2.00e002
EAS 1.00e01
DAhill (Staie
O&Ne, 1.38x-01
Supercliv Aollooo
Supertilue, Dloute
* Rupertian, 3.40a007
22
10
18
03 EVD:
30 Camulative Dstribution
0.4
0.6
0.2 Ppalich:
0.4
0.8
0.8
0.2 TAT:
04
05
0.8
6.2 04 0.6 0.8
Cumulative buson
CPC: Comulativc Distribution
Figure 5: Qualitative statistic results of Geometric Error (top) and Run Time (bottom) for known model estimation. A point on the curve with coordinate (z, y) denotes that there are 100 x z percent of image pairs which have GE or RT no more than y. The lower the better.
Applications: (1) Multi-model Fitting. We first consider the multi-class multi- model fitting problem. Unlike multi- model fitting of single type, it would be more challenging if the data contain mul- tiple models with unknown model type and model number. To this end, a pub- lic benchmark AdelaideRFM is used for evaluation. Quantitative results are re- ported in Tab. 3, where Misclassification Error (ME, %) selected to characterize the performance.
Table 3: Results of Multi-model fitting. The mean and median values of ME (%), and Running Time (RI, s) are listed. Bold and underline indicate the best and second.
Tlink RCMSA RPA
MCT MLink REM-SCAN Ours
ME ave 27.65
10.05
5.28
11.36
6.04
2.63
1.64
ME med 27.88
6.087
4.35
1,21
4.22
1.20
RT(s)
1.95
2.16
10.84
6.44
9.72
0.01
0.18
Six algorithms including T-link [29], RCMSA [36], RPA [30], MCT [31], MLink [32] and RFM- SCAN [23], are used for comparison. The table shows that our DSP can achieve the best accuracy with low computational burden.
(2) Loop Closure Detection. We next exploit 2 sequences from KITTI vision suite [20] (ie. KOO, K02) evaluate method in the loop closure detection (LCD) task, i.e., recognizing re-observations during the navigation of a robot. These two sequences have enough loop closures. Because DELG [10] is considered state-of-the-art for LCD-related tasks such as image retrieval and landmark recognition, we choose it to simultane- ously extract global and local features of images to perform LCD in a hierarchical way. Specifically, given query image, the similarity of global features under L2-metrie is first used to select its candidate frame, followed by performing model fitting between them based on local features. Only when sufficient matches are preserved would loop-closing event be triggered. Results are reported in Tab. 4, which shows that our DSP achieves the best, since our dual sparsity constraints can well address navigation of the degraded cases such only going ahead or making turn.
Table 4: Results of LCD. Metrie: the recall at 100% precision. Bold indicates the best.
RSAC USAC MSAC++ EAS OANet Ours
KOO 0.9112 0.9118 0.9105 0.9086 0.7910 0.9162 K02 0.7632 0.7796 0.7757
0.7664 0.6441 0.7882
4 Conclusion
In this paper, we proposed a unified modeling and fast optimization paradigm named DSP for robust model fitting with unknown model type and heavy outliers. enables to use as few parameters as possible to explain the inlier structure, thus achieving true model identification and parameter estimation. We introduced an alternating optimization strategy together with proximal approximation method to accurately estimate the sparse model parameters and outlier entries. In addition, our proposed method validated to exactly disambiguate the true model on both synthetic and real data. Limitation: Our proposed method can only perform for single model, but real data often contains multiple different models, such as point cloud analysis. In the future, we will integrate the multi-model fitting problem into our modeling, thus achieving four-fold task in a unified optimization.

---

Broader Impact
The proposed method enjoys great potential to improve a wide range of industrial applications including image registration and retrieval, camera pose estimation and localisation, 3D reconstruction, etc. To be specific, the proposed method achieves significant accuracy gains of model reasoning and fitting on challenging matching benchmarks, which strongly indicates that it will directly benefit many related fields including robotics and autonomous driving, where the geometric model estimation is the foundation. However, our method does have some unwelcome repercussions like many other machine learning techniques. The accurate and robust model reasoning and estimation can be illegally used for a person property without permission. It may even be weaponized to guide UAV to perform self-location then carry out a terrorism attack. But these negative impacts are more related to the fields of application rather than the technology itself. And we believe, under strict supervision, that our work will bring more benefits than harms to society.
Acknowledgments and Disclosure of Funding
Funding in direct support of this work: NSFC grant 62276192.
References
[1] Hirotogu Akaike. Information theory and extension of the maximum likelihood principle. In Selected Papers of Hirotugu Akaike, pages 199 213. 1998.
[2] Vassileios Balntas, Karel Lenc, Andrea Vedaldi, and Krystian Mikolajczyk. Hpatches: A benchmark and evaluation of handcrafted and learned local descriptors. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11725582,, 2017.
[3] Daniel Barath and Jifí Matas. Graph-cut ransac. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6733-6741, 2018.
Daniel Barath, Jiri Matas, and Jana Noskova. Magsac: marginalizing sample consensus. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pages 10197- 2019.
[5] Daniel Barath, Jana Noskova. Maksym Ivashechkin, and Jiri Matas. Magsac++, fast, reliable and accurate robust estimator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1314-1312, 2020.
[6] Daniel Barath, Jana Noskova, and Jiri Matas. Marginalizing sample consensus. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11):8420-8432, 2022.
[7] Amir Beck and Mare Teboulle. fast iterative shrinkagg-thrresholding algorithm for linear inverse problems. SIAM Journal on Imaging Sciences, 2(1):183 202, 2009.
[8] Jia- Wang Bian. Yu-Huan Wu, Ji Zhao, Yun Liu, Le Zhang, Ming- Miin Cheng, and lan Reid. An evaluation of feature matchers for fundamental matrix estimation. Proceedings of the British Machine Vision Conference, 2019.
[9] Carlos Campos, Richard Elvira, Juan J Gómez Rodríguez, José MM Montiel, and Juan Tardós. Orb- slam3: An accurate open-source library for visual, visual-inertial, and multimap slam. IEEE Transactions on Robotics, 37(6):187--1890 2021.
[10] Bingyi Cao, Andre Araujo, and Jack Sim. Unifying deep local global features for image search. In Proceedings of the European Conference on Computer Vision, pages 726 (66,743,77 2020.
[11] Ondrej Chum, Tomas Werner, and Jiri Matas. Two-view geometry estimation unaffected by a dominant plane. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 772-779, 2005.
[12] Tianjiao Ding, Yunchen Yang, Zhihui Zhu, Daniel Robinson, René Vidal, Laurent Kneip, and Manolis Tsakiris. Robust homography estimation via dual principal component pursuit. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6080-6089, 2020.
[13] Tianyu Ding, Zhihui Zhu, Tianjiao Ding, Yunchen Yang, Rene Vidal, Manolis Tsakiris, Daniel P Robinson, et al. Noisy dual principal component pursuit. In Proceedings of the International Conference on Machine Learning, page 1617-1625, 2019.
10

---

[14] Tianyu Ding. Zhihui Zhu, René Vidal, and Daniel P Robinson. Dual principal component pursuit for robust subspace learning: Theory and algorithms for holistic approach. In Proceedings the International Conference on Machine Learning, pages 2739-2748, 2021.
[15] Ehsan Flhamifar and René Vidal. Sparse subspace clustering: Algorithm, theory, and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(1 (5(11))7765-7781 2013.
[16] Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. Density- -bated spatial clustering of applications with noise. In Proceedings of the International Conference on Knowledge Discovery and Data Mining, volume 240, page 1996.
[17] Aoxiang Fan, Xingyu Jiang, Yang Wang, Junjun Jiang, and Jiayi Ma. Geometric estimation via robust subspace recovery. In Proceedings of the European Conference on Computer Vision, pages 462-478, 2020.
[18] Aoxiang Fan, Jiayi Ma, Xingyu Jiang, and Haibin Ling. Efficient deterministic search with robust loss functions for geometrie model fitting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11):82
2022.
[19] Martin A Fischler and Robert C Bolles. Random sample consensus: paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6):381-395, 1981.
[20] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(1122121111 2013.
[21] Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge University Press, 2003.
[22] Maksym Ivashechkin, Daniel Barath, and Jifí Matas. Vsac: Efficient and accurate estimator for h and f. In Proceedings of the IEEE International Conference on Cumputer Vision, pages 15243-15252, 2021.
[23] Xingyu Jiang, Jiayi Ma, Junjun Jiang, and Xiaojie Guo. Robust feature matching using spatial clustering with heavy outliers. IEEF Transactions on Image Processing, 29:736-746, 2020.
[24] Xingyu Jiang, Jiayi Ma. Guobao Xiao, Zhenfeng Shao, and Xiaojie Guo. A review of multimodal image matching: Methods and applications. Information Fusion, 73:22-7 71, 2021.
[25] Ken-ichi Kanatani. Motion segmentation by subspace separation and model selection. In Proceedings of the IEEE International Conference on Computer Vision, pages 586-591, 2001.
[26] Evgeny S Levitin and Boris T Polyak. Constrained minimization methods. USSR Computational Mathe- matics and Mathematical Physics, 6(5):1500, 1966.
[27] Yifan Lu, Jiayi Ma, Leyuan Fang, Xin Tian, and Junjun Jiang. Robust and scalable gaussian process regression and its applications. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21950-21921 21959, 2023.
[28] Jiayi Ma, Xingyu Jiang, Aoxiang Fan, Junjun Jiang, and Junchi Yan. Image matching from handcrafted to deep features: A survey, International Journal of Computer Vision, 129(1):23-79 79, 2021.
[29] Luca Magri and Andrea Fusiello. T-linkage: A contiquous relaxation of jiiikagee for multi-mode fitting. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pages 3954-3961, 2014.
[30] Luca Magri and Andrea Fusiello. Robust multiple model fitting with preference analysis and low-rank approximation. In Proceedings of the British Machine Vision Conference, 2015.
[31] Luca Magri and Andrea Fusiello. Fitting multiple heterogeneous models by multi-class cascaded t-linkage. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7460 7468, 2019.
[32] Luca Magri, Filippo Leveni, and Giacomo Boracchi. agglomerative clustering and model selection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1853-1862, 2021.
Multilink: Multi-class structure recovery via
[33] Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. Orb-slam: versatile and accurate monocular slam system. IEEE Transactions on Robotics, 31(5):1147-1166 2015.
[34] Arkadij Semenović Nemirovskij and David Borisovich Yudin. Problem complexity and method efficiency in optimization. Wiley- Intersinteeceeecce Series in Discrete Mathematics, John Wiley & Sous, New York, 1983.

---

[35] Yurii Evgen' evich Nesterov. A method of solving a convex programming problem with convergence rate 2(k"2). In Doklady Akademi Nauk, volume 269, pages 543-547, 1983.
[36] Trung T Pham, Tat-Jun Chin, Jin Yu, and David Suter. The random cluster model for robust geometric fitting, IEEE Transactions Pattern Analysis and Machine Intelligence, 1668):1658-11) 2014.
[37] Thomas Probst, Ajad Chhatkuli, Danda Pani Paudel, and Luc Van Gool. What correspondences reveal about unknown camera and motion models? In Proceedings the IEEE Conference Computer Vision and Pattern Recognition, pages 8730- 8738, 2019.
[38] Rahul Raguram. Ondrej Chum, Mare Pollefeys, Jiri Matas, and Jan-Michael Frahm. Usac: universal framework random sample consensus. IEEE Transactions on Puttern Analysis Muchine Intelligence. 35(8):2022-20) 2012.
[39] Muhamad Risqi U Saputra, Andrew Markham, and Niki Trigoni. Visual slam and structure from motion dynamic environments: A survey, ACM Computing Surveys, 51(2):1336 2018.
[40] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4938-4947, 2020.
[41] Konrad Schindler and David Suter. Two-view multibody structure-an and-motion with outliers through model selection. IEEE Transactions on Puttern Analysis and Machine Intelligence, 28(6):983-995, 2006.
[42] Gideon Schwarz. Estimating the dimension a model. The Annals of Statistics, 6(2):461-64, 1978.
[43] Weiwei Sun, Wei Jiang, Eduard Trulls. Andrea Tagliasacchi, and Kwang Moo Acne: Attentive context normalization for robust permutation--eqiiaaiann learning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 1 11286-1 1 1295, 2020.
[44] Philip HS Torr. Geometric motion segmentation and model selection. Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences. 356(1740): 1321-1344 1998.
[45] Manolis C Tsakiris and René Vidal. Dual principal component pursuit. In Proceedings of the IEEE International Conference Computer Vision Workshops, pages 10 18, 2015.
[46] Manolis C Tsakiris and René Vidal. Dual principal component pursuit. Journal of Machine Learning Research, 19:1-49, 2018.
[47] René Vidal and Paolo Favaro. Low rank subspace clustering (Irsc). Pattern Recognition Letters, 43:47-61, 2014.
[48] Yong Wang, Yuan Jiang, Yi Wu, and Zhi-Hua Zhou. Spectral clustering on multiple manifolds. IEEE Transactions Neural Networks, 1297):1149-118 2011.
[49] Kwang Moo Yi, Fduard Trulls, Yuki Ono, Vincent Lepetit, Mathieu Salzmann, and Pascal Fua. Learning to find good correspondences. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 2666-26744 2018
[50] Jiahui Zhang, Dawei Sun, Zixin Luo, Anbang Yao, Lei Zhou, Tianwei Shen, Yurong Chen, Long Quan, and Hongen Liao. Learning two-view correspondences and geometry using order-aware network. In Proceedings of the IEEE International Conference on Computer Vision, pages 5845-5854 2019.
[51] Shihua Zhang and Jiayi Ma. Convmatch: Rethinking network design for two-view correspondence learning. In Proc. AAAI Conf. Artif. Intell, pages 1-12, 2023.
[52] Zhihui 7hu, Yifan Wang, Daniel Robinson, Daniel Naiman, Rene Vidal, Manolis C Tsakiris, et al. Dual principal component pursuit: Improved analysis and efficient algorithms. In Advances in Neural Information Procensing Systems, pages
2018.
12

---

Appendices
We provide necessary proof, pseudo code, additional analysis of our method, and more details about the implementation, the used datasets and additional experimental results.
A
Notaions
We denote matrices with bold upper case letters and their elements with
lower case (zt). Thus, the matrix can be also termed as (x:) or [X] k.)) where [ means the i-th row of matrix X. We also denote X' as the
letters: X (I.z). ). We write the vector in bold lowercase
set-form matrix X E I M which consists all column vectors of X, L.e., X' = {x,}}= The y. For matrix X, the maximum
inlier product of two vectors x, y E RRth is denoted as (x.y) y) x xTy.
eigenvalue is denoted by Amax (X) and its Frobenius norm is defined as X F Σ. E, knyl
1 and its lo norm (i.e. [xx] is to compute the number of nonzero entries in vector x, while the & of x is defined as ||x|| > [z]]-
For vector x, ||x||2 (or ||x|]) denotes the Fuclidean norm |x|||
E,
B Threshold Shrinkage Method
Threshold shrinkage method is typically used to solve & norm problem, which is introduced by the Lemma from [47].
Lemma Let be given vector, for problem:
min
|x 212 + A||x||1,
(21)
x
the optimal solution is x", then the i-th element x* can be calculated as:
x = TT(a),
(22)
where Tx)+) a threshold shrinkage operation and defined as:
if X, < 0, otherwise.
X
(g)
(23)
C Proof of Solution
To solve the following linear inverse problem [7]:
x e1l2 + +112+ Al|x|111 }
x arg min
M
(24)
2
we can extend it into:
min F(x) = fxx) + (x)):
(25)
where l.g are convex functions with satisfying the following assumptions:
f : R" R smooth convex function, which is continuously differentiable with Lipschitz continuous gradient L(f):
- -V <
(26)
where L(f) is the Lipschitz constant fx)).
R" + R a continuous convex function which generally nonsmooth, such as a sparse
constraint term for our problem.
The extended problem (25) is solvable, i.e., x" arg min F + 0.
13

---

Table 5: Outline of the embedding of data and model for the classical DLT, and our DSP formulation for known and unknown model fitting. For better understanding, we assume to input only N inliers. For each formulation, the (requirements) including data type, model type, first row. And for each model, we conclude the geometry relationship GM = (D, d, v. 8).
listed in the
DLT
DSP for known model fitting
DSP for unknown model fitin
Model (req.)
(data & model Expe:D,d,r)
(data & model sype:D,d,x)
(data type:D)
m_
(u(p. V.D. pp. )ino
m
(v(P, ofp. 1ixo
m MMPT../p - :P, 1Ix9
Fund.
x = vec(F)
x= vec(F)
x= vec(F)
x
ONxI, 1.0)
ONxI. = 1,0)
ONx1. = 1,08,100)
p. ,0x3: -pp:
m
01xa. -. 2x0
m; (u/p; T 1ix9
m (u/P) opp, 1I09
Homo.
O by. -1g
Oixa, bj.
= vec(H)
X =
-hg 01xa h
x -h 01xa h
bb h. 01xa
- 01xa_
M
O2Nx1, G = (9,8, M x Owxa. G (9, 6,
M M X ONxa. G (9. 3,
P.. ,0x3s. -u
m,
m = ( o/ u/. 1/2 1)1xs
m = (v/p. oppT P. 11x9
Dixa: m 2XT
Afline
011. 012 -1, -1,0,
0,0, 0, 0000011220.. 11 1, 418
m vec(A)
X
X
021 a22 0, -1, a2s
0,0, 021. 0,0, a22. 0. -1, 02s
M x O2Nx1, -
ONxa. G (5, 3,
M x ONx2. (9,
In our subproblem, fxx 1 | x ell2. the gradient V fx) can be calculated as
9(x) X||x||1- The smallest Lipschitz constraint of
L(F) Maz (MMT).
(27)
For any L 0, fxx) can be approximated at given point y with the following form:
L
f(x) = f(y) + (VI(y), x- y)
2 x - y12
(28)
- ( const.
By ignoring constant terms related y. and let y x² k-1, can obtain the basic step in each iteration for problem (24):
L |
- Vf 12)1 + A||x|11 }
x* arg min
(x+
(29)
2
where ff(*"-1) = M(M x--1_ - et).
D Pseudo Code of Our DSP Method
We conclude the pseudo code of the implementation of DSP method in Alg. and Alg. 2. For any given data, we can directly reason out the true model from the solution of our DSP. To this end. the common embedding and formulation is outlined in Tab. If we know the exact information of true model type, we can also apply our DSP for inlier seeking and model estimation, by giving exact data embedding and basis number r, presented in the middle column Tab. 5.
Take 2D points fitting as example, i.e., the true model Line, Parabola, and Ellipse, as shown in Fig. 6. In each case, the inliers contaminated by noise of 0.01, and the inlier and outlier number are 300 and 100, respectively. We first construct the common data embedding as Φ( 14) a & 9:) = [1, .4 Vi. x? vf and the sparse basis parameter vector is A log(2N7 x [0.01, 0.1, 0.1, 1, 1 The optimization
c,,
We set the max basis number R procedure and fitting results are shown in Fig. 6. Specifically, the estimation results for these three cases
[-0.08 0.0845, 0.7135. -0.6956. 0, 0J x -0.4219, -0.1668, -0.4291, 0. 0.7915,00 0) and x 0.37722, 0.1498, 0. 0.0641, 0.6513, 036373 respectively.
14

---

Algorithm 1: Dual Sparsity Pursuit (DSP) Algorithm
Input: Observation data set S, parameters A, 7, K, B, stop threshold
Output: Sparse basis x and inlier set
Construct common embedding matrix M from S with m, = (s,,), and set e" = 0, fo 0 Initialize x" the right singular vector corresponding to the smallest singular value of M Calculate Lipschitz constraint L = Amaz (MM) )
For l: = 2, .... KK do Update e using e = TM 7M=x*~1 1) and (23);
Update the gradient Vf((' - - M(M x--1 c²):
Update x" using x* T3 (x (x*==))) and (23);
Perform sphere and orthogonal projection using Txlla and (1 BB )x
T|x - >12
then stop the iteration;
Update la with 4 )(1 / 1 1 and compute x xs x
التساي ( -x²-
Set Set = x then perform sphere and orthogonal projection using
x and (I BB )x;
12 end for 13 Return x x x", "x",and and e + e²; I Uie = 0}.
Algorithm 2: Model Reasoning & Fitting with DSP
Input: Observation data S, parameters X, % R, K, stop thresholds and Output: Sparse bases X, inlicr and model type M Construct common embedding matrix M from S with m, (s.)); Initialize X, right singular vectors corresponding to the smallest singular values of M, Eg 0,
In
0px1, 1:
Estimate (xi,e)) using Alg. to solve problem min e)) x.e
For i == 2, ... R do
Construct given bases B <- [B,
Estimate (x., e.) using Alg. to solve problem min L(M, x, e), and set = L(M, (1,,x)),
If LLM, x;, e:) r, then stop the iteration:
Set f-;;;
end for
10 Return X = X1, Xa, xx E = (e1, G2 e,];
Obtain inliers I = {j E 0}. and model type M with X ~ ((M)).
E Experiment
E.1
Implementation Details
We implement the proposed DSP method with MATLAB code. All comparative methods are im- plemented with the code provided by their authors. Their parameters are set as the original papers suggested, or referring to [6]. The experiments of RANSAC [19], EAS [18] and our DSP are conducted on desktop with 4.0 GHz Intel Core i7-6700K CPU and 16GB memory. USAC [38] 3 MAGSAC++ [6] OANet [50] and SuperGlue [40] are performed on Server and Ubuntu 16.04. And two deep learning methods are accelerated by NVIDIA TITAN GPUs. In addition, AIC [1], BIC [42] and GRIC [25] are used for performing unknown model fitting. Since they are classical model selection criteria, wherein GRIC is still used as state-of-the-ar method in vision tasks [39, 31, 32]. We perform these three criteria with the core parameters suggested in [31, 32). then equip them with robust model estimators to identify true models.
'USAC: http:p////ggttt...ooo///3///////
*MAGSAC++: htt.sa/aee.tto/////.oooooaaaaaaa $ *OANet: http://tchbb..omm//jjt/////tee
15

---

3.5
1.6
Loss LMM
2.5
L(M
Loss L(M.x.))
1.4
2
1.2
100
160
200
100
200
300
100
200
300
Iteration
Iteration
Iteration
0.8
Parameter valuee
0.5
Parameter value
0.8
Parameter
0.2
-0.5
-02
-0.5
-0.4
Iteration
Figure 6: The optimization procedure of our DSP for 2D points fitting. From left to right: model reasoning for Line, Parabola, and Ellipse model, respectively. From top bottom: loss [(M, x, e,) value with iteration, and corresponding parameter values.
E.2 Details of Datasets
Synthesized Data. To perform the robustness and model reasoning comparisons. we synthesize 900 image pairs with different outlier ratios (i.e., OR 20%, 50%, 80%) and noise levels (i.e. NL 0.2, 0.5, 0.8, 1.0, 1.5 in pixel). To be specific, these synthetic image pairs are randomly generated by two 3 x 4 camera matrices P1 K, [axxa)]( and P2 = K2 Ral R2tz]. Camera P1 is located in the origin and its image plane is parallel to XY plane. The position of camera P2 is generated by R2, t2, wherein the position of P2 is at random point inside a unit-sphere around PT. thus [t S 1. And the orientation R2 = Rx,o Ry,,R..,,,,,,, is determined by three 3D rotation matrices rotating around axes X, Y and Z by a. 3 and %, respectively (a, ,,, 7 [-7/3, T/3). We set K, = K2 with focal length Ja = fy 600 and principal points [300,300] T. To generate fundamental data, we randomly set R2 and 22, and create n, positions (s, y, =) in real world with y,y E [-1, 1]. E [3,8], which are further projected onto the first and second images by P1, P2- In this case, 20 fundamental image pairs are synthesized for each OR and NI., which creates in total 300 fundamental image pairs. As for generating homography data, we know that homography transformation is typically derived from two types of degraded cases, i.c., the target or scene in shooting is approximated plane, or the camera is only rotated around optical center without any translation. Therefore and for the first homography case, we randomly simulate n; positions (x, y, x) in real world with x, y E [-1, E [9.95, 10.05). While for another case, we just set t2 0sx1 on the basis of synthesizing fundamental data. 10 image data are respectively synthesized for these two homography types for each OR and NI., which also creates in total 300 image pairs. As for generating affine data, we directly set the last row of both P1 and P2 as [0, 0, 0, 1]. thus making them satisfy the property of affine camera. For each OR and NI., similarly synthesize 20 affine data, which creates in total 300 affine image pairs finally. For each image pair, we set the number of putative matches N n4 + n. 1000, that means consists n. 200 inliers and n. 800 outliers if OR = 50%.
Real Image Data. 8 public datasets used for test, including kusvod2, homogr?, EVD Hpatch', and the Feature Matching benchmark 10 that contains TUM, KITTI, T&T and CPC datasets. The detailed information is concluded in Tab. 6. In particular, kusvod2, homogr and EVD provide putative matches and the inlier indexes. While the rest datasets only provide ground truth model parameters, thus
"uusvdd2 homogr: http::/cmp.fett felk evut. wtt.lk...tt..c.eeeeeeeee wttpt/////aatttegeeettttt//////e xhtml $EVD: http://cmp. felk. cvut, wwut.c//vbst.
Hpatchh: httppp///ggtbb...cc/ppaaheeeee -attstee-d-tttttttt............. 10MM benchmark: https:/ htopp://jwbiiee
/ttp///binnn////////////. wwtpjjaanne//iiiieeeeee aate/m-aae bench
16

---

Table 6: Details of the used datasets. From left to right: Datasets name, ground truth(GT) Model, the number of sequences(#Seqq, the number of image pairs(#HP), resolution, baseline of the image pair, the property of imaging scenes, and the average match number(AMN) and average inlier ratio indicates all image pairs are from different imaging scenes.
Datasets
GT Model
#Seq.
#IP
Resolution
Baseline
Property
AMN
AIR
Hpatch
Homo.
59
295
varying
varying
wall or ground
3714
0.23
EVD
Homo.
15
329 x 278 to 1712 x 1712
extreme view
wall or ground 328
0.28
homogr
Homo.
16
392 x 278 to 1712 x 1368
varying
varying
873
0.47
kusvod2
Fund.
16
512 x 768 to 1944 x 2592
wide
outdoor scenes 516
0.74
TUM
Fund.
3
1000
480 x 640
short
indoor scenes 333
0.62
KITTI
Fund.
5
1000
370 x 1226
short
street views 550
0.83
T&T
Fund.
1000
1080 x 2048 or 1080 1920
wide
outdoor scenes 892
0.35
CPC
Fund.
I
1000
varying
short
internet photos 491
0.35
we use VLFEAT toolbox" "I to construct SIFT matches with ratio test 1.5, and the inlier matches are identified as the geometric error less than 2 pixels.
E.3 Additional Results
Quantitative Results of Other Four Datasets. We first give additional results of the known model fitting on the rest four datasets, including homogr, kusvod2, KITTI and TUM. Details are in Fig. which show the same trends as in other four datasets (please see the main body). that our proposed DSP achieves the best fitting accuracy and promising efficiency, followed by EAS and SAC methods.
Gromare Emer de sboet
RANBAC 0.90 nat
A RANSAG 0.99 MADRAGAs B:11 OANK a1 Ouns 5.43
Gromeette Ener (le plant)
BANGAC BA3
Geoomenns (rilee)
MANSAG
MADBaCAA
MARRACHH,B41
MAGRMAG
QANes 0.7 Oun,, 9.77
Gecoraatcc Enrer
DAAet EA2 Dara, 036
DANI, B.1 Out,, EA7
011 homogr: Comulative Distribution
62
0.6 Cumurative Distibution
144
0.6 TUM: Cumurativ Distibutton
KITTL Curulattve Destribution
RAMBAC Nouce
10"
RANRAC- 20ni01
A. RAMBSOC: S-thodt
-RANNEC: 15
MAGIAA Cibeett
Run Tima (rral
MAGGACC Fleut
Time (044)
H66066Ca. Moxic
Run Time (14))
MAAIBAC
Tiem
10"
Onhler Stauds
ORMe 33
CAANNE 1177aat
4.3exd)
Bipert 4.80
BAS. Tablutt
101
Gats 5 2.0egllinn
62
8.6
8.0
622
08 Corstative Distibution
0.0
0.8
04
0.5 0.8
homogr: Canidaaiv Dispibutionn
Cumulativ Disybutiin
Curnlaaive Distiibttion
Figure 7: Qualitative statistic results of Geometric Error(top) and Run Time (bottom) for known model estimation. A point on the curve with coordinate (x, y) denotes that there are 100 x percent of image pairs which have Error or Time more than y. The lower the better.
E.4 Applications
Multi-class Multi-model Fitting, In this part, we apply our DSP to solving multi-class multimodel fitting problem. In multi-model fitting of single type, the multiple structures belong to the same model type, such as being only line or circle, thus making it convenient for predefining the model type then conducting fitting algorithm. But this would be troublesome if the data contains multiple geometrical models of different types, such simultancously existing line, parabola, and circle, or both fundamental and homography model in two-view geometry. Existing strategy commonly borrows selection criterion like GRIC, which usually requires a long runtime 32]. Next, tend to demonstrate the superiority of our DSP in terms of simultaneously model reasoning and fitting in the task of multimodel fitting problem.
We first test DSP on synthesized point data, which consists of three inlier structures including line, parabola and ellipse together with a large number of outliers. The target is to classify each inlier structure and identify those outliers. Several manifold subspace clustering and multi-model fitting
l1 LLFat: attps://www vlfeat.ors org/
17

---

Clsttrr CuARra
Gustr
Chaster Custer
Cherter
Cluatrr
Cluose
SMMC (17.23s)
(b) DBSCAN (0.26s)
DSSCAN-SSC(26-LL))
Outlers
Outlens
Dutlees
Classer
Claster
Cuuoer
Chuster
Chaer
Chrter
(d) DBSCAN+SMMN (10.17s)
(e) MultiLink (100.67s)
(1) Ours (0.655)
Figure 8: Multi structure recovering and inlier seeking synthetic data, which consists of three inlier structures (including ellipse, parabola and line with noise level 0.05, respectively generated by 500, 500 and 300 inliers) and randomly generated outliers of number 1000, that means the total number of points is 2300. SMMC [48]. SSC [15], DBSCAN [16]. MultiLink [32] and their combinations are used for comparison. The runtime in second is attached in each sub-figure.
methods are used for comparison, such as spectral clustering on multiple manifolds (SMMC) [48], sparse subspace clustering (SSC) [15], density-based -baed clustering (DBSCAN) [16], and recently method termed MultiLink [32] that uses GRIC for model selection.
published preference-
Clustering results are shown in Fig. 8, we can find that our proposed DSP achieves the best accuracy by combining with DBSCAN. Since our DSP is merely designed for unknown sin- gle model fitting, which encourages grate multi-model problem into our outlier rejec- tion, model selection and fitting framework and achieve joint optimization in the future. Multi- Link can well classify these inlier structures as well, but several outliers that are closed to inliers are wrongly classified due its fixed thresholds. Since MultiLink asks for repeat estimations for all possible models, and uses agglomerative clus- tering as final output, it may consume a huge runtime (100 seconds for 2300 points). In ad- dition, DBSCAN can identify several potential inlier structures and filter out the most outliers with a short runtime. But the performance of DBSCAN is much sensitive to its core param- eters. In other words, a loose parameter would easily classify neighboring outliers as inlier structure, while a strict setting prefers to classify one complete structure into several small clusters, as shown in subfigure (b). In theory, SMMC and SSC can not handle outliers, and failed in clustering the data of high number of outliers, which can be seen in subfigure(a). In this regard, combine the advantages of DBSCAN and some clustering methods. We find that SMMC can well refine the results of DBSCAN to be more accurate, but SSC only work for liner structures such line plane, thus fails in our synthesized data.
inte-
Figure 9: Multi-mode fitting results on real images of our DSP method. In each scene, different colors denote different inlier structures, and the head and tail of each arrow in the motion field correspond to the positions of feature points in two images. For visibility, at most 200 randomly selected matches are presented, and the true negatives are not shown. Best view in color.
We also extend our method on real molti- fitting data, egg. AdelaideRFM 12, Fig. shows some qualitative results with three structures and four structures, which demonstrates that our DSP can accurately cluster the corrupted data into several inlier structures together with an outlier cluster. Quantitative results are reported in the main body manuscript.
adelaide. 12 Phttps://ce "ittpe::/cc.adeeladde edu, www.aaae/////aaaea hwwog/doku.. php?id-data
18


Reviews:
[{"review_id": "ZRxip48lwR", "reviewer": "Reviewer_1N35", "summary": "This paper addresses the task of robust model reasoning and fitting in an unified optimization framework that can estimate the geometric model accurately without knowing the predefined model in advance while being robust to outliers as well as highly efficient. The authors propose a novel sparse subspace recovery theory and derive corresponding propositions, thus giving a general and unified formulation for robust model reasoning and fitting. They next introduced an alternating optimization strategy together with proximal approximation method to accurately estimate the sparse model parameters and outlier entries. Extensive experiments indicate that the proposed method outperforms the selected comparison methods.", "questions": "Please refer to [Weaknesses]", "limitations": "The authors have solved most limitations of their method, and they also mentioned potential negative social impact, which are acceptable.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper solved the geometric model estimation problem from a novel perspective, that is recovering the model parameters without knowing the model type, which is interesting and valuable. The writing is good. The authors have provided clear presentation to convey their core idea. The experiment part is reliable, which well verifies the advantages of the method.", "weaknesses": "1:The authors designed an accelerated optimization approach and demonstrated its fast convergence in Fig. 2, claiming it an ‘optimal’ first order method. But there are no experimental proofs of the optimality in the paper. Authors should compare the performance of the algorithm before and after the acceleration.\n\n2:The authors mentioned their strength in efficiency, but the experiment results have not revealed this property.\n\n3:The authors propose a solution process similar to DPCP, and I think it should be added to the comparison algorithms as well.\n\n4:The experiments in the paper demonstrate the excellent performance of the DSP on 2D models and geometry models. I would like to know is the upper bound of the algorithm. Is it possible to design experiments (e.g., continuously increasing the dimension of the data space) to test the performance upper bound.\n\n5:During application test, the proposed method have successfully applied to multimodel fitting, as we can see. But why do the authors mention that they plan to integrate multimodel fitting and achieve a four-fold task in the future?\n\n6:The pose estimation or visual localization experiments are also necessary, since they are more common in this topic, comparing with multimodel fitting or Loop Closure Detection.\n\n7:Others:\n Line 113, “to predefine” should be “predefining”\n Line 159, “convert” should be “be converted”\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Q1: About the optimality if using the Acceleration Strategy (AS).\n**R1:** The optimality has not changed if using AS. We can see from Fig. 2 that, at convergence stage, solution or loss value are identical for using AS or not. This is because our used AS is not an approximation for original problem that may discard partial accuracy. Instead, it borrows the concept of Momentum or Nesterov method in gradient decent, to use an adjusted startpoint at each update step. It was proven in [7][30][31] that this strategy can achieve $O(1/k^2)$ convergence rate with theoretic guarantees, that is an ‘‘optimal’’ first order method for smooth problem.\n\n# Q2: Emphasize the efficiency.\n**R2:** We will explain the strength of efficiency as follows. First, EAS [TPAMI 2022] and our DSP are both global optimizing paradigms, but our DSP just consumes half time than EAS, due to the use of AS. Second, SAmpling Consensus (SAC) methods require sufficient time budget to hit an all-inlier subset, which is highly related to outlier ratio and noise scale. Thus, our DSP is faster for the cases of high outlier ratio, and this trend is well revealed in Fig. 3 of Supp. Mat. As for USAC, it integrates the local optimization and fast model verification in a universal framework, thus achieving fast model estimation, but most of the results are coarse or incorrect. Third, two deep methods just perform one forward propagation once trained, and they are accelerated with GPUs, thus obtain the best efficiency. \n**Without GPUs, the average execution time of OANet increases from 13ms to 117.6ms (our DSP: 70.3ms)**. In addition, compared with above methods, our DSP can estimate geometric model from contaminated data without predefining correct model type.\n\n# Q3: Comparing with DPCP\n**R3:** Our solution is merely partially similar to DPCP. DPCP provides plain formulation and solution for robust subspace learning with theoretic guarantees. But applying it to geometric model needs additional consideration, such as in DPCP-H [CVPR2020] and EAS [TPAMI2022]. In particular, EAS explores more general formulation and efficient solution for each model type, thus we take it as a representative method of DPCP and select it for comparison. As for the optimization process, because our DSP additionally considers the constraints of noise and the complexity of subspace, our formulation is more complex and hard to optimize. Thus, we explore an alternating optimization framework with proximal approximation strategy to accurately estimate the sparse subspace and outlier entries. To tackle the constraint terms, we use the similar strategy as DPCP, that performs sphere projection and orthogonal projection for orthogonal constraint $X^TX = I$.\n\n# Q4: Applying to high-dimensional model.\n**R4:** Our method mainly focuses on geometric models, particularly two-view geometry, to reason out model type and estimate model parameters from corrupted data. Thus, applying to recover high-dimensional or more complex models still need further exploration.\n\n# Q5: About multimodel fitting.\n**R5:** Our DSP have not realized multi-model fitting in a unified formulation yet, instead it asks for density-based clustering method to obtain several coarse clusters first, then apply our DSP for each cluster to reason out accurate model type and parameters, thus largely enhancing the fitting performance. In this regard, we think it is valuable to integrate estimating model number into our DSP formulation to achieve a four-fold task in the future.\n\n# Q6: Evaluation for pose estimation or visual localization.\n**R6:** In our experiment, we have conducted Fundamental matrix and Homography matrix estimation, which can directly reveal the performance of these two mentioned applications. Because the camera pose is directly decomposed from estimated $F$ matrix, i.e., with known camera intrincics $K_1, K_2$, the relation between $F$ and camera pose $R, t$ is $[t]_{\\times}R = E = K_2^{-T}FK_1^{-1}$. \n\n Following the reviewer's suggestion, we also conduct pose estimation on large scale dataset YFCC100M. This dataset has ground truth poses and sparse models obtained from an  off-the-shelf SfM tool. We follow the setting in OANet (Zhang et al., 2019) and choose 4 sequences for evaluation. We detect up to 2k matches for each data with SIFT. Similar to OANet, we use AUC of the pose error at threshold $(5^{\\circ}, 10^{\\circ},20^{\\circ})$ for evaluation, where the pose error is defined as the maximum of the angular errors in rotation and translation. Our DSP can obtain the best performance among those handcrafted methods.\n\n| Method    |  $@5^{\\circ}$  | $@10^{\\circ}$  |  $@20^{\\circ}$  |\n| ---- |  ----  | ----  | ----  |\n| RANSAC| 3.47  |9.10 |  18.60 |\n| USAC | 5.67  | 9.53  | 13.41 |\n| MAGSAC++ | 11.80  | 19.72 | 33.15|\n| EAS            | 12.18  | 22.22 | 35.67 |\n| DSP(Ours) | 14.43  |  25.32|   37.73|\n"}, {"review_id": "J5lHhA2m7I", "reviewer": "Reviewer_qrzv", "summary": "Considering that existing model estimation methods highly rely on the correct definition of model types, this paper introduces a unified optimization modeling DSP to simultaneously reason out the model type and estimate model parameters from contaminated data. For such purpose, the authors proposed Sparse Subspace Recovering (SSR) theory to modeling geometric model estimation task, that is to search a maximum of independent sparse bases under an over-embedded data space. The authors also introduced a fast and robust solver to estimate the sparse subspace parameters and error entries, and validated the advanced performance of their method on both unknown and known model estimation, and two applications.", "questions": "- Explain how to ensure the input data properly normalized.\n- Please provide mathematical explanation or proof for your SSR theory.\n- Please explain or validate the necessary of the used solution, or compare it with ADMM.\n- Please emphasize the advantages of your DSP comparing with existing model selection strategies, such as GRIC.\n- Correct the typos, see [Weaknesses]\n", "limitations": "Yes, the authors have analyzed the limitations and potential negative social impact.\n\n", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- This paper is well written and organized, and easy to follow.\n- The motivation, theory, formulation and solution are good contributions for the model fitting topic.\n- The authors solved the geometric model estimation problem from a novel perspective, that considers the model reasoning task additionally. Particularly, the authors introduced Sparse Subspace Recovering theory and formulated the unknown model fitting task into a continuous optimization objective, and explored efficient solution for it.\n- The authors designed unknown model fitting experiments, and also evaluated their method on common exact model fitting task, which is reasonable and credible. The experiments are convincing, and the results show great superiority comparing to the SOTA.\n", "weaknesses": "- Line 74, the authors claimed that if the data are properly normalized, using algebraic error is good. But how to assure this property of input data?\n- This paper proposes SSR theory, but the authors have not given mathematical explanation or proof.\n- As for the solution, why not using ADMM to solve this problem, I think ADMM is a common choice for solving L1 norm problem with Lagrange multiplier, as sparse subspace clustering (SSC) used. Please explain.\n- Line 52: How to understand ``insufficient information’’ that the GRIC used, which may cause wrong model selection for constrained motions. What are the advantages of this method comparing with those model selection criteria.\n- If using DSP or SSR theory to estimate a Fundamental matrix F, how to use the rank 2 constraint of F? That is an intrinsic property of Fundamental matrix.\n\nTypos:\n- Line 60, add citation for PSGM when it first appears.\n- Lines 163 and 171, ``sparse independent hyperplanes``, I think it is better to use ``sparse intersected hyperplanes`` or ``sparse independent bases``.\n- Line 170, $\\mathcal{R}$ should be $\\mathbb{R}$ to indicate the real number space.\n- Line 226, $e_k$ --> $e^k$.\n- Line 239, ``can successfully addressing`` --> can successfully address.\n- Line 271, ``300 image pairs of each model`` --> ``300 image pairs for each model``.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Q1: How to ensure the input data properly normalized?\n**R1:** The related distribution of the input data is their intrinsic nature, we cannot change it.  But to ensure the estimation easier, we first normalized the input points of each image into zero mean and one standard error, then we scaled each point into length 1 using $\\mathbf{m}_i \\leftarrow \\mathbf{m}_i/||\\mathbf{m}_i||_2 $  .\n\n\n# Q2: Mathematical explanation of Sparse Subspace Recovery (SSR) theory.\n**R2:** Our SSR theory claims that ‘’Geometric model fitting can be seen as a subspace recovery problem represented by the intersection of multiple sparse hyperplanes under an over embedded data space.’’,  which can be interpreted by the following mathematical form:\n\nSuppose we are given a set of clean data $\\mathcal{S} =\\\\{ \\mathbf s_i\\\\}_{i=1}^N$, \nsampled from a geometric model $\\mathcal{M}$, \n\nwhich can be parameterized by several bases $\\\\{\\theta_j\\\\}_{j=1}^r$. \nOn this basis, we can obtain compact constraint $f_j(\\mathbf{s}_i)^{T}\\theta_j = 0,~j = 1,2,\\cdots,r$, \nwhere $f_j(\\mathbf{s}_i)$ is a specific embedding for point $\\mathbf{s}_i$. \n\nOur SSR theory suggests that, it can be extended into $\\{[f_j(\\mathbf{s}_i)^{T}, g_j(\\mathbf{s}_i)^{T}][\\theta_j^T,\\mathbf{0}^T]^T=0,~j=1,2,\\cdots,r\\}$. Here $g_j(\\mathbf{s}_i)$ is another data embedding that is not contained in $f_j(\\mathbf{s}_i)$. $\\mathbf{0}$ is an all zero vector with length equal to $g_j(\\mathbf{s}_i)$. For example, a line model derives $f(\\mathbf{s}_i)= [x,y,1]^T, \\theta = [a,b,c]^T$, with $\\{f(\\mathbf{s}_i)^{T}\\theta = 0 \\}$ under ideal case, note that here $r= 1$, thus we ignore subscript $j$. Then we generate  $g(\\mathbf{s}_i) = [xy, x^2,y^2]^T$, and get an over embedded space $F(\\mathbf{s}_i)=[f_j(\\mathbf{s}_i)^{T}, g_j(\\mathbf{s}_i)^{T}] = [x,y,1,xy, x^2,y^2]$, and $\\theta' = [\\theta,0,0,0]^T$, similarly having $F(\\mathbf{s}_i)^{T}\\theta ' =0$. \n\nBased on the argument embedding $g_j(\\mathbf{s}_i)$, we can construct a common embedding $F(\\mathbf{s}_i)^{T}$ for multiple bases, such that $\\{F(\\mathbf{s}_i)^{T}\\theta_j' = 0, j =1,2,\\cdots, r\\}$, then obtain our general formulation Eq. (9) for unknown model fitting, i.e., $ \\widetilde{\\Phi}(\\mathbf{s}_i)^T\\Psi(\\mathcal{M})= \\mathbf{0}$.\n\nSSR theory can be proven easily. For each basis, since $f_j(\\mathbf{s}_i)^{T}\\theta_j = 0$ holds, $g_j(\\mathbf{s}_i)^{T}\\mathbf{0} = 0$ equally holds, thus we have\n$\\{[f_j(\\mathbf{s}_i)^{T}, g_j(\\mathbf{s}_i)^{T}][\\theta_j^T,\\mathbf{0}^T]^T = 0,~j = 1,2,\\cdots,r\\}$.\n\n\n# Q3: Why not using ADMM to solve this problem?\n**R3:** Using ADMM to solve our DSP problem requires complex computation, including the inversion operation for large matrix, which creates long execution time. Specifically, ADMM suggests an auxiliary variable $Z$ to simplify nonlinear constraint, and we have:\n\n$\\min\\limits_{(X,E,Z)} \\frac{1}{2} ||M^TX-E||_F^2 $\n$+\\gamma ||E||_2$$_1 +\\lambda ||Z||_1$\n\n$ s.t.~X^TZ = I ,X = Z.$\n\nThen, the Augmented Lagrangian expression is:\n\n$L= \\\\frac{1}{2}  ||M^TX-E||_F^2$\n$+\\gamma  \\\\|E\\\\|_2$$_1+ \\lambda \\\\|Z\\\\|_1$\n$+ <\\Lambda_1,X^TZ - I> + <\\Lambda_2, X-Z> + \\frac{\\mu_1}{2} ||X^TZ - I||_F^2 +  \\frac{\\mu_2}{2} ||X-Z||_F^2,$\n\n$=\\frac{1}{2}||M^TX-E||_F^2 +\\gamma ||E||_2$$_1+ \\lambda \\\\|Z\\\\|_1$\n$+\\frac{\\mu_1}{2} ||X^TZ - I + \\frac{\\Lambda_1}{\\mu_1} ||_F^2 + \\frac{\\mu_2}{2} ||X-Z+\\frac{\\Lambda_2}{\\mu_2}||_F^2 + const.$\n\nLetting $\\frac{\\partial L}{\\partial X} = 0$,\n we have the update formula for $X^{k+1}$  at given $(E^k,Z^k,\\Lambda_1^k,\\Lambda_2^k，\\mu_1^k,\\mu_2^k))$, \n\n$ X^{k+1} = [MM^T + (\\mu_1+\\mu_2) I]^{-1} [ME^{k} + \\mu_1^k (I-\\frac{\\Lambda_1^k}{\\mu_1^k})Z^T + \\mu_2^k(Z - \\frac{\\Lambda_2^k}{\\mu_2^k} )] $\nwhich is time consuming. In addition, parameters $(E^k,Z^k,\\Lambda_1^k,\\Lambda_2^k，\\mu_1^k,\\mu_2^k))$ also need update.\n\n\n# Q4: Our DSP V.S. GRIC.\n\n**R4:** GRIC is an early strategy to tackle unknown model fitting problem, but still widely used in many vision tasks. Given the model pool $\\mathcal{M}$, traditional pipeline first uses existing robust estimators, such as RANSAC, to estimate the parameters for each model in $\\mathcal{M}$, then uses GRIC to select the “best” one as final output. Using this estimation-then-selection strategy, the time consumption would be much large if the model pool is huge. And this greedy strategy would easily cause wrong identification, as revealed in Tab. 1. On the contrary, our proposed DSP fully considers the forms under sparse subspace, and specifically models the noise and outliers, thus solving three subproblems in a unified optimizing paradigm, such that our DSP achieves better accuracy and efficiency.\n\n\n# Q5: How to ensure rank 2 constraint of F model.\t\n**R5:** During our optimization, we do not consider the rank 2 constraint for $F$ at the beginning,  that would obtain a full rank matrix $F'$. Then we decompose $F' $ with SVD, getting $F'= U diag(s1,s2,s3)V^T$, the final estimation of rank-2 form would be $\\hat{F} = U diag(s1,s2,0)V^T$.  This process is commonly done in many robust estimators such as 8-point algorithm.  \n\n# Q6: Typos.\n**R6:** We will correct them in our next version."}, {"review_id": "pTnOZoIgUp", "reviewer": "Reviewer_cPqq", "summary": "This paper considers the robust model fitting problem in the presence of outliers, which is a fundamental problem in low-level CV. The aim is to simultaneously achieve outlier rejection, model selection, and model parameter estimation in a unified formulation. Toward this end, the authors propose to cast the joint outlier rejection, model selection, and model estimation problem into a sparse subspace recovery problem, which can cover the widely used projective transformation models for multi-view gemometry such as the fundamental, homography and affine models. The joint optimization formulation is solved by an alternating algorithm with the use of proximal approximation computation and acceleration. Experimental results on synthetic and real-word data have been provided to deminstrate the performance of the new method, including fundamental matrix and homography estimation, as well as a loop closure application.", "questions": "It is claimed that the objective with geometric error is extremely hard to optimize due to the highly non-linear nature, but in fact there exist well developed methods in the literature of multi-view geomety that can solve the formualtion with geometric error efficiently and effectively.", "limitations": "The authors have not addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This work is well motivated to joitnly achieve outlier rejection, model selection, and model estimation in a unified formulation. The proposed sparse subspace recovery formulation covers the widely used projective transformation models for multi-view gemometry, such as the fundamental, homography and affine models. The method has been evaluated on both synthetic and real-world data via various experiments.", "weaknesses": "1. The formulation only applies to algebraic error model, i.e., model estimation with algebraic distance. While algebraic distance is convenient due to its linearity, geometric distance is geometrically or statistically meaningful and can yiled better performance in projective transformation estimation over the algebraic distance. This has been demonstrated in multi-view geomety.\n\n2. Wile the proposed method is well conceived, it would not outperform the simple method that first estimates the model parameters of each candidate models and then selects the best model in terms of the fitting error. Although the provided experiments show that it outperforms AIC, BIC, GRIC selection methods, in practical applications the model is typically selected in terms of a score of the estimated model computed based on the symmetric transfer errors when the ground-truth transformation is unknown. This selection method is typically used in practice.\n\n3. The proposed algorithm has several hyper-parameters, e.g., $\\gamma$, $\\lambda$, $r$, and $\\tau$. Its performance depends on the tunning of these hyper-parameters, which diminishes its potential advantage over the simple model selection method based on the symmetric transfer errors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Q1: About using geometric distance.\n**R1:** We highly agree that minimizing geometric error (GE) could obtain better performance in accuracy, since the error entry with geometric distance is more stable. But the non-convex nature indeed makes it hard to optimize. Although there exist effective methods to search the optima, it usually consumes complex computation and long execution time, thus rarely used in real applications, such as SLAM, 3D Reconstruction, etc.\n\nTo our knowledge, GE is commonly converted into a tractable form, such as a generalized fractional form (GFF) or Sampson Distance (SD, i.e., first order approximation). For instance, Ref. [A] studies approximate solutions for GE-based objective using GFF for each match $(p,p’)$ like: \n$GE(p,p’) = \\frac{f(p,p’)} {g(p,p’)}$, where $f$, $g$ are linear functions.\nIn Sec. 6.2.1 of [A], the authors also admitted that _''finding least squares estimates based on geometric distances is intractable''._ \nActually they merely explored GE minimization for Homography and Affine matrix, as the authors explained in Sec 6.1.2 _''Fundamental matrix estimation does not have the GFF'',_ hence they still used Algebraic Error (AE) to estimate $F$ matrix. \n\nAs for SD, we take fundamental matrix $F$ as example:\n$GE(F, p, p’) = \\frac {(p’^TFp)}  {(F^Tp’)^2_1 + (F^Tp’)^2_2 + (Fp)^2_1+ (Fp)^2_2} $.\nObviously, using SD to estimate $F$ is not easy, at least more difficult than using AE. In fact, under consensus maximization framework, such as RANSAC, GE is typically used to count inliers with pixel threshold. But the model parameters are still estimated by AE. Recently, SD often serves as loss to guide the training of those deep matching methods. \n\nIn addition, by reading the experimental results from Tab. 3 and Tab. 4 of [A], we clearly find, GE-based optimization would take $>$5000ms to process around 500 point matches with 50% inlier ratio. While our DSP costs only 183ms in average for Hpatch datasets (average match number 3714 and average inlier ratio 23%), and merely causes 0.82 pixels Geometric Error. Paper [B] (Fig. 6) also reveals that, GE-based method (IBCO) [C] costs huge runtime. \n\nAnother critical point is that, the core motivation of our manuscript is to explore a unified modeling and efficient solution for unknown model fitting. But, the GE forms would vary for each specific model, thus hard to be integrated into our unified objective to achieve model reasoning. \n\nConsidering both the practicability and theory convenience, it is better to use Algebraic Error in our DSP. Surely, if the reviewer has any efficient and effective methods/literature to solve GE-based problems, please recommend them to us. It would help to advance our dual sparsity formulation in the future, i.e., modeling with GE and achieving better accuracy while maintaining the real-time property.\n\n_[A] Deterministic approximate methods for maximum consensus robust fitting. TPAMI 2021._\n\n_[B] Efficient Deterministic Search With Robust Loss Functions for Geometric Model Fitting. TPAMI 2022._\n\n_[C] Deterministic consensus maximization with biconvex programming, ECCV 2018._\n\n\n# Q2: Details of using GRIC.\n**R2:** In our experiments, we did use geometric error to support AIC, BIC and GRIC to conduct model selection. In details, for all comparing methods that use AIC, BIC, or GRIC for model identification, we first utilize those robust estimators (including RANSAC, USAC, MAGSAC++, EAS, OANet, SuperGlue) to estimate each model $\\mathcal{M}$, then use $\\mathcal{M}$ back to compute the geometric error for each match pair. In our experiments, we use Sampson Distance as geometric error with Matlab toolbox.\n\n# Q3: About the difficulty of tuning hyper parameters $\\gamma,  \\lambda, r$ and $\\tau$.\n**R3:** The hyper parameters of our DSP do not need fine tuning for each scene, since our original purpose is to explore a unified method for all unknown models under different scenes. These parameters have exact physical significance, which provides a guidance for parameter settings. \n\n* In details, $\\gamma$ is used to constrain outliers, indicating that error entry $\\mathbf{e}$ generates if two vectors are not vertical to some extent. In other words, a correct match should satisfy \n $|cos(\\theta(\\mathbf{m}_i, \\mathbf{x}))| <= \\gamma$,\n where $\\theta(\\mathbf{m}_i, \\mathbf{x}) $ denotes the angles of two vectors. For correct matches without noise, we have $\\mathbf{m}_i^T\\mathbf{x} = 0$, i.e., they are vertical.  \nUnder noise case, we set $\\gamma = 0.05$,\n which allows an inlier have\n $\\theta(\\mathbf{m}_i, \\mathbf{x}) \\in [88.85^{\\circ},91.15^{\\circ}]$. \n\nWe also follow the idea of simulated annealing, during  optimizing, we dynamically decrease $\\gamma = 0.98\\gamma$ for each 20 iterations and set $\\gamma_{min} = 0.02$.\n\n * As for parameter $\\lambda$, it balances the fitting error and model complexity. Prof. Philip HS Torr has studied this in paper “Geometric motion segmentation and model selection”, and concluded that under the assumption of independence among matches, the optimal estimation is merely contributed by 4 noisy coordinate values of each match $(u_i , v_i , u_i' , v_i ')$, thus the parameter item and error item would differ by a scale factor of $log(4N)$.\n\n* $r$ is not a hyper parameter, but a core integer variable that we need to estimate. It indicates the maximum number of basis of recovered subspace or geometric model. In our DSP, we constrain it with rank maximization and solve it progressively.  \n\n* The estimated basis number $r$ is controlled by threshold $\\tau$, but fortunately, the optima value of $\\tau$ can derive from the statistical analysis result as shown in Fig. 3.  "}, {"review_id": "r1IHAjrOjR", "reviewer": "Reviewer_WgUj", "summary": "The paper studies the geometric model fitting problem with unknown model type and heavy outliers. It proposes a unified optimization model with dual sparsity constraints that combines the outlier rejection, true model reasoning and parameter selection. Moreover, a fast numerical algorithm is proposed to solve the approximate and dimension-reduced model via separability of the objective function. Numerical experiments on synthetic and real data sets are conducted to compare the performance of the proposed method with other related works. Overall, the paper is complete from theory, algorithm, to experiments with a section of broader impact.", "questions": "1. Even with the Proposition 3, model (13) is not equivalent to model (14) which is later solved inexactly. Why was the convex relaxation  applied to (14) rather than (13)? Would that cause loss of accuracy especially the low rankness?\n2. How about the uniqueness of the solutions for the proposed model?\n3. Convergence analysis for the proposed algorithm could be provided.", "limitations": "The proposed work aims to avoid parameter selection by introducing a variable of parameters in the unified model. However, it leads to the introduction of other regularization parameters associated with the sparsity terms in the objective function, which again may need fine tuning. Some practical guidance for diverse data sets and application settings should be highlighted.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The proposed dual-sparsity optimization model has a certain novelty. \n2. A variety of numerical results are presented to justify the proposed effectiveness.", "weaknesses": "1. The existence and uniqueness of the proposed model are not discussed in detail. \n2. The description of formulation for the proposed model is not fully concise and precise, especially the treatment of the rank-term. \n3. Model sensitivity and robustness could be further discussed from the theoretical aspect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Q1: About problem (13), (14), and the rank-term?\n**R1:** For better understanding, we  should introduce  problem (13) first, which is actually modeled for finding all $r$ independent sparse bases, by supposing $r$ is known. But for our unknown model fitting task, $r$ is unknown in advance, thus needs to be estimated simultaneously.  For such purpose, we introduced Proposition 3, which converts the estimation of $r$ into $r = \\max rank(\\mathbf{X})$. Hence we integrated this into our general formulation, i.e., obtaining problem (14). This conversion is natural and necessary. And both the later convex relaxation and solution are based on problem (14), because (14) is our finally general formulation, while (13) is merely a middle formulation.\n\n# Q2. How about the uniqueness of the solutions for the proposed model?\n**R2:** We will discuss the global optimality of our proposed DSP, that implies the existence and uniqueness of solution.\n\nOur DSP formulation actually follows the minimization problem on Grassmannian $\\mathbb{G}(r,D)$. An element of $\\mathbb{G}(r,D)$ can be represented by an orthonormal matrix (multiple bases) in $\\mathbb{O}(r,D) = : \\{ X \\in \\mathbb{R}^{D\\times r}: X^T X = I_r\\}$, which is the well-known Stiefel manifold. With this understanding, we can obtain the following parameterized problem:\n\n$\\min\\limits_{X \\in \\mathbb{O}(r,D)} f(X)$,\n\nwhere $f:\\mathbb{R}^{D\\times r}\\rightarrow \\mathbb{R}$ \nis locally Lipschitz, possibly non-convex and non-smooth. In our DSP formulation, we have\n\n$f(X) = \\frac{1}{2}\\|M^T X -E\\|_F^2 + \\lambda ||X||_1 + \\gamma ||E||_2$$_1$.\n\nSince  we consider problems on the Grassmannian, we use tools from Riemannian geometry to state optimality conditions. Specifically, we generalize the definition of the Clarke subdifferential and denote $\\tilde{\\partial}f$ the Riemannian subdifferential [D] of $f$:\n\n$\\tilde{\\partial}f(X)= (I-XX^T)\\nabla f(X)$.\n\nWe say that $X$ is a critical point of our DSP problem if and only if $0=\\tilde{\\partial}f(X)$, which is a necessary condition for being a global optima. Note that $f$ is non-smooth but convex, and the Clarke subdifferential is a nonempty and convex set since a locally Lipschitz function is differentiable almost everywhere. Thus we can conclude the global optima $X^*$ exists.\n\nActually, in the community of robust subspace recovery, DPCP and its follow-ups [A]-[C]  have provided theoretical guarantees for modeling, the property of global optima and convergency. These researches can also provide basic theory for our DSP formulation, directly giving theoretic guidance for our own global optimality analysis.\n\n# Q3: Convergence analysis for the proposed algorithm could be provided.\n**R3:**  Intuitively, because our objective is convex and we use a gradient-descent-like optimization method, it must converge into the optima under proper initialization. Specifically, given a constant step size, suppose that function $f$ satisfies the $(\\alpha,\\epsilon,X^*)$-Riemannian Regularity Condition. Let $\\\\{X_k\\\\}$ be generated by our DSP with step size $\\mu_k \\equiv \\mu \\leq \\alpha\\epsilon/\\\\xi^2 $ and initial $X_0$ satisfying $dist(X_0, X^*)\\leq \\epsilon$, where $\\epsilon$ denotes an upper bound on the size of the Riemannian subgradients in a neighbrohood of $X^*$, and $\\alpha>0$ denotes a scale factor.\nThen, for all $k>0$, we have\n$dist(X_k,X^*)\\leq max\\\\{ dist(X_0,X^*) - \\mu \\alpha k/2,\\mu\\xi^2/\\alpha \\\\} $, \nwhich implies that after at most $K=2(dist(X_0,X^*)-\\mu\\xi^2/ \\alpha)/(\\mu\\alpha)$ iterates, the inequality $dist(X_k, X^*)\\leq \\mu\\xi^2/ \\alpha$ will hold for all $k\\geq K$, i.e.,converge on $K$. A larger $\\mu$ leads to a faster decrease but a larger universal upper bound of $\\mu\\xi^2/ \\alpha$. Papers [A]-[D] could provide more details of convergence analysis for such problem.\n\n# Q4: About Parameter settings.\n**R4:** The hyper parameters of our DSP do not need fine tuning for each scene, since our original purpose is to explore a unified method for all unknown models under different scenes. These parameters have exact physical significance, which provides a guidance for parameter settings. \n* In details, $\\gamma$ is used to constrain outliers, indicating that error entry $\\mathbf{e}$ generates if two vectors are not vertical to some extent. In other words, a correct match should satisfy\n $|cos(\\theta(\\mathbf{m}_i, \\mathbf{x}))| <= \\gamma$, \nwhere $\\theta(\\mathbf{m}_i, \\mathbf{x}) $ denotes the angles of two vectors. For correct matches without noise, we have $\\mathbf{m}_i^T\\mathbf{x} = 0$, i.e., they are vertical. \n Under noise case, we set $\\gamma = 0.05$, which allows an inlier have $\\theta(\\mathbf{m}_i, \\mathbf{x}) \\in [88.85^{\\circ},91.15^{\\circ}]$. \n\nWe also follow the idea of simulated annealing, during  optimizing, we dynamically decrease $\\gamma = 0.98\\gamma$ for each 20 iterations and set $\\gamma_{min} = 0.02$.\n  * As for parameter $\\lambda$, it balances the fitting error and model complexity. Prof. Philip HS Torr has studied this in paper ''Geometric motion segmentation and model selection'', and concluded that under the assumption of independence among matches, the optimal estimation is merely contributed by 4 noisy coordinate values of each match $(u_i , v_i , u_i' , v_i ')$, thus the parameter item and error item would differ by a scale factor of $log(4N)$.\n* Threshold $\\tau$ is used to control the estimation of basis number $r$, Fortunately, the optima value of $\\tau$ can derive from the statistical analysis result as shown in Fig. 3. \n\n_[A] Dual Principal Component Pursuit:Improved Analysis and Efficient Algorithms, NeurIPS 2018._\n\n_[B] A Linearly Convergent Method for Non-Smooth Non-Convex Optimization on the Grassmannian with Applications to Robust Subspace and Dictionary Learning, NeurIPS 2019._\n\n_[C] Dual Principal Component Pursuit for Robust Subspace Learning:Theory and Algorithms for a Holistic Approach, ICML 2021._\n\n_[D] Subgradient Descent Learns Orthogonal Dictionaries, ICLR 2019._"}, {"review_id": "4gDmMUTIeX", "reviewer": "Reviewer_bMPj", "summary": "This paper is about model fitting. The authors consider a scenario in which it is unknown whether correspondences between points in 2 images stem from 3D points that are (1) generally distributed, (2) lie on a plane, or (3) lie on a plane and the motion between the 2 images is not projective but affine. The latter implies that not a pinhole camera was used for projection but an affine camera. This can hold if, for instance, the distance between camera and scene is large compared to the depth variation within the scene.\n\nCase (1) amounts to the well-known fundamental matrix constraint x’^T*F*x=0 with (x, x’) being corresponding 2D homogeneous points. Here, F is a 3x3 matrix with rank(F)=2. For noise-free data, matrix F can be estimated from at least 8 point correspondences since F is determined only up to scale and each point-point correspondence yields a single equation.\nCase (2) amounts to a 3x3 homography H. Differently from the F-matrix, homographies have full-rank, in general. As fundamental matrices, they are only determined up to scale. Since each point-point correspondence yields 2 equations, H can be determined from at least 4 correspondences.\nFor a general affine transformation between the two images, case (3) implies that the last row of H equals [0,0,1]. There are no constraints on the remaining 6 entries, in general.\n\nThe authors show that by expressing cases (2) and (3) as fitting multiple subspaces, all three cases can be expressed by the same model. They propose a projected coordinate descent type of algorithm to estimate.\n", "questions": "Please correct proposition 2.", "limitations": "None", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "To me, this interpretation of F/H-matrix geometry is new. I cannot say whether this naturally arises from classical theorems. The fact that all papers I am aware of fit either an F-matrix or a homography can be taken as evidence that at least this fact is not generally known.", "weaknesses": "- proposition 2 is wrong\nMatrix Psi has rank 2 which can be seen from the fact that the third row is linearity dependent on the first two. Hence, G_H=(9,7,2,3).\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Q: About the embedding of homography matrix and its geometric relationship.\n\n**R:** We would like to thank the reviewer’s positive feedback. \n\nIt seems that the reviewer was concerning about proposition 1. As for proposition 1, we agree that the homography constraint derives two independent equations, since the last one is a linear combination of the first two. But this happens for DLT solution, that converts to fitting a hyperplane $\\theta = vec(\\mathbf{H})$. Under our subspace recovery framework, it converts to estimate subspace $\\Psi(\\mathbf{H})$, which is full rank of 3, thus the third basis indeed exists, and we have $G_{\\mathbf{H}} = (9,6,3,3)$. This is also verified and applied in related papers [A][B] (such as Eq. (2) or Tab. 1 in [A]).\n\n_[A] Robust Homography Estimation via Dual Principal Component Pursuit. CVPR 2020._\n\n_[B] Efficient Deterministic Search With Robust Loss Functions for Geometric Model Fitting. TPAMI 2022._"}]

Author rebuttal:

Meta-review: This work has been reviewed by five reviewers and received consistent positive scores (SA/WA/SA/WA/A), with four reviewers unequivocally acknowledging the novelty of the proposed dual-sparsity optimization model and excellent performance in the experimental evaluations. Overall, the AC considers that the paper makes a solid contribution to the field and recommends acceptance of the paper.
Decision: Accept (spotlight)