Title: Robust Model Reasoning and Fitting via Dual Sparsity Pursuit
Venue: NeurIPS 2023
Accepted: yes
Keywords: Model reasoning; Model fitting; Outliers; Sparse subspace learning; Feature matching
Abstract: In this paper, we contribute to solving a threefold problem: outlier rejection, true model reasoning and parameter estimation with a unified optimization modeling. To this end, we first pose this task as a sparse subspace recovering problem, to search a maximum of independent bases under an over-embedded data space. Then we convert the objective into a continuous optimization paradigm that estimates sparse solutions for both bases and errors. Wherein a fast and robust solver is proposed to accurately estimate the sparse subspace parameters and error entries, which is implemented by a proximal approximation method under the alternating optimization framework with the ``optimal'' sub-gradient descent. Extensive experiments regarding known and unknown model fitting on synthetic and challenging real datasets have demonstrated the superiority of our method against the state-of-the-art. We also apply our method to multi-class multi-model fitting and loop closure detection, and achieve promising results both in accuracy and efficiency. Code is released at: https://github.com/StaRainJ/DSP.

Robust Model Reasoning and Fitting via Dual Sparsity Pursuit

Abstract

In this paper, we contribute to solving a threefold problem: outlier rejection, true model reasoning and parameter estimation with a unified optimization modeling. To this end, we first pose this task as a sparse subspace recovering problem, to search a maximum of independent bases under an over-embedded data space. Then we convert the objective into a continuous optimization paradigm that estimates sparse solutions for both bases and errors. Wherein a fast and robust solver is proposed to accurately estimate the sparse subspace parameters and error entries, which is implemented by a proximal approximation method under the alternating optimization framework with the 'optimal' sub-gradient descent. Extensive experiments regarding known and unknown model fitting on synthetic and challenging real datasets have demonstrated the superiority of our method against the stateof-the-art. We also apply our method to multi-class multi-model fitting and loop closure detection, and achieve promising results both in accuracy and efficiency. Code is released at : https://github.com/StaRainJ/DSP .

1 Introduction

Geometric model estimation is a fundamental problem in computer vision, serving as the core of many high-level tasks including structure-from-motion [28, 39], SLAM [33, 9], and data alignment [24, 27]. The models to specific data can be explained as line/ellipse/circle for 2D point sets, plane/cylinder/sphere for 3D point sets, or fundamental/homography/affine model for point correspondences extracted from two-view images, etc. Estimating the parameters of a predefined model from inliers only is well studied [21]. But real-world inputs unavoidably contain severe noises and outliers, posing great challenges for accurate model estimation. Besides, another key problem rarely considered is how to recovery the model parameters without knowing the model type .

Recent researches mainly focus on proposing robust estimators to tackle the impact of noise and outliers, such as regarding it as a Consensus Maximization (CM) problem. This problem can be well solved with Sample Consensus (SAC) methods, such as RANSAC [19] and its variants[11, 38, 3, 6, 22]. They commonly sample a smallest inlier set, to best fit a given geometry model following a hypothesize-and-verify strategy [28]. Inside this loop, the model is actually estimated by Direct Linear Transform (DLT) method with a Least-Square (LS) solution. SAC methods can provide probabilistic guarantee of hitting an all-inlier subset. However, this scheme succeeds only if given a predefined model type with sufficient time budget [52]. Another popular strategy is to pose it in a Global Optimization framework, which formulates the fitting task as a subspace learning problem [45, 37, 13, 17, 18, 14]. Specifically, the representative method, Dual Principal Component Pursuit (DPCP) [45], tries to minimize an glyph[lscript] 1 co-sparse objective on the sphere. This is further applied to estimate specific models [12, 18], which typically embeds the model as a single hyperplane, or the intersection of multiple hyperplanes, and optimizes globally. This type of methods are admitted with

efficient implementations and strong theoretical guarantees, thus arising great research interest in recent years. Deep Learning has also stimulated numerous methods for geometric model learning, which extract deep geometric cues from sparse points using multilayer perceptrons [49, 43, 50], convolutional [51] or graph networks [40]. They can fast output potential inliers once trained, but still require robust estimator such as RANSAC as postprocess for accurate model estimation.

The above mentioned methods can only work on a fact that: one is certain about the true geometric models, then uses them to guide the formulation for parameter estimation . But in fact, we can always fit a model less constrained to obtain higher inlier count [31, 37]. Specifically, for homography data, estimating a fundamental matrix may return more consensus points, but many are outliers. While homography estimation can only find the largest plane structure in 3D scene, missing considerable inliers for full motion data. Fitting an unknown model for heavily contaminated data is much challenging and is in general NP hard. Existing methods solve it with model selection criteria [1, 42, 44], which first fit all possible models, then select the best one with a geometric verification, such as widely used GRIC metric [44, 32]. This is what is done in SfM or SLAM pipeline for fundamental matrix and homography identification [39]. However, such strategy is limited by the greedy selection strategy thus requiring huge computational burden. In addition, the used insufficient information would easily cause wrong selection of the true model for constrained motions of camera [37, 31, 32].

In this paper, we will simultaneously solve i) outlier rejection, ii) true model reasoning and iii) parameter estimation in a unified optimization modeling. To this end, we start with introducing a common paradigm for exact model fitting, then derive our sparse subspace learning theory, which makes it possible to estimate the true model without knowing the model type, and robust to outliers. On this basis, we convert the objective into a continuous optimization paradigm that estimates sparse solutions for both bases and outlier entries. Wherein a fast and robust solution is proposed, that is based on superiorities of projected sub-gradient method (PSGM) and alternating optimization strategy. Finally, the true model and the inliers are directly extracted from our solutions. Contributions: i) We are the first to propose a continuous optimization modeling for geometric model fitting with unknown model type and dominant outliers. ii) We propose sparse subspace recovery theory, which is a novel formulation for model reasoning and fitting. iii) We integrate the proximal approximation strategy and sub-gradient descent method into the alternating optimization paradigm, which solves our dual sparsity problem with a convergence rate of O (1 /k 2 ) . iv) Extensive experiments on known/unknown model fitting and two visual applications are designed to validate the superiority of our method.

2 Methodology

This paper aims to explore a valid solution for geometric model reasoning and robust fitting. Before this, we first give a brief review of the widely-used solution for exact model estimation, which helps to derive our concerned unknown model fitting problem.

2.1 Geometry Preliminaries and New Insights

In the community of model fitting, the objective with geometric error is extremely hard to optimize due to the highly non-linear nature. In contrast, if the data are properly normalized, using linearized error ( i.e., algebraic error) to construct objective would show great efficiency to find the optimal solution [18]. For algebraic objective, DLT is known as an efficient method. To be specific, suppose we are given a set of feature correspondences S = { s i = ( p i , p ′ i ) } N i =1 , where p i = ( u i , v i , 1) glyph[latticetop] and p ′ i = ( u ′ i , v ′ i , 1) glyph[latticetop] are column vectors denoting the homogeneous coordinates of feature points extracted from two-view images. Our goal is to recover the underlying geometric structure including Fundamental matrix , Homography and Affine . This two-view problem is essential in 3D vision applications, and is the main focus of this paper.

Fundamental Matrix F ∈ R 3 × 3 describes the entire epipolar geometry p ′glyph[latticetop] i Fp i = 0 . It is suggested to be represented by single equation living in the polynomial space

where is the embedding of correspondence ( p i , p ′ i ) under epipolar constraint, and vec ( F ) ∈ R 9 is the vector form of matrix F in row-major order. N ≥ 8 correspondences can uniquely determine F up to scale.

Homography H ∈ R 3 × 3 describes the pure rotation or plane projection, which claims that p ′ i and Hp i are co-linear, implying [ p ′ i ] × Hp i = 0 , where [ p ′ i ] × is a skew-symmetric matrix. In DLT solution, it usually converts to the following constraint:

where vec ( H ) ∈ R 9 , similarly. And Φ H ( p i , p ′ i ) denotes the homography embedding:

This constraint suggests that, H can be estimated from given at least 4 correspondences.

Affine Matrix A ∈ R 3 × 3 is the degraded case of H , with the last row of A being [0 , 0 , 1] . This model implies a linear transformation p ′ i = Ap i . The affine constraint is also represented as:

where vec ( A ) = [ a 11 , a 12 , a 13 , a 21 , a 22 , a 23 , 1] glyph[latticetop] , and

is the affine embedding. Using Eq. (5), the solution can be given by at least 3 correspondences.

Given sufficient inliers, DLT method provides an efficient solution for geometric model fitting, which converts to solving the following minimal least-square problem:

where M is the data embedding matrix for specific model M , i.e., m i = Φ M ( p i , p ′ i ) with M∈ { F , H , A } , as defined in Eqs. (2), (4) and (6). θ = vec ( M ) ∈ R D indicates the parameter vector of M , and ‖ θ ‖ 2 = 1 restricts it in a sphere thus avoiding the trivial solution. The optima solution θ ∗ is exactly the right singular vector corresponding to the smallest singular value of M . However, DLT only works on the outlier-free case, due to the use of glyph[lscript] 2 norm. Thus, it is usually applied in SAC paradigm by sampling a clean subset to fit a given model. Another practical formulation is using latent variable inside (7) to indicate the inlier, or using truncated or glyph[lscript] 1 loss [18]. In contrast, the glyph[lscript] 1 objective is easier to optimize due to the convex nature by solving

This is actually a subspace learning problem as introduced in DPCP series [45, 52, 13], which can be efficiently solved by a projected sub-gradient method (PSGM) [13]. Besides, [18] gives comprehensive comparisons of above mentioned losses under different model cases. These works also demonstrate the robust property of (8), which roughly states that, the estimation task can even tolerate O ( m 2 ) outliers ( m is the inlier number).

New Insights. Above formulations almost work on knowing the exact information of model type, such that the embedding matrix M is correct, i.e, choosing M from { F , H , A } to generate Φ M . Practically, predefining a correct model is much difficult. First, the camera type and motion, or the scene in shooting is not always known. Besides, we cannot manually define the correct model for each image pair in large-scale vision tasks. In this regard, this paper tends to simultaneously seek the true model and robustly estimate the model parameters, by proposing a general continuous optimization framework with dual sparsity constraints.

2.2 Dual Sparsity Formulation

In this paper, we aim to estimate the model parameters without knowing the model type. That means we should find a common data embedding ˜ Φ( · ) thus avoiding exact Φ M ( · ) for each model. Instead it converts to constructing model embedding Ψ( M ) , to generate a common constraint as:

Eq. (9) can provide great advantages for our model reasoning and fitting task. First, it avoids the requirement for exact model type to construct Φ M ( · ) before estimation. Moreover, the solution would have the form of Ψ( M ) , which directly guides the identification of different models. Eq. (9) essentially relies on the following Sparse Subspace Recovery (SSR) theory.

GLYPH(cmap:d835)

t

o

o

Embedding

Embedding

o

m

m

(b) 2D embedd. & solution of (a)

(c) 3D embedd. & sparse solution of (a)

Inliers

Outliers

Embeded inliers

Embeded outliers

Unit sphere projected points

(a)

Point model:

x-t =0

GLYPH(cmap:d835)

GLYPH(cmap:d835)

= (GLYPH(cmap:d835)

GLYPH(cmap:d835)

, 1, GLYPH(cmap:d835)

GLYPH(cmap:d835)

2

)

T

GLYPH(cmap:d835)

GLYPH(cmap:d835)

= (GLYPH(cmap:d835)

GLYPH(cmap:d835)

, 1)

T

GLYPH(cmap:d835):

GLYPH(cmap:d835) = 1

GLYPH(cmap:d835)

GLYPH(cmap:d835)

= (GLYPH(cmap:d835)

GLYPH(cmap:d835))

GLYPH(cmap:d835)

∗

GLYPH(cmap:d835)

GLYPH(cmap:d835)

= (GLYPH(cmap:d835)

GLYPH(cmap:d835))

GLYPH(cmap:d835)

'

: GLYPH(cmap:d835)

= 1, GLYPH(cmap:d835)

= GLYPH(cmap:d835)

2

GLYPH(cmap:d835):

GLYPH(cmap:d835) = 1

GLYPH(cmap:d835)

∗

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

Theorem 1 (SSR) Geometric model fitting can be seen as a subspace recovery problem represented by the intersection of multiple sparse hyperplanes under an over embedded data space.

To better interpret the proposed Theorem 1, we first give an example on 2 D point set fitting . Given a set of clean 2 D points S = { s i = ( x i , y i ) } N i =1 that are sampled from a structure of line (L), parabola (P) or ellipse (E) , i.e., M ∈ { L, P, E } . The model parameters can be estimated by exact data embedding Φ M ( x i , y i ) with DLT method (7). For ellipse estimation, Φ E ( x i , y i ) = [1 , x i , y i , x 2 i , y 2 i ] glyph[latticetop] , while for a line, Φ L ( x i , y i ) = [1 , x i , y i ] glyph[latticetop] . But in our sparse theory, with a higher-order embedding, such as fixing ˜ Φ( x i , y i ) = Φ E ( x i , y i ) , we can obtain a sparse solution θ = [ a, b, c, 0 , 0] glyph[latticetop] for a line model, and θ = [ a, b, c, d, 0] glyph[latticetop] or [ a, b, c, 0 , e ] glyph[latticetop] for a parabola model. See Fig. 2, or refer to Append.D and Fig. 6 for details. Thus, the SSR task can be formulated as:

where M is the over embedding for input data with m i = ˜ Φ( x i , y i ) = Φ E ( x i , y i ) , and we denote x as a sparse basis that indicates a hyperplane under SSR theory. The glyph[lscript] 0 norm in the objective suggests to use less parameters to fit the given data. Note that, m i can be composed by more polynomials such as x i y i , x 3 i , y 3 i or higher-order forms, but it is not necessary for the model pool { L, P, E } . Problem (10) can be illustrated by a point model fitting x = t as in Fig. 1. The middle plot shows that, the solution x ∗ or -x ∗ is ideally vertical to the embedding vector ˜ m i of inliers. With higher-order embedding [ x i , 1 , x 2 i ] glyph[latticetop] , the solution would be any vector located in the plane P that is vertical to ˜ m i and through the origin. However, with sparsity constraint, the solution reduces to be the intersection of plane P and one of coordinate planes { π xoy , π xoz , π yoz }. Since x 2 i causes higher complexity, the final solution is exactly P ⋂ π xoy , i.e., x ∗ = ± 1 √ 1+ t 2 [1 , -t, 0] glyph[latticetop] .

Denoting G M = ( D,d,r, s ) as the geometry relationship for a model M , where D is the ambient dimension of embedded data, d is the subspace dimension, r denotes the number of basis to be estimated, while s means the minimum sparsity in each basis ( i.e., the least number of zero parameters), the relations for 2D models are G L = (5 , 4 , 1 , 2) , G P = (5 , 4 , 1 , 1) , and G E = (5 , 4 , 1 , 0) . Next, we emphatically introduce the new formulation for two-view models, which is outlined in Append.D .

Extending to two-view geometry , our SSR task can directly benefit from (8), which is a special case in subspace recovery [46, 52, 14], with d = D -1 . In subspace learning theory, the learning problem is significantly easier when the relative dimension d/D is small [17, 18]. For such purpose, and together with our sparse theory, we can obtain the following new formulations. First, for F estimation, since its geometric constraint derives only one single hyperplane living in a 9 -dimensional space, thus Eq. (9) has the same form with Eq. (1). And the geometric relation is clearly G F = (9 , 8 , 1 , 0) .

Proposition 1 Given n ≥ 4 clean correspondences conforming to a homography transformation H , the model estimation can be converted to recovering a subspace with dimension no more than 6, which consists of 3 sparse hyperplanes with sparsity s ≥ 3 under the 9-D embedding ˜ Φ = Φ F ( p i , p ′ i ) .

Proof. Homography model actually derives r = 3 equations, and each of them indicates a sparse hyperplane with ˜ Φ( p i , p ′ i ) glyph[latticetop] Ψ( H ) = 0 , where Ψ( H ) ∈ R 9 × 3 is the embedding of model H , which consists of three sparse intersected hyperplanes ( i.e., orthometric bases) as:

with H = [ h glyph[latticetop] 1 ; h glyph[latticetop] 2 ; h glyph[latticetop] 3 ] . Since Ψ( H ) is of rank 3, the dimension of subspace d ≤ 6 . And the solution for each basis would have at least s = 3 zeros, thus the geometric relation is G H = (9 , 6 , 3 , 3) . glyph[square]

Proposition 2 Given n ≥ 3 clean correspondences conforming to an affine transformation A , the model estimation can be converted to recovering a subspace with dimension no more than 7, which consists of 2 sparse hyperplanes with sparsity s ≥ 5 under the 9-D embedding ˜ Φ = Φ F ( p i , p ′ i ) .

Proof. Eq. (5) derives r = 2 linear equations, and each of them indicates a sparse hyperplane with ˜ Φ( p i , p ′ i ) glyph[latticetop] Ψ( A ) = 0 , where Ψ( A ) ∈ R 9 × 2 is the embedding of model A , which consists of two sparse intersected hyperplanes as:

Because Ψ( A ) is of rank 2, the dimension of subspace is no more than 7. And the solution for each basis would have at least s = 5 zeros, thus the geometric relation is G A = (9 , 7 , 2 , 5) . glyph[square]

The above two propositions have well interpreted Theorem 1. To be specific, H and A estimation can be performed as a hyperplane fitting problem with Eq. (3) and Eq. (5). Alternatively, they can be respectively solved as recovering a 6-dimensional subspace and a 7-dimensional subspace in a common 9-dimensional data embedding space. For the purpose of our unknown model fitting, the advantages of this conversion have been explained in Eq. (9). Besides, the low-relative dimension d/D significantly makes the subspace learning task easier [17, 18]. In the following, we will give a general formulation for unknown model fitting, then explore its efficient solution.

Formulation for Unknown Model Fitting. With no outliers and noise, unknown model fitting can convert to finding all r independent sparse bases X = [ x 1 , x 2 , · · · , x r ] ∈ R D × r by solving

where M is the over embedding matrix of data with m i = ˜ Φ( s i ) . Rank r constraint of X asks for r independent bases. But r is generally unknown in advance, thus its estimation is also necessary.

Proposition 3 Given sufficient inliers with no noise, if the conformed model derives r independent bases, then the solution of SSR satisfies max(rank( X )) = r .

glyph[negationslash]

Proof. Denoting X ∗ = [ x 1 , x 2 , · · · , x r ] ∈ R D × r the optimal solution, any subset of X ∗ is also a feasible solution with rank no more than r . For ∀ x r +1 with ‖ x r +1 ‖ 2 = 1 , ( X ∗ ) glyph[latticetop] x r +1 = 0 , we have M glyph[latticetop] x r +1 = 0 . Hence, x r +1 is not a valid basis, and we can conclude max(rank( X )) = r . glyph[square]

glyph[negationslash]

Proposition 3 suggests that the target of unknown model fitting can be converted to finding a maximum number of orthometric bases with sparsity constraint. Considering the outliers and noise, the ideal constraints can be written as M glyph[latticetop] X -G -E = 0 , where G and E are noise and outlier entries, respectively. Following a common perception, G is assumed to be Gaussian, which suggests to be constricted by Frobenius norm ‖ G ‖ F . While outlier matrix E is sparse in row, and any ‖ E i, : ‖ 2 = 0 indicates that the i -th sample can be considered as an outlier, thus we use glyph[lscript] 2 , 0 to constrain this property. Then our problem is transformed into a multi-objective optimization task, including minimizing the fitting error, the parameter number, the outlier number, and maximizing the basis number. Hence, the general formulation for robust model reasoning and fitting can be written as a unified form:

where I is an identity matrix of size r × r with r = rank( X ) . It restricts that the bases are all orthometric. λ, γ, τ > 0 are hyper-parameters to balance different loss items. Problem (14) ultimately outputs an exact sparse subspace X ∈ R D × r , helping to identify the true model M with X ∼ Ψ( M ) . In the following, we will explore an efficient solution for this multi-objective problem.

2.3 Solution

Problem (14) involves glyph[lscript] 0 minimization, which is the holy grail of sparse approximation. However, glyph[lscript] 0 optimization is NP-hard, and the non-linear objective makes it even more difficult. A practical way is

to use glyph[lscript] 1 norm as the best convex approximation instead. Thus, Problem (14) can be relaxed into:

glyph[negationslash]

Another troublesome issue is to maximize rank( X ) . Consider that Problem (15) actually aims to obtain r orthometric bases [ˆ x 1 , ˆ x 2 , · · · , ˆ x r ] and a sparse outlier matrix ˆ E = [ˆ e 1 , ˆ e 2 , · · · , ˆ e r ] satisfying L ( M , ˆ x i , ˆ e i ) = 1 2 ‖ M glyph[latticetop] ˆ x i -ˆ e i ‖ 2 2 + λ ‖ ˆ x i ‖ 1 + γ ‖ ˆ e i ‖ 1 < τ, ∀ i ∈ { 1 , 2 , · · · , r } . Thus we decompose (15) into progressively estimating a new sparse bases orthometric to all given bases up to L ( M , ˆ x i , ˆ e i ) < τ not holds. For any given bases { B = ( x j ) | j = 1 , 2 , · · · , i -1 } (those have been estimated), a new sparse basis x i ( i > 1) can be estimated by solving

Since there exist two variables ( x , e ) to be estimated, it is impractical to optimize them directly with gradient descent method as used in [52, 13, 17]. A valid way is to alternatively optimize x and e in an iterative pipeline, which encourages us to compute the closed-form solution for ( x , e ) in each iteration. For such purpose, we first calculate the derivatives of objective w.r.t. variables, then make them equal to zero. In the k -th iteration and for given x k -1 , we can easily obtain

where sgn( · ) is a sign function. e k is in fact solved by a standard threshold shrinkage operation T γ introduced in the Lemma of [47]. (Refer to Append.B for details)

Given e k Update x k . Due to the coexistence of quadratic and sparse constraints, it is hard to directly optimize. Fortunately, [7] provides efficient solution for such linear inverse problem with theoretical and practical verification. We exploit its basic idea into our dual sparsity problem, that is to build at each iteration a regularization of the linearized differentiable function, then obtain

where the smallest Lipschitz constraint of the gradient ∇ f ( x ) can be calculated by L ( f ) = λ max ( MM glyph[latticetop] ) , and λ max ( · ) indicates the maximum eigenvalue of a matrix, and we denote

where ∇ f ( x k -1 ) = M ( M glyph[latticetop] x k -1 -e k ) . In this case, Problem (18) is essentially similar to solving e , and we can obtain the update formula of x with x k = T λ L ( q L ( x k -1 )) . ( Proof is in Append.C )

The above solution for x actually reduces to a subgradient method [26], which converges at a rate no worse than O (1 /k ) . However, it was proven in [34, 35] that there exists a gradient method with an O (1 /k 2 ) complexity which is an 'optimal' first order method for smooth problems [7]. The core strategy is that the iterative shrinkage operation performs at the point ˜ x k -1 which uses a specific linear combination of previous two points ( x k -1 , x k -2 ) as ˜ x k -1 = x k -1 + ( t k -1 -1 t k ) ( x k -1 -x k -2 )

Next, we further consider constraints of Problem (16). It can be easily solved by a projection operation [17, 14] in each iteration, including a sphere projection x ← x ‖ x ‖ 2 , and an orthogonal projection x ← ( I -BB glyph[latticetop] ) x , because the projector of orthogonal complement is I -BB glyph[latticetop] .

Eqs. (17) and (20) provide iterative optimization for searching sparse solution of x and e , thus we term the whole algorithm as Dual Sparsity Pursuit (DSP). Repeatedly using DSP, we can successfully address the general unknown model fitting task, and the whole process is concluded in Append.D . Note that, in two-view problem, each model has at least r = 1 basis ( e.g. F ) and at most r = 3 bases ( e.g. H ), thus we denote R as the max number of bases to be estimated, with R = 3 .

Model Reasoning Analysis. A critical problem is how to reason out the model type and estimate the parameters under a general case. Several methods [44, 41] have been early proposed to use a selection strategy [33, 31, 32], but it is not always clear due their ambiguities [37]. This would be well mitigated in our DSP. To be specific, for 2D point set fitting, we can directly distinguish the correct model from { L, P, E } through the sparsity of estimated x , as shown in Fig. 2. As for two-view models, we can also distinguish them via the sparsity and rank of

0

200

400

600

Iteration

1

1.5

2

2.5

3

w/o Acceleration:  O(1/k)

with Acceleration: O(1/k

2

)

0

200

400

600

Iteration

-0.5

0

0.5

1

Parameter value

a

a'

b

b'

c

c'

d

d'

e

e'

estimated X , referring to the model embedding Ψ( M ) or their geometric relations G M ( D,d,r, s ) . The following remark may deliver good understanding of this property, which is illustrated in Fig. 4.

Remark 1 If the true model is F , the solution for ( x 2 , e 2 ) would result in a larger fitting error than ( x 1 , e 1 ) , i.e., ‖ M glyph[latticetop] x 1 ‖ 2 glyph[lessmuch] ‖ M glyph[latticetop] x 2 ‖ 2 . In other words, the number of inliers detected by ( x 2 , e 2 ) would be much smaller than ( x 1 , e 1 ) . Since the second basis is essentially to find a homography or affine structure. On the contrary, if the true model is H or A , the detected inliers between these two solutions would have small difference, as the orthometric and sparse bases indeed exist.

3 Experiment

Implementation Details: (1) Parameter setting. In DSP, λ and γ are two hyper-parameters. Based on [44], we set λ = 0 . 005 log(4 N ) × [1 , 1 , 0 . 5 , 1 , 1 , 0 . 5 , 0 . 5 , 0 . 5 , 0 . 1] glyph[latticetop] as default (for 2D model, we set λ = log(2 N ) × [0 . 01 , 0 . 1 , 0 . 1 , 1 , 1] glyph[latticetop] ). In addition, we set γ = 0 . 06 at the beginning, then update it with 0 . 98 γ for each twenty iterations, and constrain γ min = 0 . 02 . Moreover, we set the max iteration as 2 k , and stop it if ε = ‖ x k -x k -1 ‖ 2 ≤ 1 e -6 . As for τ , it controls the number of estimated basis, i.e., r . We set ξ = L ( M , x i , e i ) / L ( M , x i -1 , e i -1 ) , and describe its distribution on all real data as in Fig. 3. Based on the best ξ , we set τ = 1 . 2 L ( M , x i -1 , e i -1 ) during the estimation of x i . (2)

Optimal  ξ = 1.2

Synthesized Data. We synthesize 300 image pairs for each model, which consist of different outlier ratio (OR = 20% , 50% , 80% ) and noise level (NL = 0 . 2 , 0 . 5 , 0 . 8 , 1 . 0 , 1 . 5 in pixel). (3) Real Image Data. 8 public datasets [8, 2] are used, and we divide them into two groups including Fund : kusvod2 , CPC , TUM , KITTI , T&T ; and Homo : homogr , EVD , Hpatch . (4) Comparisons. 6 methods are used for evaluation, including three SAC methods RNASAC [19], USAC [38], and MAGSAC++ [6]; one globally optimized method EAS [18]; and two deep methods OANet [50], SuperGlue [40]. (5) Metrics . All methods applied the normalized 4-point and 8-point algorithms for H and F estimation (which are applied as post-process [18] in EAS, DSP and two deep methods with 100 iterations). To measure the accuracy, we use Geometrical Error (GE, in pixel), which is defined as the re-projection error and Sampson distance [4, 5] for H and F estimation, respectively. We defined the failed case as that the GE is larger than 5 pixels, or the model is wrongly identified. To avoid the failed cases for a dataset, we use the medium value E med of GE as measurement. (More details are in Append.E ).

Model Reasoning and Robust Fitting Test:

(1) Test on Synthetic Data. In the simulation test, an advanced robust estimator EAS [18] is used to estimate all possible models first. Then, three widely used model selection criteria, i.e., AIC [1], BIC [42] and GRIC [25], are equipped. The accuracy ( % ) of model selection are reported in Tab. 1. AIC can achieve good performance for homography and affine models, but not for epipolar cases, particularly for high OR. BIC shows poor performance in fundamental model selection, since it prefers under-fitting in theory. GRIC fully considers the properties of vision problem performing well in most cases. As for our DSP, it can identify the

geometric models with the best accuracy even in case of high outlier ratio.

(2) Test on Real Image Data. In this part, those datasets are divided into Fund and Homo based on their model type. In fact, during real applications, it is more urgent to make sure whether the true model is F or H [41, 33], thus our used real image data are sufficient to verify the performance for real cases. In addition, considering the competitive performance in Tab. 1, we only use GRIC as comparison in model selection stage. Six representative estimators are used for comparison. That is, first using those robust estimators to obtain all possible models, then using GRIC to select the best one. Qualitative results on two representative image pairs are shown in Fig. 4, which obviously reveal the mechanism of our DSP as we assumed in Remark 1. In addition, our algebra results are somewhat coarse, but they can be well refined by a post-processing [18]. Quantitative results are evaluated in Tab. 2, including E med , Run Time (Time) and failure rate ( FR ). From this ta-

ble, we find that the other estimators typically achieve poor model selection even if using the best criterion GRIC. Three SAC methods get poor model identification, particularly for Homo data, since many image pairs are from extreme view change with high outlier ratios. OANet and SuperGlue similarly achieve low model reasoning accuracy, since some datasets are from complicated scenarios, such as EVD , T&T and CPC , which are not seen by these two deep methods in their training. Their model fitting performances directly affect the model reasoning accuracy. Overall speaking and comparing with GRIC, our DSP reasons out the true model directly from its solutions with much higher accuracy. As for running time, OANet and USAC almost achieve the best efficiency. Because OANet is a deep model with GPU acceleration (SuperGlue is slower since it takes descriptors as input to generate matches), and USAC integrates the local optimization and fast model verification strategy in its universal framework, but sacrifices the accuracy to some extent.

Known Model Estimation Test: Known model estimation is the main focus of current researches, thus we also test on those real image pairs by giving the priori of model type, and provide extensive comparisons. The statistic results are shown in Fig. 5, which only contains four challenging datasets, while others are in Append.E.3 . Statistic results show that the performance of SAC-based methods would heavily degrade with more time consuming and less accuracy on these challenging datasets. This is because they intrinsically have to sample an outlier-free subset to best fit the given model, this is much difficult for these datasets. OANet and SuperGlue achieve poor fitting accuracy due to the same reason particularly for EVD datasets. As for EAS and our DSP, they commonly optimize from global formulation, thus showing similar top accuracy. However, our DSP performs better comparing with EAS, due to the specifical modeling to restrain noise and outliers. And it is much faster, because the use of an acceleration strategy that makes the convergence rate closed to O (1 /k 2 ) .

0

0.2

0.4

0.6

0.8

1

0

2

4

6

RANSAC, 1.82

USAC, 1.14

MAGSAC++, 1.37

EAS, 1.15

OANet, 1.32

SuperGlue, 141.47

Ours, 1.09

0

0.2

0.4

0.6

0.8

1

0

2

4

6

RANSAC, 1.01

USAC, 0.85

MAGSAC++, 0.89

EAS, 0.83

OANet, 0.83

SuperGlue, 0.94

Ours, 0.82

0

0.2

0.4

0.6

0.8

1

T&T; Cumulative Distribution

0

2

4

6

RANSAC, 0.86

USAC, 0.81

MAGSAC++, 0.81

EAS, 0.90

OANet, 0.93

SuperGlue, 1.17

Ours, 0.81

0

0.2

0.4

0.6

0.8

1

CPC; Cumulative Distribution

0

2

4

6

RANSAC, 1.62

USAC, 0.87

MAGSAC++, 0.92

EAS, 0.79

OANet, 0.82

SuperGlue, 1.30

Ours, 0.75

0

0.2

0.4

0.6

0.8

1

10 1

10 2

10 3

10 4

Run Time (ms)

RANSAC, 1.06e+03

USAC, 1.06e+01

MAGSAC++, 5.01e+01

EAS, 1.88e+02

OANet, 1.48e+01

SuperGlue, 5.66e+01

0

0.2

0.4

0.6

0.8

1

10 0

10 1

10 2

10 3

10 4

10 5

Run Time (ms)

RANSAC, 3.79e+03

USAC, 5.14e+01

MAGSAC++, 8.18e+02

EAS, 2.68e+02

OANet, 1.98e+01

SuperGlue, 6.86e+01

Ours, 1.83e+02

0

0.2

0.4

0.6

0.8

1

T&T; Cumulative Distribution

10 2

10 3

10 4

10 5

Run Time (ms)

RANSAC, 9.87e+02

USAC, 2.85e+01

MAGSAC++, 4.16e+02

EAS, 9.03e+01

OANet, 1.16e+01

SuperGlue, 1.04e+02

Ours, 4.28e+01

0

0.2

0.4

0.6

0.8

1

CPC; Cumulative Distribution

10 2

10 3

10 4

10 5

10 6

Run Time (ms)

RANSAC, 1.46e+03

USAC, 2.46e+01

MAGSAC++, 2.85e+02

EAS, 8.11e+01

OANet, 1.16e+01

SuperGlue, 9.41e+01

Ours, 2.16e+01

Applications: (1) Multi-model Fitting. We first consider the multi-class multimodel fitting problem. Unlike multimodel fitting of single type, it would be more challenging if the data contain multiple models with unknown model type and model number. To this end, a public benchmark AdelaideRFM is used for evaluation. Quantitative results are re-

ported in Tab. 3, where Misclassification Error (ME, % ) is selected to characterize the performance. Six algorithms including T-link [29], RCMSA [36], RPA [30], MCT [31], MLink [32] and RFMSCAN [23], are used for comparison. The table shows that our DSP can achieve the best accuracy with low computational burden.

(2) Loop Closure Detection. We next exploit 2 sequences from KITTI vision suite [20] ( i.e. , K00, K02 ) to evaluate our method in the loop closure detection (LCD) task, i.e. , recognizing re-observations during the navigation of a robot. These two sequences have enough loop closures. Because DELG [10] is considered state-of-the-art

for LCD-related tasks such as image retrieval and landmark recognition, we choose it to simultaneously extract global and local features of images to perform LCD in a hierarchical way. Specifically, given a query image, the similarity of global features under L2-metric is first used to select its candidate frame, followed by performing model fitting between them based on local features. Only when sufficient matches are preserved would a loop-closing event be triggered. Results are reported in Tab. 4, which shows that our DSP achieves the best, since our dual sparsity constraints can well address navigation of the degraded cases such as only going ahead or making a turn.

4 Conclusion

In this paper, we proposed a unified modeling and fast optimization paradigm named DSP for robust model fitting with unknown model type and heavy outliers. It enables to use as few parameters as possible to explain the inlier structure, thus achieving true model identification and parameter estimation. We introduced an alternating optimization strategy together with proximal approximation method to accurately estimate the sparse model parameters and outlier entries. In addition, our proposed method is validated to exactly disambiguate the true model on both synthetic and real data. Limitation: Our proposed method can only perform for single model, but real data often contains multiple different models, such as point cloud analysis. In the future, we will integrate the multi-model fitting problem into our modeling, thus achieving a four-fold task in a unified optimization.

Broader Impact

The proposed method enjoys great potential to improve a wide range of industrial applications including image registration and retrieval, camera pose estimation and localisation, 3D reconstruction, etc. To be specific, the proposed method achieves significant accuracy gains of model reasoning and fitting on challenging matching benchmarks, which strongly indicates that it will directly benefit many related fields including robotics and autonomous driving, where the geometric model estimation is the foundation. However, our method does have some unwelcome repercussions like many other machine learning techniques. The accurate and robust model reasoning and estimation can be illegally used for a person or property without permission. It may even be weaponized to guide a UAV to perform self-location then carry out a terrorism attack. But these negative impacts are more related to the fields of application rather than the technology itself. And we believe, under strict supervision, that our work will bring more benefits than harms to society.

Acknowledgments and Disclosure of Funding

Funding in direct support of this work: NSFC grant 62276192.

References

Hirotogu Akaike. Information theory and an extension of the maximum likelihood principle. In Selected Papers of Hirotugu Akaike , pages 199-213. 1998.

Vassileios Balntas, Karel Lenc, Andrea Vedaldi, and Krystian Mikolajczyk. Hpatches: A benchmark and evaluation of handcrafted and learned local descriptors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5173-5182, 2017.

Daniel Barath and Jiˇ rí Matas. Graph-cut ransac. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6733-6741, 2018.

Daniel Barath, Jiri Matas, and Jana Noskova. Magsac: marginalizing sample consensus. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 10197-10205, 2019.

Daniel Barath, Jana Noskova, Maksym Ivashechkin, and Jiri Matas. Magsac++, a fast, reliable and accurate robust estimator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1304-1312, 2020.

Daniel Barath, Jana Noskova, and Jiri Matas. Marginalizing sample consensus. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(11):8420-8432, 2022.

Amir Beck and Marc Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM Journal on Imaging Sciences , 2(1):183-202, 2009.

Jia-Wang Bian, Yu-Huan Wu, Ji Zhao, Yun Liu, Le Zhang, Ming-Ming Cheng, and Ian Reid. An evaluation of feature matchers for fundamental matrix estimation. In Proceedings of the British Machine Vision Conference , 2019.

Carlos Campos, Richard Elvira, Juan J Gómez Rodríguez, José MM Montiel, and Juan D Tardós. Orbslam3: An accurate open-source library for visual, visual-inertial, and multimap slam. IEEE Transactions on Robotics , 37(6):1874-1890, 2021.

Bingyi Cao, Andre Araujo, and Jack Sim. Unifying deep local and global features for image search. In Proceedings of the European Conference on Computer Vision , pages 726-743, 2020.

Ondrej Chum, Tomas Werner, and Jiri Matas. Two-view geometry estimation unaffected by a dominant plane. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 772-779, 2005.

Tianjiao Ding, Yunchen Yang, Zhihui Zhu, Daniel P Robinson, René Vidal, Laurent Kneip, and Manolis C Tsakiris. Robust homography estimation via dual principal component pursuit. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6080-6089, 2020.

Tianyu Ding, Zhihui Zhu, Tianjiao Ding, Yunchen Yang, Rene Vidal, Manolis Tsakiris, Daniel P Robinson, et al. Noisy dual principal component pursuit. In Proceedings of the International Conference on Machine Learning , page 1617-1625, 2019.

Tianyu Ding, Zhihui Zhu, René Vidal, and Daniel P Robinson. Dual principal component pursuit for robust subspace learning: Theory and algorithms for a holistic approach. In Proceedings of the International Conference on Machine Learning , pages 2739-2748, 2021.

Ehsan Elhamifar and René Vidal. Sparse subspace clustering: Algorithm, theory, and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence , 35(11):2765-2781, 2013.

Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. Density-based spatial clustering of applications with noise. In Proceedings of the International Conference on Knowledge Discovery and Data Mining , volume 240, page 6, 1996.

Aoxiang Fan, Xingyu Jiang, Yang Wang, Junjun Jiang, and Jiayi Ma. Geometric estimation via robust subspace recovery. In Proceedings of the European Conference on Computer Vision , pages 462-478, 2020.

Aoxiang Fan, Jiayi Ma, Xingyu Jiang, and Haibin Ling. Efficient deterministic search with robust loss functions for geometric model fitting. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(11):8212-8229, 2022.

Martin A Fischler and Robert C Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM , 24(6):381-395, 1981.

Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research , 32(11):1231-1237, 2013.

Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision . Cambridge University Press, 2003.

Maksym Ivashechkin, Daniel Barath, and Jiˇ rí Matas. Vsac: Efficient and accurate estimator for h and f. In Proceedings of the IEEE International Conference on Computer Vision , pages 15243-15252, 2021.

Xingyu Jiang, Jiayi Ma, Junjun Jiang, and Xiaojie Guo. Robust feature matching using spatial clustering with heavy outliers. IEEE Transactions on Image Processing , 29:736-746, 2020.

Xingyu Jiang, Jiayi Ma, Guobao Xiao, Zhenfeng Shao, and Xiaojie Guo. A review of multimodal image matching: Methods and applications. Information Fusion , 73:22-71, 2021.

Ken-ichi Kanatani. Motion segmentation by subspace separation and model selection. In Proceedings of the IEEE International Conference on Computer Vision , pages 586-591, 2001.

Evgeny S Levitin and Boris T Polyak. Constrained minimization methods. USSR Computational Mathematics and Mathematical Physics , 6(5):1-50, 1966.

Yifan Lu, Jiayi Ma, Leyuan Fang, Xin Tian, and Junjun Jiang. Robust and scalable gaussian process regression and its applications. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 21950-21959, 2023.

Jiayi Ma, Xingyu Jiang, Aoxiang Fan, Junjun Jiang, and Junchi Yan. Image matching from handcrafted to deep features: A survey. International Journal of Computer Vision , 129(1):23-79, 2021.

Luca Magri and Andrea Fusiello. T-linkage: A continuous relaxation of j-linkage for multi-model fitting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3954-3961, 2014.

Luca Magri and Andrea Fusiello. Robust multiple model fitting with preference analysis and low-rank approximation. In Proceedings of the British Machine Vision Conference , 2015.

Luca Magri and Andrea Fusiello. Fitting multiple heterogeneous models by multi-class cascaded t-linkage. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 7460-7468, 2019.

Luca Magri, Filippo Leveni, and Giacomo Boracchi. Multilink: Multi-class structure recovery via agglomerative clustering and model selection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1853-1862, 2021.

Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. Orb-slam: a versatile and accurate monocular slam system. IEEE Transactions on Robotics , 31(5):1147-1163, 2015.

Arkadij Semenoviˇ c Nemirovskij and David Borisovich Yudin. Problem complexity and method efficiency in optimization . Wiley-Interscience Series in Discrete Mathematics, John Wiley & Sons, New York, 1983.

Yurii Evgen'evich Nesterov. A method of solving a convex programming problem with convergence rate o(kˆ2). In Doklady Akademii Nauk , volume 269, pages 543-547, 1983.

Trung T Pham, Tat-Jun Chin, Jin Yu, and David Suter. The random cluster model for robust geometric fitting. IEEE Transactions on Pattern Analysis and Machine Intelligence , 36(8):1658-1671, 2014.

Thomas Probst, Ajad Chhatkuli, Danda Pani Paudel, and Luc Van Gool. What correspondences reveal about unknown camera and motion models? In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 8730-8738, 2019.

Rahul Raguram, Ondrej Chum, Marc Pollefeys, Jiri Matas, and Jan-Michael Frahm. Usac: a universal framework for random sample consensus. IEEE Transactions on Pattern Analysis and Machine Intelligence , 35(8):2022-2038, 2012.

Muhamad Risqi U Saputra, Andrew Markham, and Niki Trigoni. Visual slam and structure from motion in dynamic environments: A survey. ACM Computing Surveys , 51(2):1-36, 2018.

Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4938-4947, 2020.

Konrad Schindler and David Suter. Two-view multibody structure-and-motion with outliers through model selection. IEEE Transactions on Pattern Analysis and Machine Intelligence , 28(6):983-995, 2006.

Gideon Schwarz. Estimating the dimension of a model. The Annals of Statistics , 6(2):461-464, 1978.

Weiwei Sun, Wei Jiang, Eduard Trulls, Andrea Tagliasacchi, and Kwang Moo Yi. Acne: Attentive context normalization for robust permutation-equivariant learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 11286-11295, 2020.

Philip HS Torr. Geometric motion segmentation and model selection. Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences , 356(1740): 1321-1340, 1998.

Manolis C Tsakiris and René Vidal. Dual principal component pursuit. In Proceedings of the IEEE International Conference on Computer Vision Workshops , pages 10-18, 2015.

Manolis C Tsakiris and René Vidal. Dual principal component pursuit. Journal of Machine Learning Research , 19:1-49, 2018.

René Vidal and Paolo Favaro. Low rank subspace clustering (lrsc). Pattern Recognition Letters , 43:47-61, 2014.

Yong Wang, Yuan Jiang, Yi Wu, and Zhi-Hua Zhou. Spectral clustering on multiple manifolds. IEEE Transactions on Neural Networks , 22(7):1149-1161, 2011.

Kwang Moo Yi, Eduard Trulls, Yuki Ono, Vincent Lepetit, Mathieu Salzmann, and Pascal Fua. Learning to find good correspondences. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2666-2674, 2018.

Jiahui Zhang, Dawei Sun, Zixin Luo, Anbang Yao, Lei Zhou, Tianwei Shen, Yurong Chen, Long Quan, and Hongen Liao. Learning two-view correspondences and geometry using order-aware network. In Proceedings of the IEEE International Conference on Computer Vision , pages 5845-5854, 2019.

Shihua Zhang and Jiayi Ma. Convmatch: Rethinking network design for two-view correspondence learning. In Proc. AAAI Conf. Artif. Intell , pages 1-12, 2023.

Zhihui Zhu, Yifan Wang, Daniel Robinson, Daniel Naiman, Rene Vidal, Manolis C Tsakiris, et al. Dual principal component pursuit: Improved analysis and efficient algorithms. In Advances in Neural Information Processing Systems , pages 2171-2181, 2018.

Appendices

We provide necessary proof, pseudo code, additional analysis of our method, and more details about the implementation, the used datasets and additional experimental results.

A Notaions

We denote matrices with bold upper case letters and their elements with double-indexed lower case letters: X = ( x ij ) . We write the vector in bold lowercase as x = ( x i ) . Thus, the matrix can be also termed as X = ( x i ) or ([ X ] i, : ) , where [ X ] i, : means the i -th row of matrix X . We also denote X as the set-form of matrix X ∈ R M × N , which consists of all column vectors of X , i.e., X = { x j } N j =1 . The inlier product of two vectors x , y ∈ R n is denoted as 〈 x , y 〉 = x glyph[latticetop] y . For a matrix X , the maximum eigenvalue is denoted by λ max ( X ) , and its Frobenius norm is defined as ‖ X ‖ F = √∑ i ∑ j | x ij | 2 . For a vector x , ‖ x ‖ 2 (or ‖ x ‖ ) denotes the Euclidean norm ‖ x ‖ 2 = √∑ i | x i | 2 , and its glyph[lscript] 0 norm ( i.e., ‖ x ‖ 0 ) is to compute the number of nonzero entries in vector x , while the glyph[lscript] 1 of x is defined as ‖ x ‖ 1 = ∑ i | x i | .

B Threshold Shrinkage Method

Threshold shrinkage method is typically used to solve glyph[lscript] 1 norm problem, which is introduced by the Lemma from [47].

Lemma 1 Let q be a given vector, for problem:

if the optimal solution is x ∗ , then the i -th element of x ∗ can be calculated as:

where T λ ( · ) is a threshold shrinkage operation and defined as:

C Proof of Solution

To solve the following linear inverse problem [7]:

where f, g are convex functions with satisfying the following assumptions:

f : R n → R is a smooth convex function, which is continuously differentiable with Lipschitz continuous gradient L ( f ) :

where L ( f ) is the Lipschitz constant of ∇ f ( x ) .

g : R n → R is a continuous convex function which is generally nonsmooth, such as a sparse constraint term for our problem.

glyph[negationslash]

The extended problem (25) is solvable, i.e., x ∗ := arg min F = ∅ .

we can extend it into:

In our subproblem, f ( x ) = 1 2 ‖ M glyph[latticetop] x -e ‖ 2 2 , g ( x ) = λ ‖ x ‖ 1 . The smallest Lipschitz constraint of the gradient ∇ f ( x ) can be calculated as

For any L > 0 , f ( x ) can be approximated at a given point y with the following form:

By ignoring constant terms related to y , and let y = x k -1 , we can obtain the basic step in each iteration for problem (24):

D Pseudo Code of Our DSP Method

We conclude the pseudo code of the implementation of our DSP method in Alg. 1 and Alg. 2. For any given data, we can directly reason out the true model from the solution of our DSP. To this end, the common embedding and formulation is outlined in Tab. 5. If we know the exact information of true model type, we can also apply our DSP for inlier seeking and model estimation, by giving exact data embedding and basis number r , as presented in the middle column of Tab. 5.

Take 2D points fitting as example, i.e., the true model is Line, Parabola, and Ellipse, as shown in Fig. 6. In each case, the inliers are contaminated by noise of 0.01, and the inlier and outlier number are 300 and 100, respectively. We first construct the common data embedding as ˜ Φ( x i , y i ) = Φ E ( x i , y i ) = [1 , x i , y i , x 2 i , y 2 i ] glyph[latticetop] , and the sparse basis or parameter vector is x = [ a, b, c, d, e ] glyph[latticetop] . We set the max basis number R = 1 , λ = log(2 N ) × [0 . 01 , 0 . 1 , 0 . 1 , 1 , 1] glyph[latticetop] . The optimization procedure and fitting results are shown in Fig. 6. Specifically, the estimation results for these three cases are x = [ -0 . 0845 , 0 . 7135 , -0 . 6956 , 0 , 0] glyph[latticetop] , x = [ -0 . 4219 , -0 . 1068 , -0 . 4291 , 0 . 7915 , 0] glyph[latticetop] and x = [ -0 . 37825 , 0 . 1498 , 0 . 0641 , 0 . 6513 , 0 . 6373] glyph[latticetop] , respectively.

E Experiment

E.1 Implementation Details

We implement the proposed DSP method with MATLAB code. All comparative methods are implemented with the code provided by their authors. Their parameters are set as the original papers suggested, or referring to [6]. The experiments of RANSAC [19], EAS [18] 2 and our DSP are conducted on a desktop with 4.0 GHz Intel Core i7-6700K CPU and 16GB memory. USAC [38] 3 , MAGSAC++ [6] 4 , OANet [50] 5 and SuperGlue [40] 6 are performed on a Server and Ubuntu 16.04. And two deep learning methods are accelerated by NVIDIA TITAN V GPUs. In addition, AIC [1], BIC [42] and GRIC [25] are used for performing unknown model fitting. Since they are classical model selection criteria, wherein GRIC is still used as a state-of-the-art method in vision tasks [39, 31, 32]. We perform these three criteria with the core parameters suggested as in [31, 32], then equip them with robust model estimators to identify true models.

3

3.5

1.6

3

2.5

1.4

2.5

2

2

1.5

1.2

1.5

1

1

0

50

100

150

200

0

100

200

300

0

100

200

300

Iteration

Iteration

Iteration

1

1

abcde

poqe

0.8

Parameter value

0.5

bcd

Parametervalue

Parametervalue

0.6

0.5

0.4

e

0

0.2

0

0

S0-

-0.2

0

50

100

150

200

-0.5

100

200

300

-0.4

0

100

200

300

Iteration

Iteration

Iteration

E.2 Details of Datasets

Synthesized Data. To perform the robustness test and model reasoning comparisons, we synthesize 900 image pairs with different outlier ratios ( i.e., OR = 20% , 50% , 80% ) and noise levels ( i.e., NL = 0 . 2 , 0 . 5 , 0 . 8 , 1 . 0 , 1 . 5 in pixel). To be specific, these synthetic image pairs are randomly generated by two 3 × 4 camera matrices P 1 = K 1 [ I 3 × 3 | 0 ] and P 2 = K 2 [ R 2 | -R 2 t 2 ] . Camera P 1 is located in the origin and its image plane is parallel to XY plane. The position of camera P 2 is generated by R 2 , t 2 , wherein the position of P 2 is at a random point inside a unit-sphere around P 1 , thus | t 2 | ≤ 1 . And the orientation R 2 = R X,α R Y,β R Z,γ is determined by three 3D rotation matrices rotating around axes X,Y and Z by α, β and γ , respectively ( α, β, γ ∈ [ -π/ 3 , π/ 3] ). We set K 1 = K 2 with focal length f x = f y = 600 and principal points [300 , 300] glyph[latticetop] . To generate fundamental data, we randomly set R 2 and t 2 , and create n i positions ( x, y, z ) in real world with x, y ∈ [ -1 , 1] , z ∈ [3 , 8] , which are further projected onto the first and second images by P 1 , P 2 . In this case, 20 fundamental image pairs are synthesized for each OR and NL, which creates in total 300 fundamental image pairs. As for generating homography data, we know that homography transformation is typically derived from two types of degraded cases, i.e., the target or scene in shooting is approximated a plane, or the camera is only rotated around optical center without any translation. Therefore and for the first homography case, we randomly simulate n i positions ( x, y, z ) in real world with x, y ∈ [ -1 , 1] , z ∈ [9 . 95 , 10 . 05] . While for another case, we just set t 2 = 0 3 × 1 on the basis of synthesizing fundamental data. 10 image data are respectively synthesized for these two homography types for each OR and NL, which also creates in total 300 image pairs. As for generating affine data, we directly set the last row of both P 1 and P 2 as [0 , 0 , 0 , 1] , thus making them satisfy the property of affine camera. For each OR and NL, we similarly synthesize 20 affine data, which creates in total 300 affine image pairs finally. For each image pair, we set the number of putative matches as N = n i + n o = 1000 , that means it consists of n i = 200 inliers and n o = 800 outliers if OR = 80% .

Real Image Data. 8 public datasets are used for test, including kusvod2 , homogr 7 , EVD 8 , Hpatch 9 , and the Feature Matching benchmark 10 that contains TUM , KITTI , T&T and CPC datasets. The detailed information is concluded in Tab. 6. In particular, kusvod2 , homogr and EVD provide putative matches and the inlier indexes. While the rest datasets only provide ground truth model parameters, thus

we use VLFEAT toolbox 11 to construct SIFT matches with ratio test 1.5, and the inlier matches are identified as the geometric error less than 2 pixels.

E.3 Additional Results

Quantitative Results of Other Four Datasets. We first give additional results of the known model fitting on the rest four datasets, including homogr , kusvod2 , KITTI and TUM . Details are in Fig. 7, which show the same trends as in other four datasets (please see the main body), that our proposed DSP achieves the best fitting accuracy and promising efficiency, followed by EAS and SAC methods.

0

0.2

0.4

0.6

0.8

1

homogr; Cumulative Distribution

0

2

4

6

RANSAC, 0.80

USAC, 0.81

MAGSAC++, 0.80

EAS, 0.77

OANet, 0.79

SuperGlue, 1.31

Ours, 0.77

0

0.2

0.4

0.6

0.8

1

kusvod2; Cumulative Distribution

0

2

4

6

RANSAC, 0.59

USAC, 0.49

MAGSAC++, 0.41

EAS, 0.44

OANet, 0.61

SuperGlue, 2.06

Ours, 0.42

0

0.2

0.4

0.6

0.8

1

KITTI; Cumulative Distribution

0

1

2

3

RANSAC, 0.43

USAC, 0.44

MAGSAC++, 0.41

EAS, 0.41

OANet, 0.42

SuperGlue, 0.65

Ours, 0.34

0

0.2

0.4

0.6

0.8

1

TUM; Cumulative Distribution

0

2

4

RANSAC, 0.53

USAC, 0.68

MAGSAC++, 0.64

EAS, 0.49

OANet, 0.51

SuperGlue, 0.57

Ours, 0.47

0

0.2

0.4

0.6

0.8

1

homogr; Cumulative Distribution

10 1

10 2

10 3

10 4

Run Time (ms)

RANSAC, 1.25e+02

USAC, 1.04e+01

MAGSAC++, 4.93e+01

EAS, 1.83e+02

OANet, 1.51e+01

SuperGlue, 8.46e+01

Ours, 1.04e+02

0

0.2

0.4

0.6

0.8

1

kusvod2; Cumulative Distribution

10 1

10 2

10 3

10 4

Run Time (ms)

RANSAC, 1.28e+01

USAC, 1.08e+01

MAGSAC++, 8.77e+01

EAS, 1.09e+02

OANet, 1.39e+01

SuperGlue, 5.71e+01

Ours, 3.01e+01

0

0.2

0.4

0.6

0.8

1

KITTI; Cumulative Distribution

10 1

10 2

10 3

10 4

Run Time (ms)

RANSAC, 3.41e+01

USAC, 2.07e+01

MAGSAC++, 1.96e+02

EAS, 8.59e+01

OANet, 1.17e+01

SuperGlue, 4.95e+01

Ours, 2.10e+01

0

0.2

0.4

0.6

0.8

1

TUM; Cumulative Distribution

10 2

10 3

10 4

Run Time (ms)

RANSAC, 1.93e+02

USAC, 1.99e+01

MAGSAC++, 1.18e+02

EAS, 7.99e+01

OANet, 1.15e+01

SuperGlue, 4.24e+01

Ours, 1.85e+01

E.4 Applications

Multi-class Multi-model Fitting. In this part, we apply our DSP to solving multi-class multimodel fitting problem. In multi-model fitting of single type, the multiple structures belong to the same model type, such as being only line or circle, thus making it convenient for predefining the model type then conducting fitting algorithm. But this would be troublesome if the data contains multiple geometrical models of different types, such as simultaneously existing line, parabola, and circle, or both fundamental and homography model in two-view geometry. Existing strategy commonly borrows selection criterion like GRIC, which usually requires a long runtime [31, 32]. Next, we tend to demonstrate the superiority of our DSP in terms of simultaneously model reasoning and fitting in the task of multimodel fitting problem.

We first test DSP on synthesized point data, which consists of three inlier structures including line, parabola and ellipse together with a large number of outliers. The target is to classify each inlier structure and identify those outliers. Several manifold subspace clustering and multi-model fitting

(a) SMMC (17.23s)

(b) DBSCAN (0.26s)

(c) DBSCAN+SSC(26.41s)

(d) DBSCAN+SMMC (10.17s)

(e) MultiLink (100.67s)

(f) Ours (0.65s)

Cluster1

Outliers

Outliers

Cluster2

Cluster1

Cluster1

Cluster3

Cluster2

Cluster2

Cluster3

Cluster3

Cluster4

Cluster5

Cluster6

Outliers

Outliers

Outliers

Cluster1

Cluster2

Cluster1

Cluster2

Cluster2

Cluster1

Cluster3

Cluster3

Cluster3

methods are used for comparison, such as spectral clustering on multiple manifolds (SMMC) [48], sparse subspace clustering (SSC) [15], density-based clustering (DBSCAN) [16], and a recently published preference-based method termed MultiLink [32] that uses GRIC for model selection.

Clustering results are shown in Fig. 8, we can find that our proposed DSP achieves the best accuracy by combining with DBSCAN. Since our DSP is merely designed for unknown single model fitting, which encourages us to integrate multi-model problem into our outlier rejection, model selection and fitting framework and achieve joint optimization in the future. MultiLink can well classify these inlier structures as well, but several outliers that are closed to inliers are wrongly classified due to its fixed thresholds. Since MultiLink asks for repeat estimations for all possible models, and uses agglomerative clustering as final output, it may consume a huge runtime (100 seconds for 2300 points). In addition, DBSCAN can identify several potential inlier structures and filter out the most outliers with a short runtime. But the performance of DBSCAN is much sensitive to its core parameters. In other words, a loose parameter would

easily classify neighboring outliers as inlier structure, while a strict setting prefers to classify one complete structure into several small clusters, as shown in subfigure (b). In theory, SMMC and SSC can not handle outliers, and failed in clustering the data of a high number of outliers, which can be seen in subfigure(a). In this regard, we combine the advantages of DBSCAN and some clustering methods. We find that SMMC can well refine the results of DBSCAN to be more accurate, but SSC can only work for liner structures such as line or plane, thus fails in our synthesized data.

We also extend our method on real multi-model fitting data, e.g. AdelaideRFM 12 . Fig. 9 shows some qualitative results with three structures and four structures, which demonstrates that our DSP can accurately cluster the corrupted data into several inlier structures together with an outlier cluster. Quantitative results are reported in the main body manuscript.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is about model fitting. The authors consider a scenario in which it is unknown whether correspondences between points in 2 images stem from 3D points that are (1) generally distributed, (2) lie on a plane, or (3) lie on a plane and the motion between the 2 images is not projective but affine. The latter implies that not a pinhole camera was used for projection but an affine camera. This can hold if, for instance, the distance between camera and scene is large compared to the depth variation within the scene.\n\nCase (1) amounts to the well-known fundamental matrix constraint x’^T*F*x=0 with (x, x’) being corresponding 2D homogeneous points. Here, F is a 3x3 matrix with rank(F)=2. For noise-free data, matrix F can be estimated from at least 8 point correspondences since F is determined only up to scale and each point-point correspondence yields a single equation.\nCase (2) amounts to a 3x3 homography H. Differently from the F-matrix, homographies have full-rank, in general. As fundamental matrices, they are only determined up to scale. Since each point-point correspondence yields 2 equations, H can be determined from at least 4 correspondences.\nFor a general affine transformation between the two images, case (3) implies that the last row of H equals [0,0,1]. There are no constraints on the remaining 6 entries, in general.\n\nThe authors show that by expressing cases (2) and (3) as fitting multiple subspaces, all three cases can be expressed by the same model. They propose a projected coordinate descent type of algorithm to estimate.\n\nSTRENGTHS:\nTo me, this interpretation of F/H-matrix geometry is new. I cannot say whether this naturally arises from classical theorems. The fact that all papers I am aware of fit either an F-matrix or a homography can be taken as evidence that at least this fact is not generally known.\n\nWEAKNESSES:\n- proposition 2 is wrong\nMatrix Psi has rank 2 which can be seen from the fact that the third row is linearity dependent on the first two. Hence, G_H=(9,7,2,3).\n\nQUESTIONS:\nPlease correct proposition 2.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the geometric model fitting problem with unknown model type and heavy outliers. It proposes a unified optimization model with dual sparsity constraints that combines the outlier rejection, true model reasoning and parameter selection. Moreover, a fast numerical algorithm is proposed to solve the approximate and dimension-reduced model via separability of the objective function. Numerical experiments on synthetic and real data sets are conducted to compare the performance of the proposed method with other related works. Overall, the paper is complete from theory, algorithm, to experiments with a section of broader impact.\n\nSTRENGTHS:\n1. The proposed dual-sparsity optimization model has a certain novelty. \n2. A variety of numerical results are presented to justify the proposed effectiveness.\n\nWEAKNESSES:\n1. The existence and uniqueness of the proposed model are not discussed in detail. \n2. The description of formulation for the proposed model is not fully concise and precise, especially the treatment of the rank-term. \n3. Model sensitivity and robustness could be further discussed from the theoretical aspect.\n\nQUESTIONS:\n1. Even with the Proposition 3, model (13) is not equivalent to model (14) which is later solved inexactly. Why was the convex relaxation  applied to (14) rather than (13)? Would that cause loss of accuracy especially the low rankness?\n2. How about the uniqueness of the solutions for the proposed model?\n3. Convergence analysis for the proposed algorithm could be provided.\n\nLIMITATIONS:\nThe proposed work aims to avoid parameter selection by introducing a variable of parameters in the unified model. However, it leads to the introduction of other regularization parameters associated with the sparsity terms in the objective function, which again may need fine tuning. Some practical guidance for diverse data sets and application settings should be highlighted.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the robust model fitting problem in the presence of outliers, which is a fundamental problem in low-level CV. The aim is to simultaneously achieve outlier rejection, model selection, and model parameter estimation in a unified formulation. Toward this end, the authors propose to cast the joint outlier rejection, model selection, and model estimation problem into a sparse subspace recovery problem, which can cover the widely used projective transformation models for multi-view gemometry such as the fundamental, homography and affine models. The joint optimization formulation is solved by an alternating algorithm with the use of proximal approximation computation and acceleration. Experimental results on synthetic and real-word data have been provided to deminstrate the performance of the new method, including fundamental matrix and homography estimation, as well as a loop closure application.\n\nSTRENGTHS:\nThis work is well motivated to joitnly achieve outlier rejection, model selection, and model estimation in a unified formulation. The proposed sparse subspace recovery formulation covers the widely used projective transformation models for multi-view gemometry, such as the fundamental, homography and affine models. The method has been evaluated on both synthetic and real-world data via various experiments.\n\nWEAKNESSES:\n1. The formulation only applies to algebraic error model, i.e., model estimation with algebraic distance. While algebraic distance is convenient due to its linearity, geometric distance is geometrically or statistically meaningful and can yiled better performance in projective transformation estimation over the algebraic distance. This has been demonstrated in multi-view geomety.\n\n2. Wile the proposed method is well conceived, it would not outperform the simple method that first estimates the model parameters of each candidate models and then selects the best model in terms of the fitting error. Although the provided experiments show that it outperforms AIC, BIC, GRIC selection methods, in practical applications the model is typically selected in terms of a score of the estimated model computed based on the symmetric transfer errors when the ground-truth transformation is unknown. This selection method is typically used in practice.\n\n3. The proposed algorithm has several hyper-parameters, e.g., $\\gamma$, $\\lambda$, $r$, and $\\tau$. Its performance depends on the tunning of these hyper-parameters, which diminishes its potential advantage over the simple model selection method based on the symmetric transfer errors.\n\nQUESTIONS:\nIt is claimed that the objective with geometric error is extremely hard to optimize due to the highly non-linear nature, but in fact there exist well developed methods in the literature of multi-view geomety that can solve the formualtion with geometric error efficiently and effectively.\n\nLIMITATIONS:\nThe authors have not addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nConsidering that existing model estimation methods highly rely on the correct definition of model types, this paper introduces a unified optimization modeling DSP to simultaneously reason out the model type and estimate model parameters from contaminated data. For such purpose, the authors proposed Sparse Subspace Recovering (SSR) theory to modeling geometric model estimation task, that is to search a maximum of independent sparse bases under an over-embedded data space. The authors also introduced a fast and robust solver to estimate the sparse subspace parameters and error entries, and validated the advanced performance of their method on both unknown and known model estimation, and two applications.\n\nSTRENGTHS:\n- This paper is well written and organized, and easy to follow.\n- The motivation, theory, formulation and solution are good contributions for the model fitting topic.\n- The authors solved the geometric model estimation problem from a novel perspective, that considers the model reasoning task additionally. Particularly, the authors introduced Sparse Subspace Recovering theory and formulated the unknown model fitting task into a continuous optimization objective, and explored efficient solution for it.\n- The authors designed unknown model fitting experiments, and also evaluated their method on common exact model fitting task, which is reasonable and credible. The experiments are convincing, and the results show great superiority comparing to the SOTA.\n\nWEAKNESSES:\n- Line 74, the authors claimed that if the data are properly normalized, using algebraic error is good. But how to assure this property of input data?\n- This paper proposes SSR theory, but the authors have not given mathematical explanation or proof.\n- As for the solution, why not using ADMM to solve this problem, I think ADMM is a common choice for solving L1 norm problem with Lagrange multiplier, as sparse subspace clustering (SSC) used. Please explain.\n- Line 52: How to understand ``insufficient information’’ that the GRIC used, which may cause wrong model selection for constrained motions. What are the advantages of this method comparing with those model selection criteria.\n- If using DSP or SSR theory to estimate a Fundamental matrix F, how to use the rank 2 constraint of F? That is an intrinsic property of Fundamental matrix.\n\nTypos:\n- Line 60, add citation for PSGM when it first appears.\n- Lines 163 and 171, ``sparse independent hyperplanes``, I think it is better to use ``sparse intersected hyperplanes`` or ``sparse independent bases``.\n- Line 170, $\\mathcal{R}$ should be $\\mathbb{R}$ to indicate the real number space.\n- Line 226, $e_k$ --> $e^k$.\n- Line 239, ``can successfully addressing`` --> can successfully address.\n- Line 271, ``300 image pairs of each model`` --> ``300 image pairs for each model``.\n\nQUESTIONS:\n- Explain how to ensure the input data properly normalized.\n- Please provide mathematical explanation or proof for your SSR theory.\n- Please explain or validate the necessary of the used solution, or compare it with ADMM.\n- Please emphasize the advantages of your DSP comparing with existing model selection strategies, such as GRIC.\n- Correct the typos, see [Weaknesses]\n\nLIMITATIONS:\nYes, the authors have analyzed the limitations and potential negative social impact.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the task of robust model reasoning and fitting in an unified optimization framework that can estimate the geometric model accurately without knowing the predefined model in advance while being robust to outliers as well as highly efficient. The authors propose a novel sparse subspace recovery theory and derive corresponding propositions, thus giving a general and unified formulation for robust model reasoning and fitting. They next introduced an alternating optimization strategy together with proximal approximation method to accurately estimate the sparse model parameters and outlier entries. Extensive experiments indicate that the proposed method outperforms the selected comparison methods.\n\nSTRENGTHS:\nThis paper solved the geometric model estimation problem from a novel perspective, that is recovering the model parameters without knowing the model type, which is interesting and valuable. The writing is good. The authors have provided clear presentation to convey their core idea. The experiment part is reliable, which well verifies the advantages of the method.\n\nWEAKNESSES:\n1:The authors designed an accelerated optimization approach and demonstrated its fast convergence in Fig. 2, claiming it an ‘optimal’ first order method. But there are no experimental proofs of the optimality in the paper. Authors should compare the performance of the algorithm before and after the acceleration.\n\n2:The authors mentioned their strength in efficiency, but the experiment results have not revealed this property.\n\n3:The authors propose a solution process similar to DPCP, and I think it should be added to the comparison algorithms as well.\n\n4:The experiments in the paper demonstrate the excellent performance of the DSP on 2D models and geometry models. I would like to know is the upper bound of the algorithm. Is it possible to design experiments (e.g., continuously increasing the dimension of the data space) to test the performance upper bound.\n\n5:During application test, the proposed method have successfully applied to multimodel fitting, as we can see. But why do the authors mention that they plan to integrate multimodel fitting and achieve a four-fold task in the future?\n\n6:The pose estimation or visual localization experiments are also necessary, since they are more common in this topic, comparing with multimodel fitting or Loop Closure Detection.\n\n7:Others:\n Line 113, “to predefine” should be “predefining”\n Line 159, “convert” should be “be converted”\n\nQUESTIONS:\nPlease refer to [Weaknesses]\n\nLIMITATIONS:\nThe authors have solved most limitations of their method, and they also mentioned potential negative social impact, which are acceptable.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-11"}