Diverse consensuses paired with motion estimation-based multi-model fitting

Authors: wenyu  yin
shuyuan  lin
yang  lu
hanzi  wang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 06:20:55

Abstract: Multi-model fitting aims to robustly estimate the parameters of various model instances in data contaminated by noise and outliers. Most previous works employ only a single type of consensus or implicit fusion model to represent the correlation between data points and model hypotheses. This approach often results in unrealistic and incorrect model fitting in the presence of noise and uncertainty. In this paper, we propose a novel method of diverse Consensuses paired with Motion estimation-based multi-Model Fitting (CMMF), which leverages three types of diverse consensuses along with inter-model collaboration to enhance the effectiveness of multi-model fusion. We design a Tangent Consensus Residual Reconstruction (TCRR) module to capture motion structure information of two points at the pixel level. Additionally, we introduce a Cross Consensus Affinity (CCA) framework to strengthen the correlation between data points and model hypotheses. To address the challenge of multi-body motion estimation, we propose a Nested Consensus Clustering (NCC) strategy, which formulates multi-model fitting as a motion estimation problem. It explicitly establishes motion collaboration between models and ensures that multiple models are well-fitted. Extensive quantitative and qualitative experiments are conducted on four public datasets (i.e., AdelaideRMF-F, Hopkins155, KITTI, MTPV62), and the results demonstrate that our proposed method outperforms several state-of-the-art methods.

Main paper: 
 introduction: 
 Multi-model fitting is a fundamental but challenging computer vision task with many potential applications, including image matching [9, 22] , motion segmentation [1, 19] , image stitching [21, 24] , point cloud registration [15, 18] , and geometric entity detection [14, 17] . In essence, multi-model fitting is the process of estimating the parameters of various models from a set of input data containing noise and outliers. Typically, this input data consists of feature points on object surfaces tracked across image pairs and video sequences. Therefore, the multi-model fitting problem can be reformulated as estimating moving objects across various image pairs or video sequences. In the state-of-the-art algorithms, the task of finding an unknown number of model instances is achieved Figure 1 : Left: Differences among our CMMF method, ALC [46] , and Subset [61] . ALC and Subset, which are some of the fastest SOTA methods, consider only one type of consensus, while our method achieves a more accurate clustering performance in multi-model fitting. Right: Qualitative results of CMMF on the AdelaideRMF dataset [58] .by classifying data points into moving objects, each representing a particular model instance. Thus, the primary challenge lies in devising methodologies that can effectively estimate the parameters of multiple models and concurrently segment input data, all in the absence of prior knowledge regarding the accurate number of structures. Additionally, this task becomes more complex when input data is contaminated with both gross and pseudo outliers.Over the past decades, the requirement to explore moving object trajectories in motion estimation has ceaselessly inspired research into robust multi-model fitting methods. Among such, subspacebased [57] , optimization-based [2] , and affinity-based [13] methods are three representatives. Subspace-based methods [31, 36] treat the point constraint as a labeling problem. In a subspace, each point is described as a linear combination of its corresponding points. However, these methods cannot be extended to non-linear correspondence combinations to define models. Typically, optimizationbased methods [3, 4] employ a hypothesis-and-validation strategy. The hypothesis stage generates a set of candidate model hypotheses using a random or guided sampling technique. The validation stage selects model hypotheses and applies graph notation techniques to minimize an energy function. To cover all the required model instances, these methods often necessitate generating numerous model hypotheses to account for potential models. Nonetheless, it is still difficult to guarantee obtaining a model and decreases computational costs. In contrast, affinity-based methods [44, 49] can be formulated as a clustering problem. This problem is solved by projecting high-order affinity between data points onto a graph and then applying spectral clustering techniques. However, these methods are sensitive to outliers and noise. They become expensive when calculating all potential high-order affinities.In view of these problems, we propose a novel motion estimationbased multi-model fitting method that integrates three types of diverse consensuses. The proposed method comprehensively explores the intricate relationships among data points, hypotheses, and their mutual interactions across multiple models. By redefining the problem in terms of motion estimation, we can achieve more accurate and reliable results, even in the presence of noisy or incomplete data. Specifically, the tangent consensus residual reconstruction module rebuilds the shape and structure information at the pixel level. It applies the tangent consensus technique to identify valuable model hypotheses from residual information. This module ensures that only the relevant and accurate hypotheses are considered for further analysis. Additionally, we establish cross consensus relationships between points and hypotheses by generating a consensus affinity matrix. This matrix strengthens the correlations among inliers and reduces the sensitivity of the model to outliers. This step significantly improves the robustness of our method against noisy or incomplete data. Finally, we introduce a nested consensus clustering strategy to analyze the potential connections between multiple parameter models from a motion perspective. This strategy ensures the consistency and coherence of multiple model matrices, resulting in more accurate and reliable multi-model fitting.The key contributions of this paper can be summarized as follows:• We propose a novel multi-model fitting method based on motion estimation that explores three types of diverse consensuses and the potential motion interactions among multiple models. This method effectively reduces the sensitivity to noise and outlier data, improves the accuracy of model selection, and enhances the multi-model fusion capability. • We propose a tangent consensus residual reconstruction module to optimize the correspondence between data points, which goes beyond the traditional Sampson metric by leveraging the properties of tangent consensus. • We propose a nested consensus clustering strategy that guides the motion estimation of underlying models. This strategy combines the interaction of motion through nested consensus, which in turn improves the accuracy of parameter estimation. • We evaluate the proposed method on four publicly available datasets. Both qualitative and quantitative results validate the applicability and superiority of our method over several stateof-the-art methods in terms of clustering accuracy and fitting performance. 
related work 2.1 analysis-based methods: 
 Analysis-based model fitting methods have been proposed to examine the relationship between data points and model hypotheses. These methods can be categorized into consensus analysis-based methods [50, 65] and preference analysis-based methods [59, 67] . Consensus analysis-based methods calculate the number of inliers to match the model hypotheses and select the significant hypotheses as the model instance for estimation. The conventional method RANSAC [12] iteratively performs two processes: model hypothesis generation and model validation. In this method, the model hypotheses with the maximum consensus set are chosen as the estimated model instance, which is primarily used to process single-structure data. To address multi-structure data, sequential RANSAC [25] , MultiRANSAC [68] , AKSWH [56] , RansaCov [41] , and other representative model fitting methods are proposed to improve RANSAC. Preference analysis-based methods derive preference information to compute the residuals between data points and a set of model hypotheses, such as RHA [66] , J-Linkage [52] , T-Linkage [40] , RPA [42] and KF [7] . More specifically, preference information is typically utilized for data representation, which enhances the similarity matrix for data. The data are separated by combining the similarity matrix of the data with clustering methods. Subsequently, the inliers obtained from the segmentation are used to estimate the model instance. However, analysis-based model fitting methods are highly sensitive to the inlier threshold, and the accuracy of data characterization also needs to be improved. When there are a large number of outliers and multiple model structures in data, the computational efficiency of model fitting will be significantly affected. These limitations restrict their usefulness in complex scenes.Multi-model fitting methods based on deep learning have emerged in recent years, owing to their powerful learning and expressive capabilities. For instance, CONSAC [28] aims to infer search strategies directly from data. This method introduces neural network-guided model estimation based on previously estimated instances to select a subset of various metrics, which enables sequentially seeking model instances. LDA [60] formulates the multi-model fitting problem as one of learning deep feature embeddings that are clustering-friendly.In other words, this network embeds the points belonging to the same clusters together. ULRF [54] claims that robust model fitting can be effectively solved by using an unsupervised learning framework, which efficiently trains and explores search trees through the use of backbone networks. FSNet [5] generates simple matrix hypotheses and predicts the angular translation and rotation errors of image pairs. It is a framework that employs correspondences to formulate model hypotheses and integrates epipolar geometry into the attention layer. However, it does not use epipolar geometry to score the hypotheses throughout the RANSAC cycle. Moreover, these methods solve multi-model fitting problems by utilizing the priors learned from data, which constrains their interpretability. Additionally, it may also be necessary to retrain task-specific models when applying them to fit various multi-structure and multi-type geometric models.Motion estimation-based model fitting methods utilize the motion estimation of objects in images or videos to fit models. Subspacebased methods, such as ORK [6] , initially reject outliers by removing data with low vector norms from the main subspace. Then, they restore the number of clusters using the N-cut and K-means algorithm to obtain the model parameters. ELSA [64] enhances the local subspace and selects a model based on the probability density function, which calculates the number of motions using an eigenvalue spectral threshold to achieve the model fitting results. Optimization-based methods typically construct a function and optimize this function. JESS [63] offers a sparse optimization method that iteratively corrects motion segmentation and removes outliers. This method reconstructs model fitting using the desired motion structure. To address the challenge of unknown model numbers, HOM [30] designs a pseudo-boolean formulation to optimize multi-graph decomposition and construct a high-order loss function. Affinity-based methods estimate models by constructing affinity matrices and performing clustering. MSSC [29] integrates motion information from all frames in a video sequence into a correlation matrix. This method applies spectral clustering to estimate the number of motions and fit multiple models. MCMS [23] is designed for consistent affinity segmentation across all geometric models. It obtains common structure information through block diagonals to select consistent data. However, these methods often face challenges in handling video sequences and image pairs with camera motion, occlusion, and strong perspective effects.In this work, unlike some conventional multi-model fitting methods that only utilize a single type of consensus and implicitly fuse models, we propose a method that integrates the advantages of analysis-based and motion estimation-based methods. It owns both the comprehensive conciseness of model selection and the flexibility of clustering. 
methodology 3.1 residual reconstruction with tangent consensus: 
 For multi-model fitting of two-view images and video sequences, identifying the correct point correspondences is essential for sampling the minimum subset of data and generating a set of precise model hypotheses. In model fitting, the Sampson error [39] is commonly utilized to approximate the geometric distance between a point and a model hypothesis to identify this correspondence. However, it only evaluates certain geometric quantities and fails to approximate the true reprojection error, thereby ignoring shape and motion structure information at the image level. To address this limitation, we design a tangent consensus residual reconstruction module to screen residuals and select valuable model hypotheses at the pixel level, as shown in Figure 2 .Let 𝑃 = {𝑝 1 , 𝑝 2 , ..., 𝑝 𝑛 } be a set of input data containing 𝑛 feature points. We randomly select a minimal number of 𝑚 points to generate three model hypothesis matrices represented by A, H, and F, which denote the affine matrix A, the homography matrix H, and the fundamental matrix F, respectively. The minimum numbers of points required to generate these models are 3, 4, and 8, respectively.Suppose that we are given a set of correspondences 𝐶 = {𝑝 𝑖 , 𝑝 𝑗 } ∈ R 2×2 with a few outliers, where 𝑖 and 𝑗 are the index of the data point. The residual with respect to each hypothesis and data point is derived from the traditional Sampson error:EQUATION)where E is the essential matrix. This residual is approximated by the geometric error [48] , which is the distance between the nearest pair of points that satisfy the epipolar constraint 𝐼 (•):EQUATIONwhere 𝜏 = {𝑡 𝑖 , 𝑡 𝑗 } ∈ R 2×2 represents the correct correspondence, and 𝑡 𝑖 and 𝑡 𝑗 correspond to 𝑝 𝑖 and 𝑝 𝑗 , respectively. We then reconstruct the residual derived from the tangent consensus error 𝜀 𝑡𝑐 for matching the motion estimation as follows:EQUATIONwhere Ψ(•) is the tangent inverse function, which depends on the forward projection function Φ(•). The forward projection function is adopted from the Kannala-Brandt model [26] :EQUATIONwhere 𝛼 and 𝛽 are the inherent parameters. The forward projection function solely defines forward projection to restore the shape of an object on the image plane. However, during iterative solving for unprojection, it fails to obtain the structural information about the object. This limitation impacts the estimation of moving objects. Note that Φ(•) supports right inverses, but they only differ in lengths assigned to the unprojected bearing vectors [51] . Nevertheless, for motion estimation, a more complex model is required to capture the structural details. Considering an arbitrary central camera model, it is necessary to optimize the continuously differentiable bearing vector at the pixel level tangent planes. Therefore, the tangent inverse function is designed for 𝐶:EQUATIONwhere 𝐽 𝑖 and 𝐽 𝑗 ∈ R 2×2 are the Jacobians of 𝑝 𝑖 and 𝑝 𝑗 with respect to the epipolar constraint, and × denotes the cross product. Inspired by [47] , we linearize the epipolar constraint:EQUATION)Thus we substitute and minimize Eq. ( 3) to obtain:EQUATIONThe tangent consensus residual reconstruction module reflects the efficient optimization of the epipolar constraint on the differentiable tangent plane of the points. Extracting motion structure information on a pixel basis is meaningful since the original image has a tangent measure. As long as we can determine the Jacobian of the projection function in each corresponding connection, it works with arbitrary central camera models. Notably, this reconstruction module is cost-effective, as it requires computing only a single forward projection and Jacobian. Through the tangent consensus error, we reconstruct the residual between each model hypothesis and data point. The residual vector based on the tangent consensus error is defined as follows:EQUATIONThe reconstructed residual not only measures the geometric distance corresponding to the point but also approximates the true reprojection error. This is because the forward projection function can recover the shape information, meanwhile, the tangent inverse function can obtain the structural information. Both of these functions provide indispensable correlations for finding the affinity matrix.The affinity between two features can be characterized by alignment based on the consistency of points across all hypotheses. However, the model hypotheses generated by random sampling often include a large number of irrelevant or incorrect hypotheses. This can significantly impact the performance of motion estimation and the accuracy of the affinity matrix, especially in scenarios involving multiple models.To address the above issues, we propose a cross consensus affinity framework to describe the affinity between points and underlying model hypotheses. First, we sort the residual r of the tangent consensus reconstruction in ascending order. Then, we obtain a permutation 𝜼 = {𝜂 1 , 𝜂 2 , ..., 𝜂 𝑛 }, which expresses the preference relationship between model hypotheses and data points. If two points 𝑝 𝑖 and 𝑝 𝑗 are inliers from the same structure, they will have a lot of shared hypotheses at the top of their preference lists. Motivated by [45] , we calculate the cross kernel 𝑘 of two points 𝑝 𝑖 and 𝑝 𝑗 as follows:EQUATIONwhere 𝜌 is a bandwidth, 𝑥 is the element number of 𝜼, and 𝜂 𝑎∼𝑏 Next, for each pair of points in the three model hypothesis matrices, the cross kernel matrix K is computed as follows:EQUATION)where 𝜆 is defined as the set of three model hypothesis matrices (i.e., A, H, and F).Then, the adaptive weighted density estimation [8] is utilized to assess the weighted score 𝑠 of a model hypothesis, which corresponds to the three model hypothesis matrices:EQUATION)where 𝑖 is the number of model hypotheses, K is the Epanechnikov kernel, 𝜅 is the inlier scale that IKOSE calculated [56] , and 𝜌 is a bandwidth. The weighted scores will assign a high score to meaningful model hypotheses via Eq. (11) . The cumulative weighted score for each point can be expressed as:EQUATION)where 𝑗 is the number of points. The cross consensus affinity Ω 𝜆 can be constructed based on the cross relationship of the points and the weighted scores of the model hypotheses:EQUATION)The cross consensus affinity between two points from the same moving object should be greater, while the affinity between two points from different moving objects should be smaller. At last, an 𝜖-neighborhood method [29] is used to sparsify the cross consensus affinity Ω 𝜆 . After obtaining the cross consensus affinity matrices of multiple models, the next step is to find partitions for multiple motions. We design a nested consensus clustering strategy to recover the clustering of different moving objects. Unlike previous fusion or accumulation methods, we utilize the potential relationships between different models. In the case of multi-model fitting for motion estimation, we are aware that the fundamental matrix F can be described as a family of F = ⟨𝝃 ⟩ * H parameterized by a vector 𝝃 , where ⟨𝝃 ⟩ is the skew-symmetric matrix [16] , and ( * ) represents the cross multiplication. This implies that for a given pair of points within a homography matrix H, they are considered inliers with respect to a specific fundamental matrix. Conversely, if two points are outliers with regard to a fundamental matrix, they fail to satisfy the constraints of a homography matrix. Similarly, the homography matrix H can be defined as H = ⌈𝜁 ⌋ • A, where ⌈𝜁 ⌋ represents the perspective transformation, and (•) denotes the dot product. This means that through a series of translation and rotation transformations, the inliers of the affine matrix can be transformed into the inliers of the homography matrix, and vice versa. If a point is not an inlier of a certain homography matrix, it cannot be an inlier of the affine matrix.Based on the potential relationships between the moving object models mentioned above, Ω A , Ω H , and Ω F are the sparse affinity matrices of A, H, and F, respectively. We can define a nested consensus clustering strategy for the existing affinitiesEQUATIONwhere Q represents the spectral embedding, Λ denotes the degree matrix of the sparse affinity matrix Ω, 𝛾 ∈ R, I is the identity matrix, and Π is the nested consensus matrix. When 𝜋 𝑖 𝑗 = 𝜐, the nested consensus promotes a large inner product Q T 𝜆𝑖 Q 𝜆 𝑗 , where Q 𝜆𝑖 denotes the 𝑖-th column. Therefore, it is desirable for points 𝑖 and 𝑗 to belong to the same cluster. There is no nested consensus for 𝜋 𝑖 𝑗 = 0. A different cluster assignment between 𝑖 and 𝑗 is encouraged by the nested consensus when 𝜋 𝑖 𝑗 = -𝜐. This nested consensus will help affinity matrices to be further denoised or repaired.We relax Π to continuous values and optimize Π using the affinity reconstructed from the spectral embedding G = QQ T . Then Eq. ( 14) is rewritten as:EQUATIONwhere ⊙ denotes element-wise multiplication and I(•) represents the indicator function. We assume that three model matrices are arranged as follows: the affine (𝜋 = -1) followed by the homography (𝜋 = 0) and the fundamental matrix (𝜋 = 1). According to the principle of nested consensus, H and A can be nested in F, and A can be nested in H. Finally, a separate K-means step is taken. Normalized Q is input to group the points into motion groups. This step helps obtain multi-model fitting results. Figure 2 depicts an overview of the proposed method. An overview of the complete steps is provided in Algorithm 1. 
experiments 4.1 experimental setup: 
 Datasets. We assess the proposed method using four challenging datasets:• AdelaideRMF-F [58] is a two-view multi-model fitting dataset that includes 19 pairs of images. Each image contains at least two to four objects that may be scaled, rotated, or distorted, and may also exhibit shadows or occlusions between objects. These factors contribute to the challenging nature of this dataset. • Hopkins155 [53] consists of two motion groups in 120 video sequences and three motion groups in 35 video sequences. It includes incomplete motion trajectories of objects, which makes motion estimation difficult. • KITTI [61] comprises a series of photos captured by a camera mounted on a moving car. It contains two to five moving objects in total, including the background. The photos have a resolution of 1226 x 370 pixels. This benchmark showcases the interaction of multiple actions, intricate backgrounds, and significant camera translation. • MTPV62 [32] is an extension of the Hopkins155 dataset [53] , involving 12 real outdoor fragments and 50 Hopkins 155 fragments. Nine of the clips exhibit strong perspective effects, making them suitable for robust perspective testing. Additionally, it provides feature trajectory information. Evaluation of Two-View Model Fitting. We first evaluate the proposed method on the AdelaideRMF-F dataset [58] . We compare the proposed method with ten state-of-the-art methods. Among these, PEARL [20] , T-linkage [40] , and RPA [42] are hypothesize-cluster methods. MLink [43] , RansaCov [41] , Prog-X [3] , and LDA [60] are analysis-based methods. CBG [34] , D2Fitting [33] , and QUMF [11] are optimization-based methods. As shown in Table 1 , the proposed method achieves the best performance in terms of ACE and MCE criteria with improvements of 0.43% and 0.89%, respectively. These improvements are attributed to the effectiveness of our tangent consensus structural details reconstruction and the multi-model nested consensus learning. Figure 3 shows some qualitative results obtained by the proposed method. We observe consistently better multi-model fitting results by our method. Specifically, for the 'breadtoy' and 'carchipscub' image pairs, our method demonstrates robustness against occlusion and shadows. Furthermore, for the 'breadcartoychips' and 'cubebreadtoychip' image pairs, our method accurately detects entire objects with multiple sub-classes. This is because our method utilizes three diverse consensuses between feature points and hypotheses, which improves the accuracy of multi-model fitting.Evaluation of Motion Estimation. We further compare our method on the KITTI, Hopkins155, and MTPV62 datasets with seven state-of-the-art methods. Subspace-based single-model methods include GPCA [55] , SSC [10] , BDR [38] , and ALC [46] . Fusionbased multi-model methods include Subset [61] , MCMS [23] , and HMFMS [35] . Figure 1 summarizes the core differences between them. As shown in Table 2 , the subspace-based single-model methods exhibit larger ACE values. In contrast, ACE is significantly improved by the fusion-based multi-model methods. Our proposed method consistently outperforms all other competing methods in terms of ACE, with performance improvements of 1.16%, 0.12%, and 0.11%, respectively. It is noteworthy that the KITTI dataset presents various actions and intricate background effects. However, our proposed CMMF method effectively addresses these challenges by leveraging the nested consensus of different models. To tackle the issue of missing object motion trajectories in the Hopkins155 dataset and the strong perspective problem in the MPTV62 dataset, our method reconstructs the residual between points and hypotheses to obtain structural information of moving objects. It further enhances motion correlation between models using a cross consensus affinity framework. This method effectively preserves pixel-level motion structure information. Therefore, the proposed CMMF method offers significant performance advantages over fusion-based methods. Some qualitative estimation results obtained by our method are shown in Figure 4 . In the 'KITTI104-108' and 'KITTI198-201' video sequences, the proposed method not only correctly recognizes cars and a trash can but also properly marks walls in dark areas. In the 'KITTI579-582' and 'KITTI738-742' video sequences, our proposed method accurately labels multiple moving objects. This is because our method reduces residual-induced instability and enhances estimation performance. It achieves this by exploring the correlation of different moving objects in challenging scenarios.Influence of Tangent Consensus Residual Reconstruction Module. We demonstrate the effectiveness of the proposed tangent consensus residual reconstruction. This is achieved by approximating the true reprojection error between the points and hypotheses during the residual reconstruction of the different errors, as shown in Figure 5 . We utilize Kendall's 𝛿 Rank correlation coefficient [27] to analyze a series of checkerboard-like [37] error measures. The coefficient 𝛿 ∈ [-1, 1] is calculated between two residual sortings.The same sorting results in 𝛿 approaching 1, and 𝛿 approaching -1 in the opposite case. Competitors are divided into two groups: one group is used for geometric errors [48] , such as algebraic error (AE) and cosine error (CE), and the other group is used for image errors [47] , such as symmetric epipolar error (SEE), Sampson error (SE), and tangent consensus error (TCE). We observe that the most accurate approximation of the true reprojection error for each row and column is provided by the tangent consensus error, excluding the self error. The closer 𝛿 is to 1, the greater the likelihood that two sortings come from the same structure. Our tangent consensus residual reconstruction represents a pixel-level difference in an image pair. It not only restores shape information through forward projection but also obtains structural information through inverse projection. This improves the correlation between points and hypotheses. These observations indicate that the proposed CMMF method offers an effective residual reconstruction manner for multi-model fitting to approximate the true reprojection error.Influence of Cross Consensus Affinity Framework. We conduct an ablation study to explore the contribution of cross consensus affinity framework in our proposed method. As shown in Table 3 , the comparison reveals that augmenting the traditional ordered residual kernels (ORK) [6] and kernels add (KA) [61] with cross consensus notably improves results on the MTPV62 and Hop-kins155 datasets. Method (a) utilizes ORK to encapsulate hypothesis affinity in a kernel. Method (b) combines KA with a superposition of multiple affinities. Our method (c) captures affinity through cross consensus. It assigns weights to different hypotheses for sampling, providing elasticity for severe sampling imbalances. This significantly boosts the performance because it is easier to reduce the impact of incorrect related information.Influence of Nested Consensus Clustering Strategy. To investigate the effectiveness of the proposed nested consensus clustering strategy, we conduct extensive ablation experiments on the Hopkins155, MTPV62, and KITTI datasets. We apply the parameter 𝛾 to promote convergence and select 𝛾 = 10 -2 can make the problem more easily convergent and improve the accuracy of our method. Specifically, we compare the single-model method (i.e., Affine, Homography, and Fundamental), the regularized method (CoReg) [62] , and the multiplicative decomposition method (MCMS) [23] . As shown in Table 4 , the proposed method consistently improves results across all methods and metrics. The nested consensus clustering strategy plays a critical role in our method. This is because most of the other competitive methods only seek one consensus spectral feature. We believe that segmenting information solely based on one consensus may not be sufficient to eliminate outliers. It may also lead to unreliable or semantically irrelevant movement. Therefore, we construct the nested consensus clustering strategy, which integrates the nested consensus of multiple models. It enhances motion connections between multiple models and improves the accuracy of multi-model fitting. 
conclusion: 
 In this paper, we propose a highly effective method for multi-model fitting through diverse consensuses paired with motion estimation. Our method comprises three key components: the tangent consensus residual reconstruction module, the cross consensus affinity framework, and the nested consensus clustering strategy. We leverage diverse types of consensuses to analyze potential correlations between data points and model hypotheses, and we demonstrate how the proposed method can be used on potential correlations, thereby contributing to the understanding of motion collaboration between multiple models. Our method is evaluated on four challenging datasets, with the results showing its superiority over other state-of-the-art approaches. It is highly competent in practical vision tasks, excelling in fitting multiple models and achieving high clustering accuracy, even in scenarios where prior knowledge of the number of motions is lacking. 
 
 Appendix:  


Decision: All reviewers lean towards acceptance of this paper. The AC concurs with the reviewers and recommends this paper for publication. The authors should address all the issues raised in the reviews in the final version of their paper.

Reviews: review: paper_summary: paper_summary: The authors introduce a new method called CMMF, which enhances multi-model fitting by using three diverse consensuses and inter-model collaboration. The method includes a Tangent Consensus Residual Reconstruction (TCRR) module for capturing pixel-level motion structure, a Cross Consensus Affinity (CCA) framework to strengthen correlations between data points and model hypotheses, and a Nested Consensus Clustering (NCC) strategy for handling multi-body motion estimation. Extensive experiments on public datasets demonstrate that CMMF outperforms state-of-the-art methods in terms of accuracy and robustness in the presence of noise and outliers.
strength_weakness: strength_weakness: - In general, this paper is well-structured and articulates the methodology clearly, supported by detailed technical descriptions and illustrative figures (e.g., Fig. 1)
- The experimental results exhibit the promising registration performance of the proposed method across different settings, and present significant performance advantages than other methods.
- The authors offer the sufficient visualization comparisons to support their claims.
limitations: limitations: - The authors should include detailed speed comparisons to provide a fair and comprehensive evaluation.
- The paper lacks comparisons with several highly relevant state-of-the-art multi-model fitting methods, such as [1].

[1] Quantum Multi-Model Fitting, CVPR'2023
paper_summary: paper_summary: The paper presents a novel approach to multi-model fitting in computer vision, which is a crucial task for applications like image matching, motion segmentation, and point cloud registration. The proposed method addresses the challenge of estimating parameters for multiple models within noisy data by introducing three key components corresponding to three consensus types. It explicitly establishes motion collaboration between models and ensures that multiple models are well-fitted. Experimental results show superior performance compared to several SOTA methods.
strength_weakness: strength_weakness: - Combining three types of consensus makes sense to boost the overall performance. The paper introduces a novel multi-model fitting approach that leverages diverse consensus mechanisms and motion estimation, which is an interesting idea from traditional single-consensus methods. 
- Very good experimental results. By combining three different consensus mechanisms (Tangent Consensus, Cross Consensus Affinity, and Nested Consensus Clustering), the method can handle complex scenarios with multiple moving objects more effectively. The proposed TCRR module and CCA framework enhance the method's robustness against noise and outliers, which are common in real-world vision tasks.
limitations: limitations: - My main concern is about the efficiency. The proposed method involves complex calculations, which might lead to high computational overhead, making it less suitable for real-time applications or very large datasets. Comparing the runtime performance with existing methods on the same datasets could provide a clearer picture of its practical applicability.  Can you explain the time overhead? Also, I think the deep learning-based methods are missing in the evaluation. It would be great to involve at least one.
- The paper does not provide sufficient details on how to reproduce the results. e.g. how to get the data points? Is there any preprocess before applying the proposed method? The authors should include a detailed description of the data point extraction process. This would help in understanding the preprocessing steps and their impact on the overall performance of the method. Also, will the source code be released?

Minor questions
- Is there any parameter used in the proposed method? As the proposed method is going through three consensus components, the performance of the proposed method may be sensitive to the selection of model parameters. The authors should provide an analysis of the sensitivity of the method to various parameter settings.
- could you explain more about the motion estimation? As the intro states the proposed method is based on motion estimation but the method seems to combine three types of consensus into a pipeline, I cannot see any functionality of motion-based estimation. Is that about the F, H, A matrix? or what?
paper_summary: paper_summary: The paper addresses the challenge of multi-model fitting, where traditional methods often rely on a single consensus or implicit fusion model, leading to unrealistic fitting in the presence of noise and uncertainty. The proposed method, named CMMF (Computational Multi-Model Fitting), enhances multi-model fusion by incorporating three diverse consensus mechanisms along with inter-model collaboration.
The method is evaluated using four challenging datasets (AdelaideRMF-F, Hopkins155, KITTI, MTPV62). The proposed method is compared with eight state-of-the-art methods, demonstrating the best performance in terms of ACE and MCE.
strength_weakness: strength_weakness: 1. The introduction of a novel multi-model fitting method that explores three different types of consensuses and potential motion interactions among multiple models, reducing sensitivity to noise and outliers and improving model selection accuracy.
2. The design of a Tangent Consensus Residual Reconstruction (TCRR) module that optimizes correspondence between data points using tangent consensus, going beyond the simple Sampson metric.
3. The proposal of a Nested Consensus Clustering (NCC) strategy that guides motion estimation of underlying models, improving both estimation accuracy and parameter accuracy through motion interaction via nested consensus.
4. Extensive quantitative and qualitative experiments on four public datasets demonstrate the proposed method's superiority over state-of-the-art methods in terms of clustering accuracy and fitting performance.
limitations: limitations: 1. The proposed method involves numerous iterative and matrix operation steps; however, the authors have not analyzed the computational complexity of the algorithm. It would be important to provide a theoretical analysis of the algorithmic complexity, as well as a comparison of the time cost with other algorithms.

2. How are the n input feature points extracted, and can different extraction methods affect the results of this paper? Are there different extraction methods on different datasets? This question is important because these feature points are the input part of the entire algorithm.

3. The proposed method involves the setting of multiple parameters, but the paper does not mention the process of selecting and adjusting these parameters, as well as their impact on the final results.

4. In the experiments, Evaluation Metrics used ACE and MCE. It is not explained how these two errors, ACE and MCE, are calculated. The authors should clarify why these two metrics are used and what their formulations are. Some theoretical reasoning or citations are necessary.

5. The authors have summarized Deep Learning-Based Methods in the related work section. However, the experimental part does not include a comparison with these methods.

6. The authors have compared some methods in the experimental section that are rather outdated, with some papers published a decade ago. The authors need to include comparisons with newer methods from the past few years.