Title: Sensitivity in Translation Averaging
Venue: NeurIPS 2023
Accepted: yes
Keywords: sensitivity

translation averaging

structure from motion

3D computer vision
Abstract: In 3D computer vision, translation averaging solves for absolute translations given a set of pairwise relative translation directions. While there has been much work on robustness to outliers and studies on the uniqueness of the solution, this paper deals with a distinctly different problem of sensitivity in translation averaging under uncertainty. We first analyze sensitivity in estimating scales corresponding to relative directions under small perturbations of the relative directions. Then, we formally define the conditioning of the translation averaging problem, which assesses the reliability of estimated translations based solely on the input directions. We give a sufficient criterion to ensure that the problem is well-conditioned. Subsequently, we provide an efficient algorithm to identify and remove combinations of directions which make the problem ill-conditioned while ensuring uniqueness of the solution. We demonstrate the utility of such analysis in global structure-from-motion pipelines for obtaining 3D reconstructions, which reveals the benefits of filtering the ill-conditioned set of directions in translation averaging in terms of reduced translation errors, a higher number of 3D points triangulated and faster convergence of bundle adjustment.

Sensitivity in Translation Averaging

Abstract

In 3D computer vision, translation averaging solves for absolute translations given a set of pairwise relative translation directions. While there has been much work on robustness to outliers and studies on the uniqueness of the solution, this paper deals with a distinctly different problem of sensitivity in translation averaging under uncertainty. We first analyze sensitivity in estimating scales corresponding to relative directions under small perturbations of the relative directions. Then, we formally define the conditioning of the translation averaging problem, which assesses the reliability of estimated translations based solely on the input directions. We give a sufficient criterion to ensure that the problem is well-conditioned. Subsequently, we provide an efficient algorithm to identify and remove combinations of directions which make the problem ill-conditioned while ensuring uniqueness of the solution. We demonstrate the utility of such analysis in global structure-frommotion pipelines for obtaining 3D reconstructions, which reveals the benefits of filtering the ill-conditioned set of directions in translation averaging in terms of reduced translation errors, a higher number of 3D points triangulated and faster convergence of bundle adjustment.

1 Introduction

The goal of the translation averaging problem is to recover absolute translations given a redundant set of pairwise relative translation directions. This problem has been relatively less studied compared to the case when pairwise displacements are available [31, 12, 6, 51]. These methods belong to the category of averaging or map synchronization [43, 46], which can be modelled as a network with known pairwise relations between nodes and unknown node values to be estimated. For our specific problem of translation averaging, G = ( V , E ) represents a network, with N nodes, V , denoting the absolute translations T i ∈ R 3 , i ∈ V , and M edges, E , denoting the pairwise relative direction measurements between the nodes v ij ∈ S 2 , ( i, j ) ∈ E . Here, all the relative directions v ij 's are in the same coordinate frame. Ideally, v ij should be the unit vector in the direction of T j -T i . Since the measurements made are directions, these networks are also called bearing-based networks. This problem finds a place in global methods [23, 24, 18, 52] for solving Structure-from-Motion (SfM) [28] in 3D computer vision, where a network of cameras is present with relative translation directions measured between the cameras.

Solving translation averaging is a challenging problem since it requires estimating translation scales in a context of dissimilarity in the input (directions) and output (absolute translations). Due to this dissimilarity, the solutions are defined upto a global scale and a choice of origin. The wellposedness of the bearing-based network problems has been studied under the lens of parallel rigidity theory [45, 19, 36, 42, 56, 2]. This theory determines whether a given set of input directions will have a unique solution (upto a global scale and an origin) for absolute translations.

Distinction from parallel rigidity and outlier detection: This paper takes the first step to deal with sensitivity in translation averaging under uncertainty, i.e. determining under what configuration of

input directions the solution is reliable. This can be seen as a perturbation analysis of the translation averaging solution with respect to small changes in the input directions. Although there have been empirical studies on small-scale datasets (3 to 20 cameras) on the effect of perturbing input directions [48, 40, 32], to the best of our knowledge, analysis on large-scale networks, both theoretical and empirical, has not been done for bearing-based networks. This analysis is independent of parallel rigidity [2] since parallel rigidity deals with uniqueness of the solution, while we deal with change in solution for small input perturbations. Parallel rigidity can be equivalently described with the algebraic rank of a specific matrix [2], which is similar to the analysis of uniqueness of a solution given a matrix obtained from a linear system of equations. Sensitivity analysis is similar in spirit to the conditioning of a matrix while solving a linear system of equations ( Ax = b problem) where the reliability of a solution is studied. Sensitivity is also different from outlier detection in the data because it only deals with small perturbations of the given input directions and not with the noise/outlier levels. The issue of sensitivity remains relevant even without outliers.

Our contributions: We first analyze the sensitivity in estimating edge scales in translation averaging by small perturbations to input directions on the smallest solvable bearing-based network. Then, we characterize the conditioning of the translation averaging problem based only on input directions, without any perturbation, for the smallest solvable network i.e. 3 edges between all 3 nodes, and extend it to a general network. We provide a sufficient condition for the network to be wellconditioned. We propose an efficient algorithm to identify ill-conditioned parts of the network, filter them and extract the maximal parallel rigid graph without explicit computation that checks for parallel rigidity. Finally, we show the usefulness of such a filter in the context of SfM with improved translation accuracy, more 3D points triangulated and faster convergence of bundle adjustment after removing ill-conditioned parts of the network.

2 Literature Review

In this section, we briefly discuss the relevant literature on translation averaging and parallel rigidity.

Translation averaging in SfM : In SfM, the input data contains relative translation directions that are not aligned to a common reference frame. To get the input directions in a common global frame, absolute rotations for each node are estimated using rotation averaging methods [27, 10, 11, 21, 47]. Many translation averaging methods have been proposed over the last two decades. Govindu [23] minimized the cross-product between the input directions and directions obtained from absolute translations. Jiang et al. [32] considered triplets of nodes and used the constraints of a triangle to formulate the problem. Wilson et al. [55] minimized the deviation between the input directions and directions estimated from absolute translations. Tron et al. [54] minimized the squared relative displacements and solved it in a distributed manner. Ozyesil et al. proposed the Least Unsquared Deviations (LUD) method, extending [54], with L 1 loss for robustness which made the problem as a convex program. Arrigoni et al. [5] minimized the squared error of the orthogonal projection of the estimated relative translations onto input directions. Goldstein et al. [22] also minimized the orthogonal projection using ADMM but used an L 1 loss for robustness. Zhuang et al. [57] relaxed the cost in [55] by comparing estimated relative translations to that of the observed directions and called it Bilinear Angle-based Translation Averaging (BATA). Other methods include using two-view and three-view geometry [28] of the cameras to set up the problem [1, 40], estimating edge scales through cycles in a network before solving for absolute translations [3, 4] or through point correspondence constraints [13, 14], iteratively refining input directions [37], averaging matrices obtained from two view geometry [33, 34], and exploiting the structure of the matrix generated from pairwise displacements [15].

Parallel rigidity : Several works about parallel rigidity are present in the literature arising from different communities: computer vision [42, 3, 2], robotics [36], computer-aided design [45] and decision control [20, 19, 53, 56]. The node-based formulation is the classical way to approach parallel rigidity, which deals with absolute translations (also called point formation) [45, 20, 19]. The edge-based formulation is a more recent approach which reasons about parallel rigidity based on edge lengths in terms of the cycles in the network [35, 3, 53]. Readers are referred to [2] for an excellent survey on parallel rigidity.

3 Sensitivity in Scale Estimation from Directions

In this section, we study the sensitivity in estimating edge scales in translation averaging. At first, we define the notion of a consistent set of directions for a network G , which will be useful for further discussion.

Definition 1 (Consistent Directions) . A set of relative directions, v ij , ( i, j ) ∈ E in a bearing-based network G , are said to be consistent if there exist absolute translations, T i , i ∈ V , such that T j -T i ‖ T j -T i ‖ = v ij .

We consider the smallest possible bearing-based network which is solvable, i.e. a network G ∆ of 3 nodes, V ∆ = { 1 , 2 , 3 } , with all possible edges, E ∆ = { (1 , 2) , (2 , 3) , (3 , 1) } . When these edges are consistent, they form a triangle. Let V be the matrix containing relative directions v ij , ( i, j ) ∈ E ∆ in its columns and s be the vector containing edge scales s ij . Then, the least squares problem to estimate the scales can be written as

where the unit norm constraint on s is used to fix the global scale. The solution to the problem in Eqn. 1 is given by the eigenvector corresponding to the smallest eigenvalue of V T V .

To analyze the sensitivity of the estimated scales, we perturb each direction in G ∆ by a small 3D rotation δ R ij ∈ SO (3) , therby ensuring that the perturbed vectors always lie on the unit sphere. Let n ij ∈ S 2 and δθ ij > 0 be the rotation axis and angle for δ R ij , respectively. Then, the small rotation can be approximated to a first order, using Rodrigues' rotation formula, as δ R ij ≈ I + δθ ij [ n ij ] × , where [ n ij ] × ∈ R 3 × 3 is a matrix such that [ n ij ] × h = n ij × h for any h ∈ R 3 , and I is the 3 × 3 identity matrix. For small perturbations to v ij by δ R ij , the following theorem holds:

Theorem 1. For a set of consistent directions, v ij , ( i, j ) ∈ E ∆ , in G ∆ , the absolute change in any eigenvalue of V T V , denoted as | δλ | , when the directions v ij are perturbed by small rotations δ R ij , with n ij and δθ ij > 0 being the rotation axis and angle, is bounded by

where TI (∆) = { (1 , 2 , 3) , (2 , 3 , 1) , (3 , 1 , 2) } , v n ij ⊥ is the component of v orthogonal to n ij , φ n ij ⊥ ( k,i ) , ( k,j ) is the angle between v n ij ⊥ ki and v n ij ⊥ kj , and φ ( k,i ) , ( k,j ) is the angle between v ki and v kj .

The proof of Thm. 1 uses eigenvalue perturbation theory [30]. Please refer to the appendix for the proof. From Thm. 1, it can be observed that the bound for absolute change in any eigenvalue is inversely dependent on the square of the sine of angles of the triangle formed by the unperturbed directions. Since the scale estimate, s , is the eigenvector corresponding to the smallest eigenvalue of V T V , Thm. 1 reveals that scales are sensitive to small perturbation of directions when at least one angle in the triangle is small. The term in the square root in Eqn. 2 is bounded due to the dot product of unit norm vectors and is strictly greater than zero since the unperturbed directions are consistent. We note that the bound is also directly proportional to the sine of the angle between the directions projected onto the orthogonal space of the rotation axis. This suggests that the effect of perturbation is maximum when the rotation axis is orthogonal to the directions. This is the same as the case in Cor. 1 when the perturbed directions are consistent.

Corollary 1. For a set of consistent directions, v ij , ( i, j ) ∈ E ∆ , in G ∆ , the absolute change in any eigenvalue of V T V , denoted as | δλ | , when the directions v ij are perturbed by small rotations δ R ij , with n ij and δθ ij > 0 being the rotation axis and angle, and n ij being othogonal to v ki and v kj for all ( i, j, k ) ∈ TI (∆) , is bounded by

A

A'

B

A

A'

B

C

AA'

B

(a) Well-conditioned triangle

(b) Ill-conditioned triangle (Type-I) (c) Ill-conditioned triangle (Type-II)

Cor. 1 reveals that the scale estimates are unstable with small angles, even in the case when the perturbed directions are consistent. We call triangles with at least one small angle as skewed triangles. In Fig. 1, we show conditioning of the triangle under different scenarios, with green and red depicting the unperturbed and perturbed directions, respectively. For a well-conditioned triangle, a small change in direction leads to a small change in the absolute translation. But for an ill-conditioned triangle (skewed triangle), a small change in the direction leads to a large change in the absolute translation. Fig. 1a shows a well-conditioned triangle, Fig. 1b shows a triangle with one small angle due to which it is ill-conditioned (Type I) and Fig. 1c shows a triangle with two small angles making it ill-conditioned (Type II) . Such skewed triangles are known to be problematic in different fields. In numerical analysis [25, 8], the skewed triangles are termed as 'needle-like' triangles, and measuring their area and angles have numerical issues. In computational topology and computer graphics [17, 38], skewed triangles (also called sliver triangles) are often avoided, as in Delaunay triangulation, or are filtered out. We note that none of them deal with directions as input. Our aim is to quantify the conditioning of the triangle and the translation averaging problem based solely on the input directions, without actual perturbation, and filter out skewed triangles, which are discussed in subsequent sections.

4 Conditioning of Translation Averaging

In this section, we first analyze the conditioning of the bearing-based network with 3 nodes and 3 edges, i.e. G ∆ (as defined in Sec. 3). We define an angle matrix , A G ∆ ∈ R 3 × 3 , as follows:

where φ ( k,i ) , ( k,j ) is the angle between v ki and v kj (as defined in Thm. 1). The rows and columns of A G ∆ signify edges in G ∆ and its entries are the angular differences between directions with angles being measured within the triangle using the common node between two edges as the reference point for measuring directions. Since φ ( k,i ) , ( k,j ) = φ ( k,j ) , ( k,i ) , A G ∆ is symmetric, and since φ ( k,i ) , ( k,i ) = 0 , A G ∆ has zero diagonal entries. We quantify conditioning of the translation averaging problem on G ∆ as conditioning of the matrix A G ∆ based on the following theorem:

Theorem 2. Consider the bearing based-network of 3 nodes and 3 edges, G ∆ = ( V ∆ , E ∆ ) , and the corresponding angle matrix A G ∆ . The conditioning of the matrix A G ∆ signifies the skewness of the triangle formed using the directions in E ∆ .

The proof of Thm. 2 involves proving A G ∆ being non-singular for non-zero angles using its determinant and checking for the closeness of columns in relation to the angle values. We choose this approach instead of checking determinant since the later is not a good measure of closeness of a matrix to singularity [39]. Please refer to the appendix for the complete proof. Thm. 2 reveals that the conditioning of G ∆ can be characterized by the condition number of matrix A G ∆ .

We generalize the angle matrix and Thm. 2 for any bearing-based network G . We assume all edges in G are a part of at least one triplet (a triplet consists of 3 nodes with all possible edges between them). We use the terms triplet and triangle interchangeably since a triplet forms a triangle for our specific problem. If this assumption is not satisfied, we remove the edges in G , which are not a part of any triplet and consider its maximum connected component. This will enable us to use G ∆ as the basic building block of G and thus characterizing the conditioning of G .

We now expand the angle matrix to a general bearing-based network G in which every edge is a part of at least one triplet. We denote the angle matrix for the general network as A G ∈ R M × M ( M is the number of edges in G ), where the rows and columns signify edges in G , and its entries, a ij,kl G , corresponding to the row for the edge ( i, j ) and the column for the edge ( k, l ) , are defined as follows:

In A G , we measure the angles between the directions formed by triplet. In such cases, to ensure that the directions are measured within the triangle, similar to A G ∆ , the common node, c , is taken as reference and the directions are measured from c to other nodes c ′ 1 and c ′ 2 . From Eqn. 5, it is clear that the diagonal entries are zero ( φ ( k,i ) , ( k,i ) = 0 ) and the matrix is symmetric ( φ ( k,i ) , ( k,j ) = φ ( k,j ) , ( k,i ) ). We note that A G is not the same as that of the distance matrix used in Multi-Dimensional Scaling (MDS) [7, 9]. In instances when a node appears in more than one triplet, angles between the edges belonging to the different triplets are not computed, but in MDS, such angles would also be considered. This restricts the angle matrix A G to contain information only about triangles and not of other structures. Such a construction helps to extend the findings from A G ∆ to A G . We define the conditioning of the translation averaging problem as the condition number of angle matrix A G . The following theorem states a sufficiency condition for a well-conditioned angle matrix:

Theorem 3. Consider a bearing-based network G , with all edges contributing to triplets. The angle matrix A G , corresponding to G , is well conditioned if the minimum angle (or equivalently all the angles) in all the triangles formed by the triplets are sufficiently large.

The proof for 3 is similar to the proof for Thm. 2. Here, we check for different combinations of columns based on the structure of G and check for their closeness. Please refer to the appendix for the proof.

We note that we do not need G to be parallel rigid for the angle matrix A G to be well conditioned, implying that sensitivity and parallel rigidity are different aspects of the bearing-based network. However, we need G to be parallel rigid to ensure a unique solution. In general, computing the maximal parallel rigid component is expensive. Since we remove edges in G not contributing to triplets, this can affect its parallel rigidity. We avoid such expensive computation since we deal with triplets. We construct a triplet network G T = ( V T , E T ) , as done in [32], where nodes V T denote a triplet in G and edges E T connect the nodes if an edge is common between the triplets in G . By construction, it can be seen that disconnected components in G will be disconnected in G T since there cannot be shared edges among disconnected components. The following theorem enables extracting the maximal parallel rigid graph when G contains all edges contributing to triplets:

Theorem 4. Given a bearing-based network G , with all edges contributing to triplets forming triangles, and its corresponding triplet network G T , the maximal parallel rigid component of G can be determined by the edges in G contributing to the largest connected component of G T .

Please refer to the appendix for proof of Thm. 4. Based on Thms. 3 and 4, we develop an algorithm to identify and remove skewed triangles from the translation averaging problem while ensuring that the network is parallel rigid, which is discussed in the next section.

5 Proposed Method

In this section, we show how to efficiently identify skewed triangles in G . First, we get the list of nodes and edges contributing to each triplet in G , for which efficient implementations exist [52, 26]. The brute force way to identify skewed triangles is to compute the angles in each triangle using the relative directions v ij and mark triangles with the minimum angle less than a threshold. The number

of triplets depends on the sparsity of the network G , which, in general, is large. This makes the brute force method time-consuming. In contrast with the brute force approach, we use vectorized operations to construct the angle matrix A G and filter skewed triangles, making it time efficient. It is observed that the vectorized version is ∼ 100 times faster than the parallelized version (with 20 threads) of the brute force method for ∼ 10 5 triplets coming from ∼ 10 4 edges, and this gain increases significantly with an increase in the number of triplets and edges. A time comparison between the brute force method and our method is given in the appendix. In real-world data, the large-scale bearing-based networks are sparse, making A G sparse. We present our method for sparse networks in the following steps and for dense networks in the appendix.

Step 1: The first step is to identify which entries are non-zero in the angle matrix A G . Since we are considering only triplets, we require a matrix that captures which edges are part of a triplet. In computational topology [29, 16], higher order relationships in G = ( V , E ) (other than node-tonode relationship via edges) are studied using simplices. A k -simplex is a subset of the vertex set V with ( k +1) elements. A finite collection of simplices, such as nodes ( 0 -simplices), edges ( 1 -simplices), and triangles ( 2 -simplices), is called a simplicial complex. The boundary matrix B k of a simplicial complex encodes which ( k -1) -simplex contributes to k -simplex. In our case, we need the relationship between 1 -simplices (edges) and 2 -simplices (triangles). The ( i, j ) th element of the boundary matrix B 2 ∈ R M × W , with M edges and W triplets, is given as:

It is easy to compute B 2 since we have the list of edges contributing to each triplet. The row and column indices of non-zero elements in B 2 B T 2 give us the edge pairs participating in triplets. Only these elements in A G are to be computed (except diagonals in B 2 B T 2 which are non-zero). The corresponding dot products between the edges are computed in a vectorized fashion and stored.

Step 2: We need to identify the correct signs for the dot products to ensure that the dot products reflect the cosine of the angles of the triangles. Since the dot products are cos φ ( i,j ) , ( k,l ) for edges ( i, j ) , ( k, l ) ∈ E in triplets, there will be one common node c ∈ { i, j } ∩ { k, l } . The directions v ij and v kl should be measured with reference to the common node to get the angle of the triangle that includes those edges. We need the dot product between v cc ′ 1 and v cc ′ 2 , where c ′ 1 ∈ { i, j } \ { c } and c ′ 2 ∈ { k, l } \ { c } . Using v T cc ′ 1 v cc ′ 2 = v T c ′ 1 c v c ′ 2 c and v ij = -v ji , we premultiply each dot product by m ij,kl , where m ij,kl is determined as follows:

Since edges ( i, j ) , ( k, l ) are part of a triplet, the third case in Eqn. 7 will not arise, but it will be helpful in the dense network case. We premultiply m ij,kl to the corresponding dot products. We take the inverse cosine of the sign-corrected dot products to get the angles and allocate A G .

Step 3: It is easy to extract angles from A G since edges corresponding to a triplet are known and A G provides an easy way to index edges based on its rows and columns. Let Trp be the set of triplets in G . We check if the angles of the triangles are greater than a minimum threshold and remove the triplets having skewed triangles. This creates two sets of triplets Trp Ret and Trp Rem denoting retained and removed triplets, both being mutually exclusive. Let the edges contributing to the two triplet sets be E ret and E rem , respectively, such that E = E ret ∪E rem . We note that E ret ∩E rem is not necessarily empty. For a parallel rigid G , the two edge sets are not mutually exclusive since there will be common edges between the triplets of E ret and E rem , as a consequence of Thm. 4. Let the filtered network be denoted as ˜ G F = ( ˜ V F , ˜ E F ) . There are two ways to remove edges: one is to remove edges aggressively such that ˜ E F = E \ E rem and the other way is non-aggressive i.e. ˜ E F = E ret . Since all operations are vectorized, this makes the whole process of extracting the angles and removal of skewed triangles time efficient. Next, we construct the triplet network G T (as discussed in Sec. 4) from ˜ G F and extract the largest connected component of G T , to ensure that the network is connected and is parallel rigid. The edges of ˜ G F contributing to the largest connected component of G T gives the final network G F = ( V F , E F ) . The whole process is summarized in Algo 1. In our experiments, the network G is sparse, and we choose the non-aggressive way of removing the edges, which ensures

Algorithm 1: Removal of Skewed Triangles from Sparse Networks

Input: Bearing-based network G = ( V , E ) , containing only triplets and is parallel rigid, and the triplet list of G .

Output: Bearing-based network G F = ( V F , E F ) , containing only triplets and is parallel rigid, without skewed triangles.

1 Compute the non-zero elements in angle matrix A G using the boundary matrix B 2 (Eqn. 6).

2 Compute the dot product for the non-zero elements in A G .

3 Get the signs of the dot product i.e. m ij,kl using Eqn. 7.

4 Multiply dot products with their corresponding signs m ij,kl .

5 Take the inverse cosine of dot products.

6 Allocate the matrix A G . Extract the angles of the triplets from A G .

7 Filter out the triplets with the minimum angle less than a threshold to get ˜ G F = ( ˜ V F , ˜ E F ) .

8 Construct the triplet network G T using ˜ G F .

9 Get the largest connected component of G T .

10 Get the edges of ˜ G F contributing to the largest connected component of G T to get

G F = ( V F , E F ) , which is connected and is parallel rigid.

that the connected component of G F is large. We note that with non-aggressive pruning, we still ensure that the nodes are estimated reliably since the nodes are also a part of atleast one triplet, which belong to the set of non-skewed triangles.

6 Experiments

We consider SfM datasets provided in 1DSfM [55] for the experiments. It provides the relative motions and a reference reconstruction using Bundler [49, 50]. Since Bundler was published more than ten years ago, we use COLMAP [44] to generate the pairwise relative rotations and translations and use COLMAP's solution as the ground truth to get a better reconstruction. We take COLMAP solution and align it to the solution provided in 1DSfM to get absolute translations in meters. We compute absolute rotations using [11] and use them to align the relative translation directions into a global coordinate frame. The output of this process gives us a bearing-based network. Then, we extract triplets from the network and ensure parallel rigidity. Since the filtering process is independent of the cost function to solve the problem, we consider two representative cost functions. Revised LUD (an improvement on [41], provided by [57]), compares relative displacements, and BATA [57], compares relative directions. Please refer to appendix for the problem formulations. Our code is implemented in MATLAB. All experiments are performed on a PC with Intel Xeon Silver 4210 processor with 128 GB RAM. Finally, in each table, Mean-ATE and RMS-ATE denote the mean and RMS absolute translation errors, respectively, w/o and w/ filter denotes the network before and after removing skewed triangles and bold entries denote better performance between the solutions of the two networks.

6.1 Analysis of Real Data

In this subsection, we first provide an illustrative example using COLMAP [44] reconstruction and then analyze the real data. Fig. 2 shows an illustrative example of the existence of such skewed triangles in the SfM problem. For the Alamo dataset, most images capture the front part of the museum, and thus, the cameras are densely connected in the network. The blue triangle depicts a triplet which is Type-I ill-conditioned triangle and the green triangle shows Type-II ill-conditioned triangle.

Now, we analyze the real data to understand the frequency of occurrence of skewed triangles and their difference with outlier data. For the analysis, we compute the errors in relative directions with respect to ground truth. To check whether atleast one outlier is present in the triplet, we check the maximum error of relative directions in a triplet. Since we identify skewed triangles with the minimum angle in each triplet, we compare the maximum error of relative directions with the minimum angle between directions for each triplet. In Fig. 3, we show scatter plots between the two compared quantities for all the triplets for three datasets with different edge densities. At first, we observe that there are a

(a) Piccadilly (

ed

= 1

.

16%

)

(b) Tower of London (

ed

= 3

.

91%

)

(c) Union Square (

ed

= 2

.

76%

)

180

180

180

160

160

160

Max error (in degrees)

140

(saap u) oa xew

140

Max error (in degrees)

140

120

120

120

100

100

100

80

80

60

60

40

40

40

20

20

20

D

0

20

40

60

80

100

120

140

160

180

0

0

20

40

60

80

100

12 0

140

160

180

0

20

40

60

80

100

120

140

160

180

Min angle (in degrees)

Min angle (in degrees)

Min angle (in degrees)

considerable number of skewed triangles in the network since there are many points in the scatter plot which are close to 0 ◦ in the x-axis (representing the minimum angle in triplets). Next, we can also see that the minimum angle between edges in the triplets is independent of the maximum error in the triplets, implying that the presence of an outlier and the skewness of a triangle have no relation. Also, it can be seen that triplets with minimum angle > 60 ◦ are clear outliers and all of them have a high angular error with respect to ground truth. Based on this observation, we first provide results on outlier-free data, and then we show the results on the real data with outliers in the next subsections.

6.2 Outlier-Free Data

In this subsection, we examine the impact of skewed triangles on the quality of the translation estimate independent of outliers. First, we extract the component of the network for which the ground truth is available. For this experiment, to remove the effect of errors coming from absolute rotation estimates, we use ground truth rotations to align the relative directions to a common reference frame. We consider the edges as outliers if the relative direction on the edge differs from its ground truth equivalent by more than 10 ◦ and remove them. Then, we extract the triplets from the network and ensure parallel rigidity. This gives us a network with no outliers but still contains skewed triangles. We call it the unfiltered network. Now, we use Algo. 1 to remove skewed triangles (minimum angle < 5 ◦ ) and denote the output as the filtered network and compare the solutions from the two networks.

In Table 1, we list the number of nodes and edges removed due to the removal of skewed triangles and check the absolute translation errors obtained using BATA. It can be seen with the removal of a small number of nodes and edges, the mean and RMS errors of the absolute translations decrease consistently (see appendix for the number of nodes and edges of the networks). We also check the errors of the nodes in the unfiltered network, which were removed due to the removal of skewed triangles. It can be seen that the mean and RMS errors of the removed nodes are high compared to the overall errors implying that absolute translations at these nodes are not well estimated. Absolute translation error obtained using Revised LUD is provided in the appendix, which also shows that removal of skewed triangles leads to improved translation estimates. This shows the impact of skewed triangles on the translation averaging and the benefits of removing them.

6.3 Real Data

In this subsection, we deal with real data in the context of SfM, which is obtained as described at the beginning of this section. We note that this data contain outliers, and our aim is to understand the impact of skewed triangles in the presence of outliers. We denote the network obtained as unfiltered and remove the skewed triangles (minimum angle < 5 ◦ ) using Algo. 1, calling it the filtered network. In Table 2, we provide details of the unfiltered and filtered network. It can be observed that the removal of a small fraction of nodes and edges from skewed triangles leads to a significant decrease in the condition number of the angle matrix (with matrix-2 norm), κ 2 ( A G ) , for all datasets expect ND. We employ non-aggressive pruning due to which some skewed triangles can remain in the filtered network making the condition number κ 2 ( A G ) increase for ND and the same for YKM. We reiterate that non-aggressive pruning still ensures that the nodes are well estimated (see Step 3 in Sec. 5). This shows that the conditioning of the translation averaging improves for most datasets, even with the non-aggressive removal of skewed triangles.

In Table 3, we present the absolute translation errors obtained using BATA without and with filtering skewed triangles. It can be observed that the mean and RMS errors improve for most of the datasets after the removal of skewed triangles. Also, errors of the removed nodes in the unfiltered network are high, which indicates that the absolute translations at these nodes are not reliable in the unfiltered network. We note that from Tables 2 and 3, the filtering time is ∼ 1% of the time taken for translation averaging, which shows the practicality of Algo. 1. Next, we perform 3D reconstruction using

Theia [52] with the absolute translation solutions obtained in Table 3. Theia removes the triangulated 3D points which have reprojection errors greater than 15 pixels. It can be seen from Table 4 that more 3D points are triangulated in the filtered networks for most of the datasets. We note that filtered networks have lesser nodes than unfiltered networks, which implies removal of nodes coming from skewed triangles leads to better conditioning for 3D point triangulation. We also see that the bundle adjustment converges faster for the filtered networks indicating absolute translations are more stable in the filtered networks compared to unfiltered networks.

Limitation: Our sensitivity analysis of translation averaging is based on the triplets in the network. Although we do not lose much in terms of 3D reconstruction in SfM, further work is required to understand the sensitivity of a general bearing-based network.

7 Conclusion

This paper deals with sensitivity in translation averaging under input uncertainty. We study sensitivity in estimating edge scales in bearing-based networks which suggests skewed triangles are unstable. We define the conditioning of the translation averaging problem and provide a sufficient criterion to ensure that the problem is well-conditioned. Then, we propose an efficient algorithm to remove skewed triangles from the network while ensuring parallel rigidity. We demonstrate the effectiveness of our filtering scheme using structure-from-motion data without and with outliers leading to better absolute translation estimates, more 3D points triangulated and faster convergence of bundle adjustment for filtered networks.

Acknowledgments and Disclosure of Funding

Lalit Manam is supported by a Prime Minister's Research Fellowship, Government of India. This research was supported in part by a Core Research Grant from Science and Engineering Research Board, Department of Science and Technology, Government of India.

References

Mica Arie-Nachimson, Shahar Z Kovalsky, Ira Kemelmacher-Shlizerman, Amit Singer, and Ronen Basri. Global motion estimation from point matches. In 2012 Second international conference on 3D imaging, modeling, processing, visualization & transmission , pages 81-88. IEEE, 2012.

Federica Arrigoni and Andrea Fusiello. Bearing-based network localizability: a unifying view. IEEE transactions on pattern analysis and machine intelligence , 41(9):2049-2069, 2018.

Federica Arrigoni, Andrea Fusiello, and Beatrice Rossi. On computing the translations norm in the epipolar graph. In 2015 International Conference on 3D Vision , pages 300-308. IEEE, 2015.

Federica Arrigoni, Andrea Fusiello, and Beatrice Rossi. Camera motion from group synchronization. In 2016 Fourth International Conference on 3D Vision (3DV) , pages 546-555. IEEE, 2016.

Federica Arrigoni, Beatrice Rossi, and Andrea Fusiello. Robust and efficient camera motion synchronization via matrix decomposition. In International Conference on Image Analysis and Processing , pages 444-455. Springer, 2015.

Federica Arrigoni, Beatrice Rossi, and Andrea Fusiello. Spectral synchronization of multiple views in se (3). SIAM Journal on Imaging Sciences , 9(4):1963-1990, 2016.

Richard Beals, David H Krantz, and Amos Tversky. Foundations of multidimensional scaling. Psychological review , 75(2):127, 1968.

Sylvie Boldo. How to compute the area of a triangle: a formal revisit. In 2013 IEEE 21st Symposium on Computer Arithmetic , pages 91-98. IEEE, 2013.

Ingwer Borg and Patrick JF Groenen. Modern multidimensional scaling: Theory and applications . Springer Science & Business Media, 2005.

Avishek Chatterjee and Venu Madhav Govindu. Efficient and robust large-scale rotation averaging. In Proceedings of the IEEE International Conference on Computer Vision , pages 521-528, 2013.

Avishek Chatterjee and Venu Madhav Govindu. Robust relative rotation averaging. IEEE transactions on pattern analysis and machine intelligence , 40(4):958-972, 2017.

Robert Connelly. Generic global rigidity. Discrete & Computational Geometry , 33(4):549, 2005.

Hainan Cui, Shuhan Shen, and Zhanyi Hu. Robust global translation averaging with feature tracks. In 2016 23rd International Conference on Pattern Recognition (ICPR) , pages 3727-3732. IEEE, 2016.

Zhaopeng Cui, Nianjuan Jiang, Chengzhou Tang, and Ping Tan. Linear global translation estimation with feature tracks. In Proc. ECCV , volume 3, pages 61-75, 2014.

Qiulei Dong, Xiang Gao, Hainan Cui, and Zhanyi Hu. Robust camera translation estimation via rank enforcement. IEEE transactions on cybernetics , 2020.

H. Edelsbrunner and J. Harer. Computational Topology: An Introduction . Applied Mathematics. American Mathematical Society, 2010.

Herbert Edelsbrunner, Xiang-Yang Li, Gary Miller, Andreas Stathopoulos, Dafna Talmor, Shang-Hua Teng, Alper Üngör, and Noel Walkington. Smoothing and cleaning up slivers. In Proceedings of the thirty-second annual ACM symposium on Theory of computing , pages 273-277, 2000.

Olof Enqvist, Fredrik Kahl, and Carl Olsson. Non-sequential structure from motion. In 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops) , pages 264-271. IEEE, 2011.

Tolga Eren, Walter Whiteley, and Peter N Belhumeur. Using angle of arrival (bearing) information in network localization. In Proceedings of the 45th IEEE Conference on Decision and Control , pages 4676-4681. Ieee, 2006.

Tolga Eren, Walter Whiteley, A Stephen Morse, Peter N Belhumeur, and Brian DO Anderson. Sensor and network topologies of formations with direction, bearing, and angle information between agents. In 42nd IEEE International Conference on Decision and Control (IEEE Cat. No. 03CH37475) , volume 3, pages 3064-3069. IEEE, 2003.

A. Eriksson, C. Olsson, F. Kahl, and T. Chin. Rotation averaging and strong duality. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 127-135, 2018.

Thomas Goldstein, Paul Hand, Choongbum Lee, Vladislav Voroninski, and Stefano Soatto. Shapefit and shapekick for robust, scalable structure from motion. In European Conference on Computer Vision , pages 289-304. Springer, 2016.

Venu Madhav Govindu. Combining two-view constraints for motion estimation. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 , volume 2, pages II-II. IEEE, 2001.

Venu Madhav Govindu. Lie-algebraic averaging for globally consistent motion estimation. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004. , volume 1, pages I-I. IEEE, 2004.

Fred G Gustavson, José E Moreira, and Robert F Enenkel. The fused multiply-add instruction leads to algorithms for extended-precision floating point: applications to java and high-performance computing .

Aric A. Hagberg, Daniel A. Schult, and Pieter J. Swart. Exploring network structure, dynamics, and function using networkx. In Gaël Varoquaux, Travis Vaught, and Jarrod Millman, editors, Proceedings of the 7th Python in Science Conference , pages 11 - 15, Pasadena, CA USA, 2008.

Richard Hartley, Khurrum Aftab, and Jochen Trumpf. L1 rotation averaging using the weiszfeld algorithm. In CVPR 2011 , pages 3041-3048. IEEE, 2011.

R. I. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision . Cambridge University Press, ISBN: 0521540518, second edition, 2004.

A. Hatcher. Algebraic Topology . Algebraic Topology. Cambridge University Press, 2002.

Roger A Horn and Charles R Johnson. Matrix analysis . Cambridge university press, 2012.

Xiangru Huang, Zhenxiao Liang, Chandrajit Bajaj, and Qixing Huang. Translation synchronization via truncated least squares. Advances in neural information processing systems , 30, 2017.

Nianjuan Jiang, Zhaopeng Cui, and Ping Tan. A global linear method for camera pose registration. In Proceedings of the IEEE international conference on computer vision , pages 481-488, 2013.

Yoni Kasten, Amnon Geifman, Meirav Galun, and Ronen Basri. Algebraic characterization of essential matrices and their averaging in multiview settings. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5895-5903, 2019.

Yoni Kasten, Amnon Geifman, Meirav Galun, and Ronen Basri. Gpsfm: Global projective sfm using algebraic constraints on multi-view fundamental matrices. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3264-3272, 2019.

Bastian Katz, Marco Gaertler, and Dorothea Wagner. Maximum rigid components as means for directionbased localization in sensor networks. Lecture Notes in Computer Science , 4362:330, 2007.

Ryan Kennedy, Kostas Daniilidis, Oleg Naroditsky, and Camillo J Taylor. Identifying maximal rigid components in bearing-based localization. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 194-201. IEEE, 2012.

Lalit Manam and Venu Madhav Govindu. Correspondence reweighted translation averaging. In European Conference on Computer Vision, Proceedings, Part XXXIII , pages 56-72. Springer, 2022.

de Berg Mark, Cheong Otfried, van Kreveld Marc, and Overmars Mark. Computational geometry algorithms and applications . Spinger, 2008.

Carl D Meyer. Matrix analysis and applied linear algebra , volume 71. Siam, 2000.

Pierre Moulon, Pascal Monasse, and Renaud Marlet. Global fusion of relative motions for robust, accurate and scalable structure from motion. In Proceedings of the IEEE International Conference on Computer Vision , pages 3248-3255, 2013.

Onur Ozyesil and Amit Singer. Robust camera location estimation by convex programming. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2674-2683, 2015.

Onur Ozyesil, Amit Singer, and Ronen Basri. Stable camera motion estimation using convex programming. SIAM Journal on Imaging Sciences , 8(2):1220-1262, 2015.

Deepti Pachauri, Risi Kondor, Gautam Sargur, and Vikas Singh. Permutation diffusion maps (pdm) with application to the image association problem in computer vision. Advances in Neural Information Processing Systems , 27, 2014.

Johannes Lutz Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4104-4113, 2016.

Brigitte Servatius and Walter Whiteley. Constraining plane configurations in computer-aided design: Combinatorics of directions and lengths. SIAM Journal on Discrete Mathematics , 12(1):136-153, 1999.

Yanyao Shen, Qixing Huang, Nati Srebro, and Sujay Sanghavi. Normalized spectral map synchronization. Advances in neural information processing systems , 29, 2016.

Yunpeng Shi and Gilad Lerman. Message passing least squares framework and its application to rotation synchronization. In International Conference on Machine Learning , pages 8796-8806. PMLR, 2020.

Kristy Sim and Richard Hartley. Recovering camera motion using l \ infty minimization. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06) , volume 1, pages 1230-1237. IEEE, 2006.

Noah Snavely, Steven M Seitz, and Richard Szeliski. Photo tourism: exploring photo collections in 3d. In ACM siggraph 2006 papers , pages 835-846. 2006.

Noah Snavely, Steven M Seitz, and Richard Szeliski. Modeling the world from internet photo collections. International journal of computer vision , 80:189-210, 2008.

Yifan Sun, Jiacheng Zhuo, Arnav Mohan, and Qixing Huang. K-best transformation synchronization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10252-10261, 2019.

Christopher Sweeney, Tobias Hollerer, and Matthew Turk. Theia: A fast and scalable structure-from-motion library. In Proceedings of the 23rd ACM international conference on Multimedia , pages 693-696, 2015.

Roberto Tron, Luca Carlone, Frank Dellaert, and Kostas Daniilidis. Rigid components identification and rigidity control in bearing-only localization using the graph cycle basis. In 2015 American Control Conference (ACC) , pages 3911-3918. IEEE, 2015.

Roberto Tron and René Vidal. Distributed image-based 3-d localization of camera sensor networks. In Proceedings of the 48h IEEE Conference on Decision and Control (CDC) held jointly with 2009 28th Chinese Control Conference , pages 901-908. IEEE, 2009.

Kyle Wilson and Noah Snavely. Robust global translations with 1dsfm. In European Conference on Computer Vision , pages 61-75. Springer, 2014.

Shiyu Zhao and Daniel Zelazo. Localizability and distributed protocols for bearing-based network localization in arbitrary dimensions. Automatica , 69:334-341, 2016.

Bingbing Zhuang, Loong-Fah Cheong, and Gim Hee Lee. Baseline desensitizing in translation averaging. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4539-4547, 2018.

Sensitivity in Translation Averaging Appendix

Lalit Manam

Indian Institute of Science Bengaluru, India - 560012

lalitmanam@iisc.ac.in

8 Proofs for Theorems

Before we prove the theorems, we first state the results, which will be helpful in proving Thm. 1. We first state a result from first-order eigenvalue perturbation theory [30].

Result 1. Given a matrix X ∈ R M × M and a small additive perturbation on it by a matrix δ X ∈ R M × M , the absolute change in eigenvalue, denoted as | δλ ( X ) | , due to the perturbation is bounded by

where w , z ∈ C M , and are the left and right eigenvectors of X corresponding to the same eigenvalue, respectively.

Since w and z are eigenvectors, ‖ w ‖ 2 = 1 and ‖ z ‖ 2 = 1 . For a symmetric matrix X , the left and the right eigenvectors corresponding to any eigenvalue are the same due to which | w H z | = | w H w | = 1 . Hence, for symmetric matrix X , Result 1 can be rewritten as

where σ max ( δ X ) is the maximum singular value of δ X . Next, we provide our result on the singular value decomposition of a specific matrix, which will be used further.

Result 2. Given a matrix X ∈ R 3 × 3 such that

the singular value decomposition of X is given as X = USW T , where

Now, we prove the theorems stated in the main paper.

Venu Madhav Govindu

Indian Institute of Science Bengaluru, India - 560012 venug@iisc.ac.in

Theorem 1. For a set of consistent directions, v ij , ( i, j ) ∈ E ∆ , in G ∆ , the absolute change in any eigenvalue of V T V , denoted as | δλ | , when the directions v ij are perturbed by small rotations δ R ij , with n ij and δθ ij > 0 being the rotation axis and angle, is bounded by

where TI (∆) = { (1 , 2 , 3) , (2 , 3 , 1) , (3 , 1 , 2) } , v n ij ⊥ is the component of v orthogonal to n ij , φ n ij ⊥ ( k,i ) , ( k,j ) is the angle between v n ij ⊥ ki and v n ij ⊥ kj , and φ ( k,i ) , ( k,j ) is the angle between v ki and v kj .

Proof. Our goal is to check for the absolute change in the eigenvalue of V T V , denoted as | δλ | . V T V is a symmetric matrix. Using Result 1 and Eqn. 9, finding the matrix-2 norm of the perturbation on V T V will give us the bound on the change in its eigenvalue.

Let us first look at the case when one edge is perturbed and then extend it later to all edges. We note that E ∆ = { (1 , 2) , (2 , 3)(3 , 1) } . Without loss of generality, we perturb v 31 and denote the perturbed vector as v P 31 . Since the perturbation is made by a small rotation δ R 31 , we can write the perturbed vector as

where δ ω 31 = δθ 31 n 31 . From Eqn. 11, it can be seen that the perturbation can be treated as an additive perturbation v P 31 = v 31 + δ v 31 , where δ v 31 = δ ω 31 × v 31 . Let the perturbed matrix of the directions be V P such that V P = V + δ V 31 , where δ V 31 = [ 0 0 δ v 31 ] . Then, ( V P ) T V P can be written as

where δ 31 ( V T V ) = V T δ V 31 + δ V T 31 V . After simplification of δ 31 ( V T V ) , we get

It can be seen that δ 31 ( V T V ) has the same structure as that of the matrix in Eqn. 10. So, using Result 2, we get

Since the directions are consistent, the three directions are coplanar. So, there exists α and β such that

We obtain α and β by premultiplying Eqn. 15 by v T 12 and v T 23 to get the following set of equations

Solving Eqns. 16 and 17 and using the fact that v T ij v ij = 1 , we get

Also, using Eqn. 15, we get

Using Eqns. 23 and 24, we get

Premultiplying n T 31 in Eqn. 25, we get

since, by construction, v n 31 ⊥ 23 × v n 31 ⊥ 12 = ‖ v n 31 ⊥ 23 × v n 31 ⊥ 12 ‖ · n 31 . Using Eqn. 26 into Eqn. 22, we get

Using the values of α and β from Eqns. 18 and 19, we get

where φ n 31 ⊥ (2 , 1) , (2 , 3) is the angle between v n 31 ⊥ 21 and v n 31 ⊥ 23 , and φ (2 , 1) , (2 , 3) is the angle between v 21 and v 23 .

Now, we perturb all three edges in G ∆ . Then, Eqn. 12 will change to

Using Result 1, we get,

Using Eqns. 20 and 21 in Eqn. 14, we get

Let us decompose the directions as v = v n ‖ + v n ⊥ , where v n ‖ and v n ⊥ denote parallel and perpendicular components of v to n , which can be obtained as

Following Eqn. 28 for each edge perturbation in Eqn. 30, we get

Corollary 1. For a set of consistent directions, v ij , ( i, j ) ∈ E ∆ , in G ∆ , the absolute change in any eigenvalue of V T V , denoted as | δλ | , when the directions v ij are perturbed by small rotations δ R ij , with n ij and δθ ij > 0 being the rotation axis and angle, and n ij being othogonal to v ki and v kj for all ( i, j, k ) ∈ TI (∆) , is bounded by

Proof. The unperturbed directions v ij are coplanar since the directions are consistent. Given that the rotation axis n ij for all the rotations δ R ij are orthogonal to v ki and v kj for all ( i, j, k ) ∈ TI (∆) , the rotation axis n ij is orthogonal to v kl , for all ( i, j ) , ( k, l ) ∈ E ∆ . This ensures that the perturbed directions are still coplanar and, thus, are consistent. Since n ij is orthogonal to all the directions v ij , the component of directions parallel to the axis are zero, i.e. v n ‖ = 0 . So, v n ⊥ = v (refer to Eqns. 23 and 24) and sin φ n ij ⊥ ( k,i ) , ( k,j ) = sin φ ( k,i ) , ( k,j ) . Also, ‖ v n ⊥ ij ‖ = ‖ v ij ‖ = 1 for all ( i, j ) ∈ E ∆ . Hence, using these in Eqn. 2, we get

Theorem 2. Consider the bearing based-network of 3 nodes and 3 edges, G ∆ = ( V ∆ , E ∆ ) , and the corresponding angle matrix A G ∆ . The conditioning of the matrix A G ∆ signifies the skewness of the triangle formed using the directions in E ∆ .

Proof. The determinant of A G ∆ is given as | A G ∆ | = 2 · φ (1 , 2) , (1 , 3) · φ (2 , 1) , (2 , 3) · φ (3 , 1) , (3 , 2) . The determinant will be zero if and only if atleast one of the angles in the triangle is zero. So, the matrix is non-singular when all the angles are non-zero.

It is known that the determinant is not a good measure of the closeness of the matrix to singularity [39]. So, we consider the closeness of any two columns in the matrix in terms of the cosine of the angles between the column vectors of A G ∆ . Let a ij G ∆ be the column of A G ∆ corresponding to the edge ( i, j ) ∈ E ∆ . In G ∆ , there is one node common for any pair of edges. So, the cosine of the angle between any two columns is given as cos ∠ ( a ij , a jk ) = φ ( i,j ) , ( i,k ) · φ ( k,i ) , ( k,j ) .

G ∆ G ∆ √ φ 2 ( i,j ) , ( i,k ) + φ 2 ( j,i ) , ( j,k ) · √ φ 2 ( j,i ) , ( j,k ) + φ 2 ( k,i ) , ( k,j ) It can be clearly seen that if the angle at the common node j is small, i.e. φ ( j,i ) , ( j,k ) → 0 radian, then the two columns are very close or cos ∠ ( a ij G ∆ , a jk G ∆ ) → 1 . Small values of φ ( j,i ) , ( j,k ) make A G ∆ nearly singular and its condition number high. If all values of φ ( j,i ) , ( j,k ) are sufficiently large, then cos ∠ ( a ij G ∆ , a jk G ∆ ) between any two columns will not be close to 1 making A G ∆ well conditioned with a low condition number. Thus, the condition number of A G ∆ is reflective of the skewness of the triangle formed from the directions in E ∆ . glyph[squaresolid]

Theorem 3. Consider a bearing-based network G , with all edges contributing to triplets. The angle matrix A G , corresponding to G , is well conditioned if the minimum angle (or equivalently all the angles) in all the triangles formed by the triplets are sufficiently large.

Proof. Let us assume that all the angles in all the triangles formed by the triplets in G are sufficiently large. In A G , no two columns can have exactly the same set of non-zero elements. This is evident since the diagonal entries are zero. In real-world scenarios, G is a sparse network, and thus A G is sparse. If two columns have exactly the same set of non-zero elements, then it means the two edges share exactly the same set of triplets. This is attained only when the two edges are connecting the same two nodes. But this is not possible since multiple edges between nodes are not allowed. Also, every column will have atleast two non-zero elements since every edge is a part of atleast one triplet and angles measured with the common nodes between edges will be present. This makes the angle matrix non-singular when all the angles in all the triangles are sufficiently large.

Next, we look at the conditioning of the matrix. Let a ij G be the column of A G corresponding to the edge ( i, j ) ∈ E . We check the cosine of the angle between any two columns of A G to understand the conditioning of A G . We define triplets to be adjacent if they share a common edge, otherwise they are non-adjacent . There are only three cases which arise between any two edges ( i, j ) and ( k, l ) , ( i, j ) , ( k, l ) ∈ E , and are listed below:

( i, j ) and ( k, l ) belong to the same triplet,

( i, j ) and ( k, l ) belong to adjacent triplets,

( i, j ) and ( k, l ) belong to non-adjacent triplets.

Since each column in A G represents an edge, there are only the three cases, stated above, which occur for the pair of columns. For all three cases, the cosine of angles between two columns is checked. Let Trp be the set of all triplets in G .

Case 1: Edges ( i, j ) and ( k, l ) belong to the same triplet.

glyph[negationslash]

where c ∈ { i, j } ∩ { k, l } is the common node and c ′ 1 , c ′ 2 ∈ { i, j } ∪ { k, l } \ { c } with c ′ 1 = c ′ 2 are the other two nodes in the triplet.

It can be seen that for this case, if all the angles are sufficiently large, then cos ∠ ( a ij G , a kl G ) cannot be close to 1, and thus the two columns are not similar.

Case 2: Edges ( i, j ) and ( k, l ) belong to adjacent triplets tr 1 , tr 2 ∈ Trp .

Part I: When the edges ( i, j ) and ( k, l ) share a common node of the triplets tr 1 and tr 2 :

where c ∈ { i, j } ∩ { k, l } is the common node, ( r, c ) ∈ E is the common edge between tr 1 and tr 2 , and c ′ 1 ∈ { i, j } \ { c } , c ′ 2 ∈ { k, l } \ { c } are the other two nodes in the two triplets.

Part II: When the edges ( i, j ) and ( k, l ) are connected by the common edge of the triplets tr 1 and tr 2 :

glyph[negationslash]

where ( c ′ 1 , c ′ 2 ) ∈ E is the common edge between tr 1 and tr 2 such that c ′ 1 , c ′ 2 ∈ { i, j, k, l } with c ′ 1 = c ′ 2 , and c ′′ 1 , c ′′ 2 ∈ { i, j, k, l } \ { c ′ 1 , c ′ 2 } with c ′′ 1 = c ′′ 2 are the other two nodes in the triplet such that ( c ′ 1 , c ′ 2 , c ′′ 1 ) , ( c ′ 1 , c ′ 2 , c ′′ 2 ) ∈ Trp .

glyph[negationslash]

It can be seen that for both parts in this case, if all the angles are sufficiently large, then cos ∠ ( a ij G , a kl G ) cannot be close to 1, and thus the two columns are not similar.

Case 3: Edges ( i, j ) and ( k, l ) belong to non-adjacent triplets tr 1 , tr 2 ∈ Trp .

By the construction of the matrix using Eqn. 5, there would be no common non-zero elements in the two columns. So,

Thus, if all the angles in all the triangles formed by the triplets in G are sufficiently large, then the angle matrix A G is non-singular, and the columns are not similar implying that A G is well-conditioned. glyph[squaresolid]

Theorem 4. Given a bearing-based network G , with all edges contributing to triplets forming triangles, and its corresponding triplet network G T , the maximal parallel rigid component of G can be determined by the edges in G contributing to the largest connected component of G T .

Proof. It is known that the union of parallel rigid components is parallel rigid if the components share atleast one edge [2]. Each triplet forming a triangle in G is parallel rigid by itself. Also, by construction, each connected component of the triplet network G T will have shared edges among triplets forming triangles, making every component parallel rigid independently. Thus, the maximal parallel rigid graph of G can be determined by the edges in G participating in the maximum connected component of G T . glyph[squaresolid]

9 Proposed Method for Dense Networks

In Sec. 5 of the main paper, we presented the algorithm to remove skewed triangles from a sparse network G . Here, we consider the case when G is dense (nearly all possible edges exist), making A G dense. We provide the method for removing skewed triangles in the following steps.

Step 1: First, construct a matrix V with columns as the directions in G . The elements of V T V give the dot product between all possible directions. Then, construct the boundary matrix using Eqn. 6. The row and column indices of non-zero elements of the matrix B 2 B T 2 give us the edge pairs participating in triplets which are the non-zero elements in the angle matrix A G . So, the zero elements of the matrix B 2 B T 2 can be used to mask the elements in V T V . The net matrix generated is ˜ C G .

Step 2: We identify the correct signs for the elements in ˜ C G to ensure that the dot products reflect the cosine of the angles of the triangles. We first create a matrix M consisting of m ij,kl (Eqn. 7) as the elements corresponding to the row for ( i, j ) th edge and column for ( k, l ) th edge. Then, the Hadamard product (element-wise multiplication) of ˜ C G and M gives C G . Once the matrix C G is formed, we take element-wise inverse cosine to get the angles as the entries and thus the angle matrix A G .

Step 3: This step is the same as that provided in the main paper.

We summarize the method in Algo. 2.

10 Formulations for Solving Translation Averaging

In Sec. 6 of the main paper, we used two formulations, Revised LUD [41, 57] and BATA [57], for the experiments. Here, we provide the optimization problems of those methods for ready reference.

Algorithm 2: Removal of Skewed Triangles from Dense Networks

Input: Bearing-based network G = ( V , E ) , containing only triplets and is parallel rigid, and the triplet list of G .

Output: Bearing-based network G F = ( V F , E F ) , containing only triplets and is parallel rigid, without skewed triangles.

1 Compute the non-zero elements in angle matrix A G using the boundary matrix B 2 (Eqn. 6).

2 Construct the matrix V , whose columns consists of directions v ij , ( i, j ) ∈ E .

3 Get dot products between all possible directions using V T V .

4 Use zero elements in B T 2 B 2 to mask elements in V T V , and denote it as ˜ C G .

5 Get the signs of the dot product for non-zero elements of ˜ C G , i.e. m ij,kl using Eqn. 7.

6 Construct a matrix M using m ij,kl .

7 Perform element-wise multiplication of ˜ C G with M to get C G .

8 Get the inverse cosine for every element in C G to get A G .

9 Extract the angles of the triplets from A G .

10 Filter out the triplets with the minimum angle less than a threshold to get ˜ G F = ( ˜ V F , ˜ E F ) .

11 Construct the triplet network G T using the filtered network ˜ G F .

12 Get the largest connected component of G T .

13 Get the edges of ˜ G F contributing to the largest connected component of G T to get G F = ( V F , E F ) , which is connected and is parallel rigid.

Revised LUD [41, 57] (compares relative displacements):

BATA [57] (compares relative directions):

The zero centroid and dot product constraints in Eqns. 31 and 32 fix the origin and the global scale ambiguities, respectively. ρ denotes the Cauchy loss function. λ ij and γ ij are non-negative variables that are ideally equal to baseline and inverse baseline for the edge ( i, j ) , respectively.

11 Additional Results

As mentioned in the main paper, our filtering scheme uses only input directions and does not favour any specific cost function used for translation averaging. In Sec. 6 of the main paper, we presented the results using BATA [57]. In the following subsection, we present the results using Revised LUD [41, 57] and observe a similar trend as seen for the BATA results.

11.1 Outlier-Free Data

In Sec. 6.2 of the main paper, we provided experimental results on outlier-free data. Here, in Table 5, we provide the details of the 1DSfM [55] datasets without outliers. It can be seen that only a small fraction of the nodes and edges are removed due to the removal of skewed triangles. Also, it can be seen that for many datasets, the condition number of the angle matrix (with matrix-2 norm) decreases. For remaining datasets, it either increases or is similar for both the unfiltered and filtered networks. As mentioned in Sec. 5 of the main paper, we perform non-aggressive pruning, due to which some

skewed triangles are still present. We re-emphasize that non-aggressive pruning still ensures that the nodes are estimated reliably since the nodes are also a part of atleast one triplet, which belongs to the set of non-skewed triangles.

In Table 6, we provide the results obtained from Revised LUD. It can be seen that the absolute translation errors decrease for most of the datasets after the removal of skewed triangles. Moreover, errors of the removed nodes in the unfiltered network are high compared to the overall errors, indicating that the removed nodes are not estimated reliably in the unfiltered network. We also observe that the time taken to filter skewed triangles (from Table 5) is ∼ 1% of the time taken for translation averaging by Revised LUD (Table 6), which shows the efficiency of Algo. 1 of the main paper.

11.2 Real Data

In Sec. 6.3 of the main paper, we provided the details of the datasets and the translation averaging results using BATA [57]. In Table 7, we present the results using Revised LUD. It can be seen that similar to the trend in BATA results, the absolute translation estimates improve for most datasets. Also, the errors of the nodes removed due to filtering are high compared to the overall errors in the unfiltered network, indicating that the filtered nodes are not estimated reliably in the unfiltered network.

Next, we apply 1DSfM outlier filter (Algo. 2, [55]) on the datasets to study the complementary benefits of removal of skewed triangles after applying an outlier filter. We reiterate that the removal of skewed triangles and outlier edges are two distinct aspects of the problem. In Table 8, we provide the network details after applying 1DSfM outlier filter without and with the removal of skewed triangles and compare their performance in Table 9. It can be seen that applying the 1DSfM outlier filter and then removing skewed triangles with our filter leads to better translation estimates. Moreover, our filter + 1DSfM outlier filter removes the nodes which are poorly estimated than only 1DSfM outlier filter, as seen from removed node errors in the unfiltered network in Table 9. Since both outliers and skewed triangles are different issues, combining filters for both types improves accuracy.

In Sec. 5, we mentioned the improved speed using Algo. 1 compared to the brute force method. In Table 10, we compare the time taken by the brute force method to remove skewed triangles to that of our method in Algo. 1. In the brute force method, all three angles in every triangle are computed, and then the skewed triangles are identified and removed. We used 20 threads in parallel for the brute force method, while our method is based only on vectorized operations without any parallelization. It can be seen that Algo. 1 is significantly faster than the brute force method with ∼ 100 times faster for most datasets and ∼ 1000 times faster for large scale datasets. This shows the efficiency of Algo. 1.

Finally, we show the differences in reconstructions obtained without and with using our filter. In Fig. 4, the arch in the Notre Dame reconstruction is misplaced due to improper translation estimates, as seen in the lateral view. Using our filter removes such a misplaced arch. In Fig. 5, the wall in the

Piazza del Popolo reconstruction is misplaced, but such an effect is not seen after applying our filter. This reveals that removing skewed triangles helps get more accurate reconstructions.

(a) Without Algo. 1

(b) With Algo. 1

(c) Reference reconstruction

O

(a) Without Algo. 1

(b) With Algo. 1

O

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a method to efficiently remove view triplets from a pose graph where the minimum angle falls below a threshold. This is relevant to global SfM algorithms, where such triplets (i.e., triangles) lead to highly uncertain translation scale estimation. The method's performance has been demonstrated on the 1DSfM dataset.\n\nSTRENGTHS:\n- The proposed method is a nice solution to a simple idea. Even though I feel the description has been written in an overcomplicated way, I like the idea and the method. \n- The problem is relevant to global SfM methods where the translation averaging part is currently one of the main bottlenecks of achieving good accuracy. \n- The proposed solution is ~100 times faster than the trivial brute-force solution.\n\nWEAKNESSES:\nI have two major problems with the proposed method, and both are regarding the provided experiments:\n- First, the proposed method is not compared to any other filters (e.g., 1DSfM), which really makes it hard to understand how much it actually improves the SOTA (if it does). I know that the proposed solver can be easily combined with 1DSfM, but I am not convinced it leads to noticeable improvements in that case. To understand this, the authors should provide results of other filters with and without the proposed one. \n- Second, the authors use the 1DSfM dataset in their experiments where the addressed problem of skewed triangles is not really present. This can be seen in Table 2, which shows that actually only a few edges are removed from the pose graphs (it is surprising that removing such few edges can lead to an improvement in the final global SfM accuracy). I think it would make more sense to showcase the method on datasets where SfM is actually often failing due to the coinciding translation directions, e.g., on KITTI, which presents the trajectory of a moving vehicle. There, scale estimation is challenging. On the 1DSfM dataset, most global methods work reasonably well. Making global SfM work on such cases would have an impact.\n\nWithout comparison to baselines, the paper is a clear rejection to me. In case such a comparison is provided, I am willing to improve my rating. \n\nMinor things/comments:\n- The brute-force solution's timing should be compared to the proposed method. I know that the authors mentioned this in the text, but it would be helpful to show the actual run-time, given that the speed-up is the main contribution of the paper, while accuracy-wise, the brute-force method should be similarly good.\n- L109 \"being the rotation axis angle\" -> \"being the rotation axis and angle respectively\"\n- Fig.1 How can the minimum angle in a triangle be close to 180°? Maybe I misunderstood the figure.\n- L294 BATA is first mentioned here (aside from a half sentence in the introduction). It should be written down that BATA is used for estimating the global positions from the directions.\n- Eq.2 is quite something that could be visualized by a figure and it would help a lot in understanding it.\n\nQUESTIONS:\n- I don't call this a major issue, but it is definitely something that should be discussed in the manuscript. It is unclear why the authors removed skewed triangles instead of deleting a single edge. The choice of triangle removal should be justified.\n\nLIMITATIONS:\nDiscussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzed the sensitivity problem in translation averaging. Built upon the theoretical analysis of the skew triangles in a bearing network, this paper also proposes an efficient algorithm to identify and remove edges that can make the translation averaging problem ill-conditioned. The proposed algorithm is integrated into a global SfM pipeline. The experiments are conducted on the 1DSfM dataset, with different translation averaging solvers (RevisedLUD and BATA) used to solve for global positions. The quantitative results show the proposed algorithm effectively reduced the condition number of the angle matrix, and the translation errors are reduced consistently.\n\nSTRENGTHS:\nThis paper analyzed the sensitivity problem in translation averaging, which is different from the parallel rigidity problem and has not been considered before. Strict proofs are given for the theorems proposed in the paper. The proposed edge filtering algorithm is very efficient and effective.\n\nWEAKNESSES:\nThe main limitation is that the analysis and the proposed algorithm are based on the triplets in the graph, where the requirements are not always held in the real world. The sensitivity analysis in the translation averaging problem is useful, however, is less important than the parallel rigidity problem in my point of view. The paper gives sufficient quantitative results to support their method but is lack qualitative results.\n\nQUESTIONS:\nI have no more questions regarding the details of the proposed method. My question is on the experiment part. The authors evaluated their method on the 1DSfM dataset, which has been always used in many structure-from-motion methods. However, I think the dataset is out of date though it is very popular a decade ago. I would like to know did the authors evaluate their method on other more realistic datasets, such as some SLAM datasets and aerial datasets.\n\nLIMITATIONS:\nThe limitation is that the proposed algorithm is based on the triplets in the graph. I would like to see an improved version of this paper can be extended to more general graphs. For now, I have to say the method contributes not significantly to the NeurIPS community since it has limited application and is less important than the parallel rigidity analysis.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe translation averaging problem is considered, i.e recover absolute translations from pairwise relative translation directions. The paper focuses on analyzing the change in solution with small changes in the input relative directions. The smallest problem (3 nodes) is initially considered which allows to understand that skewed triangles (triplets) are problematic. In order to move to a general problem, the authors propose to consider the case of a network containing only triplets. It allows them to defined the conditioning of the translation averaging problem as the condition number of the \"angle matrix\". They prove a problem is well conditioned if the minimum angle in all triangles are sufficiently large. Thus, they propose a method to remove such triangles. They experimentally demonstrate that including the proposed method within a classical sfm pipeline allows to obtain better absolute translations, as well as more 3D points triangulated and faster BA.\n\nSTRENGTHS:\n1. Analyzing the sensitivity is a non-trivial problem.\n2. Several theorems are proven that allow to identify skewed triangles as problematic.\n3. The theorems are important to motivate the proposed skewed-triangle-removal algorithm.\n4. The experiments consider both the case of outlier free data and real data.\n5. The paper is well written and easy to read.\n\nWEAKNESSES:\nThe proposed algorithm consists in removing information to obtain a problem that is better conditioned. But it is said in the experiments that the skewness of a triangle is not related to the presence of outliers. So if I understood correctly, the removed triangles are not necessarily outliers and thus may contain important information. Thus the proposed algorithm may harm the final reconstruction. Could you please comment on this?\n\nQUESTIONS:\nPlease see \"weaknesses\"\n\nLIMITATIONS:\n.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose the sensitivity theory for the Translation Average problem (i.e., input is a large number of coordinate vertex point pairs in relative directions observations and the output is the absolute vertex coordinates with consistent scales), which can be used to efficiently identify the inputs that would make the problem ill-condition (i.e., too small relative direction angles and too large uncertainty to form valid constraints) and remove them to improve the overall algorithm accuracy and convergence speed.\n\nSTRENGTHS:\n1.The paper has solid theory, elegant derivation, and creative use of mathematical tools to analyze its uncertainty.\n2.The experiments fully confirm the authors’ claim which effectively improve the accuracy of the algorithm and accelerate the convergence speed.\n\nWEAKNESSES:\n1.The motivation of this article is not sufficient, I can understand that by comparing the direction of the angle we can determine whether it is an ill condition problem or not, but I do not know whether this situation often occurs in practical applications (from the experimental point of view, the improvement of accuracy is very limited).\n2.The article is rather obscure and difficult to understand, especially section4 and section5.\n3.Lack of qualitative experiments to demonstrate the importance of such problems and the superiority of the method.\n\nQUESTIONS:\n1.I completely missed the meaning of what is shown in Figure 1, could author reintroduce it\n2.Whether this uncertainty can be explicitly modeled as uncertainty of Gaussian distribution and incorporated into the optimization framework for solving absolute vertex positions.\n\nLIMITATIONS:\n1.I think the biggest problem of the paper is that this problem is not a very important problem, first of all, there must be similar methods inside other SFM works to avoid the uncertainty caused by too small parallax. Secondly, whether this extreme case is more common in real scenes is doubtful. Thirdly, if a vertex is observed by more than one other vertex, then the observation in other directions can be used to constraint the absolute vertex solution. So, the authors should explain again why it is an important question which I think may be proved using qualitative experiments.\n2.Some parts of the paper are too obscure, it is suggested to add more intuition descriptions, and visual diagrams to aid understanding.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}