Title: Tensor-Based Synchronization and the Low-Rankness of the Block Trifocal Tensor
Venue: NeurIPS 2024
Accepted: yes
Keywords: synchronization

tensor decomposition

structure from motion

multilinear rank

multiview geometry

trifocal tensor

higher-order scene information
Abstract: The block tensor of trifocal tensors provides crucial geometric information on the three-view geometry of a scene. The underlying synchronization problem seeks to recover camera poses (locations and orientations up to a global transformation) from the block trifocal tensor.  We establish an explicit Tucker factorization of this tensor, revealing a low multilinear rank of $(6,4,4)$ independent of the number of cameras under appropriate scaling conditions. We prove that this rank constraint provides sufficient information for camera recovery in the noiseless case. The constraint motivates a synchronization algorithm based on the higher-order singular value decomposition of the block trifocal tensor. Experimental comparisons with state-of-the-art global synchronization methods on real datasets demonstrate the potential of this algorithm for significantly improving location estimation accuracy. Overall this work suggests that higher-order interactions in synchronization problems can be exploited to improve performance, beyond the usual pairwise-based approaches.

Tensor-Based Synchronization and the Low-Rankness of the Block Trifocal Tensor

Abstract

The block tensor of trifocal tensors provides crucial geometric information on the three-view geometry of a scene. The underlying synchronization problem seeks to recover camera poses (locations and orientations up to a global transformation) from the block trifocal tensor. We establish an explicit Tucker factorization of this tensor, revealing a low multilinear rank of (6 , 4 , 4) independent of the number of cameras under appropriate scaling conditions. We prove that this rank constraint provides sufficient information for camera recovery in the noiseless case. The constraint motivates a synchronization algorithm based on the higher-order singular value decomposition of the block trifocal tensor. Experimental comparisons with stateof-the-art global synchronization methods on real datasets demonstrate the potential of this algorithm for significantly improving location estimation accuracy. Overall this work suggests that higher-order interactions in synchronization problems can be exploited to improve performance, beyond the usual pairwise-based approaches.

1 Introduction

Synchronization is crucial for the success of many data-intensive applications, including structure from motion, simultaneous localization and mapping (SLAM), and community detection. This problem involves estimating global states from relative measurements between states. While many studies have explored synchronization in different contexts using pairwise measurements, few have considered measurements between three or more states. In real-world scenarios, relying solely on pairwise measurements often fails to capture the full complexity of the system. For instance, in networked systems, interactions frequently occur among groups of nodes, necessitating approaches that can handle higherorder relationships. Extending synchronization to consider measurements between three or more states, however, increases computational complexity and requires sophisticated mathematical models. Addressing these challenges is vital for advancing various technological fields. For example, higher-order synchronization can improve the accuracy of 3D reconstructions in structure from motion by leveraging more complex geometric relationships. In SLAM, it enhances mapping and localization precision in dynamic environments by considering multi-robot interactions. Similarly, in social networks, it could lead to more accurate identification of tightly-knit groups. Developing efficient algorithms to handle higherorder measurements will open new research avenues and make systems more resilient and accurate.

In this work, we focus on a specific instance of the synchronization problem within the context of structure from motion in 3D computer vision, where each state represents the orientation and location of a camera. Traditional approaches rely on relative measurements encoded by fundamental matrices, which describe the relative projective geometry between pairs of images. Instead, we consider higher-order relative measurements encoded in trifocal tensors, which capture the projective information between triplets of images. Trifocal tensors uniquely determine the geometry of three views, even in the collinear case [1], making them more favorable than triplets of fundamental matrices

for synchronization. To understand the structure and properties of trifocal tensors in multi-view geometry, we carefully study the mathematical properties of the block tensor of trifocal tensors. We then use these theoretical insights to develop effective synchronization algorithms.

Directly relevant previous works. In the structure from motion problem, synchronization has traditionally been done using incremental methods, such as Bundler [2] and COLMAP [3]. These methods process images sequentially, gradually recovering camera poses. However, the order of image processing can impact reconstruction quality, as error may significantly accumulate. Bundle adjustment [4], which jointly optimizes camera parameters and 3D points, has been used to limit drifting but is computationally expensive.

Alternatively, global synchronization methods have been proposed. These methods process multiple images simultaneously, avoiding iterative procedures and offering more rigorous and robust solutions. Global methods generally optimize noisy and corrupted measurements by exploiting the structure of relative measurements and imposing constraints. Many global methods solve for orientation and location separately, using structures on SO (3) and the set of locations. Solutions for retrieving camera poses from pairwise measurements have been developed for camera orientations [5, 6, 7, 8, 9, 10], camera locations [11, 12, 13], and both simultaneously [14, 15, 16, 17]. Some methods explore the structure on fundamental or essential matrices [18, 19, 20].

Several attempts to extract information from trifocal tensors include works by: Leonardos et al. [21], which parameterizes calibrated trifocal tensors with non-collinear pinhole cameras as a quotient Riemannian manifold and uses the manifold structure to estimate individual trifocal tensors robustly; Larsson et al. [22], which proposes minimal solvers to determine calibrated radial trifocal tensors for use in an incremental pipeline, handling distorted images with constraints invariant to radial displacement; and Moulon et al. [23], which introduces a structure from motion pipeline, retrieving global rotations via cleaning the estimation graph and solving a least squares problem, and solving for translations by estimating trifocal tensors individually by linear programs. To our knowledge, no prior works develop a global pipeline where the synchronization operates directly on trifocal tensors.

Contribution of this work. The main contributions of this work are as follows:

Weestablish an explicit Tucker factorization of the block trifocal tensor when its blocks are suitably scaled, demonstrating a low multilinear rank of (6 , 4 , 4) . Moreover, we prove that this rank constraint is sufficient to determine the scales and fully characterizes camera poses in the noiseless case.

Wedevelop a method for synchronizing trifocal tensors by enforcing this low rank constraint on the block tensor. We validate the effectiveness of our method through tests on several real datasets in structure from motion.

2 Low-rankness of the block trifocal tensor

We first briefly review relevant background material in Section 2.1. Then we present the main new construction and theoretical results in Section 2.2.

2.1 Background

2.1.1 Cameras and 3D geometry

Given a collection of n images I 1 ,...,I n of a 3D scene, let t i ∈ R 3 and R i ∈ SO (3) denote the location and orientation of the camera associated with the image I i in the global coordinate system. Moreover, each camera is associated with a calibration matrix K i that encodes the intrinsic parameters of a camera, including the focal length, the principal points, and the skew parameter. Then, the 3 × 4 camera matrix has the following form, P i = K i R i [ I 3 × 3 , -t i ] and is defined up to nonzero scale. Three-dimensional world points X are represented as R 4 vectors in homogeneous coordinates, and the projection of X into the image corresponding to P is x = PX . 3D world lines L can be represented via Plücker coordinates as an R 6 vector. Then the projection of L onto the image corresponding to P is l = P L , where P is the 3 × 6 line projection matrix. It can be written as P = [ P 2 ∧ P 3 ; P 3 ∧ P 1 ; P 1 ∧ P 2 ] where P i is the i -th row of the camera matrix P and wedge denotes exterior product. Explicitly the ( i,j ) element of the line projection matrix can be calculated as the determinant of the submatrix, where the i -th row

is omitted and the column are selected as the j -th pair from [(1 , 2) , (1 , 3) , (1 , 4) , (2 , 3) , (2 , 4) , (3 , 4)] . The elements on the second row are multiplied by -1 .

To retrieve global poses, relative measurement of pairs or triplets of images is needed. Let x i and x j be any pair of corresponding keypoints in images I i and I j respectively, meaning that they are images of a common world point. The fundamental matrix F ij is a 3 × 3 matrix such that x T i F ij x j =0 . It is known that F ij encodes the relative orientation R ij = R i R T j and translation t ij = R i ( t i -t j ) through F ij = K -T i [ t ij ] × R ij K -1 j . The essential matrix corresponds to the calibrated case, where K i = I 3 × 3 for all i .

2.1.2 Trifocal tensors

Analogous to the fundamental matrix, the trifocal tensor T ijk is a 3 × 3 × 3 tensor that relates the features across images and characterizes the relative pose between a triplet of cameras P i ,P j ,P k . The trifocal tensor T ijk corresponding to cameras P i ,P j ,P k can be calculated by

2.1.3 Tucker decomposition and the multilinear rank of tensors

Wereview basic material on the Tucker decomposition and the multilinear rank of a tensor. We refer to [35] for more details while adopting its notation. Let T ∈ R I 1 × I 2 ×···× I N be an order N tensor. The modei flattening (or matricization ) T ( i ) ∈ R I i × ( I 1 ...I i -1 I i +1 ...I N ) is the rearrangement of T into a matrix by taking modei fibers to be columns of the flattened matrix. By convention, the ordering of the columns in the flattening follows lexicographic order of the modes excluding i . Symbols ⊗ and ⊙ denote the Kronecker product and the Hadamard product respectively. The norm on tensors is defined as ∥ T ∥ = ∥ T (1) ∥ F . The i -rank of T is the column rank of T ( i ) and is denoted as rank i ( T ) . Let R i =rank i ( T ) . Then the multilinear rank of T is defined as mlrank( T ) = ( R 1 ,R 2 ,...,R N ) . The i -mode product of T with a matrix U ∈ R m × I i is a tensor in R I 1 ×···× I i -1 × m × I i +1 ×···× I N such that

Then, the Tucker decomposition of T ∈ R I 1 × I 2 ×···× I N is a decomposition of the following form:

where G∈ R Q 1 ×···× Q N is the core tensor, and A n ∈ R I n × Q n are the factor matrices. Without loss of generality, the factor matrices can be assumed to have orthonormal columns. Given the multilinear rank of the core tensor ( R 1 ,...,R N ) , the Tucker decomposition approximation problem can be written as

A standard way of solving (2) is the higher-order singular value decomposition ( HOSVD ). The HOSVDis computed with the following steps. First, for each i calculate the factor matrix A i as the R i leading left singular vectors of T ( i ) . Second, set the core tensor G as G = T × 1 A T 1 × 2 ···× N A T N . Though the solution from HOSVD will not be the optimal solution to (2), it satisfies a quasi-optimality property: if T ∗ is the optimal solution, and T ′ the solution from HOSVD, then

2.2 LowTucker rank of the block trifocal tensor and one shot camera retrieval

Suppose we are given a set of camera matrices { P i } n i =1 with n ≥ 3 and scales fixed on each camera matrix. Define the block trifocal tensor T n to be the 3 n × 3 n × 3 n tensor, where the 3 × 3 × 3 sized ijk block is the trifocal tensor corresponding to the triplet of cameras P i ,P j ,P k . Weassume for all blocks that have overlapping indices, the corresponding 3 × 3 × 3 tensor is also calculated using the formula (1). We summarize key properties of T n in Proposition 1 and Theorem 1. The proof of Proposition 1 is by direct computation and can be found in Appendix A.3.

Proposition 1. We have the following observations for the block trifocal tensor T n . For all distinct i,j ∈ [ n ] , we have the following properties:

(i) T n iii =0 3 × 3 × 3

(ii) The T n jii blocks are rearrangements of elements in the fundamental matrix F ij up to signs.

(iii) The T n iji and T n iij blocks encode the epipoles.

(iv) The horizontal slices T n ( i, : , :) of T n are skew symmetric.

(v) When all cameras are calibrated, three singular values of T n (1) are equal.

Theorem 1 (Tucker factorization and low multilinear rank of block trifocal tensor) . The block trifocal tensor T n admits a Tucker factorization, T n = G× 1 P× 2 C× 3 C , where G∈ R 6 × 4 × 4 , P∈ R 3 n × 6 , and C ∈ R 3 n × 4 . If the n cameras that produce T n are not all collinear, then mlrank ( T n ) = (6 , 4 , 4) . If the n cameras that produce T n are collinear, then mlrank ( T n ) ⪯ (6 , 4 , 4) .

Proof. We can explicitly calculate that T n = G× 1 P× 2 C× 3 C . The details of the calculation are in Appendix A.2. The specific forms for G , C , P are the following. The horizontal slices of the core are

The factor matrices are C =[ P 1 ,P 2 ,...,P n ] T ∈ R 3 n × 4 and P =[ S 1 ,S 2 ,...,S n ] T ∈ R 3 n × 6 , where P i are the camera matrices and S i are the corresponding line projection matrices.

̸

Now, we suppose that the n cameras are not collinear. We first show that C and P both have full rank. From [1], the null space of a camera matrix P i is generated by the camera center. For the sake of contradiction, suppose that rank( C ) < 4. Then there exists x ∈ R 4 such that x =0 and C x =0 . This means that P i x =0 for all i =1 ,...,n . Then, x is the camera centre for all cameras, which means that the cameras are centered at one point and are collinear. Similarly, every vector in the null space of the line projection matrix S i is a line that passes through the camera centre [1]. For the sake of contradiction, suppose that rank( P ) < 6. Then there exists x ∈ R 6 such that x =0 and P x =0 . This implies that S i x =0 for all i =1 ,...,n , which means that x is a line that passes through all of the camera centers. Again the cameras are collinear, which is a contradiction. Next we write the flattening of the block trifocal tensor as T n (1) = PG (1) ( C ⊗C ) T . Then P ∈ R 3 n × 6 has rank 6, and ( C ⊗C ) T ∈ R 16 × 9 n 2 has rank 16. Given the specific form of G , where G (1) ∈ R 6 × 16 it is easy to check rank( G (1) ) = 6. Thus,

̸

rank( T n (1) ) = 6. Similarly, we can show that rank( T n (2) ) = 4, and rank( T n (3) ) = 4. This implies that the multilinear rank of the block trifocal tensor is (6 , 4 , 4) when the n cameras are not collinear.

When the n cameras are collinear, the individual factors in each flattening may be rank deficient, so that rank( T n (1) ) ≤ 6 , rank( T n (2) ) ≤ 4 , and rank( T n (3) ) ≤ 4 . This implies mlrank( T n ) ⪯ (6 , 4 , 4) .

The theorem inspires a straightforward way of retrieving global poses from the block trifocal tensor, which we summarize in the following claim.

Proposition 2 (One shot camera pose retrieval) . Given the block trifocal tensor T n produced by cameras P 1 ,P 2 ,...,P n , the cameras can be retrieved from T n up to a global projective ambiguity using the higher-order SVD. The cameras will be the leading 4 singular vectors of T n (2) or T n (3) .

Using the higher-order SVD on T n , we can get a Tucker decomposition of the block trifocal tensor T n = ˆ G × 1 ˆ P × 2 ˆ C × 3 ˆ C ′ . Though the Tucker factorization is not unique [35], as we can apply an invertible linear transformation to one of the factor matrices and apply the inverse onto the core tensor, this invertible linear transformation can be interpreted as the global projective ambiguity for projective 3Dreconstruction algorithms. Thus, the cameras can be retrieved by taking the leading four singular vectors of the mode-2 and mode-3 flattenings of the block tensor.

Very importantly however, in practice each trifocal tensor block in T n can be estimated from image data only up to an unknown multiplicative scale [1]. The following theorem establishes the fact that the multilinear rank constraints provide sufficient information for determining the correct scales. In the statement ⊙ b denotes blockwise scalar multiplication, thus the ( i,j,k ) -block of λ ⊙ b T n is λ ijk T n ijk ∈ R 3 × 3 × 3 .

Theorem 2. Let T n ∈ R 3 n × 3 n × 3 n be a block trifocal tensor corresponding to n ≥ 4 calibrated or uncalibrated cameras in generic position. Let λ ∈ R n × n × n be a block scaling with λ ijk nonzero iff i,j,k are not all equal. Assume that λ ⊙ b T n ∈ R 3 n × 3 n × 3 n has multilinear rank (6 , 4 , 4) where ⊙ b denotes blockwise scalar multiplication. Then there exist α,β,γ ∈ R n such that λ ijk = α i β j γ k whenever i,j,k are not all the same.

Sketch. The idea is to identify certain submatrices in the flattenings of λ ⊙ b T n which must have determinant 0 , and use these to solve for λ . Aproof is in Appendix A.4. We remark that the proof technique extends that of [36, Theorem 5.1], which showed a similar result for a matrix problem.

Theorem 2 is the basic guarantee for our algorithm development below. We stress that the ambiguities brought by α,β,γ are not problematic for purposes of recovering the camera matrices by Proposition 2. Indeed, ( α ⊗ β ⊗ γ ) ⊙ b T n = G× 1 ( D α P ) × 2 ( D β C ) × 3 ( D γ C ) where D α ∈ R 3 n × 3 n is the diagonal matrix with each entry of α triplicated, etc. Hence the camera matrices can still be recovered up to individual scales (as expected) and a global projective transformation, from the higher-order SVD.

3 Synchronization of the block trifocal tensor

In this section, we develop a heuristic method for synchronizing the block trifocal tensor T n by exploiting the multilinear rank of T n from Theorem 1. Let ˆ T n denote the estimated block trifocal tensor, and T n the ground truth. Assume that there are n images and a set of trifocal tensor estimates ˆ T ijk where ( i,j,k ) ∈ Ω and Ω is the set of indices whose corresponding trifocal tensor is estimated. Note that each estimated trifocal tensor ˆ T ijk will have an unknown scale λ ijk ∈ R ∗ associated with it. We always assume that we observe the iii blocks, as they will be 0 . Weformulate the block trifocal tensor ˆ T n by plugging in the estimates ˆ T ijk and setting the unobserved positions ( ( i,j,k ) ̸∈ Ω ) to 3 × 3 × 3 tensors of all zeros. Let W Ω ∈{ 0 , 1 } 3 n × 3 n × 3 n denote the block tensor where the ( i,j,k ) blocks are ones for ( i,j,k ) ∈ Ω and zeros otherwise. Let W Ω C denote the opposite. In our experiments, we observe that the HOSVD is quite robust against noise for retrieving camera poses, which arises e.g., from numerical sensitivities when first estimating relative poses [37]. Therefore we develop an algorithm that projects ˆ T n onto the set of tensors that have multilinear rank of (6 , 4 , 4) while completing the tensor and retrieving an appropriate set of scales. Specifically, we can write our problem as

where Λ ∈ R 3 n × 3 n × 3 n , each 3 × 3 × 3 block is uniform, Λ ijk blocks are zero for ( i,j,k ) ̸∈ Ω , and Λ satisfies a normalization condition like ∥ Λ ∥ 2 =1 to avoid its vanishing. However, we drop this normalization constant in our implementation as we never observe Λ vanishing in practice. (For convenience, we formulate this section with the notation of Λ ∈ R 3 n × 3 n × 3 n and Hadamard multiplication, rather than λ ∈ R n × n × n and blockwise scalar multiplication from Theorem 2.) Furthermore in problem (4), P τ denotes the exact projection onto the set Γ= { T ∈ R 3 n × 3 n × 3 n : mlrank ( T ) =(6 , 4 , 4) } . Note that though HOSVD provides an efficient way to project onto Γ , it is quasi-optimal and not the exact projection. The exact projection is much harder to calculate, and in general NP-hard. The algorithm below adopts an alternating projection strategy to estimate the best set of scales.

3.1 Higher-order SVD with a hard threshold (HOSVD-HT)

The key idea for our algorithm is to use the relative scales on the rank truncated tensor as a heuristic to retrieve scales for the estimated block tensor. There are two main challenges for calculating the rank truncated tensor. First, the exact projection P τ onto Γ is expensive and difficult to calculate. Second, many blocks in the block tensor will be unknown if the corresponding images of the block lacks corresponding point and directly projecting the uncompleted tensor will be inaccurate. We apply an HOSVDframework with imputations to tackle the challenges. Regarding the first challenge, HOSVD is a simple, efficient, and quasi-optimal (3) projection onto Γ . Though inexact, it is a reliable approximation. For the second challenge, the tensor ˆ T n must be completed. We adopt the matrix completion idea of HARD-IMPUTE [38], where the matrix is filled-in iteratively with the rank truncated matrix obtained using the hard-thresholded SVD. In other words, we complete the missing blocks with the corresponding blocks in the rank truncated tensor. We define three hyperparameters l 1 ,l 2 ,l 3 that correspond to the thresholding parameters of the hard-thresholded SVD on modes 1 , 2 , 3 of the block tensor respectively. Specifically, for each modei flattening T n ( i ) , we calculate the full SVD T n ( i ) = USV T . Since our tensor will scale cubically with the number of cameras, we suggest using a randomized SVD. Werefer to [39] for different randomized strategies. Assume the singular values σ i on the diagonal of S are sorted in descending order, as usual. We return the factor matrix A i as the top a left singular vectors in U , where a =max { i : S ii >l i } . Our adapted truncation method is summarized by Algorithm 1.

Algorithm 1 HOSVD-HT

Input: ˆ T n ∈ R 3 n × 3 n × 3 n : the estimated block tensor; l 1 ,l 2 ,l 3 ∈ R : the thresholds for modes 1 , 2 , 3 respectively

Output: ˆ T r ∈ R 3 n × 3 n × 3 n : the rank truncated tensor.

for i = 1 to 3

do

Perform the randomized SVD on the modei flattening such that ˆ T n ( i ) ← USV T

A

i

first

a

i

columns of

end for

From now on, we refer to hard-thresholded HOSVD as HOSVD-HT and denote the operation as P ht .

3.2 Scale recovery

HOSVD-HTprovides an efficient way for projecting ˆ T n onto the set of tensors with with truncated rank. To recover scales, we use the rank truncated tensor's relative scale as a heuristic to adjust the scale on our estimated block trifocal tensor ˆ T ( n ) . For each step, we solve

where we drop the normalization condition on Λ because in practice it is not needed. We solve (5) for each observed block separately. Denoting P ht (Λ ( t ) ⊙ ˆ T n ) as ( ˆ T n r ) ( t ) , we have

U

Recall that our strategy for completing the tensor is to impute the tensor with the entries from the rank truncated tensor using HOSVD-HT. Specifically, given the current imputed tensor ( ˆ T n ) ( t ) , we calculate P ht (( ˆ T n ) ( t ) ) and the new scales Λ ( t +1) . Then update with

3.3 Synchronization algorithm

Now we summarize our synchronization framework in Algorithm 2. We have observed that the

Algorithm 2 Synchronization of the block trifocal tensor

algorithm can overfit, as the recovered scales will experience sudden and huge leaps. Our stopping criteria for the algorithm is when we observe sudden jumps in the variance of the new scales or when we exceed a maximum number of iterations. Another challenge in structure from motion datasets is that estimations may be highly corrupted. The HOSVD framework mainly consists of retrieving a dominant subspace from each flattening. Thus, it is natural to replace the SVD on each flattening with a more robust subspace recovery method, such as the Tyler's M estimator (TME) [40] or a recent extension of TME that incorporates the information of the dimension of the subspace in the algorithm [41]. We refer to Appendix A.5.2 for more details and provide an implementation there.

4 Numerical experiments

Weconduct experiments of Algorithm 2 on two benchmark real datasets, the EPFL datasets [42] and the Photo Tourism datasets [11]. We observe that the algorithm performs better in the calibrated setting, and since the calibration matrix is usually known in practice, we restrict our scope of experiments to calibrated trifocal tensors. We compare against three state-of-the-art synchronization based on two view measurements, NRFM [18] and LUD [12]. NRFM relies on nonconvex optimization and requires a good initialization. We test NRFM with an initialization obtained from LUD and with a random initialization. We also test BATA [43] initialized with MPLS [9]. We refer to A.6 in the appendix for a comprehensive summary of numerical results including rotation and translation estimation errors. Weinclude our code in the following github repository: TrifocalSync.

4.1 EPFLdataset

For EPFL, we follow the experimental setup and adopt code from [44] and test an entire structure from motion pipeline. We first describe the structure from motion pipeline for EPFL experiements.

Step 1 (feature detection and feature matching). We obtain matched features across pairs of images using a modern deep learning based feature detection and matching algorithm, GlueStick [45]. Though we do not implement this in our experiments, there have been methods developed to further screen corrupted keypoint matches or obtain matches robustly, such as [46, 47, 48]. Key points across a triplet of cameras is matched from pairs and is included only if it appears in all the pair combinations of the three images.

Step 2 (estimation and refinement of trifocal tensors). With the triplet matches, we calculate the trifocal tensors with more than 11 correspondences. To have an even sparser graph, one can skip the

estimation of trifocal tensors and rely on the imputation for images that have less than a number bigger than 11 point correspondences. This can further speed up the trifocal tensor estimation process. We apply STE from [41] to find 40% of the correspondences as inliers, then use at most 30 inlier point correspondences to linearly estimate the trifocal tensor. To refine the estimates, we apply bundle adjustment on the inliers and delete triplets with reprojection error larger than 1 pixel.

· Step 3 (synchronization). We synchronize the estimated block trifocal tensor with a robust variant of SVDusing the framework described in Algorithm 2. The robustness comes from replacing SVD with a robust subspace recovery method [41]. More details can be found in Appendix A.5.2. Recall that the cameras we retrieve are up to a global projective ambiguity. When comparing with ground truth poses, we first align our estimated cameras with the ground truth cameras by finding a 4 × 4 projective transformation. Then we round the cameras to calibrated cameras and compare.

Wetest our full pipeline on two EPFL datasets on a personal machine with 2 GHz Intel Core i5 with 4 cores and 16GB of memory. To test NRFM [18], LUD [12] and BATA [43] initialized with MPLS [9], we estimate the corresponding essential matrices using the GC-ransac [49]. We did not include blocks corresponding to two views in our trifocal tensor pipeline. The mean and median translation errors are summarized in Figure 1 here and more comprehensive results can be found in Table 1 and Table 2 in the appendix.

20

our

NRFM(LUD)

LUD

MeanTranslationError

15

INRFM

BATA(MPLS)

10

5

0

FP11HZP8HZ25EN10CS19CS30

20

our

NRFM(LUD)

LUD

MedianTranslationError

15

INRFM

BATA(MPLS)

10

5

0

FP11HZP8HZ25EN10CS19CS30

The EPFL datasets generally have a plethora of point correspondences, so that the trifocal tensors are estimated accurately. When the dataset focuses on a single scene, our algorithm retrieves locations competitively. Our algorithm achieves the best location estimation for 4 out of 6 datasets. The translation error bars are not visible for FP11, HZP8, EN10 due to the accuracy that we achieve. However, our pipeline is incapable of accurately processing CastleP19 and CastleP30. The main reason is that our algorithm relies on having a very dense observation graph to ensure high completion rate. CastleP19 and CastleP30 are datasets where the camera scans portions of the general area sequentially, so that not many triplets have overlapping features. Our method is not suitable for this type of dataset. However, it is possible to apply our algorithm in parallel on groups of neighboring frames, so that the completion rate is high in each group. Then the results can be merged to obtain a larger reconstruction. Rotations for the two view methods are estimated via rejecting outliers from iteratively applying [10]. Wealso compare against [43] for location estimation, where we initialize with a state-of-the-art global rotation estimation method [9]. Our algorithm achieves superior rotation estimation for only 2 out of the 6 datasets. See Table 1 and 2 in the appendix for comprehensive errors.

4.2 Photo Tourism

We conduct experiments on the Photo Tourism datasets. The Photo Tourism datasets consist of internet images of real world scenes. Each scene has hundreds to thousands of images. The datasets

[11] provide essential matrix estimates, and we estimate the trifocal tensors from the given essential matrices. To limit the computational cost for tensors, we downsample the datasets by choosing cameras with observations more than a certain percentage in the corresponding block frontal slice while maintaining a decent number of cameras. Note that this may not be the optimal way of extracting a dense subset in general. The maximum number of cameras we select for each dataset is 225 cameras. The largest dataset Piccadilly has 2031 cameras initially. We randomly sample 1000 cameras and then run our procedure. For Roman Forum and Piccadilly, the two view methods further deleted cameras from the robust rotation estimation process or parallel rigidity test. We rerun and report the trifocal tensor synchronization algorithm with the further downsampled data. We initialize the hard thresholding parameters for HOSVD-HT by first imputing the trifocal tensor with small random entries and then calculating the singular values for each of the flattenings. We take l i to be the tertile singular value for each modei flattening. We then keep this parameter fixed for the synchronization process. Recall that the jii blocks in the block trifocal tensor correspond to elements in the essential matrix E ij . Wealso include these essential matrix estimations in the block trifocal tensor. The Photo Tourism experiments were run on an HPC center with 32 cores, but the only procedure that can benefit from parallel computing in a single experiment is the scale retrieval. Mean and median translation errors are summarized in Figure 2. Fully comprehensive results can be found in Tables 3 and 4 in Appendix A.6.

70

Our

NRFM (LUD init.)

60

LUD

NRFM (Rand init.)

BATA(MPLS)

MeanTranslationError

50

40

30

20

10

0

arkt

Piazza delPopolo

EllisIsland

Towerof London

Vienna

Alamo

Cathedral

RomanForum

Notre

Dame

Union

Piccadilly

50

Our

45

NRFM (LUD init.)

LUD

40

NRFM (Rand init.)

BATA(MPLS)

MedianTranslationError

35

15

10

5

s

EllisIsland

Towerof London

Vienna

Cathedral

RomanForum

UnionSqus

Piccadilly

Our method is able to achieve competitive translation errors on 8 of the 14 datasets tested. Similar to the observation in the EPFL experiments, our algorithm performs well when the viewing graph is dense, or in other words, when the estimation percentage is high. We achieve better locations in 6 out of 8 datasets where the estimation percentage exceeds 60%, and better locations in only 2 out of 6 datasets where the estimation percentage falls below 60%. We achieve reasonable rotation estimations for 10 out of 14 datasets, but not as good as LUD. See Table 4 for a comprehensive result. Since the block trifocal tensor scales cubically with respect to the number of cameras, our algorithm runtime is longer than most two view global methods. This could be alleviated by synchronizing dense subsets in parallel and merging the results to construct a larger reconstruction.

Additional remark: Trifocal tensors can be estimated from line correspondences or a mix of point and line correspondences, while fundamental matrices are estimated from only point correspondences. There are many situations where accurate point correspondences are in short supply but there is a plethora of clear and distinct lines. For example, see datasets in a recent SfM method using lines [50]. We demonstrate the potential of our method to be adapted to process datasets with only lines or very few points. Due to the limited availability of well annotated line datasets, we provide a small synthetic experiment that simulates a case where only lines correspondences are present. We first generate 20 random camera matrices, then we generate 25 lines that are projected on and shared across all images. We add about 0.02 percent of noise in terms of the relative frobenius norms between the line equation parameters and the noise. We estimate the trifocal tensor of three different views from line correspondences linearly. One remark is that our synchronization method works well only when the signs of the initial unknown scales are mostly uniform. We manually use ground truth trifocal tensors

to correct the sign of the scale. This has not been an issue in the previous experiments due to bundle adjustment for EPFL and the overall good estimations in Photo Tourism. In practice, the sign of the scale on a trifocal tensor can be corrected via triangulation of points or reconstruction of lines, and correcting the sign using the depths of the reconstructed points or intersecting line segments. We synchronize the trifocal tensors with Algorithm 2 and were able to achieve a mean rotation error of 0.61 degrees, median rotation error of 0.49 degrees, mean location error of 0.76, and median location error of 0.74.

5 Conclusion

In this work, we introduced the block tensor of trifocal tensors characterizing the three-view geometry of a scene. We established an explicit Tucker factorization of the block trifocal tensor and proved it has a low multilinear rank of (6 , 4 , 4) under appropriate scaling. We developed a synchronization algorithm based on tensor decomposition that retrieves an appropriate set of scales, and synchronizes rotations and translations simultaneously. On several real data benchmarks we demonstrated state-of-the-art performance in terms of camera location estimation, and saw particular advantages on smaller and denser sets of images. Overall, this work suggests that higher-order interactions in synchronization problems have the potential to improve performance over pairwise-based methods.

There are several limitations to our tensor-based synchronization method. First, our rotation estimations are not as strong as our location estimations. Second, our algorithm performance is affected by the estimation percentage of trifocal tensors within the block trifocal tensor. One could incorporate more robust completion methods and explore new approaches for processing sparse triplet graphs. Further, our block trifocal tensor scales cubically in terms of the number of cameras and becomes computationally expensive for large datasets. We can develop methods for extracting dense subgraphs, synchronizing in parallel, then merging results to obtain a larger reconstruction, similarly to the distributed algorithms of [51] and [52]. Moreover, our synchronization method's success depends on accurate trifocal tensor estimations, and it motivates further work on robust estimation of multi-view tensors. Algorithm 2 could also be made more robust by adding outlier rejection techniques. Finally we plan to extend our theory by proving convergence of our algorithm and exploring structures for even higher-order tensors, such as quadrifocal tensors.

References

Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision . Cambridge University Press, 2003.

Noah Snavely, Steven Seitz, and Richard Szeliski. Photo Tourism: Exploring photo collections in 3D. In Proceedings of the ACM Special Interest Group on Computer Graphics and Interactive Techniques Conference, SIGGRAPH 2006 , pages 835-846, 2006.

Johannes Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016 , pages 4104-4113, 2016.

Bill Triggs, Philip McLauchlan, Richard Hartley, and Andrew Fitzgibbon. Bundle adjustment -Amodernsynthesis. In Bill Triggs, Andrew Zisserman, and Richard Szeliski, editors, Vision Algorithms: Theory and Practice , pages 298-372, Berlin, Heidelberg, 2000. Springer Berlin Heidelberg.

Venu Madhav Govindu. Lie-algebraic averaging for globally consistent motion estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2004 , volume 1, pages 1-8, 2004.

Richard Hartley, Jochen Trumpf, Yuchao Dai, and Hongdong Li. Rotation averaging. International Journal of Computer Vision , 103:267-305, 2013.

Avishek Chatterjee and Venu Madhav Govindu. Robust relative rotation averaging. IEEE Transactions on Pattern Analysis and Machine Intelligence , 40(4):958-972, 2018.

Avishek Chatterjee and Venu Madhav Govindu. Efficient and robust large-scale rotation averaging. In Proceedings of the IEEE International Conference on Computer Vision, ICCV 2013 , pages 521-528, 2013.

Yunpeng Shi and Gilad Lerman. Message passing least squares framework and its application to rotation synchronization. In Proceedings of the International Conference on Machine Learning, ICML 2020 , pages 8796-8806, 2020.

Mica Arie-Nachimson, Shahar Kovalsky, Ira Kemelmacher-Shlizerman, Amit Singer, and Ronen Basri. Global motion estimation from point matches. In Proceedings of the International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission, 3DIMPVT 2012 , pages 81-88, 2012.

Kyle Wilson and Noah Snavely. Robust global translations with 1DSfM. In Proceedings of the European Conference on Computer Vision, EECV 2014 , pages 61-75, 2014.

Onur Ozyesil and Amit Singer. Robust camera location estimation by convex programming. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015 , pages 2674-2683, 2015.

Thomas Goldstein, Paul Hand, Choongbum Lee, Vladislav Voroninski, and Stefano Soatto. ShapeFit and ShapeKick for robust, scalable structure from motion. In Proceedings of the European Conference on Computer Vision, EECV 2016 , pages 289-304, 2016.

David Rosen, Luca Carlone, Afonso Bandeira, and John Leonard. SE-Sync: A certifiably correct algorithm for synchronization over the special Euclidean group. International Journal of Robotics Research , 38(2-3):95-125, 2019.

Federica Arrigoni, Beatrice Rossi, and Andrea Fusiello. Spectral synchronization of multiple views in SE(3). SIAM Journal on Imaging Sciences , 9(4):1963-1990, 2016.

Mihai Cucuringu, Yaron Lipman, and Amit Singer. Sensor network localization by eigenvector synchronization over the Euclidean group. ACMTransactions on Sensor Networks , 8(3), 2012.

Jesus Briales and Javier Gonzalez-Jimenez. Cartan-Sync: Fast and global SE(d)-synchronization. IEEE Robotics and Automation Letters , 2(4):2127-2134, 2017.

Soumyadip Sengupta, Tal Amir, Meirav Galun, Tom Goldstein, David Jacobs, Amit Singer, and Ronen Basri. A new rank constraint on multi-view fundamental matrices, and its application to camera location recovery. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017 , pages 4798-4806, 2017.

Yoni Kasten, Amnon Geifman, Meirav Galun, and Ronen Basri. Algebraic characterization of essential matrices and their averaging in multiview settings. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019 , pages 5895-5903, 2019.

Yoni Kasten, Amnon Geifman, Meirav Galun, and Ronen Basri. GPSfM: Global projective SFM using algebraic constraints on multi-view fundamental matrices. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019 , pages 3264-3272, 2019.

Spyridon Leonardos, Roberto Tron, and Kostas Daniilidis. A metric parametrization for trifocal tensors with non-colinear pinholes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015 , pages 259-267, 2015.

Viktor Larsson, Nicolas Zobernig, Kasim Taskin, and Marc Pollefeys. Calibration-free structure-from-motion with calibrated radial trifocal tensors. In Proceedings of the European Conference on Computer Vision, EECV 2020 , pages 382-399, 2020.

Pierre Moulon, Pascal Monasse, and Renaud Marlet. Global fusion of relative motions for robust, accurate and scalable structure-from-motion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2013 , pages 3248-3255, 2013.

Onur Ozyesil, Vladislav Voroninski, Ronen Basri, and Amit Singer. A survey of structure from motion. Acta Numerica , 26:305-364, 2017.

Joe Kileel. Minimal problems for the calibrated trifocal variety. SIAM Journal on Applied Algebra and Geometry , 1(1):575-598, 2017.

Timothy Duff, Kathlen Kohn, Anton Leykin, and Tomas Pajdla. PLMP-point-line minimal problems in complete multi-view visibility. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019 , pages 1675-1684, 2019.

Ricardo Fabbri, Timothy Duff, Hongyi Fan, Margaret Regan, David da Costa de Pinho, Elias Tsigaridas, Charles Wampler, Jonathan Hauenstein, Peter Giblin, Benjamin Kimia, Anton Leykin, and Tomas Pajdla. Trifocal relative pose from lines at points. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(6):7870-7884, 2023.

David Nister and Henrik Stewenius. A minimal solution to the generalised 3-point pose problem. Journal of Mathematical Imaging and Vision , 27(1):67-79, 2007.

Ali Elqursh and Ahmed Elgammal. Line-based relative pose estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2011 , pages 3049-3056, 2011.

Yubin Kuang and Kalle Astrom. Pose estimation with unknown focal length using points, directions and lines. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2013 , pages 529-536, 2013.

Zuzana Kukelova, Joe Kileel, Bernd Sturmfels, and Tomas Pajdla. A clever elimination strategy for efficient minimal solvers. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017 , pages 4912-4921, 2017.

Pedro Miraldo, Tiago Dias, and Srikumar Ramalingam. A minimal closed-form solution for multi-perspective pose estimation using points and lines. In Proceedings of the European Conference on Computer Vision, ECCV 2018 , pages 474-490, 2018.

Joe Kileel, Zuzana Kukelova, Tomas Pajdla, and Bernd Sturmfels. Distortion varieties. Foundations of Computational Mathematics , 18:1043-1071, 2018.

Joe Kileel and Kathlén Kohn. Snapshot of algebraic vision. arXiv preprint arXiv:2210.11443 , 2022.

Tamara Kolda and Brett Bader. Tensor decompositions and applications. SIAM Review , 51(3):455-500, 2009.

Tommi Muller, Adriana Duncan, Eric Verbeke, and Joe Kileel. Algebraic constraints and algorithms for common lines in cryo-EM. Biological Imaging , pages 1-30, Published online 2024.

Hongyi Fan, Joe Kileel, and Benjamin Kimia. On the instability of relative pose estimation and RANSAC's role. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2022 , pages 8935-8943, 2022.

Rahul Mazumder, Trevor Hastie, and Robert Tibshirani. Spectral regularization algorithms for learning large incomplete matrices. Journal of Machine Learning Research , 11:2287-2322, 2010.

Nathan Halko, Per-Gunnar Martinsson, and Joel Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM Review , 53(2):217-288, 2011.

David Tyler. A distribution-free M-estimator of multivariate scatter. Annals of Statistics , pages 234-251, 1987.

Feng Yu, Teng Zhang, and Gilad Lerman. A subspace-constrained Tyler's estimator and its applications to structure from motion. arXiv preprint arXiv:2404.11590 , 2024.

Christoph Strecha, Wolfgang Von Hansen, Luc Van Gool, Pascal Fua, and Ulrich Thoennessen. On benchmarking camera calibration and multi-view stereo for high resolution imagery. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2008 , pages 1-8, 2008.

Bingbing Zhuang, Loong-Fah Cheong, and Gim Hee Lee. Baseline desensitizing in translation averaging. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR2018) , June 2018.

Laura Julia and Pascal Monasse. A critical review of the trifocal tensor estimation. In Proceedings of the Pacific Rim Symposium on Image and Video Technology, PSIVT 2017, Revised Selected Papers 8 , pages 337-349. Springer, 2018.

Rémi Pautrat, Iago Suárez, Yifan Yu, Marc Pollefeys, and Viktor Larsson. Gluestick: Robust image matching by sticking points and lines together. In Proceedings of the IEEE/CVF International Conference on Computer Vision, CVPR 2023 , pages 9706-9716, 2023.

Yunpeng Shi, Shaohan Li, Tyler Maunu, and Gilad Lerman. Scalable cluster-consistency statistics for robust multi-object matching. In Proceedings of the International Conference on 3DVision, 3DV 2021 , pages 352-360, 2021.

Yunpeng Shi, Shaohan Li, and Gilad Lerman. Robust multi-object matching via iterative reweighting of the graph connection laplacian. Advances in Neural Information Processing Systems , 33:15243-15253, 2020.

Shaohan Li, Yunpeng Shi, and Gilad Lerman. Fast, accurate and memory-efficient partial permutation synchronization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2022 , pages 15735-15743, 2022.

Daniel Barath and Jiˇ rí Matas. Graph-cut ransac. In Proceedings of the IEEE conference on computer vision and pattern recognition, CVPR 2018 , pages 6733-6741, 2018.

Shaohui Liu, Yifan Yu, Rémi Pautrat, Marc Pollefeys, and Viktor Larsson. 3d line mapping revisited. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023 , pages 21445-21455, 2023.

Shaohan Li, Yunpeng Shi, and Gilad Lerman. Efficient detection of long consistent cycles and its application to distributed synchronization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024 , pages 5260-5269, 2024.

Andrea Porfiri Dal Cin, Luca Magri, Federica Arrigoni, Andrea Fusiello, and Giacomo Boracchi. Synchronization of group-labelled multi-graphs. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021 , pages 6433-6443. IEEE, 2021.

Joe Harris. Algebraic Geometry: A First Course , volume 133. Springer Science & Business Media, 1992.

Ying Sun, Prabhu Babu, and Daniel Palomar. Regularized Tyler's scatter estimator: Existence, uniqueness, and algorithms. IEEE Transactions on Signal Processing , 62(19):5143-5156, 2014.

A Appendix / supplemental material

A.1 Derivation of the trifocal tensor

To provide a better intuition for the trifocal tensor, we briefly summarize the derivation of the trifocal tensor from [1] and [21] under the general setup of uncalibrated cameras.

Let P i = K i R i [ I, -t i ] be the form of the camera matrix for P 1 ,P 2 ,P 3 . Let L be a line in the 3D world scene, and l 1 ,l 2 ,l 3 the corresponding projections in the images I 1 ,I 2 ,I 3 respectively. Each l i back projects to a plane π i = P T i l i in R 3 , and since l i correspond to the same L in the 3D world scene, π =[ π 1 ,π 2 ,π 3 ] must be rank deficient and its kernel will generically be spanned by L . Then,

will also be rank-deficient, implying that the columns of π ′ are linearly dependent. This means that there exist α,β such that π ′ 1 = απ ′ 2 + βπ ′ 3 . Wecanchoose α = -( t 1 -t 3 ) T R T 3 K T 3 l 3 , β =( t 1 -t 2 ) T R T 2 K T 2 l 2 , so that

Then, the canonical trifocal tensor centered at camera 1 is defined as

where e i ∈ R 3 is the i -th standard basis vector. The trifocal tensor will be the tensor { T 1 ,T 2 ,T 3 } , where the T i 's are stacked along the first mode. The line incidence relation is then ( l 1 ) i = l T 2 T i l 3 . Other combinations of point and line incidence relations are also encoded by the trifocal tensor; see [1] for details. The construction for calibrated cameras is the same, just with P i in calibrated form.

A.2 Proof details for Theorem 1

Weinclude a detailed calculation for the Tucker factorization of the block trifocal tensor. Recall that each individual trifocal tensor corresponding to the cameras a,b,c can be calculated as

The last equality can be easily checked since G is sparse. For example, when k = 1 , P ( a ) i 1 is the determinant of the submatrix dropping the i -th row and keeping columns 1 and 2, which is det[ a m 1 a m 2 ; a n 1 a n 2 ] . The only nonzero elements in the first horizontal slice are G (1 , 4 , 3) = -1 and G (1 , 3 , 4) = 1 . Then, the nonzero elements in the sum when k = 1 will be exactly P ( a ) i 1 ∑ 4 w =1 b qw ∑ 4 j =1 c rj G 1 wj = P ( a ) i 1 ( b q 3 c r 4 -b q 4 c r 3 ) .

Then, since P will be the stackings of P ( P i ) , C is the stacking of camera matrices in Theorem 1, each ijk block in T n will be calculated by exactly the corresponding i , j , k blocks in P , C , C respectively using the calculations above.

A.3 Proof details for Proposition 1

Proof for (i). Wehave

since P i is a 3 × 4 matrix and the submatrix above will always have two identical rows.

Proof for (ii): Consider the wqr element of the jii block trifocal tensor, T n jii . It can be written as

Thus, when q = r , clearly ( T n jii ) wqr =0 as we will have identical rows again. When q = r , we first observe that ( T n jii ) wqr = -( T n jii ) wrq since we just swap two rows. Second,

̸

where m ∈{ 1 , 2 , 3 }\{ q,r } . This is exactly the bilinear relationship in [1] defining the fundamental matrix ( F ji ) mw element up to a possible negative sign.

Proof for (iii): We can only show this for T n iij blocks from symmetry. The elements in T n iij blocks can be calculated as

Elements are nonzero only when w = q , and they correspond to determinants of matrices with three rows from one P i and one row from P j . By [1], these are exactly the elements of the epipoles. When w =1 , the order of the rows in the determinant corresponding to camera i is (2 , 3 , 1) , when w =2 , the order is (1 , 3 , 2) and there is a negative sign in front of the determinant, and when w =3 , the order is (1 , 2 , 3) . Since the first and last case are even permutations of the rows of P i , and the second case is corrected by a negative sign, ( T n iji ) ww : is exactly the epipole.

Proof for (iv): On a horizontal slice, the camera along the 1st mode is fixed, and blocks symmetric across the diagonal is calculated by cameras, which the 2nd and 3rd mode cameras are swapped. Then, we will simply be swapping rows in (1), which means that we will simply be changing signs for elements symmetric across the diagonal, implying skew symmetry.

Proof for (v): Now assume that we have a block trifocal tensor whose corresponding cameras are all calibrated. Let P be the line projection matrix, C = [ P 1 , P 2 , ... , P n ] T is the stacked camera matrix, and G is the core tensor. The flattening in the 1st mode can be written as T n (1) = PG (1) ( C⊗C ) T , where T n (1) is a 3 n × 9 n 2 matrix. For the proof, we calculate the eigenvalue of T n (1) ( T n (1) ) T = PG (1) ( C⊗C ) T ( C⊗C ) G T (1) P T

The second and third line uses two Kronecker product properties: ( A ⊗ B ) T = A T ⊗ B T and ( A ⊗ B )( C ⊗ D )= AC ⊗ BD as long as AC and BD are defined.

Wefirst calculate ( C T C⊗C T C ) .

Weassume that the cameras are centered at the origin, i.e. ∑ n i =1 t i =0 . Then we have

so that

Wehave an explicit form for G (1) :

Let X = C T C and let X ij denote the ij entry in C T C . Let a = ∑ n i =1 ∥ t i ∥ 2 .

Wefirst show that G (1) ( C T C⊗C T C ) G T (1) is diagonal by direct computation:

Wethen calculate the spectral decomposition of T n (1) . With a slight abuse of notation, let P k i denote the k -th column of P i . The 3 n × 6 rank6 stacked line projection matrix would have columns ordered according to

and since the second row in P for each camera is P 3 i ∧ P 1 i it holds

Or equivalently, the stacked wedge products between columns. Let P = USV T be the thin singular value decomposition of P , so that U is a 3 n × 6 orthonormal matrix, S is a 6 × 6 diagonal matrix where all diagonal entries are nonzero, and V is a 6 × 6 orthonormal matrix.

Then,

Since V is orthonormal, SV T G (1) ( C T C ⊗C T C ) G T (1) V S is still a diagonal matrix. We just need to establish the fact that three of the diagonal entries are the same.

For one camera, P T P equals

where the matrix is symmetric and we reduce redundancy by omitting the entries below the diagonal. For n cameras,

where P a i · P b i means the dot product between the a th and b th column

The SVD of P T P is P T P = HDH T , where H is 6 × 6 orthonormal matrix, D is 6 × 6 diagonal matrix. However, since we have an nI 3 × 3 submatrix in P T P , we deduce that n appears as an eigenvalue 3 times for P T P , where we can use the determinant identity for block matrices. We check that this indeed holds by a computer calculation, generating random instances of P i 's and calculating the eigenvalues for P T P .

As a result, in the thin SVD of P , we have P = USV T where S = √ D , V = H . Then in

we see that SV T G (1) ( C T C⊗C T C ) G T (1) V S is a diagonal matrix where three of the entries are the same. By the uniqueness of the eigenvalues, we see that we have a spectral decomposition of T n (1) ( T n (1) ) T , so that three of the singular values of T n (1) are equal.

A.4 Proof details for Theorem 2

Proof. Note blockwise multiplication by a rank1 tensor with nonzero entries preserves multilinear rank, since it is a Tucker product by invertible diagonal matrices. Therefore, without loss of generality we may assume λ i 11 = λ 1 j 1 = λ 11 k =1 for all i,j,k ∈{ 2 , 3 ,...,n } . Below we will prove it follows λ ijk = c if exactly one of i,j,k equals 1 , and λ ijk = c 2 if none of i,j,k equal 1 and the indices are not all the same, for some constant c ∈ R ∗ . This will immediately imply the theorem, because taking α = β =(1 cc...c ) and γ =( 1 c 11 ... 1) achieves λ ijk = α i β j γ k whenever i,j,k are not all the same.

Weconsider the matrix flattenings T n (2) and ( λ ⊙ b T n ) (2) in R 3 n × 9 n 2 of the block trifocal tensor and its scaled counterpart, with rows corresponding to the second mode of the tensors. By Theorem 1 and assumptions, the flattenings have matrix rank 4 , thus all of their 5 × 5 minors vanish. The argument consists of considering several carefully chosen 5 × 5 submatrices of ( λ ⊙ b T n ) (2) to prove the existence of a constant c as above. Index the rows and columns of the flattenings by ( j,r ) and ( iq,ks ) respectively, for i,j,k ∈ [ n ] and q,r,s ∈ [3] , so that e.g., (( λ ⊙ b T n ) (2) ) ( j,r ) , ( iq,ks ) = λ ijk ( T n ijk ) qrs .

Step 1: The first submatrix of ( λ ⊙ b T n ) (2) we consider has column labels ( i 1 , 11) , ( i 1 , 21) , ( i 1 , 31) , ( i 1 , 12) , ( i 2 , 11) and row labels (1 , 1) , (1 , 2) , (1 , 3) , ( i, 1) , ( i, 2) , where i ∈{ 2 ,...,n } . Explicitly, it is

which we abbreviate as

̸

with asterisk denoting the corresponding entry in T n (2) . As a function of λ ii 1 ,λ 1 ii , the determinant of (12) is a degree ≤ 2 polynomial, which must be divisible by λ ii 1 and λ ii 1 -λ 1 ii (because if λ ii 1 =0 then clearly the bottom two rows of (12) are linearly independent, and if λ ii 1 -λ 1 ii =0 we have a submatrix of T n (2) with the bottom two rows scaled uniformly). So the determinant of (12) is a scalar multiple of λ ii 1 ( λ ii 1 -λ 1 ii ) . Note that the multiple is a polynomial function of the cameras P 1 and P i . Weclaim that generically the multiple is nonzero; and to see this, it suffices to exhibit a single instance of (calibrated) cameras where the determinant of (12) does not vanish identically for all λ ii 1 ,λ 1 ii duetothepolynomiality (e.g., see [53]). We check that this indeed holds by a computer calculation, generating numerical instances of P 1 and P i randomly. Thus the vanishing of the minor in (12) implies λ ii 1 ( λ ii 1 -λ 1 ii )= 0 , whence λ ii 1 = λ 1 ii since λ ii 1 =0 . Ananalogous calculation with ( λ ⊙ b T n ) (3) gives λ i 1 i = λ 1 ii .

Step 2: Next consider the submatrix of ( λ ⊙ b T n ) (2) with column labels ( j 1 , 11) , ( j 1 , 21) , ( j 1 , 31) , ( j 1 , 12) , (1 j, 11) and row labels (1 , 1) , (1 , 2) , (1 , 3) , ( i, 1) , ( i, 2) , where i,j ∈{ 2 ,...,n } are distinct. It looks like

with asterisks denoting entries of T n (2) . Similarly to the previous case, the determinant of (13) must be a scalar multiple of λ ji 1 ( λ ji 1 -λ 1 ij ) where the scale depends polynomially on P 1 ,P i ,P j . By a computer computation, we find that the scale is nonzero for random instances of cameras (alternatively, note the polynomial system in step 1 is a special case of the present one). It the scale is generically nonzero, hence λ ji 1 = λ 1 ij . Ananalogous calculation with ( λ ⊙ b T n ) (3) gives λ i 1 j = λ 1 ij .

Step 3: Consider the submatrix of ( λ ⊙ b T n ) (2) with column labels ( j 1 , 11) , ( j 1 , 21) , ( j 1 , 31) , ( j 1 , 12) , (1 k, 11) and row labels (1 , 1) , (1 , 2) , (1 , 3) , ( i, 1) , ( i, 2) , for i,j,k ∈{ 2 ,...,n } distinct. It looks like

The determinant of (14) is a scalar multiple of λ ji 1 ( λ ji 1 -λ 1 ik ) . By a direct computer computation as before, it is a nonzero multiple generically (alternatively, note the polynomial system in step 1 is a special of the present one). We deduce λ ji 1 = λ 1 ik . Ananalogous calculation with ( λ ⊙ b T n ) (3) gives λ i 1 j = λ 1 kj .

In particular, combining with step 2 it follows λ 1 ij = λ 1 ji , because λ 1 ij = λ k 1 j = λ 1 kj = λ ik 1 = λ 1 ki = λ j 1 i = λ 1 ji . From this, step 1 and step 2, we have that the λ -scale does not depend on the ordering of its indices, provided there is a 1 among the indices.

Step 4: Consider the submatrix of ( λ ⊙ b T n ) (2) with column labels ( j 1 , 11) , ( j 1 , 21) , ( j 1 , 31) , ( j 1 , 12) , (1 i, 11) and row labels (1 , 1) , (1 , 2) , (1 , 3) , ( i, 1) , ( i, 2) , for i,j ∈{ 2 ,...,n } distinct. It looks like

The determinant of (15) is a scalar multiple of λ ji 1 ( λ ji 1 -λ 1 ii ) . By a direct computer computation, it is a nonzero multiple generically (alternatively, note the polynomial system in step 1 is a special case of the present one). We deduce λ ji 1 = λ 1 ii .

Putting together what we know so far, all λ -scales with a single 1 -index agree. Indeed, this follows from λ 1 ii = λ ji 1 = λ ij 1 = λ 1 jj so all λ -scales with a single 1 -index and two repeated indices agree, combined with λ ji 1 = λ 1 ii and the last sentence of step 3. Let c ∈ R ∗ denote this common scale.

Step 5: Consider the submatrix of ( λ ⊙ b T n ) (2) with column labels (1 i, 11) , (1 i, 21) , (1 i, 31) , (1 i, 12) , ( ij, 11) and row labels (1 , 1) , (1 , 2) , (1 , 3) , ( i, 1) , ( i, 2) , for i,j ∈{ 2 ,...,n } distinct. It looks like

As a function of c and λ iij , the determinant of (16) is a scalar multiple of c ( c 2 -λ iij ) (the second factor is present because it corresponds to scaling the bottom two rows and rightmost column of a 5 × 5 submatrix of T (2) each by c , which preserves rank deficiency). By a direct computer computation, we find that the scale is nonzero for a random instance of P 1 ,P i ,P j , therefore it is nonzero generically. It follows c 2 = λ iij . Ananalogous calculation with ( λ ⊙ b T n ) (3) gives c 2 = λ iji .

Step 6: Consider the submatrix of ( λ ⊙ b T n ) (2) with column labels (1 i, 11) , (1 i, 21) , (1 i, 31) , (1 i, 12) , ( ji, 11) and row labels (1 , 1) , (1 , 2) , (1 , 3) , ( i, 1) , ( i, 2) , for i,j ∈{ 2 ,...,n } distinct. It looks like

Similarly to the previous step, the determinant of (17) must be a scalar multiple of c ( c 2 -λ jii ) . By a direct computer computation, it is a nonzero multiple generically. We deduce c 2 = λ jii .

Step 7: Consider the submatrix of ( λ ⊙ b T n ) (2) with column labels (1 i, 11) , (1 i, 21) , (1 i, 31) , (1 i, 12) , ( ik, 11) and row labels (1 , 1) , (1 , 2) , (1 , 3) , ( j, 1) , ( j, 2) , for i,j,k ∈{ 2 ,...,n } distinct. It looks like

The determinant of (18) is a scalar multiple of c ( c 2 -λ ijk ) . By a direct computer computation, it is a nonzero multiple generically (alternatively, note the polynomial system in step 5 is a special case of the present case). We deduce c 2 = λ ijk .

At this point, by steps 5,6,7 we have that all λ -scales with no 1 -indices and not all indices the same must equal c 2 . Combined with the second paragraph of step 4, this shows c satisfies the property announced at the start of the proof. Therefore the proof is complete.

A.5 Implementation details

A.5.1 Estimating trifocal tensors from three fundamental matrices

Given three cameras P 1 ,P 2 ,P 3 and the corresponding fundamental matrices F 21 ,F 31 ,F 32 , we can calculate the trifocal tensor T ijk using the following procedure detailed in [1]. Specifically, from

F 21 calculate an initial estimate of the cameras P ′ 1 ,P ′ 2 . Then, P T 3 F 32 P ′ 2 and P T 3 F 31 P ′ 1 should be skew-symmetric matrices. This gives 20 linear equations in terms of the entries in P 3 , which can be used to solve for the trifocal tensor. Note that there are no geometrical constraints when calculating P 3 , and there will be no guarantee of the quality of the estimation.

A.5.2 Higher-order regularized subspace-constrained Tyler's estimator (HOrSTE) for EPFL

Wedescribe the robust variant of SVD that we used for the EPFL experiments in Section 4. Numerically, it performs more stably and accurately than HOSVD-HT, yet it is an iterative procedure and each iteration requires an SVD of the 3 n × 9 n 2 flattening. This becomes computationally expensive when n becomes large and the number of iterations are also large. However, since the number of cameras for the EPFL dataset are below 20 cameras, the computational overhead is not too great.

In HOSVD, a low dimensional subspace is estimated using the singular value decomposition and taking the R n leading left singular vectors for each moden flattening. The Tyler's M Estimator (TME) [40] is a robust covariance estimator of a D dimensional dataset { x i } N i =1 ⊂ R D . It minimizes the objective function

such that Σ is positive definite and has trace 1. The TME estimator can be applied to robustly find an R n dimensional subspace by taking the R n leading eigenvectors of the covariance matrix of TME. To compute the TME, [40] proposes an iterative algorithm, where

TMEdoesn't exist when D>N , but a regularized TME has been proposed by [54]. The iterations become

where α is a regularization parameter, and I is the D × D identity matrix. TME does not assume the dimension of the subspace d is predetermined. In the case when d is prespecified, [41] improves the TME estimator by incorporating the information into the algorithm and develops the subspace-constrained Tyler's Estimator (STE). For each iteration, STE equalizes the trailing D -d eigenvalues of the estimated covariance matrix and uses a parameter 0 <γ< 1 to shrink the eigenvalues. The iterative procedure for STE is summarized into 3 steps:

Calculate the unnormalized TME, Z ( k ) = ∑ N i =1 ( x i x T i /x T i (Σ ( k -1) ) -1 ) x i ) .

Perform the eigendecomposition of Z ( k ) = U ( k ) S ( k ) ( U ( k ) ) T , and set the trailing D -d eigenvalues as γ ∑ D i = d +1 σ i / ( D -d ) .

Calculate Σ ( k ) = U ( k ) S ( k ) ( U ( k ) ) T /tr ( U ( k ) S ( k ) ( U ( k ) ) T ) , which is the normalized covariance matrix. Repeat steps 1-3 until convergence.

Similar to the regularized TME, STE can also be regularized to succeed in situations where there are fewer inliers, and can improve the robustness of the algorithm. The regularized STE differs from STE in only the first step, which is replaced by

1.* Calculate the unnormalized regularized TME, Z ( k ) = 1 1+ α D N ∑ N i =1 x i x T i x T i (Σ ( k -1) ) -1 ) x i + α 1+ α I

We apply the regularized STE to the HOSVD framework, and call the resulting projection the higher-order regularized STE (HOrSTE). It is performed via the following steps:

For each n , calculate the factor matrices A n as the R n leading left singular vectors from regularized STE applied to T ( n ) .

Set the core tensor G as G = T × 1 A T 1 × 2 A T 2 ···× N A T N .

A.6 Additional numerical results

In this section, we include comprehensive results for the rotation and translation errors for the EPFL and Photo Tourism experiments. Table 1 and 2 contains all results for EPFL datasets. Table 3 contains the location estimation errors for Photo Tourism. Table 4 contains the rotation estimation errors for Photo Tourism. In Table 4, we only report the rotation errors for LUD for all the methods that we compared against, as they are mostly the same since they used the same rotation averaging method.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Please check Section 3 and 4 for the main claims of our paper and the experimental results. An accurate summary of the contributions is found at the end of the introduction section, and also conveyed in the abstract.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NAanswer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please check our conclusion in Section 5. The last paragraph describes the limitations of the work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings,

model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Please see the proofs in Appendix A.3, A.2, A.4 and statements in Section 2. The statements are precise, and accompanied by complete proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Yes, we mention the numerical details and processing of our algorithm in section Section 4 for each experiment. We also include code in the supplementary material with instructions on how to run them. Our code depends on other publicly available code. Instructions can be found in supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) Werecognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes]

Justification: Please feel free to check our code in the supplementary material and the README.MDfile for instructions. A public version will be available in the future.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

Whileweencouragethereleaseofcodeanddata, weunderstandthat this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please see our experiment details in Section 4. In particular, we included the way we chose hyperparameter and specific numbers when applicable.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Though we didn't set a random seed for some experiments, there were insignificant differences between different runs of the method.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please see our experiment details in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We only used open source code and have complied to the code of ethics. Please check code in supplementary material.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Our method is focused on a general framework for tensor based structure from motion and we run experiments on open source datasets. No negative social impacts are related to our work. Having a tensor based synchronization method opens up more research directions and could help apply structure from motion to develop applications where point correspondences are scarce.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper doesn't pose such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

Werecognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cited the authors whose code in our code, and respected all relevant licenses. We include links to code in our submitted code as well. Please refer to the code and the experiment details.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

Theauthors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. NewAssets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide code for our methods and instructions for running the methods in the supplementary material.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

AccordingtotheNeurIPSCodeofEthics,workersinvolvedindatacollection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper doesn't involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a camera pose synchronization method based on trifocal tensors. The method first estimates trifocal tensors given a set of calibrated images. A HOSVD runs on the trifocal tensors to project them into a set of tensors with multilinear rank (6, 4, 4). The edge scales are then obtained from projected tensors.\n\nSTRENGTHS:\n- The paper proposes a strong theoretical work for trifocal tensor synchronization.\n\nWEAKNESSES:\nI guess this is the case when the weak experiments are overwhelmed by the strong theory. \nI propose accepting the paper since I believe the theoretical part of the paper will be useful for the community. \nHowever, I don't think the method in its current form can compete with SOTA SfM methods. \n\nWeaknesses in the experiments:\n- The EPFL dataset (I used to know it as the Strecha dataset) is extremely simple, with all recent SfM methods having a few cm errors in the end. The fact that the proposed method fails on CastleP19 and CastleP30 shows that the proposed method does not really work in practice. CastleP19 or CastleP30 are easy with lots of features and, thus, accurate essential matrices. \n- I welcome the experiments on the PhotoTourism dataset, but the baselines used in the paper do not really allow me to judge how the method works. LUD is an old method, significantly outperformed by BATA. Also, it is unclear what rotation averaging method do the authors use. While LUD is a translation averaging method, as I remember, it also has some rotation averaging in their code, which is actually not particularly accurate. The good baseline would be rotation averaging from the Theia library (alternating IRLS + L1) and BATA translation averaging.\n- \"We observe that the algorithm performs better in the calibrated setting, and since the calibration matrix is usually known in practice, we restrict our scope of experiments to calibrated trifocal tensors\". While I know this is often done in global SfM papers, it goes very much against the SfM goals. For example, the authors can only apply their method on the PhotoTourism dataset since someone has run Bundler/COLMAP beforehand to get camera intrinsics, making it pointless to run another SfM afterward. I guess the proposed method fails entirely without calibration otherwise the authors would have shown something. Am I correct? \n- Note: I would not use MATLAB's built-in RANSAC. Use PoseLib or GC-RANSAC. It will likely improve the accuracy of the proposed method as well, given that the relative poses become significantly more accurate. \n- The method is extremely slow. Theia runs 1-2 orders of magnitude faster while being more accurate. \n\nMinor things:\n- L81 \"projection of X onto the image\" -> \"projection of X into the image\"\n- Missing citation for SfM with lines [a]\n\n[a] Liu, S., Yu, Y., Pautrat, R., Pollefeys, M. and Larsson, V., 2023. 3d line mapping revisited. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 21445-21455).\n\nQUESTIONS:\n- L137 The scales are fixed to what?\n- Please try re-running some experiments with E matrices obtained from PoseLib or GC-RANSAC. Maybe, it would improve the accuracy.\n\nLIMITATIONS:\nDiscussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the characterization of the block tensor of trifocal tensors. The paper shows that under the assumption that the scales are known and the block tensor is complete, the global camera parameters can be extracted using Tucker factorization. Since those assumptions don't apply to real-world problems, the authors suggest an iterative algorithm for estimating the scales and completing the block tensor.\nEven though the paper's contribution to real-world applications is limited in terms of runtime, number of input images, and accuracy, in my opinion, it is an important paper due to its contribution to our understanding of multiple-view geoemtry.\n\nSTRENGTHS:\nThe papers suggest a novel approach for global camera pose estimation, an important and challenging task in computer vision.\nThe suggested algorithm is based on solid theory and contributes to our understanding of trifocal tensors.\n\nWEAKNESSES:\nThe main weakness of the paper is its contribution to real-world applications in terms of runtime, accuracy, and input size.\nDue to the usage of block trifocal tensors, the algorithm is limited in the number of input cameras (up to 225), and its runtime is longer than most baselines.\nThe accuracy presented by the suggested algorithm is low. The rotation error for Phototourism is higher for every scene except Piccadilly. In addition, a comparison to more recent works for global pose recovery is missing, e.g. [18, 19, 20] mentioned in the paper, and even deep-based methods such as Neurora, PoGO-Net, ESFM, and GASFM.\n\nThe authors mentioned, \"There are many situations where accurate point correspondences are in short supply but there is a plethora of clear and distinct lines.\" However, they didn't provide any examples of such a situation.\n\nQUESTIONS:\nDo your results include a global bundle adjustment step as post-processing?\nTable 3 in the appendix doesn't include Bold notation.\n\nLIMITATIONS:\nYes, the authors mentioned their runtime and dataset size limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to recover rotation and translation from the trifocal tensor, which encodes the three-view geometric parameters. The basic idea is based on Tucker factorization of the trifocal tensor, revealing a low multilinear rank of (6, 4, 4), independent of the number of cameras. A synchronization algorithm based on higher-order SVD is then introduced. The proposed method has been compared with existing methods and demonstrated certain benefits.\n\nSTRENGTHS:\nWhen we have point correspondences in three views and estimate the trifocal tensor based on three-view geometry, the motion parameters are generally more accurate than those obtained from two-view geometry. Motivated by this, the trifocal tensor serves as an important alternative to two-view geometry. The derivation is based on a noise-free case, and the paper demonstrates that the rank constraint provides sufficient information for camera motion recovery.\n\nThe proposed method was tested on two datasets, EPFL and Phototourism, and was shown to outperform some existing methods on certain sequences. The main advantage lies in the accuracy of translation estimation.\n\nWEAKNESSES:\nThe paper claims, 'We observe that the algorithm performs better in the calibrated setting, and since the calibration matrix is usually known in practice, we restrict our scope of experiments to calibrated trifocal tensors.' When focusing on the calibrated trifocal tensor, there are some algorithms that can directly obtain the rotation and translation from four points in three views, e.g., 'Nister et al. Four points in two or three calibrated views: Theory and practice. IJCV 2006.' Using such algorithms, we do not need to decompose the trifocal tensor for the calibrated case, making decomposition more relevant for the uncalibrated case.\n\nIn the main paper, only comparisons for the translation errors are presented, with rotation errors shown in the supplemental material. Based on the supplemental results, it appears that the proposed method does not perform well for rotation estimation. The derivation and rank constraint are based on the noise-free case. Conducting synthetic evaluations with noisy data could help readers understand the recovered rotation and translation accuracy in the presence of increasing image noise.\n\nQUESTIONS:\nThe balance between rotation and translation recovery suggests that the proposed method is better for translation recovery, while existing methods excel in rotation recovery. Are there any explanations for this? Additionally, if we apply optimization using points, what are the final results?\n\nIn general, errors originate from point correspondences. The paper presents statistics using SIFT features, but do these statistics hold if we use modern learning-based features?\n\nLastly, do we achieve better results than with two-view geometry?\n\nLIMITATIONS:\nThe paper provides a discussion of its limitations in the conclusion section, addressing most of the concerns. However, the paper is somewhat difficult to read and follow.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel method for camera synchronization using a tensor-based approach, specifically focusing on the block trifocal tensor. The authors discovered that the block tensor of trifocal tensors has a (6, 4, 4) core tensor in the Tucker factorization (HOSVD), independent of the number of cameras under mild constraints. The authors propose a Higher-Order Singular Value Decomposition (HOSVD) framework with imputation techniques to handle missing data and achieve robust camera synchronization. The methodology is tested on various datasets, demonstrating competitive performance in terms of translation and rotation errors compared to existing methods.\n\nSTRENGTHS:\nThe paper contributes significantly to the field of multi-view geometry and camera synchronization through the introduction of a tensor-based approach using the block trifocal tensor and HOSVD. The originality of applying a (6, 4, 4) core tensor in the Tucker factorization and the robustness provided by the STE are noteworthy. \n\nThe technical claims are generally sound, supported by experimental evidence and detailed derivations.\n\nThe theoretical parts of the paper are well-presented, providing clear and detailed explanations of the methodology and derivations.\n\nWEAKNESSES:\nThe novelty could be further strengthened by exploring and comparing additional tensor decomposition methods and imputation techniques. Addressing scalability concerns and providing a more detailed comparison with existing methods would also enhance the contribution.\n\nThe robustness of the imputation method for missing data, which involves using small random entries, needs further validation. This approach might introduce biases, and its impact on the final results should be carefully evaluated. Additionally, while the use of Subspace-constrained Tyler’s Estimator (STE) is innovative, its integration and performance, especially in handling outliers and noise, require more comprehensive analysis and justification.\n\nCertain sections are dense and could benefit from additional clarity, particularly the tensor notations and some derivation steps. The paper would also be significantly improved by including more visualizations to illustrate the results of camera synchronization, such as comparisons of estimated camera poses against ground truth and multi-view reconstruction results. The lack of these visual aids makes it challenging to fully grasp the practical impact of the proposed method.\n\nHandling of Rotations: The discrepancy in performance between the translational and rotational parts of the camera poses might be related to the rotation parameterization used. The paper employs robust rotation averaging and estimation methods but does not specify the parameterization of rotations. Considering parameterizations such as quaternions, Lie algebra representations with the exponential map, or the Cayley transform could improve the accuracy of the rotational estimates and enhance the overall synchronization performance.\n\nImputation Method: The imputation of missing data with small random entries might introduce biases. Alternative imputation techniques, such as Higher-Order Alternating Least Squares (HOALS), Tensor Nuclear Norm Minimization (TNNM), or Expectation-Maximization (EM) for tensors, should be considered for potentially more robust and accurate results.\n\nHandling of Outliers: The method's robustness to outliers could be further enhanced by integrating advanced outlier detection techniques such as Robust Principal Component Analysis (RPCA), RANSAC, the Huber loss function, or autoencoder-based approaches. These techniques could provide more robust results by effectively separating or mitigating the influence of outliers on the tensor decomposition process.\n\nSensitivity Analysis: The paper does not provide a sensitivity analysis to show how the method performs under different conditions, such as varying levels of noise, different amounts of missing data, or different types of outliers. This analysis would demonstrate the robustness of the method under various real-world conditions.\n\nVisualization of Results: The paper could benefit from more visualizations that illustrate the results of the camera synchronization and the accuracy of the imputed data. This would help in understanding the practical impact of the proposed method. Examples would be camera poses compared to GT, multiview reconstructions results from the estimated camera poses.\n\nThe paper is technically solid and presents a valuable contribution to the field. However, addressing the identified weaknesses and providing additional validation for the imputation method and scaling factor calculation would significantly strengthen the work.\n\nQUESTIONS:\nNotation Clarification: Can the authors provide a more detailed and consistent explanation of the tensor notations used, particularly for the trifocal and quadrifocal tensors?\nAlternative Imputation Techniques: Have the authors considered other imputation methods for handling missing data? How do these compare in terms of robustness and accuracy?\nAlternative Rotation Parameterization Techniques: Have the authors considered alternatives for the rotation parametrization? How do these compare in terms of reconstruction accuracy of the camera rotations?\nMissing Visualization of Results: Have the authors considered more visualizations that illustrate the results of the camera synchronization and the accuracy of the imputed data. Examples include visual comparisons of camera poses against ground truth and multi-view reconstruction results from the estimated camera poses?\nOutlier Handling: What specific steps are taken to identify and mitigate the impact of outliers in the data? How much do outlilers influence the result anyway when using HOSVD?\n\nLIMITATIONS:\nThe authors have acknowledged some limitations, such as the scalability of the block trifocal tensor with the number of cameras. Additional limitations that should be addressed include:\n\n    Robustness to Noise: The method's sensitivity to noise and outliers should be discussed in more detail, along with potential mitigation strategies.\n    Experimental Scope: The experiments are limited to specific datasets. Broader testing on more diverse datasets would strengthen the claims.\n    Computational Efficiency: The computational demands of the proposed method should be analyzed in more detail, especially for large-scale applications.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}