## 2 Bundle Adjustment for Motion Estimation Using HFR Cameras  
### Bundle Adjustment Reconstructs Motion's Trajectory  
The key idea of BA is that there exist bundles of light rays from the camera center and a projected position in images at each moment, and all parameters adjust simultaneously and iteratively together to obtain the optimal bundles of all times. Reprojection errors in bundle adjustment come from all deviations between the observed position in the projected images and the reprojected position at the same image at each time. The motion and the  
Figure 1: Continuous-time motion estimation using bundle adjustment (BA) and B-spline representation. trajectory are estimated along with the image sequences whereas the cameras are static to the world coordinate. We define the reproject error \(F_{ij}\) as the following equation:  
\[F_{ij}=c_{ij}-b_{ij}=\left[ \begin{array}{c}c_{ij,u}\\ c_{ij,v}\end{array} \right]-\left[ \begin{array}{c}b_{ij,u}\\ b_{ij,v}\end{array} \right] \tag{1}\]  
where \(b_{ij}\) is the observed position, \(c_{ij}\) is the reprojected position, \(i\) is the identification of the recording camera, \(j\) is the time stamp. The reprojected function can be expressed in short as:  
\[c_{ij}=\Pi\Big{(}PW_{j},\phi_{iw},t_{i}^{iw},y_{i}\Big{)} \tag{2}\]  
where \(PW_{j}\) is the position of the world coordinate at the time stamp \(j\); \(\phi_{iw}\) is the rotation vector presented in lie algebra \(\mathfrak{so}\) (3) from the world coordinate to camera \(i\) coordinate; \(t_{i}^{iw}\) is the translation vector from the world to camera \(i\), presented in camera \(i\) coordinate; \(y_{i}\) is the intrinsic parameter of camera \(y_{i}=[f_{i},k_{1i},k_{2i}]^{T}\) whereas \(f_{i}\) is the focal length of the camera \(i\); \(k_{1i}\) and \(k_{2i}\) are the distortion parameters of the camera. Equation (2) follows the pinhole camera model with the distortion expression. It can be detailed by the following equations:  
\[ \begin{split} PC_{ij}=\exp(\hat{g}_{iw})PW_{j}+t_{i}^{iw}\\ c_{ij}^{\prime}=\left[\begin{array}{c}\frac{PC_{ij,x}}{PC_{ij,x}}\\ \frac{PC_{ij,x}}{PC_{ij,x}}\end{array}\right]\\ c_{ij}=f_{i}(1+k_{1i}\Big{|}\Big{|}c_{ij}^{\prime}\Big{|}_{2}+k_{2i}\Big{|} \Big{|}c_{ij}^{\prime}\Big{|}_{4})c_{ij}^{\prime}\end{split}  \tag{3}\]  
The optimized parameters can be expressed as, with m recording cameras and n captured positions, \(\varphi=\big{[}\cdots,\phi_{iw},t_{i}^{iw},y_{i},\cdots,PW_{j},\cdots\big{]}_{ mn}^{T}\). All single reprojection errors \(F_{ij}\) form a \(mn\times 1\) reprojection vector \(F=\left[F_{11}^{T},F_{21}^{T},\ldots,F_{m1}^{T},\ldots F_{ij}^{T},\ldots,F_{mn }^{T}\right]^{T}\). The optimization process for the bundle adjustment is finding the minimum of all reprojection errors. In this paper, we use the sum of the squares constructed by a large number of nonlinear functions as the cost function to form a nonlinear least-square problem. \[\underset{(\varphi)}{\text{argmin}}\frac{1}{2}\underset{i=1}{\overset{m}{ \sum}}\underset{j=1}{\overset{n}{\sum}}F_{ij}^{T}F_{ij} \tag{4}\]  
To solve this unconstrained optimization Equation (4), basing on [PERSON]'s theorem, a combination of linear search and trust region method Levenberg-Marquardt (LM) algorithm is deployed in this paper. For each iteration of the LM algorithm, the increment of optimized \(i\) can be calculated by the following equation:  
\[\Big{[}\mathcal{J}^{T}\mathcal{J}+\lambda D\Big{]}\Delta\varphi=-\mathcal{J}^ {T}F \tag{5}\]  
where \(\mathcal{J}\) is the Jacobian matrix, calculated analytically by the partial derivatives of the reprojection error; damping factor \(\lambda\) controls the LM algorithm search for the result following the linear search or trust region; D is the damping matrix. For the original LM algorithm, D is set as the identity matrix. In this paper, we set D as a diagonal matrix; each element at the diagonal is the eigenvalue of the matrix \(\mathcal{J}^{T}\mathcal{J}\).