# Depth Estimation of a Deformable Object via a Monocular Camera
## 4 3D Reconstruction at \(i\)th Time  
This section introduces how to quickly reconstruct the dynamic object with a single RGB camera. For each frame by the camera, we first extract all the feature points. Through matching these feature points to those on the reconstructed point cloud, we are able to know to which sub-point-clouds these feature points individually correspond. Finally, the poses of the sub-point-clouds can be estimated by the correspondences between the feature points on the point cloud and the image feature points. This pose estimation problem can be easily handled by solving a nonlinear optimization. The reconstruction is therefore reduced to the reorganization of the sub-point-clouds with updated poses. Denote the image captured at \(i\)th instant (\(i>N\)) as \(\left\{m_{i}^{j}\right\}_{j=1..M}\). Through the SIFT algorithm, we can extract the feature points and their corresponding descriptor vectors. Through Section 2.2, a labeled 3D point cloud is reconstructed, to which the descriptor vectors are attached. Thus, we can use the descriptors from the captured image and from the cloud to find their correspondence, which can be denoted as a two-tuple set \(\left\{\left(m_{i}^{f_{i}},P_{f_{i}}^{*}\right)\right\}_{f_{i}=j_{1}..j_{F}}\). In this set, \(m_{i}^{f_{i}}\) and \(P_{f_{i}}^{*}\) respectively represent the feature point from the image and the cloud point, whose descriptor vectors are similar. In addition, \(\left\{\left(m_{i}^{f_{i}},P_{f_{i}}^{*}\right)\right\}_{f_{i}=j_{1}..j_{F}}\) yields a bijective map \(\varphi_{i}:\left\{m_{i}^{f_{i}}\right\}_{f_{i}=j_{1}..j_{F}}\rightarrow\left\{ P_{f_{i}}^{*}\right\}_{f_{i}=j_{1}..j_{F}}\). Suppose \(\mathcal{T}=\left\{\left[p\right]:p\in\Omega\right\}=\left\{\left\{q\in\Omega :q\sim p\right\}:p\in\Omega\right\}\) as the basis for the topology of space \(\Omega=\left\{P_{f}^{*}\right\}_{j=1..M}\). We can get the basis for topology (denoted as \(\mathcal{T}_{i}\)) of the subset \(\left\{P_{f_{i}}^{*}\right\}_{f_{i}=j_{1}..j_{F}}\subset\left\{P_{f_{i}}^{*} \right\}_{j=1..M}\) by  
\[\mathcal{T}_{i}=\left\{T_{c}\cap\left\{P_{f_{i}}^{*}\right\}_{f_{i}=j_{1}..j_{F }}\left|T_{c}\in\mathcal{T}\right.\right\}. \tag{16}\]  
We can further acquire the basis for the topology (denoted as \(\mathcal{T}_{m_{i}}\)) of the set \(\left\{m_{i}^{f_{i}}\right\}_{f_{i}=j_{1}..j_{F}}\) by  
\[\mathcal{T}_{m_{i}}=\left\{\varphi_{i}^{-1}\left(T_{m_{i}}\right)\left|T_{m_{i }}\in\mathcal{T}_{i}\right.\right\}. \tag{17}\]  
Based on our design, each element in \(\mathcal{T}\) is a rigid component of \(\Omega\). Thus, when the object represented by \(\Omega\) moves stochastically, the elements in \(T_{c}\in\mathcal{T}\) have the same rigid transformation, i.e., there exists a single pair \(\left(R_{i}^{c},t_{i}^{i}\right)\) for \(T_{c}^{i}\) such that  
\[T_{c}^{i}=R_{i}^{c}T_{c}+t_{c}^{i}, \tag{18}\]  
where \(R_{i}^{c}\) is a three-dimensional rotation matrix, \(t_{i}^{c}\) is a three-dimensional translation vector and \(R_{i}^{c}T_{c}^{i}+t_{c}^{i}\) is defined as  
\[R_{i}^{c}T_{c}^{i}+t_{c}^{i}=\left\{R_{i}^{c}p+t_{c}^{i}\left|p\in T_{c}^{i},\,T_{c}^{i}\in\mathcal{T}\right.\right\}, \tag{19}\]  
In the third step, when capturing a new image, we extract the SIFT features and match these features to those attached to the point cloud in the first phase. These matched features implicate the correspondence between the image and the point cloud. Based on the topological segmentation executed on the point cloud, the correspondence between the image and each topological part can also be computed. Then, the pose and position of each topological part can be deduced, and the reconstruction result can thus be obtained through a simple re-organization of these topological parts. The whole flow chart of the proposed approach is illustrated in Figure 3.  
Figure 3: The framework of the proposed method. **Construction of rotation matrix.