ColVO: Colonoscopic Visual Odometry Considering Geometric and Photometric Consistency

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 11:51:39

Abstract: Locating lesions is the primary goal of colonoscopy examinations. 3D perception techniques can enhance the accuracy of lesion localization by restoring 3D spatial information of the colon. However, existing methods focus on the local depth estimation of a single frame and neglect the precise global positioning of the colonoscope, thus failing to provide the accurate 3D location of lesions. The root causes of this shortfall are twofold: Firstly, existing methods treat colon depth and colonoscope pose estimation as independent tasks or design them as parallel sub-task branches. Secondly, the light source in the colon environment moves with the colonoscope, leading to brightness fluctuations among continuous frame images. To address these two issues, we propose ColVO, a novel deep learning-based Visual Odometry framework, which can continuously estimate colon depth and colonoscopic pose using two key components: a deep couple strategy for depth and pose estimation (DCDP) and a light consistent calibration mechanism (LCC). DCDP utilization of multimodal fusion and loss function constraints to couple depth and pose estimation modes ensures seamless alignment of geometric projections between consecutive frames. Meanwhile, LCC accounts for brightness variations by recalibrating the luminosity values of adjacent frames, enhancing ColVO's robustness. A comprehensive evaluation of ColVO on colon odometry benchmarks reveals its superiority over state-of-the-art methods in depth and pose estimation. We also demonstrate two valuable applications: immediate polyp localization and complete 3D reconstruction of the intestine. The code for ColVO is available at https://github.com/xxx/xxx.

Main paper: 
 introduction: 
 The primary objective of colonoscopy examinations is lesion localization. Traditional colonoscopy procedures typically rely on capturing two-dimensional (2D) images from colonoscopy videos to identify the location of lesions within the planar region. However, due to limitations in 2D observation such as restricted field-of-view (Fov) and potential occlusions, physicians often rely on their expertise to estimate the approximate location of lesions within the intestinal tract, making it difficult to avoid subjective errors. With the emergence of large-scale cross-modal colonoscopy datasets and integrated deep learning (DL) methods, DL-based three-dimensional (3D) perception techniques have the potential to enhance lesion localization accuracy by recovering 3D spatial information of the intestine. However, most existing methods [19, 21, 24, 40] focus only on single-image depth estimation, overlooking the geometric continuity between frames of colonoscopy videos. Consequently, they can only reflect the spatial position of lesions within a local frame and cannot accurately determine their precise global coordinates within the entire colon. To solve this problem, some work [12, 29, 30, 32, 38] have attempted to joint camera motion estimation with depth estimation by minimizing photometric errors between consecutive frames to obtain global 3D spatial information of the intestine. However, these methods fail to consider specific challenges in the colon, including sparse texture features and lighting fluctuations. The former affects the sufficient and effective feature extraction in network models, thereby impacting depth and pose estimation consistency and accuracy. The latter arises from the correlated motion between the camera and light source during colonoscopy examinations and the non-Lambertian reflectance properties of tissues, both of which significantly affect the constraints of photometric errors on network models. Consequently, these methods cannot balance the accuracy of depth and pose estimation.In this paper, we propose ColVO, a novel colonoscopic visual odometry framework for precise spatial localization of lesions by simultaneously enhancing depth and camera motion estimation in colon environments. To address the challenges in the colon, we first introduce the deep couple strategy for depth and pose estimation (DCDP). DCDP incorporates cross-modal RGB and inferred depth features from the depth estimator through multilevel fusion in the pose estimator. The estimated depth and pose are jointed through a reprojection-based photometric loss function to constrain the network. By leveraging the scene geometry features from the depth maps, DCDP improves pose estimation accuracy and ensures geometric consistency in low-texture conditions. To tackle the illumination variation in the intestinal environment, we present the light consistent calibration mechanism (LCC). The LCC calculates the light state of each frame based on the surface normal and camera pose. By reconstructing the photometric loss function using LCC and mask for non-Lambertian regions, photometric consistency between adjacent frames can be aligned to enhance the continuity and accuracy of depth and pose estimation under nonuniform illumination conditions. The main contributions of our paper are as follows:• We design a novel VO framework tailored for lesion localization in colonoscopy (ColVO) that estimates depth and camera motion simultaneously. To meet the clinical application of precise polyp localization and 3D colon reconstruction, our ColVO framework focuses on geometric consistency and photometric consistency to enhance the accuracy of depth and pose estimation. • We introduce DCDP, a novel technique that couples depth and pose estimation in a single network and uses cross-modal fusion of RGB and depth features to enhance the pose estimator. • We propose LCC, a novel technique that calibrates the pixel values of adjacent frames based on the light state and masks out the non-Lambertian regions to enhance the photometric consistency. • We conduct experiments on two synthetic datasets and one real dataset and show that ColVO achieves state-of-the-art accuracy in depth and pose estimation in the colon environment. We also show that ColVO enables complete and coherent 3D reconstruction of the colon, accurate colonoscope trajectory prediction, and precise lesion localization. 
related work 2.1 monocular depth estimation method: 
 Traditional depth sensors such as stereo, structured light, or timeof-flight cameras are ineffective when used in narrow intra-body environments, especially in colon scenes [30] . Therefore, algorithmbased monocular depth estimation using colonoscopic images is preferred. However, these methods encounter challenges in colon environments, including lighting variations and texture scarcity. Oda et al. [21] addressed the issue of light reflections and textures in monocular endoscopic depth estimation by directly removing them using Lambertian surface translation. Similarly, Visentini-Scarzanella et al. [33] employed a mapping technique for input bronchoscopy frames. However, these crude approaches hinder the network model from effectively capturing the textured features of the colon and fail to address the fundamental challenges posed by the intra-body environment in depth estimation tasks. The scarcity of texture-rich images makes it challenging to provide valuable information for guiding network training. Although some studies [3, 8, 10, [13] [14] [15] [16] have considered using sparse depth information, easily obtained through traditional structure-from-motion or simultaneous localization and mapping (SLAM) methods, to assist in image-based depth estimation. However, these methods introduce challenges such as inconsistent distribution of valid pixels when combined with fullresolution RGB images. Inspired by depth modality's effectiveness in enhancing depth estimation, our method leverages the fusion of inferred depth with RGB information to improve depth and pose estimation in the colon. Although existing DL-based depth estimation algorithms can generate satisfactory depth maps for individual images, their effectiveness in endoscopy is often limited to the local spatial region of a single frame [19, 24, 42] , disregarding the correlation between consecutive frames. Consequently, when attempting to stitch together the depth maps of each frame to generate a complete 3D model of the colon, misalignment may occur in the space projection of adjacent frames, resulting in deficiencies in the reconstructed gastrointestinal model and the inability to obtain the location of lesions within the entire colon.Localization lesion in the colon requires predicting both dense depth and camera pose for each frame. To achieve this, some works design VO framework jointly training depth estimation network (DepthNet) and pose estimation network (PoseNet). Turan et al. [30] proposed Deep EndoVO, which is the first monocular VO approach for endoscopy through DL techniques. However, Deep EndoVO does not strictly belong to the end-to-end VO as the depth map is derived from traditional visual geometry methods. Similarly, DL technology is only adopted in the DepthNet of Endo-Depth-and-Motion [25] , resulting in a disjoint between DepthNet and the Pose estimator and preventing close integration during training. Subsequently, some studies have utilized view synthesis between consecutive frames as a photometric error function to constrain both depth and pose, thereby achieving unsupervised/self-supervised VO. The premise for photometric error is constant illumination and unchanged pixel brightness with respect to the view direction. Unsupervised VO must incorporate illumination variations in the endoscopy environment. Turan et al. [32] presented the first unsupervised endoscopic VO Endo Odometry Learner and designed a soft reliability mask to address factors that disrupt view synthesis, such as non-Lambertian surfaces. However, [32] lacked sufficient details on the soft reliability mask. Ling et al. [12] also created masks to remove highlighted regions according to the values of saturation and intensity. Zhang et al. [45] extended classic photometric loss with feature matching to compensate for changing contrast and brightness. Ozyoruk et al. [22] leveraged brightness-aware photometric loss to improve the robustness under fast illumination changes in endoscopic videos. These methods primarily rely on masks or adjustments in lighting from the image surface to mitigate illumination effects. In contrast, our approach tackles the underlying cause of illumination variations in the intestinal environment, namely, the motion of the light source accompanying the camera, and recalibrates the illumination. Notably, large illumination changes and a lack of texture in colon scenes can result in unsatisfactory performance of the DL when trained without supervision on colonoscopic videos [26, 38, 45, 46] .Current VO frameworks typically treat DepthNet and PoseNet as separate parallel networks and overlook the interconnectedness between the two sub-networks. In contrast, our ColVO leverages the potential benefits of the inferred depth map from DepthNet to improve pose estimation accuracy. Additionally, some endoscopic VO methods [12, 25, 30, 32] are not specifically designed for the colon. Some methods [18, 45] can only achieve local mapping to reconstruct the partial colon, thus failing to meet the demand for global lesion localization. 
method: 
 In this section, we present our ColVO network which takes consecutive colon video frames as input and simultaneously generates dense depth maps and colonoscopic trajectories. We first describe the entire ColVO framework, and then explain the learning objectives of ColVO.Our goal is to jointly train DepthNet and PoseNet using monocular colonoscopic videos. As shown in Fig. 2 , given two consecutive colon frames with dimensions 𝐻 ×𝑊 ×𝐶 at time 𝑡 and 𝑡 +1, DepthNet first generates dense depth maps with dimensions 𝐻 ×𝑊 ×1. Then, the predicted depth maps, denoted as ( D𝑡 , D𝑡+1 ), are concatenated with the image pairs and fed into the PoseNet to estimate the relative pose parameterized as 6-DoF transformation matrices T𝑡→𝑡+1 between consecutive frames. DepthNet follows the U-Net encoder-decoder architecture [27] , while PoseNet adopts an encoder-regressor structure. The DCDP strategy leverages the RGB-depth incremental fusion in PoseNet for improved geometric consistency between consecutive frames. To deal with drastic illumination changes and weak texture in the colon, ColVO integrates both supervised and self-supervised constraints to train DepthNet and PoseNet. Supervised signals are derived from ground-truth (GT) depth maps and pose information, while self-supervised signals are based on photometric loss. As the movement of light sources with the colonoscope and non-Lambertian reflection of colon tissues readily disrupt the light consistency in the photometric loss, the LCC mechanism is designed for reconstructing the photometric loss to ensure consistent lighting conditions.To achieve accurate lesion spatial localization, we need to estimate both dense depth and precise colonoscope trajectories from monocular images. However, these two tasks are interdependent and challenging in the colon environment. On one hand, the pose estimation relies on the image features, which are often limited and affected by the dynamic lighting in the colon. On the other hand, the depth estimation depends on the geometric consistency and continuity, which can be disrupted by the inaccurate pose estimation. Therefore, we propose a deep coupled VO framework that jointly trains DepthNet and PoseNet in a sequential and progressive manner. Unlike the traditional parallel dual-stream structures [2, 6, 12, 30, 45, 47] that use only image inputs for pose estimation, our pose estimation sub-network leverages both RGB and inferred depth inputs from DepthNet. Inspired by Jiang et al 's work [11] , we use a multilayer incremental fusion strategy [11] to fuse cross-modal features from intermediate layers of the encoder. This strategy allows the PoseNet to benefit from the rich and relatively robust geometric features from DepthNet branch. We use ResNet-18 [43] as the encoder for both streams, but we only share the weights except for the batch normalization (BN) layer, thereby ensuring that the features before BN are in the same latent space, so that they can be exchanged between streams. To preserve complementary features between RGB and depth modalities, we use a channel exchange (CE) strategy [36] that swaps important feature elements in each stage of feature fusion.EQUATIONwhere D = 𝐷𝑒𝑝𝑡ℎ𝑁 𝑒𝑡 (𝐼 ), 𝑅𝑒𝑙𝑢 is the activation function. After encoding each modality into the generic feature space, we use a pose regressor to predict a 6-dimensional relative pose representation from each feature.EQUATIONwhere 𝐸 ′ 1 , 𝐸 ′ 2 are generated by multiple Eq. 1 processes. The final output is the refined pose estimation from the fourth stage. We also apply the training loss proposed in [11] to prevent the model from reaching singular solutions.While the deep coupling of depth and pose estimators in the network architecture enhances the geometric consistency between consecutive frames, the DL-based VO task ensures photometric consistency between consecutive frames by minimizing photometric loss based on estimated depth D𝑡 and pose T𝑠→𝑡 . In addition to the classical image pixel error [47] , we also adopt image dissimilarity error SSIM [2, 37] in photometric loss to effectively handle illumination changes.EQUATIONwhere 𝑆𝑆𝐼𝑀 (𝐼 𝑡 , Î𝑠→𝑡 ) and ∥𝐼 𝑡 -Î𝑠→𝑡 ∥ stand for the element-wise similarity and pixel-level similarity between the target image 𝐼 𝑡 and the synthetic target image Î𝑠→𝑡 , respectively. Î𝑠→𝑡 = 𝐼 𝑠 ⟨𝑝𝑟𝑜 𝑗 ( D𝑠 , T𝑠→𝑡 , 𝐾)⟩, ⟨•⟩ represents the sampling operator, 𝑝𝑟𝑜 𝑗 (•) returns the 2D coordinates of the depths in 𝐷 𝑡 when reprojected into the viewpoint of 𝐼 𝑠 . Following [2] , we use 𝛼 = 0.15 in our framework.As the classical VO systems task assumes a fixed absolute position of the light source in global environments, the synthesized images maintain constant illumination and unchanged pixel brightness with the target images. However, the light source moves with the camera in the colonoscopic videos, leading to the photometric variation of the same spatial position between consecutive frames (the upper right corner of Fig. 2 ). This means that a certain difference remains between Î𝑠→𝑡 and 𝐼 𝑡 even if the depth and pose estimation are completely correct, which disfavors neural network learning. To address the issue of photometric inconsistency in ColVO, we propose an LCC mechanism considering light source mobility. LCC is defined as the ratio of luminosity generated by the reflection of light rays from the same spatial point on the colon surface between continuous time 𝑡 and 𝑠. EQUATIONwhere 𝑚 represents the luminosity value and is denoted as 𝑚 = 𝑒l𝜌n according to photometric stereo methods [4, 7, 41] . 𝑒 and 𝜌 are constants and represent light source intensity and the surface diffuse reflection coefficient, respectively. ì 𝑙 represents the light direction and is denoted as ì 𝑙 = ì 𝑙 𝑙𝑖𝑔ℎ𝑡 -ì 𝑝, where ì 𝑙 𝑙𝑖𝑔ℎ𝑡 is the vector from the camera position to the light source position and ì 𝑝 is the orientation vector from the camera to the spatial point. ì 𝑛 stands for the surface normal vector and is calculated through principal component analysis (PCA) [28] using the predicted depth map. All the vectors that appear are in the camera coordinate system.By implementing LCC, we can recalibrate the luminosity values of adjacent frames and align the colon frames, originally captured under different illumination conditions, to a uniform lighting standard. This crucial step guarantees consistent illumination between frames in colonoscopic videos. Moreover, leveraging the LCC mechanism, we reconstruct the photometric loss function 𝐿 𝑝 (𝐼 𝑡 , 𝐿𝐶𝐶 • Î𝑠→𝑡 ) in ColVO to further enhance depth and pose estimation accuracy.According to [18, 26, 38] , neural networks trained solely on selfsupervision using colonoscopic videos can be susceptible to challenges posed by illumination variations and textureless features in the colon. To address this, ColVO incorporates both supervised and self-supervised learning into estimating depth and pose from colonoscopic videos. The supervised aspect L 𝑠𝑢 of ColVO involves minimizing the Euclidean distance between the predicted depth and pose and their GTs or references, which can be obtained using advanced sensors such as CT models [33, 44] and OptiTrack Prime [30, 31] . The self-supervised aspect L 𝑠𝑒𝑙 𝑓 of ColVO involves enforcing photometric consistency and smoothness constraints on the reconstructed images and depth maps. The total loss L is a weighted sum of the supervised and self-supervised losses, as shown below:EQUATIONEQUATIONEQUATIONwhere L 𝑔𝑑 and L 𝑔𝑥 are the L1 loss in depth and pose, respectively.𝐷 𝑡 and 𝑇 𝑠→𝑡 are the estimated depth and pose, while 𝐷 𝑡 and 𝑇 𝑠→𝑡 are the GTs. L 𝑛 (𝑓 𝑥 ) is the regularization term of PoseNet [11] , which is used to prevent the network from falling into a local optimum. 𝜆 is the parameter for each loss. Photometric consistency is usually satisfied under the assumption that the camera is moving and the scene is stationary. However, in the colon, the colon wall can perform irregular movements and the watery colon surface can reflect the light and violate the Lambertian reflectance assumption. To address these challenges, ColVO uses two masking techniques: auto mask M 𝑝 [6] and temporal mask M 𝑡 [6] . The auto mask removes the pixels that do not conform to the photometric consistency assumption, while the temporal mask removes the pixels that do not have corresponding matches between the preceding and the following frames. The final photometric loss is as follows:EQUATION)where 𝐿𝐶𝐶 is the local color correction mechanism proposed by ColVO to handle the illumination variations in the colon. 
experiments 4.1 experimental setup: 
 Datasets:We used two typical benchmark datasets, namely the VR-Caps dataset (VCD) and the colonoscopy simulator dataset (CSD), to train and evaluate our ColVO model. The VR-Caps dataset was collected from a virtual capsule endoscopy system [9] that provides realistic 3D models of the colon with real in-body texture mapping. We split the VCD dataset, which consists of 17,057 RGB-D images with poses, into a training set of 8,676 continuous colon image sequences and a validation set of 8,381 colon image sequences, following [34, 39] . The CSD dataset was collected from a colonoscopy simulator [44] that offers a more challenging colon environment with rich vascular texture features. The CSD dataset contains two paths with a total of 3500 images. We use one path with 1748 images as the training set and the other path with 1752 images as the validation set. In addition, we also reported the results on a real colon dataset [22] named olympus dataset (OD) to evaluate the generalization ability of our model.Evaluation Metrics: We adopted the classic VO criterion [47] to evaluate the depth and pose. We used the error metrics (RMSE, abs.REL) and the accuracy metrics (𝛿 1 , 𝛿 2 ) to evaluate the depth estimation performance. We used the Absolute Trajectory Error (ATE) and Rotation Error (RE) to evaluate the error of the estimated position and rotation.Implementation and Settings: We implemented our models using PyTorch [23] . Both the DepthNet and PoseNet received input of size 320×320 pixels. We trained the model for 20 epochs with a batch size of 12. We set the learning rate to 10 -4 and reduced it to 10 -5 after 15 epochs. Following [6, 11] , we set the hyperparameters in Eq. 8 as follows: 𝜆 1 = 0.1, 𝜆 2 = 0.001, 𝜆 3 = 0.0002.We evaluated our ColVO method against traditional VOs [5, 20] and existing end-to-end DL-based VO methods [1, 2, 17, 22, 29, 47] . The traditional multiview geometry-based VO methods include ORB-SLAM [20] and DSO [5] . As shown in Fig. 3 , ORBSLAM failed to accurately predict colonoscope trajectories due to its reliance on strict feature detection and matching, which is impractical in feature-scarce colon environments. Similarly, the optical flow-based DSO algorithm also performs poorly in pose estimation. We trained these VO methods using open-source code on the VCD and the CSD dataset. However, we observed that these unsupervised VO methods failed to learn features effectively from challenging colon videos, resulting in extremely inaccurate depth and pose estimation. Therefore, we specialized them by incorporating supervised information (Eq. 5) to adapt to the challenges of the colon environment for fair comparisons. Fig. 3 and Fig. 4 show the qualitative results of the depth and pose estimation comparison with the SoTA methods, and Table 1 contains the quantitative results. Experimental results revealed that integrating supervised signals with MonoDepth2 [6] yielded competitive results in terms of depth and pose estimation. Therefore, we selected this method as the baseline method for comparison. Quantitative analysis (Table 1 ) revealed that the ColVO model exhibited exceptional overall performance, especially in position estimation. This advantage translates into achieving more precise colonoscope trajectories and improved 3D perception.Visualizing the depth estimation results (Fig. 4 ), the ColVO method produced more accurate and realistic depth maps than the existing methods. Classical VO systems, such as SfMLearner [47] and SC-Depth [2] , performed poorly in estimating the correct distances in the colon due to their focus on human activity scenes. Although the latest VO methods, such as DualRefine [1] and SRD-Depth [17] , exhibited considerable capability in adapting to the colon environment, they still produced relatively coarse depth maps with noise and artifacts when faced with the more challenging CSD dataset. As for endoscopic VO systems, Endo-SfMLearner [22] and AF-SfMLearner [29] showed better overall performance in depth map prediction due to their specialized modules for handling lighting variations and low-texture images. Nonetheless, these systems suffered from suboptimal error performance partly due to their error maps demonstrating significant absolute errors. However, regarding depth estimation, ColVO's performance is slightly inferior to Endo-SfMLearner. This discrepancy arises because ColVO emphasizes improving the precision of the global camera trajectory to meet the clinical requirement of precise polyp localization. Consequently, there is a trade-off in sacrificing some local depth estimation performance. The qualitative depth prediction results are consistent with the quantitative results shown in the error metrics RMSE and abs.REL, and accuracy metrics (𝛿 1 , 𝛿 2 ) in Table 1 .According to the visualization of the colonoscope trajectory in Fig. 3 , the trajectory predictions of ColVO are close to the ground truth and exhibit less drift or jitter compared to other methods. Most existing VO methods primarily focus on depth estimation, often utilizing PoseNet only to assist in depth prediction. As a result, these methods are unable to accurately estimate colonoscope trajectories that align with the reference. In clinical applications, doctors typically focus more on the location of polyps, hence the importance of endoscopic pose estimation exceeds that of depth estimation. As evident from Table 2 , although our method's depth estimation is slightly less accurate than the best one, its pose estimation is the best, aligning well with practical clinical applications.To better understand how the components of our ColVO model contribute to the overall performance in depth and pose estimation, we performed an ablation study by removing the corresponding modules from the complete ColVO model. Previous experiments demonstrated that the VCD dataset posed greater challenges to the VO task, especially in terms of colonoscope trajectory estimation. Therefore, we conducted ablation experiments on the VCD dataset. As listed in Table 2 , our ColVO method outperformed the baseline method in both depth and pose estimation, demonstrating positive and effective contributions of the designed modules in ColVO. Table 2 lists the performance degradation when specific modules are removed from our method, resulting in a decrease in performance in either depth or pose estimation, or both simultaneously. The visualization of trajectories in Fig. 5 further highlights this effect, providing a clearer depiction of the results. The purple trajectory representing the baseline exhibits the most severe deviation compared to the red trajectory of ColVO.We analyze the benefits of each module as follows: • Benefits of DCDP. Experiments in Table 2 and Fig. 5 demonstrated that this strategy led to a remarkable decline in pose error (ATE: 2.1% ↓, RE: 89.5% ↓) and a significant enhancement in scale consistency. In terms of depth estimation, the RMSE shows a slight increase. This is because the DCDP module prioritizes the accuracy of the overall camera trajectory rather than solely focusing on improving the accuracy of local depth estimation. Therefore, the DCDP can fulfill the specific clinical demand for exact polyps localization. • Benefits of LCC. From the baseline to -𝐷𝐶𝐷𝑃 row (only LCC) in Table 2 , the RMSE decreased by 5.79%, and the abs.REL by 5.95%, signifying a substantial depth accuracy enhancement due to LCC. Furthermore, pose-related errors witnessed a significant decline, with the ATE declining by 10.8%, and the RE by 23.9% compared to the baseline. Additionally, LCC is a versatile solution for environments with active light sources, like pipeline and tunnel robot operations. • Benefits of supervised signal. Supervision helps ColVO to directly overcome lighting variations and texture scarcity in the colon. As listed in Table 2 , without direct supervision, both depth and pose estimation results drastically decline. • Benefits of mask. The auto mask M 𝑝 and temporal mask M 𝑡 can eliminate the negative effects caused by the frame-to-frame image changes and the colon's self-motion, greatly improving the ColVO's performance.We validated the generalization performance of ColVO in real Olym-pusCam colon dataset using model trained on VR-Caps dataset. CT visualization in the bottom left corner of Fig. 6 revealed that the real colon exhibited a straight and elongated structure. The predicted depth map accurately captured the detailed depth map of the colon, successfully estimating even small abnormalities within the colon, particularly at node 3 ○ . Additionally, the predicted trajectory closely aligned with the actual trends and variations of the colon, especially at node 2 ○. Similarly, we make a comparison between ColVO and other methods. Consequently, these methods fail to reconstruct satisfactory 3D colon models. In contrast, ColVO generates clearer and more accurate depth maps, even in polyp regions. For example, SfMLearner, SC-Depth and Endo-SfMLearner fail to produce accurate depth predictions. Although AF-SfMLearner, DualRefine and SRD-Depth show better depth maps, their trajectory predictions are unsatisfactory. The 3D colon model with complete structure and correct shape demonstrates ColVO's superior generalization performance.According to [35] , scale is defined as the median of a depth map. The significant scale fluctuation between adjacent frames will cause the corresponding depth maps fail to be accurately stitched together to produce a complete and accurate 3D colon model. Fig. 7 indicted that merging DepthNet and PoseNet in a simplistic and independent manner can actually result in depth scale estimation outcomes that are even worse than those achieved by DepthNet alone. In contrast, DCDP effectively reduced scale fluctuations and trajectory errors between frames, leading to more accurate ego-motion estimation and enhanced geometric consistency.To demonstrate the effectiveness of our ColVO, we implemented a 3D colon reconstruction visualization in Fig. 1 by stitching together the dense depth maps of each frame using the colonoscopic trajectory. We did not incorporate any additional design such as surface mesh, the dense point cloud generated by ColVO provided intuitive and original results. In comparison to other colon VOs [13, 18, 45] , which focus only on partial colon reconstruction over short timescales, the ColVO is capable of reconstructing the complete 3D colon model along with long timescales colonoscopic trajectory. Although there may be a few noise points present, our colon models have a clear shape and intact structure.We demonstrated a valuable and practical application of our ColVO in polyp localization, as depicted in Fig. 1 . Our ColVO enables immediate and direct acquisition of the 3D position of lesions during colonoscopy. This capability holds significant importance in the diagnosis and treatment of gastrointestinal diseases. After the polyps were detected, we utilized the estimated depth values and endoscope pose by ColVO to calculate the spatial positions of the polyps. By comparing with the GT values, most of the predictions can achieve a level of accuracy within millimeters. 
conclusion and future work: 
 This paper proposed ColVO, a novel model for simultaneous dense depth estimation and camera pose prediction in challenging colon environments. Our model leveraged two key innovations: the DCDP module, which fuses depth information with RGB and enforces geometric consistency to enhance the PoseNet performance, and the LCC mechanism, which adapts itself to the dynamic light source and improves the photometric consistency. We conducted extensive experiments on synthetic and real colon datasets and demonstrated that our model achieved superior results and clinical benefits compared to existing methods. In the future, we will focus on semantic cues such as surgical instruments, bodily fluids, polyps, and cancerous lesions can potentially serve as valuable feature cues to enhance ColVO performance. 
 
 Appendix:  


Decision: This paper initially got mixed reviews. The authors' rebuttal addressed the reviewers' concerns and all reviewers reached a consensus on the acceptance of the paper.

Reviews: review: paper_summary: paper_summary: This paper aims to address the challenge of lesion localization, which involves jointly estimating depth and pose. Current methods treat pose and depth as separate tasks, but this paper suggests coupling them for better accuracy. Additionally, to tackle brightness fluctuations, the paper introduces a light-consistent calibration mechanism to filter out unreliable pixels. Experimental results confirm the effectiveness of these proposed methods.
strength_weakness: strength_weakness: 1. The paper proposes two novel and effective models for vendor odometry.
2. The evaluation is thorough, and the baselines appear adequate.
limitations: limitations: 1. I have observed that coupling pose and depth estimation leads to better results, but I still have concerns about the motivation behind coupling depth and pose estimation. While it is logical that better depth prediction improves pose prediction and vice versa, there is a concern that worse depth leads to worse poses, and vice versa. It would be helpful if the paper could provide more reasons and analysis to support the idea that depth and pose should be learned jointly. 

2. Regarding equation (4) for the LCC term, I find it difficult to interpret. As I understand, p represents the viewing vector and l_light represents the light direction. However, I am unable to clearly interpret the geometry of the light transport. It would be beneficial to include a figure for better understanding. 

3. In the minor issues, I suggest replacing "Fov" with "FoV" in line 92.
paper_summary: paper_summary: The paper poses two questions: 1) Existing methods treat colon depth and colonoscope pose estimation as independent tasks and design them as parallel sub-task branches.  2) the light source in the colon environment moves with the colonoscope, leading to brightness fluctuations among continuous frame images. Then they propose the framework ColVO which continuously estimates colon depth and colonoscopic pose using two key components, DCDP and LCC. Experiments with SoTA are shown to support the method.
strength_weakness: strength_weakness: 1. clear description of the method, including the method to simultaneously train the two separate tasks.
2. Better visualizations of the proposed method is shown, which supports the idea.
limitations: limitations: 1. About the posed Q1, the authors want.to bridge the gap with simultaneous training between two tasks, while the connection between the two tasks is not well modeled. Simply combining or concatenating them together in this task seems not to be a novel idea.
2. The proposed model seems to work better on pose estimation, while the performance on depth estimation is not the best as shown in Table. Although the authors claim that the two tasks can benefit each other, why the model does not work the best on depth estimation? This needs further explanation.
3. About the ablation study, if there are no errors in the table, the model also works well when deleting DCDP and LCC. Specifically, the "-DCDP" model works better on depth estimation, while failing to estimate accurate poses; and the "-LCC" model works well better on pose estimation, while failing to accurate depth. The authors do not provide enough about the results. Overall, the supervision signal seems the most important.
3. Various typos and problems should be corrected.
paper_summary: paper_summary: The paper present ColVO, a method that estimates depth and camera position simultaneously. The ColVO focuses on geometric and photometric consistency to enhance accuracy of both depth and pose estimation. The ColVO include two key components：a deep couple strategy for depth and pose estimation (DCDP) and a light consistent calibration mechanism (LCC). The ColVO achieves SOTA accuracy in depth and pose estimation on synthetic dataset and real dataset.
strength_weakness: strength_weakness: - The proposed ColVO framework enhance depth and pose estimation accuray by introducing geometric and photometric consistency.
- SOTA is achieved on synthetic dataset and real dataset.
- The paper is generally well written.
- The problem raised by the author that the light source in the colon environment will cause brightness fluctuations between consecutive frame images as the colonoscope moves, shows that the author has a full understanding and knowledge of the application scenario he wants to study.
limitations: limitations: - Can the author make the real OlympusCam colon dataset open source?

- The author set the hyperparameters as follows:$\lambda_1=0.1, \lambda_2=0.001, \lambda_3=0.0002$. These hyperparameters correspond to forcing photometric consistency and regularization term of PoseNet, which prevent the network from falling into a local optimum. However, in the ablation experiment, the impact of regularization term of PoseNet on the entire network was not analyzed. In addition, can the author give an approximate description of how the ranges of the above three hyperparameters are determined? Are the hyperparameters used in simulated data and real data consistent? Are these hyperparameters affected by the pixel numbers of the input image? How much impact will these hyperparameter values have on the final result? Can the author add a comparative analysis under different hyperparameter values?