## 2 Materials and Methods
### Methods
#### 2.2.2 Model Definition  
(1) Network Structure: As given in Figure 5, details of the proposed channel R-CNN are provided. DenseNet is applied as the base model, which outputs the feature maps of the surgery frames. Then, the features are fed into two parallel channels to generate the instance segmentation masks and the rough cutting curve. In channel 1#, the Mask R-CNN structure is applied. Feature maps from the DenseNet are input into the region proposal network (RPN) [49] to obtain the alternative proposals, which are the RoIs. Then, the proposed Detection Network Head (DNH) is applied to output the specific category, accurate bounding box, and mask graph. As for channel 2#, the proposed Thorax Network Head (TNH) produces a rough cutting curve to navigate the operation of the thoracoscopic surgery, depending on the feature maps from DenseNet. Finally, the refined cutting curve is generated based on the outputs of the DNH and TNH, and an improved leapfrog algorithm [16] provides the path planning for the prediction. Due to the multi-channel architecture of the proposed model, we name it the channel R-CNN. (2) Objective Function: the proposed system aims to navigate the thoracoscopic surgery by outputting the refined cutting curve. The DNH and TNH models accomplish this task. Specific loss functions are designed: for channel 1#, \(L_{\text{det}}\) evaluates whether the RoIs are located accurately, \(L_{\text{cat}}\) is applied to measure the classification accuracy for different thoracic components, and \(L_{mask}\) determines whether the bounding boxes reg  
Figure 5: Overview of the proposed channel R-CNN structure. DenseNet is applied as the base model, and feature maps are extracted and fed into the two parallel channels (channel 1# and 2#). The Detection Network Head (DNH) is applied to output the specific category, accurate bounding box, and mask graph; the Thorax Network Head (TNH) produces a rough cutting curve. Finally, the refined cutting curve is generated based on the outputs of the DNH and TNH using the improved leapfrog algorithm. For channel 2#, a cutting prediction loss \(L_{cut}\) is proposed for the TNH, which evaluates the segmentation performance of the cutting path, as follows:  
\[L_{cut}=\sum\limits_{k=1}^{N}{(-q_{k}\log(p_{k})-(1-q_{k})\log(1-p_{k}))} \tag{1}\]  
where \(N\) represents the total number of applied frames in the model training; \(p_{k}\in(0,1)\), \(p_{k}=n_{in}/n_{total}\), and \(n_{in}\) represents the pixel number in the two-dimensional confidence interval of the annotation path; \(n_{total}\) represents the total pixel number of the cutting curve generated by the TNH. \(p_{k}\) is output by the sigmoid function of the proposed model. \(q_{k}=1\) when the created curve has an intersection with the ground truth; if not, \(q_{k}=0\). Therefore, the loss function \(L\) of the proposed channel R-CNN consists of two parts (channel 1# and 2#), by which the rough cutting curve is generated. \[L=\underbrace{L_{\text{det}}+L_{cut}+L_{mask}}_{DNH}+\underbrace{L_{cut}}_{ TNH} \tag{2}\]  
(3) Cutting path replanning: after generating the segmentation mask and the rough cutting curve from the DNH and TNH, we refined the initial cutting path based on the previously published work, using the improved leapfrog algorithm [16]. The mask image is effectively combined with the initial cutting indication information by the refined process, in order to avoid the key areas that would be likely to affect the patients' bodily function if they are cut up accidentally, as given in Figure 6. The image which combines the segmentation mask and the rough cutting line is marked as the directed graph \(G=(V,E)\), \(V\) represents the pixel nodes, and \(E\) stands for the detected edges (the mask \(E_{M}\) and the rough cutting lines \(E_{R}\)). Then, the frogs are randomly generated to construct the initial population, which depicts the potential path among pixel nodes, and they are recorded as \(U=(U_{1},U_{2},\dots,U_{d})\); here, \(d\) represents the dimension of path space solution.