# Semantic 3D Reconstruction for Robotic Manipulators with an Eye-In-Hand Vision System
## 3 Overview of the Proposed Method
### Object Recognition and Fusion Segmentation
#### 3.1.2 Point Cloud Segmentation Based on the Geometric Feature Method  
Although mask R-CNN has a relatively high recognition accuracy, the image segmentation accuracy is still insufficient, so it is difficult to achieve high-precision 3D reconstruction by merely adopting 2D image segmentation. In order to improve the accuracy of segmentation, we also take advantage of the 3D point cloud segmentation method. Firstly, the point clouds have been decomposing into many small patches by way of supervoxel segmentation to implement over-segmentation and then perform clustering analysis using the locally convex connected patches (LCCP) method [41].  
The aforementioned LCCP method merely utilizes position and normal as not relying on the point cloud color. Suppose \(\overrightarrow{p_{i}}\) and \(\overrightarrow{p_{j}}\) represent two adjacent supervoxels, \(\textit{cov}(\overrightarrow{p_{i}},\overrightarrow{p_{j}})\) represents whether the connection between two supervoxels is convex. Extended Convexity Criterion and Sanity Criterion can be expressed with \(\text{CC}_{e}\left(\overrightarrow{p_{i}},\overrightarrow{p_{j}}\right)\) and \(\textit{SC}(\overrightarrow{p_{i}},\overrightarrow{p_{j}})\), respectively [41].  
Since the conventional LCCP method is not able to recognize two objects when the surface of different objects is tangential, it is necessary to differentiate objects by means of color information. In consideration of this problem, we improve the LCCP method by adding a parameter named the Point Color Criterion (PCC). We define \(\gamma\) as the maximum value of color-difference between two adjacent supervoxels, that is:  
\[\gamma(\overrightarrow{p_{i}},\overrightarrow{p_{j}})=\max\left(\left|R_{ \overrightarrow{p_{i}}}-R_{\overrightarrow{p_{j}}}\right|,\left|\text{G}_{ \overrightarrow{p_{i}}}-\text{G}_{\overrightarrow{p_{j}}}\right|,\left| \text{B}_{\overrightarrow{p_{i}}}-\text{B}_{\overrightarrow{p_{j}}}\right|\right) \tag{1}\]  
where \(\gamma(\overrightarrow{p_{i}},\overrightarrow{p_{j}})\) is larger than the threshold value \(\gamma_{\textit{thresh}}\), the two supervoxels are recognized as two different objects. \(\gamma_{\textit{thresh}}\) is an important parameter, which depends on the color difference between the objects. It is generally set to be a small value. Therefore, even if the color differences between the objects are small, the algorithm can also distinguish between them. However, too small a \(\gamma_{\textit{thresh}}\) will cause over-segmentation. The color criterion of point cloud can be defined as:  
\[\text{PCC}(\overrightarrow{p_{i}},\overrightarrow{p_{j}}):=\begin{cases} \text{true}&\gamma(\overrightarrow{p_{i}},\overrightarrow{p_{j}})<\gamma_{ \textit{thresh}}\\ \text{false}&\text{otherwise}\end{cases} \tag{2}\]  
As a result, the LCCCP method is judged by the criteria:  
\[\text{conv}(\overrightarrow{p_{i}},\overrightarrow{p_{j}})=\text{CC}_{e} \left(\overrightarrow{p_{i}},\overrightarrow{p_{j}}\right)\wedge\text{SC}( \overrightarrow{p_{i}},\overrightarrow{p_{j}})\wedge\text{PCC}( \overrightarrow{p_{i}},\overrightarrow{p_{j}}) \tag{3}\]