# Filtering Organized 3D Point Clouds for Bin Picking Applications
## 3 New Filtering Method  
As mentioned in the introduction, for organized point clouds, each Cartesian coordinate of a dataset can be represented as a matrix of size \(N_{row}\times N_{col}\), for example, \(\mathbf{Z}=z(row,col)\). In the underlying RGB image, each of its pixels has three assigned valid numerical values (for red, green, and blue components). For the organized 3D point cloud, the data structure is more complicated as some pixels may not have valid entries, as shown in Figure 4. This happens when the calculation of the depth value corresponding to a given pixel fails. The reason for such failures depends on the underlying sensor technology. For example, for time-of-flight sensors, it could be the oversaturation or undersaturation of the receiver. For triangulation-based systems, it could be a failure to find pixel correspondence due to object occlusions and shadows. Figure 4: Example of different formats of data output: (**a**) RGB image; (**b**) binary map, white pixels correspond to NaN pixels. The locations of such pixels in an organized point cloud are usually marked with NaN (Not a Number) or filled with \((0,0,0)\), which is the origin of the coordinate frame in which the 3D points are output. To take advantage of the organized format and apply well-known 2D techniques to the \(\mathbf{Z}\) image, NaN pixels must be first converted. If the direction of the \(z\)-axis is such that all valid 3D points have negative z components (i.e., \(z(n_{valid})<0\)), then the converted point cloud \(\widetilde{\mathbf{P}}_{n}\) is defined as  
\[\widetilde{\mathbf{P}}_{n}=\left(\widetilde{\mathrm{x}}_{n},\widetilde{\mathrm{y }}_{n},\widetilde{z}_{n}\right)=\left\{ \begin{array}{cc}\mathbf{P}_{n}&\text{ for }n=n_{valid}\\ (0,0,z_{min})&\text{ for }n=n_{NaN}\end{array} \right. \tag{1}\]  
where \(z_{min}\) is the minimum of all valid \(z(n_{valid})\). Then, a \(z\)-cut at the level \(d>0\) can be defined as the binary matrix \(\mathbf{Z}_{d}\) of the size \(N_{row}\times N_{col}\), such that  
\[\mathbf{Z}_{d}(row,col)=\left\{ \begin{array}{cc}1&\text{if }\widetilde{z}(row,col) \geq z_{max}-d\\ 0&\text{otherwise}\end{array} \right. \tag{2}\]  
where \(z_{max}\) is the maximum of all valid \(z(n_{valid})\). An example z-cut at \(\mathbf{Z}_{d}\) is shown in Figure 5. As can be seen, the binary mask consists of many disjointed clusters \(\mathbf{C}_{d}(k)\), \(k=1,\dots,\ K\), each characterized by its area \(\mu_{d}(k)\), expressed as a number of connected pixels in a single cluster \(\mathbf{C}_{d}(k)\). The proposed filtering method is based on the observations, which are illustrated in Figure 6. Any two clusters \(\mathbf{C}_{d}(i)\) and \(\mathbf{C}_{d+\Delta d}(j)\) belonging to two z-cuts \(\mathbf{Z}_{d}\) and \(\mathbf{Z}_{d+\Delta d}\) may be in one of the three relative positions on the binary maps:  
1. they do not have any common points, e.g., \(\mathbf{C}_{d}(i_{1})\cap\mathbf{C}_{d+\Delta d}(j_{2})=\emptyset\) as in Figure 6;
2. one is a subset of another, e.g., \(\mathbf{C}_{d}(i_{2})\subset\mathbf{C}_{d+\Delta d}(j_{2})\);
3. they are the same, e.g., \(\mathbf{C}_{d}(i_{1})\cap\mathbf{C}_{d+\Delta d}(j_{1})=\mathbf{C}_{d}(i_{1})\cup\mathbf{C}_{ d+\Delta d}(j_{1})\), and they have the same area \(\mu\) (units are number of pixels). Each cluster \(\mathbf{C}_{d}(i)\) determines a corresponding set of 3D points \(\mathbf{G}_{d}(i)\), which is a subset of the entire organized point cloud \(\widetilde{\mathbf{P}}\) (i.e., \(\mathbf{G}_{d}(i)\subset\widetilde{\mathbf{P}}\)). Thus, two identical clusters \(\mathbf{C}_{d}(i)\) and \(\mathbf{C}_{d+\Delta d}(j)\) define a subset of 3D points \(\mathbf{G}_{d}(i)\equiv\mathbf{G}_{d+\Delta d}(j)\) such that they are separated from the remaining points \(\widetilde{\mathbf{P}}\smallsetminus\mathbf{G}_{d}(i)\) by at least \(\Delta d\). These points  
Figure 5: Example of z-cut and the resulting binary map \(\mathbf{Z}_{d}\) for the depth map shown in Figure 1. White pixels are 3D points with \(\widetilde{z}>-550\) mm.