## 2 Materials and Methods
### Recognition of Rocks Using Machine Vision-Based Image Processing Technology
Figure 6 depicts the results of the successive steps of converting the grayscale image in Figure 5 into a binary image and then processing it. A threshold value that included the color value of the rock and removed surrounding elements was selected and the image was converted into a binary image (Figure 6a). Using a low-pass filter and other techniques, the pre-processing algorithm removed the fine noise from the binary image (Figure 6b), and the morphology algorithm removed all objects in all regions except the rocks (Figure 6c). Finally, the coordinates of the extracted object were obtained via particle analysis (Figure 6d). Figure 5: Example of RGB image processing: (**a**) RGB image is obtained, (**b**) ROI is selected, (**c**) brightness, gamma, and contrast are adjusted, and (**d**) it is converted into a grayscale image. Figure 6: Example of binary image processing: (**a**) grayscale image is converted into a binary image, (**b**) low-pass filter is utilized, (**c**) small objects are removed, and (**d**) the object’s coordinates are calculated. Figure 7 depicts the results obtained from the successive steps of the processing sequence in the case of the depth image obtained from the RGB-D camera. First, the depth image was captured and visualized in real-time. Then, the distance values were converted to distance units in the real world and recorded in the form of a matrix (Figure 7a). The value corresponding to each pixel of the depth image was displayed. Points with distances that corresponded to the upper portion of the hopper were retained, while points with all other distance values (i.e., greater than that of the top of the hopper) were removed (Figure 7b). Finally, using the Morphology algorithm, all points apart from those corresponding to the smart hopper were deleted (Figure 7c). Figure 8 illustrates the simplified lattice form of the processed depth image of a smart hopper. The binary image of the top of the hopper obtained from the depth image and the binary image of the rock obtained via RGB image processing were treated as matrices of the same size and multiplied. As each pixel of a binary image can be represented by 1 or 0, multiplication of two such matrices produces a matrix whose (i, j)th entry is 1 if and only if the ith and jth entries of the first and second multiplicand matrices, respectively, are 1. Thus, the multiplication set the values of pixels corresponding to rocks below the hopper to 0 as they did not coincide with the pixels demarcating the hopper, and it set those of rocks above the hopper that coincided with the hopper's area to 1. Thus, rocks that had descended through the hopper inlet and were photographed and identified by the proposed system while they were being pulverized by the gyratory crusher were excluded from the blockage removal process, and the robotic arm did not try to dislodge them. Figure 9 presents the results obtained by following the algorithm to exclude rocks located below the hopper after determining the positions of rocks on the hopper. Figure 9a,b depict images of the hopper and the rocks expressed in binary form following the completion of image processing. Represented as a matrix, the red pixels take the value 1, while  
Figure 8: Detection of target objects on the hopper’s boundary using RGB and depth images. Figure 7: Example of depth image processing: (**a**) depth image is obtained, (**b**) it is converted into a binary image, and (**c**) border objects are removed. the black ones take the value 0. In Figure 9c, the depth and RGB images are superimposed to determine whether the locations of the detected rocks are above or below the hopper. In the example depicted in the figure, one of the two detected rocks is jammed above the hopper, while the other has already passed through the hopper inlet. These two rocks can be differentiated by multiplying the matrix forms of the two binary images; in the product, only pixels that correspond to the first rock take the value 1, while those of the second rock and all others take the value 0.