## 2 Materials and Methods
### Recognition of Rocks Using Machine Vision-Based Image Processing Technology  
Figure 4 depicts the machine vision-based image-processing algorithm for the RGB and depth images captured by the RGB-D camera. First, the RGB image was captured using the color sensor, and the region where the hopper was located was selected to be the Region of Interest (ROI). Then, image characteristics, such as brightness, gamma, and contrast, were adjusted based on the environment of the laboratory. Next, a pre-processing operation was performed to convert RGB images into grayscale images, and then into binary images. A suitable threshold was selected based on the characteristics of the experimental environment, e.g., camera angle, resolution, and illumination, to identify the shape of the rock accurately. Finally, the binarized image was pre-processed using a low-pass filter, which removed all elements other than rocks from the image. Thus, the coordinates of the rocks were obtained. \begin{table}
\begin{tabular}{c c c} \hline
**Equipment** & **Model** & **Specification** \\ \hline \multirow{4}{*}{Main Controller} & Laptop PC & Intel Core i7-9750H CPU 4.50 GHz (Intel, Santa Clara, CA, UAS), 16 GB RAM, \\  & Windows 10 (Microsoft & NVIDIA GeForce 1650 4 GB (NVIDIA, Santa Clara, CA, USA) \\ \hline \multirow{4}{*}{RGB-D Camera} & \multirow{4}{*}{Kinect V1 (Microsoft & RGB Camera resolution: 640 \(\times\) 480 \\  & Corporation, Redmond, WA, USA)} & 1R Camera resolution: 320 \(\times\) 240 \\  & & \\ \cline{1-1}  & & \
\begin{tabular}{c} Depth image Field of view: \(62^{\circ}\)\(\times\) 48.6\({}^{\circ}\) \\ Depth image Field of view: \(57^{\circ}\)\(\times\) 43\({}^{\circ}\) \\ Measuring range: 0.4 m\(-\)4 m \\ \end{tabular} \\ \hline \multirow{4}{*}{Robot Arm} & Niryo One (NIRYO, Wambrechies, France) & Num. of axis: 6 \\  & & Max reach: 440 mm \\ \cline{1-1}  & &
\begin{tabular}{c} Base joint range: \(-175^{\circ}\)\(\sim\) +175\({}^{\circ}\) \\ Repeatability: \(\pm\)1 mm \\ \end{tabular} \\ \hline \end{tabular}
\end{table} 
Table 1: Specification of sensors, the controller used in this study. Figure 3: System architecture of data processing for the proposed smart hopper system. Similarly, depth images were captured using the IR Sensor and the IR Emitter sensor. The area where the hopper was located was selected to be the ROI, and the distance value of the selected area of each depth image was converted into a distance unit in the real world, which was recorded in a matrix. A threshold value was selected to filter for distance values corresponding to the upper portions of the hopper. After filtering the image using the threshold, it was converted into a binary image. The binary RGB and depth images were combined into one by representing each pixel of the two binary images using 0 or 1 and multiplying them as matrices. This produced a binary image where only points corresponding to rocks on top of the hopper had a value of 1. Thus, information about the rocks located in the upper portions of the hopper was retained, while that of rocks below the hopper was ignored. Finally, the center of each rock above the hopper was calculated. The vision processing algorithms of National Instrument's LabVIEW and the Vision Assistant software were used to process RGB and depth images in this study. Figure 5 depicts the results obtained from the successive steps of the processing sequence in the case of an RGB image. The RGB image was captured in real-time (Figure 5a), and only the image of the hopper was extracted via the Image Mask function (Figure 5b). In addition, the brightness, gamma, and contrast of the image were adjusted based on the experimental environment using the Brightness function (Figure 5c). Finally, the RGB image was converted into a grayscale image emphasizing the green plane using the Color Plane Extraction function (Figure 5c,d). Figure 4: The procedure of machine vision-based processing of the RGB and depth images.