# Detection of Waste Containers Using Computer Vision
## 3 Performance Evaluation
### Reference Benchmark Scenario  
All of the following implementations were done on a standard ASUS NJ750 with Ubuntu 18.04.1 LTS. The images were all obtained using a Xiaomi Pocophone F1 and resized to 960 \(\times\) 720 in jpg format. We used a total of 375 images. Table 1 demonstrates how they were divided and used. The pictures were taken with different perspectives, illumination settings, and distances relative to containers. Table 1 demonstrates the picture distribution in terms of illumination settings. The pictures taken were of an emulation of a trash recollection setting, which is detailed as follows. The distance from a container to the camera was never greater than the width of an average road (approximately 10 m). In terms of the camera angle, if the front face of a container is considered to be at 90\({}^{\circ}\), the pictures were taken approximately between 40\({}^{\circ}\) and 140\({}^{\circ}\). This study focuses on four different types of containers for which there are some small differences in appearance. A focus on identifying containers with a different appearance is outside the scope of this preliminary study. These pictures also include several examples where there is partial occlusion of the containers to mimic a real scenario. Figures 1, 2, 7, 8, and 9 demonstrate the variety of camera-object distances, camera angles, illuminations, and obstructions. For both of the implementations, it was necessary to install the OpenCV 3.2.0 library [23]. For the implementation of VLAD, we used an open-source implementation of VLAD in Python called PyVLAD [24]. This implementation had some small advantages over the original [13], such as minor fixes to the code and the addition of multi-threading in the description phase. For YOLOv2 and YOLOv3, the starting point is to prepare the data. This means labeling each container on every \begin{table}
\begin{tabular}{c c c c}  & **Night** & **Day** & **Total** \\ \hline Multiple containers per image (for training the CNN) & 60 & 84 & 144 \\ Without containers (for training the CNN) & 63 & 81 & 144 \\ Individual containers (to create VLAD) & 0 & 56 & 56 \\ Multiple containers per image (for CNN prediction/querying VLAD) & 9 & 22 & 31 \\ Total & 132 & 243 & 375 \\ \hline \end{tabular}
\end{table} 
Table 1: The distribution of images. picture with the correct bounding box and class. Figure 1 shows an example of the labeling process. The labeling was done using the labellmg open-source tool [25]. After the labeling, we used a translation of darknet [22] to tensorflow, called darkflow [26], for training and testing the first implementation of YOLOv2. For the second implementation of YOLOv2 and the implementation of YOLOv3, we used a different version of darknet [27]. It is important to point out that all of the implementations of neural networks had approximately the same amount of training time. Neural networks almost need to be trained on a graphical processing unit (GPU) due to a GPU's superiority when performing parallel computation. The use of a GPU requires the installation of CUDA [28] and cuDNN [29]; more precisely, CUDA 9.0 and cuDNN 7.1.4. All parameters were left at their defaults or were rearranged to match the computational power available. These changes had no repercussions for the results. The metrics used to measure the VLAD results were created specifically for this implementation, and the details can be found in the results. For CNNs, the metric used was the mean average precision (mAP) [30]. For the first implementation of YOLOv2, in order to obtain some of the information about the mAP and the corresponding graphs, the open-source software mAP was used [31]. For the second implementation of YOLOv2 and the implementation of YOLOv3, the mAP was calculated while the CNN was being trained. This permitted us to plot a graph with the changes in mAP at every fourth iteration. The mean average loss is a metric that is used to identify how well a CNN is being trained and how much longer it needs to be trained.