# An Energy Saving Road Sweeper Using Deep Vision for Garbage Detection
## 4 Experimental Results  
The system has been deployed in the real world, and has proven its correctness. The system has been tested for days of runtime on-board a sweeper truck. It has kept a steady \(FPS=1\) during all the tests, with no sign of slowdown or overheat. The \(y\) accuracy of the system is considerably good, even if only the vehicle speed feedback has been used for the moving 2D garbage buffer (and not the aforementioned steering angle). When driving straight, the \(y\) accuracy of the system is in the order of some centimeters with a CanBus vehicle refresh rate of 20 Hz. The accuracy of the network, and the whole work, clearly depends on the quality and representative nature of the training dataset, but can be finely tuned with the weights of the Categorical Cross-Entropy. In our experiments with a dataset of over 400 images for training and about 80 testing images, the sensitivity and specificity of the system were 91% and 94%, respectively. It is worth noting that power saving efficiency is directly proportional with the specificity of analysis, while the effective cleanup quality is directly related to the sensitivity term. It is also important to note that the test set (over which the accuracy is computed) has been purposely selected as hard cases for the network (partially over/under exposed pictures, hard backgrounds, wrong 3D perspective transformations). This has been done both to keep the better quality pictures for the training set (pictures which, is worth remembering, are limited in quantity since they need to be manually annotated) and also to highlight the performances for a \"worst case scenario\" for the network. As weights for categorical cross entropy, we used \(w_{dirt}=13\), \(w_{clean}=1\) since in most real street pictures, even dirty ones, garbage is under-represented (covers at most 10% of the picture area). Figure 11 shows an ROC curve that we created training different networks with different \(w_{dirt}\) parameters. This curve guided us in choosing the right \(w_{dirt}\) (the one that maximizes Sensitivity + Specificity). From the curve, it is evident that not using a weighted categorical cross-entropy (\(w_{dirt}=1\)) would greatly harm the accuracy of the selection process, in particular the Sensitivity term. This is another often overlooked aspect of training a neural network with an unbalanced dataset. In Table 2 we also computed the metrics of IOU and DICE. These metrics, typically used for segmentation tasks, describe the ability of the network to correctly classify the class of \(dirt\) with spatial awareness. In general, an IOU above 50% can be considered good, and we note that this percentage is well reached by our system. From the table we can see that using a lower \(w_{dirt}\) could further increase these metrics, at the price of a degraded sensitivity. For this application, we claim that sensitivity is more important, since IOU and DICE are heavily influenced by false positives, which are less of an issue in our domain: a few false positive percentage points only cause the roads to be \"overcleaned\" by that small percentage. We also run tests to demonstrate the learning capabilities of the network with different topologies, proving that our Mini U-Net represents a good trade-off of inference time and representation power. The topology has been changed only for the three innermost (bottleneck) layers, respecting the U-shape typical of this architecture, and imposing a limit to the innermost features of 512 and 1024, respectively (see Figure 12). The first test we run determined how the learning capabilities of the network change by varying its topology. In particular, we tested the two most relevant aspects for a cleaning scenario: specificity and sensitivity--see Table 3 for the results. The 256-features network exhibits performance very similar to the 512-sized one, which, surprisingly, is even better than the one with 1024 bottleneck features (the full, traditional, U-Net topology).