# An Energy Saving Road Sweeper Using Deep Vision for Garbage Detection
## 3 Description of The System
### Neural Network for Garbage Detection  
Detecting garbage and trash on a road from pictures calls for semantic segmentation. This system has the additional requirement of real-time detection, so the problem becomes also an instance of real-time segmentation. A well-known and simple neural network for semantic segmentation is the so-called U-Net [20]. While the original U-Net is not real-time, a reduced version of the U-Net can, in the present year (2020), segment a 1 Megapixel image in under a second on a mobile GPU (as we will show in Section 4). The first idea for this system was performing an end-to-end image-to-image transformation, from the original picture taken by the camera to a heatmap highlighting each piece of trash. The approach is similar to the task of segmenting road views for vehicles and pedestrians (e.g., CityScapes [26]). Differently from other approaches, we choose to train our system to detect only two classes: garbage and clean road (binary classification). First, we created a dataset of input road scenes and their expected output ground truths (highlighting garbage in red, while depicting the background in blue). Then, we trained an U-Net expecting the network to learn to replicate that behavior--i.e., detecting trash in the pictures. As input we used the original, unchanged pictures from the camera (perspective views like Figure 3, that yielded segmentations like the one in Figure 4). The approach proved to work quite well, but we noticed dimensionality inconsistencies (e.g., the same object would have been overly detected when near, and not detected at all when far). Moreover, after the garbage detection phase, an inverse perspective transform was necessary to convert the trash position from perspective pixel coordinates to orthogonal 3D world coordinates. This convinced us to perform the inverse perspective transform directly on the input image, instead of on the results, and use that orthogonal, top-view image of the street for the network classification (since the road is assumed perfectly horizontal). See Figures 3 and 5 for an example of the transformation. Using this method, we were also immediately able to establish a precise pixels to millimeters correspondence that eased every consecutive operation. The easiest way to obtain the perspective transformation matrix is to manually tape measure, in the real world, the four corner coordinates of the sweeper work area and highlight them in a camera picture (red boxes in Figures 3 and 5). These corner points will be mapped to:  
\[(d_{near},-\frac{h_{wid}}{2}),(d_{near},\frac{h_{wid}}{2}),(d_{far},-\frac{h_{wid }}{2}),(d_{far},\frac{h_{wid}}{2})\]  
by solving a linear system that yields the perspective transformation matrix. Thus, using that matrix we will create a dataset of orthogonal birds-eye street views (Figure 5) that, after supplying their respective ground truth labels (Figure 6), we will use to train the network. The only remaining aspect to establish is a suitable pixel/millimeter ratio for our transformed pictures. The smallest piece of trash we want to detect is a cigarette stub, so we choose a sufficient ratio of 1:1 pixel/cm (i.e., one pixel corresponds to a 1 cm \(\cdot\) 1 cm area in the real word). Higher resolutions would improve the quality of analysis but also notably slow down the performance of the system. This ratio will yield images of at least 300 \(\times\) 700 pixels, given the aforementioned \(d_{far}\) - \(d_{near}\) = 7 and \(h_{wid}\) = 3 (in meters). In fact, we chose to stick with 500 \(\times\) 700 images, having some additional lateral view in the higher part of the image due to perspective (the area outside the red box in Figure 5). This additional  
Figure 4: The semantic segmentation performed directly on the input (perspective) image (Figure 3). This approach is similar to the one of popular datasets like Cityscapes. Initial experiments with this setting showed its effectiveness, but the system has been further improved via orthogonal perspective transformation.