# Dual Image-Based CNN Ensemble Model for Waste Classification in Reverse Vending Machine
## 4 Experimental Results and Discussions
### Single Image-Based Classification  
As a baseline experiment, we classified the object with only the top view image. This experiment was conducted in two steps. First, we trained the model without the non-target dataset and classified the object into two classes, PET and Can classes. Since the models used in the investigation do not produce the non-target class, we applied the decision threshold of 0.8 to distinguish the non-target objects in the secondary validation. Table 3 shows the results of this experiment. SqueezeNet and MobileNetV2 showed the highest and balanced accuracy among the candidate models, but their classification performance for non-target objects was unsatisfactory. Furthermore, when we further divide non-target objects into fraud and IAO, the accuracy on fraud objects is significantly lower than IAO. This result indicates that CNNs trained with our dataset also show erroneous classification results on non-target objects including fraud objects, as in the previous RVM study [14]. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Step** & **Network** & **Input Image** &
\begin{tabular}{c} **Decision** \\ **Threshold** \\ \end{tabular} & **Class** \\ \hline (1) & Single CNN & Top view only & 0.8 & PET, Can \\ (2) & Single CNN & Top view only & 0.5 & PET, Can, Non-target \\ (3) & Single CNN & Front view only & 0.5 & PET, Can, Non-target \\ (4) & CNN Ensemble Model & Top view and Front view at once & 0.5 & PET, Can, Non-target \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Details of our experiments to find optimized classifiers. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Model** & **\#Params** & **PET** & **Can** &
\begin{tabular}{c} **Non-Target** \\ **Fraud** \\ \end{tabular} & **IAO** \\ \hline ResNet-18 & 11.2M & 86.36\% & 87.88\% & 22.86\% & 60.00\% \\ DenseNet-121 & 7.0M & 97.73\% & 77.27\% & 10.00\% & 71.67\% \\ SqueezeNet & 0.7 M & 93.94\% & 92.42\% & 14.29\% & 18.33\% \\ MobileNetV2 & 2.2 M & 93.94\% & 90.91\% & 24.29\% & 43.33\% \\ EfficientNet-B0 & 4.0 M & 89.39\% & 45.45\% & 34.29\% & 78.33\% \\ EfficientNet-B1 & 6.5 M & 65.15\% & 33.33\% & 45.71\% & 75.00\% \\ EfficientNet-B2 & 7.7M & 76.52\% & 42.42\% & 40.00\% & 73.33\% \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Performance of CNNs trained with top view data, classifying objects into PET and Can classes. Heuristic threshold has been applied to distinguish non-target objects. In the following experiment, we added the non-target class to improve the poor classification performance of non-target objects. Table 4 shows the results of the experiment. It is observed that the EfficientNet family and DenseNet have a lower classification accuracy of the Can class than the PET class, while SqueezeNet and MobileNetV2 show equally high accuracy for both classes. On the other hand, in the classification accuracy of the non-target class, all models showed better results than the first experiment. However, it is still not enough for a real-world RVM system. Comparing the results of SqueezeNet and MobileNetV2 in Tables 3 and 4, we observe that adding a negative class improves the robustness against intentional frauds and unexpected inputs. However, it also degrades the accuracy of target object classification. Therefore, we further investigated using model visualization of which parts of the images the training models focus on to make the classification decision. As the model visualization method, we used the Gradient-weighted Class Activation Mapping (Grad-CAM) [54]. Figure 9,b illustrate the Grad-CAM visualization results of correctly classified objects and incorrectly classified objects with the MobileNetV2 model, respectively. It shows that the trained MobileNetV2 concentrates on the neck and bottom part of the PET bottle rather than the label. So, it is challenging to classify a bottle with a short neck or a large label covering a whole part of the container. Contrarily, as for the aluminum cans, the model mainly focuses on the label but sometimes focuses on the shadows.