# Computer Vision Based Planogram Compliance Evaluation
## 3 Experiments and Analysis of the Results
### Results
It could be argued that these accuracies are a bit skewed in our favor, as we used a part of the whole GP-180 test set as a validation sethowever, as seen in Table 4, the performance using only the part of the data unseen during the validation is approximately the same as--and often even slightly better than--the full-set performance. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & \multicolumn{3}{c}{**GP-180 All**} & \multicolumn{3}{c}{**GP-180 Test**} & \multicolumn{1}{c}{**Highest Acc.**} \\ \cline{2-6}
**Method** & \(\mathbf{K=1}\) & \(\mathbf{K=5}\) & \(\mathbf{K=1}\) & \(\mathbf{K=5}\) & **at Epoch** \\ \hline \multirow{2}{*}{[23]} & Untrained & 0.787 & – & – & – & – \\ \cline{2-6}  & Best & **0.842** & **0.942** & – & – & – \\ \hline Our DIHE, untrained & 0.727 & 0.882 & 0.730 & 0.878 & – \\ \hline Our DIHE, trained with GP-180 & 0.809 & **0.936** & 0.811 & 0.930 & 9 \\ \hline Our DIHE, trained with internal data & **0.812** & **0.936** & **0.813** & **0.931** & 3 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Product classification performance with the GP-180 dataset of [13]. “GP-180 all” refers to all of the test annotations in GP-180, while “GP-180 test” refers to our test split. \(K\) refers to the number of nearest neighbors used to determine classification correctness, \(K=1\) means a classification is considered correct if the correct class is the nearest neighbor of the query, while \(K=5\) means that a classification is considered correct if the set of five nearest neighbors of the query contains the correct class. Both the results achieved by [23] and the best results achieved by us are highlighted in bold. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Method** & \(\mathbf{AP}_{\text{COCO}}\) & \(\mathbf{AP}_{0.50}\) & \(\mathbf{AP}_{0.75}\) & \(\mathbf{AR}_{300}\) \\ \hline Our GLN, optimized hyperparameters & **0.287** & **0.537** & **0.271** & **0.505** \\ \hline \multicolumn{4}{c}{+ Tanh activation} & 0.233 & 0.469 & 0.202 & 0.415 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Product proposal generation performance with the GP-180 dataset [13]. The IoU threshold used for AP is given as a subscript; \(\text{AP}_{\text{COCO}}\) is average precision calculated with and averaged over COCO IoU thresholds. The best results for each metric are highlighted in bold. We used a DIHE model from each of the respective datasets in our further planogram compliance evaluation performance tests. We kept the model trained with Grocery Products as a part of our tests despite the one trained with our internal dataset outperforming it even with GP-180. The quantitative planogram compliance evaluation performance of our approach is somewhat underwhelming, as shown in Table 5. However, this is mostly due to the classifier performance not being up to par. Figure 12 visualizes the planogram compliance evaluation performance for both the worst- and the best-performing images of our internal dataset using our best model (see Tables 5 and 6). It is evident that even with the worst performing image (Image 1, the top image in the figure) the RANSAC reprojection of the planogram matches the locations of most of the products, yet the classifier does not recognize the products correctly. Overall, the results show that our approach is a promising option for tackling the difficult automated planogram compliance evaluation problem. However, some further research into the classifier's accuracy is needed in order to achieve an operationally acceptable level of performance.