# Computer Vision Based Planogram Compliance Evaluation
## 3 Experiments and Analysis of the Results
### Results  
As evident from Table 1, hyperparameter optimization gives significant improvements with the SKU-110K dataset to both the average precision and the average recall--1.8 and 4.4 percentage points, respectively--and our hyperbolic tangent activation in the Gaussian subnet gives an additional average precision boost of 0.7 percentage points for a total advantage of 2.5 percentage points over [21]. As shown in Tables 2 and 3, the performance of our models with the GP-180 dataset does not get anywhere near the performance of [21]. The suspected reason for poor GP-180 performance is differences in implementation: notably, we did not include anything like random crops or resizes of the training data in our training procedure. Nevertheless, we do not consider the sub-par performance of the model with the GP-180 set too big of a problem, as the test images in the set do not really correspond to real-life planogram compliance evaluation cases. \begin{table}
\begin{tabular}{c c c c c} \hline
**Method** & **AP\({}_{\text{COCO}}\)** & **AP\({}_{\text{0.50}}\)** & **AP\({}_{\text{0.75}}\)** & **AR\({}_{\text{300}}\)** \\ \hline
[21] & 0.521 & **0.891** & 0.562 & 0.569 \\ \hline Our GLN, optimized hyperparameters & 0.539 & 0.874 & **0.571** & **0.613** \\ \hline + Tanh activation & **0.546** & 0.874 & **0.571** & 0.612 \\ \hline \end{tabular}
\end{table} 
Table 1: Product proposal generation performance with the SKU-110K dataset of Goldman et al. [20]. The IoU threshold used for AP is given as a subscript; AP\({}_{\text{COCO}}\) is average precision calculated with and averaged over COCO IoU thresholds. The best results for each metric are highlighted in bold. \begin{table}
\begin{tabular}{c c c c c} \hline
**Method** & **AP\({}_{\text{COCO}}\)** & **AP\({}_{\text{0.50}}\)** & **AP\({}_{\text{0.75}}\)** & **AR\({}_{\text{300}}\)** \\ \hline
[21] & **0.506** & **0.862** & **0.548** & **0.634** \\ \hline Our GLN, optimized hyperparameters & 0.303 & 0.594 & 0.263 & 0.476 \\ \hline + Tanh activation & 0.296 & 0.569 & 0.252 & 0.417 \\ \hline \end{tabular}
\end{table} 
Table 2: Product proposal generation performance with classless annotations of the GP-180 dataset, given by [PERSON] et al. [38]. The IoU threshold used for AP is given as a subscript; AP\({}_{\text{COCO}}\) is average precision calculated with and averaged over COCO IoU thresholds. The best results for each metric are highlighted in bold. As the two product proposal generator models are slightly better than each other in different areas--the one with hyperbolic tangent activation with SKU-110K-like data, and the one with rectified linear unit activation with Grocery Products data--we kept both of the models in the loop when validating product detection and planogram compliance evaluation performance. During our DIHE implementation process, we noticed that achieving the classification accuracy reported by [23] is quite challenging. The DIHE training starts to overfit quite easily, and the best validation set performance is often achieved after just a few epochs of training. In addition to this, the out-of-the-box, no fine-tuning performance of the pre-trained VGG16 provided by PyTorch is five percentage points worse than that of the Tensorflow weights used by [23], meaning that more fine-tuning is necessary to achieve the same accuracy with PyTorch. Both of these facts are shown in Table 4.  
Despite these hurdles, the DIHE trained with our internal dataset almost reached the performance of [23], with nearest neighbor classification performance 3 percentage points behind their results. When considering a classification correct when the correct class is contained within the five nearest neighbors of the query image, our classification performance got even closer to that of [23], the gap being only 0.6 percentage points.