# Computer Vision Based Planogram Compliance Evaluation
## 3 Experiments and Analysis of the Results
### Metrics  
For evaluating product proposal generation and product classification performance, we use the industry standard metrics. Our product proposal generation metrics include mean average precision (AP) calculated at intersection of union (IoU) of 0.50 and 0.75 [36], the mean of APs calculated at IoUs of 0.50 to 0.95 in steps of 0.05 (APCOCO) [8], and average recall (AR) with 300 top detections [8]. The product classification metric we use is the recall at one and five top classifications. To the best of our knowledge, there do not seem to be any established evaluation metrics for planogram compliance evaluation performance despite prior work on the subject. The existing research seems to assume that the test image perfectly matches the planogram and to then just calculate standard object detection performance metrics based on how many of the facings were detected. We argue that for a real-world use case, having test images that do not fully match the planogram is more interesting than having only one-hundred-percent compliant images. Therefore, using the planogram as ground truth and evaluating product detection performance based on it--as has been often carried out by prior work--is not sufficient. Due to this, we propose a new metric, the _normalized planogram compliance error_\(E_{PC}\). Given the set of facings expected by the planogram \(E\), and the set of facings that are actually present in their expected positions in the test image \(F\subseteq E\), we can calculate _planogram compliance_\(PC\) simply as  
\[PC(F,E)=\frac{|F|}{|E|}\,. \tag{15}\]  
Let \(D\subseteq E\) be the set of facings that were detected by the planogram compliance evaluation model to be in their expected positions, and \(G\subseteq E\) the set of facings that are truly in their correct positions in the image, where \(G\) is determined via manual annotation. Utilising these, we can calculate the detected planogram compliance \(PC(D,E)\) and the ground-truth planogram compliance \(PC(G,E)\). Further, we can set the _planogram compliance error_\(PE\) as  
\[PE(D,G,E)=PC(D,E)-PC(G,E)\,. \tag{16}\]  
The planogram compliance error is a value between \(-1\) and \(1\), where the negative values signify underestimation by the model and vice versa. To further make the metric one-sided, we can calculate the square of planogram compliance error. By calculating the squared planogram compliance error for all of our test images and then calculating the mean of all the squared planogram compliance errors, we arrive at the _mean-squared planogram compliance error_, \(MSE_{PC}\),  
\[MSE_{PC}(R)=\frac{1}{|R|}\sum_{(D,G,E)\in R}PE(D,G,E)^{2}\,, \tag{17}\]  
where \(R\) is a set of tuples consisting of detected facings, ground truths, and planograms for each of the test images. The \(MSE_{PC}\) would otherwise be a good metric as it is, but its scale is somewhat awkward. Let us say we have two independent variables \(X\) and \(Y\), both of which are uniformly random between \(0\) and \(1\). The variable \(X\) could represent a randomly guessed planogram compliance, while the variable \(Y\) represents the ground-truth planogram compliance of some random image. The expected value of the squared error between these two variables--in our example, that is, the squared planogram compliance error--is  
\[ \begin{split} E[(X-Y)^{2}]&=E[X^{2}-2 XY+Y^{2}]\\ &=E[X^{2}]-2E[X]E[Y]+E[Y^{2}]\,.\end{split}  \tag{18}\]  
The expected value of a random variable \(Z\) that is uniformly random between \(0\) and \(1\) is \(E[Z]=\frac{1}{2}\), and the expected value of its square \(E[Z^{2}]=\frac{1}{3}\). Plugging these into Equation (18), we obtain  
\[ \begin{split} E[(X-Y)^{2}]&=E[X^{2}]-2E[X]E[Y]+E[ Y^{2}]\\ &=\frac{1}{3}-2\cdot\frac{1}{2}\cdot\frac{1}{2}+\frac{1}{3}\\ &=\frac{1}{6}\,.\end{split}