# Computer Vision Based Planogram Compliance Evaluation
###### Abstract
The planogram compliance evaluation problem is not solved by just detecting the products. Additionally, the detected arrangement needs to be compared to the planogram, and any deviations noted. This is not trivial, either: the exact positions where the products given in the planogram should be in the image are rarely known. Planogram compliance evaluation is not as well researched as retail product detection. The seminal articles [1; 24] came out in 2015, with approaches consisting of the utilization of detecting subsections of shelves [1] and distance metrics of the current shelf image with an image of the shelf taken in its ideal, planogram-compliant state [24]. Other notable approaches to the problem include viewing it as a recurring pattern detection problem [25], evaluating compliance one shelf at a time with the help of directed acyclic graphs (DAGs) [14], and viewing the product arrangements as graphs and solving for a maximum common subgraph between them [13]. The evaluation methods used in existing research on planogram compliance evaluation seem to assume that the test images match the planograms and to then just calculate standard object detection performance metrics based on how many of the facings were indeed detected. We argue that for a real-world use case, having test images that do not fully match the planogram is more interesting than having only a hundred percent compliant images. To enable evaluating planogram compliance evaluation methods with datasets that correct this deficiency, we propose a novel metric called normalized planogram compliance error \(E_{\text{PC}}\). Some of the recent developments bordering retail product detection have not been to our knowledge yet utilized in a full retail product detection pipeline. On the one hand, the SKU-110K dataset [20] has enabled the training of product proposal generators, deep neural networks that locate the products without classifying them, on an unprecedented scale. The dataset has already been successfully utilized in networks such as the Gaussian layer network (GLN) [21] and the dynamic refinement network [22]. On the other hand, the domain invariant hierarchical embedding (DIHE) [23] introduced the idea of utilizing GANs to train an encoder for KNN classification to be used for product classification purposes. To enable planogram compliance evaluation on large, real-world shelves and to take planogram compliance evaluation to the neural network era, we propose a novel retail product detection pipeline using GLN for product detection, DIHE for product classification, and RANSAC for pose estimation [26]. Our proposed pipeline is suitable for use in all brick-and-mortar stores regardless of the retail vertical. As the conventional metrics used for evaluating the performance of object detection are not suitable for planogram compliance detection, we propose a novel metric. We summarize our main contributions as follows:  
* We propose a novel retail product detection pipeline by combining a product proposal generator [21] with a product classification approach [23], and compare the product proposal generation, product classification, and retail product detection performance of the resulting framework to existing approaches. * We utilize the product detection pipeline and RANSAC pose estimation [26] to evaluate planogram compliance on real retail data consisting of full planograms and images that are not fully planogram compliant, while previous research has focused on small fully compliant subsections of planograms
* We propose a novel metric, the mean-squared planogram compliance error, for benchmarking planogram compliance detection performance.