# Computer Vision Based Planogram Compliance Evaluation
###### Abstract
Other difficulties of retail product detection when compared to generic object detection and object instance detection include the very high number of object classes (in the order of several thousand for small shops and tens of thousands for hypermarkets); the closeness of the appearance of the object classes (for example, for different flavors of the same product from the same brand), requiring fine-grained classification; and the frequently changing assortment-making models that require slow retraining infeasible [11]. Due to these peculiarities, the results achieved by [10]--and, indeed, results achieved with any pure generic object detection or instance detection methods--were quite poor. Retail product detection approaches that perform well enough to be utilized in practice have emerged only recently. Local invariant feature-based approaches have been utilized to solve the problem well beyond the breakthrough of CNNs for generic object detection. Notable approaches that utilize invariant features, in addition to the seminal [10], include combining corner detection, color information, and Bag of Words [12]; utilizing BRISK features and Hough transform estimation [13]; and detecting products one shelf level at a time with SURF features and dimensionality information-based refinement [14]. The first papers discussing the use of deep learning for retail product detection were published only in 2017. Initially, deep learning was utilized in hybrid approaches, that is, in combination with local invariant features. In notable hybrid models, deep learning was used for product classification [12] showing the power of deep learning over more traditional methods in the case of complex (that is, realistic) scenarios and [15] presenting a non-parametric probabilistic model for initial detection with CNN-based refinement trying to overcome the lack of training data. An attention mechanism was added by [PERSON] et al. [16], improving the detection accuracy and adaptability to new product classes without retraining. Starting in 2018, fully deep-learning-based product detection pipelines have started to emerge. These include combining a class-agnostic object detector with an encoder network utilized for classification into a model that, after being trained once, was able to fit new stores, products, and packages of existing products [11] and combining RCNN [5] object detection and ResNet-101 [17] classification with a specialized non-maximal suppression approach for rejecting region proposals for unlikely objects arising from overlapping products in images [18]. In addition to end-to-end retail product detection, research has been carried out towards _product proposal generation_--detecting the objects without classifying them. The main obstacle with this research was, until 2019, the lack of quality training data, with attempts to circumvent this via, e.g., 3-D renderings [19]. In 2019, however, Ref. [20] introduced the SKU-110K dataset, including images collected from thousands of supermarket stores and being of various scales, viewing angles, lighting conditions, and noise levels and thus enabled the training of deep product proposal generators. The baseline product proposal generation approach of [20] consisted of combining RetinaNet with a Gaussian-based non-maximal suppression. Later utilizations of SKU-110K include baking the Gaussian non-maximal suppression into the product proposal generator network itself, thus resulting in a multi-task learning problem [21], and addressing object rotations and outlier training samples with engineered-for-purpose neural network models [22]. The other end of end-to-end retail product detection--that is, _retail product classification_--has also seen some research separate from the full problem. One notable approach is that of [23], which combines training an encoder for k-nearest neighbor (KNN) product classification with utilizing generative adversarial networks (GANs) for data augmentation and a distance measure that takes into account the product hierarchy.