# Computer Vision Based Planogram Compliance Evaluation
## 2 Materials And Methods
### Product Classification  
We use an embedding approach, the _domain invariant hierarchical embedding_ (DIHE, Ref. [23]), as our product classification model. It consists of three networks: the _encoder_, the _generator_, and the _discriminator_. The encoder network learns an embedding, and a function from images into vectors, with the distance between two output vectors signifying the similarity between the respective images. This embedding can be used to classify images via nearest neighbor search: the query image is converted into a vector with the encoder, and it is assigned a class based on the nearest embedding of a reference image. The encoder is the only network that is used beyond the training stage. The generator and discriminator are only used to augment the data when training the encoder. The purpose of this is to overcome the lack of data inherent with retail product image sets and to combat the problems caused by _domain shift_, that is, the training images being clear and iconic while the test images are noisy crops from an image of a retail store shelf. We selected the VGG16 [31] as the encoder network pre-trained with ImageNet, following the example of [23]. We use _maximum activation of convolutions_ (MACs)--the maximum value produced by select convolution layers--as the output of our encoder network, due to [23] achieving their best results with this approach. The encoder architecture is visualized in Figure 8.  
For the generator and discriminator, we adopt the pix2 pix GAN, Ref. [32] again, following the example of [23]. The GAN architecture is visualized in Figures 9-11. Keeping with the methodology of [23], we use _domain invariant hierarchical embedding loss_ (\(\mathcal{L}_{\text{DIHE}}\)) to train our encoder:  
\[\mathcal{L}_{\text{DIHE}}(i_{p},i_{n},h_{p},h_{n})=\mathcal{L}_{\text{triplet} }(G(i_{p}),i_{p},i_{n},\alpha_{\text{hierarchial}}(h_{p},h_{n}))\,, \tag{7}\]  
where \(i_{p}\) is an image belonging to the class of interest, \(i_{n}\) an image belonging to a different (randomly drawn) class, \(h_{p}\) the product hierarchy of \(i_{p}\), \(h_{n}\) the product hierarchy of \(i_{n}\), \(G\) the generator network of the GAN, and \(\alpha_{\text{hierarchial}}\) a function for calculating the triplet loss marginal. \(\alpha_{\text{hierarchial}}\) is specified as  
\[\alpha_{\text{hierarchial}}(h_{p},h_{n})=\alpha_{\text{min}}+(1-\frac{|h_{p} \cap h_{n}|}{|h_{p}|})\cdot(\alpha_{\text{max}}-\alpha_{\text{min}})\,, \tag{8}\]  
where \(\alpha_{\text{min}}\) and \(\alpha_{\text{max}}\) are hyperparameters that define the minimum and maximum values of \(\alpha_{\text{hierarchial}}\). Figure 8: VGG16 encoder utilizing MAC descriptors, built from the PyTorch implementation of said network [29, 31, 33]. The input is an image with shape \(3\times H\times W\). The model outputs a 1024-element MAC descriptor, with the first 512 elements determined by the maximum activations of the last layer of the second-to-last convolutional block, and the remaining 512 elements determined by the maximum activations of the last convolution layer. Downsampling is implemented via stride 2 when max pooling. The loss is based on the standard triplet ranking loss \(\mathcal{L}_{\text{triplet}}\)[34]. Given an encoder network \(E\), a desired marginal \(\alpha\), a distance function \(d(a,b)\), and three images--the anchor image \(i_{a}\); the positive image \(i_{p}\), which belongs to the same class as the anchor image; and the negative image \(i_{n}\), which belongs to a different class from the anchor image--a triplet ranking loss (\(\mathcal{L}_{\text{triplet}}\)) may be calculated as  
\[\mathcal{L}_{\text{triplet}}(i_{a},i_{p},i_{n},\alpha)=\max(0,\alpha+d(E(i_{a} ),E(i_{p}))-d(E(i_{a}),E(i_{n})))\,. \tag{9}\]  
The standard triplet ranking loss expects multiple training samples for each class. In the product detection problem setting, however, we have only one image per class.