# Computer Vision Based Planogram Compliance Evaluation
## 2 Materials And Methods
### Product Proposal Generation  
We follow the example of [21] in choosing ResNet-50, a fully convolutional object classification network, pre-trained with the ImageNet dataset [7], as the backbone for our product proposal generator. The backbone architecture is visualized in Figures 2 and 3. We build a feature pyramid network (FPN, Ref. [4]) on top of this backbone, and augment the architecture further by making our network training a multi-task learning problem via the introduction of a Gaussian layer and subnet [21]. The architecture of the resulting network is visualized in Figures 4-7. We further explore substituting the ReLU activation of the final layer of the Gaussian subnet with Tanh. Our intuition here is that as the prediction target of the Gaussian subnet is limited to the range \([0,1]\), it makes sense to bar the neural network from producing values outside of this range. This alternative is visualized in Figure 6b. Figure 1: A visualization of the proposed approach. The inputs consist of the input image and the reference planogram. The purple box marked with \(P\) represents the product proposal generator, and the brown box marked with \(C\) represents the product classifier. The result shows which of the products expected in the reference planogram were included in the input image (green numbers) and which were not found (red numbers). The input image and the images that the reference planogram consists of are from the Grocery Products dataset of [PERSON] [28]. As suggested by [21], we use a weighted sum of three different loss functions, each calculated from different outputs of our product proposal generator network. Our full loss function is  
\[\mathcal{L}=\alpha_{\text{focal}}\mathcal{L}_{\text{focal}}+\mathcal{L}_{L_{1}}+ \alpha_{\text{MSE}}\mathcal{L}_{\text{MSE}}\,. \tag{1}\]  
In the above equation, \(\mathcal{L}_{\text{focal}}\) is _focal loss_[30]. We use it for foreground-background classification loss. It is calculated as  
\[\mathcal{L}_{\text{focal}}(p_{t})=-\alpha_{t}(1-p_{t})^{\gamma}\log(p_{t})\,, \tag{2}\]  
where  
\[\alpha_{t}= \begin{cases}\alpha&\text{if the detection belongs to the foreground},\\ 1-\alpha&\text{otherwise}\end{cases} \,. \tag{3}\]  
\(\alpha\in[0,1],\gamma\geq 0\) are hyperparameters,  
\[p_{t}= \begin{cases}p&\text{if the detection belongs to the foreground},\\ 1-p&\text{otherwise}\end{cases} \,, \tag{4}\]  
and \(p\in[0,1]\) is the estimated probability that the detection belongs to the foreground [30]. \(\mathcal{L}_{L_{1}}\) is \(L_{1}\) loss, or, in other words, mean absolute error. We use it as our bounding box regression loss. It is defined as  
\[\mathcal{L}_{L_{1}}(x,\hat{x})=\left|x-\hat{x}\right|, \tag{5}\]  
where \(x\) is the ground truth, and \(\hat{x}\) is the prediction. Figure 2: The bottlenecks in the PyTorch [29] implementation of ResNet-50. The input width is \(W\), height is \(H\), and it consists of \(F\), \(F\) - 2 or \(F\) - 4 feature maps, depending on the position in the network. The output from the bottleneck consists of \(F\) - 4 feature maps of size \(H\times W\) in the case of basic bottleneck or \(W/2\times H/2\) in the case of downsampling bottleneck. The identity connection (the blue arrow from input to before the final ReLU activation) is implemented as simple summation if the input is of the same shape as the output. Otherwise a \(1\times 1\) convolution, possibly with a stride of 2, is applied before the summation. The downsampling in the downsampling bottleneck is implemented via a stride of 2 in the middle, \(3\times 3\) convolution layer. The PyTorch implementation differs from the original ResNet-50, introduced by [PERSON] et al. [17], by downsampling with a stride of 2 in the \(3\times 3\) convolution layer instead of in the first, \(1\times 1\) convolution layer. Figure 3: The PyTorch implementation of the ResNet-50 backbone, as used in RetinaNet [17, 29, 30]. The input width is \(W\), height is \(H\), and the number of input channels is 3.