# Deep Pixel-Level Matching via Attention for Video Co-Segmentation
## 3 Our Framework
### Our Attention Module  
The pipeline of our attention module is shown in Figure 2. \(v_{a,1}\) and \(v_{b,1}\), respectively, denote the feature maps that are reshaped and inflated from \(f_{a}\) and \(f_{b}\); \(v_{a,2}\) and \(v_{b,2}\), respectively, denote the feature maps that are only inflated in channel-level from \(f_{a}\) and \(f_{b}\). \(D\) is the spatial attention map that is concatenated after the softmax layers, \(V\) is the feature maps that is generated by concatenating \(v_{a,2}\) and \(v_{b,2}\) and \(V^{\prime}\) is the dense attention feature which is obtained by fusing \(D\) and \(V\). Our module receives two feature maps (\(f_{a}\) and \(f_{b}\)) belonging to two input frames and outputs the segmentation results of these two frames. Feature maps \(f_{a}\) and \(f_{b}\) contain the foreground objects' features. **Pixel-level matching and features fusion. ** In order to compute the difference between the feature values in \(f_{a}\) with the feature values in \(f_{b}\), we apply the reshape layers and tile layers. These two layers transform the size of the feature maps from \(1\times n\times n\) to \(n^{2}\times n\times n\). Considering the case of \(f_{a}\) in Figure 2 as an example, its size is \(1\times n\times n\), and it is first reshaped into a new map with \(n^{2}\) channels by a reshape layer; in this new map, each channel only contains one feature value from \(f_{a}\). Then, the new map is inflated along the width and height into the new feature map \(v_{a,1}\) with size \(n^{2}\times n\times n\) by two tile layers. In \(v_{a,1}\), all elements' values in the matrix in the same channel are the same. Similarly, feature map \(v_{b,1}\) is also generated from \(f_{b}\) in the same way. In addition, \(f_{a}\) and \(f_{b}\) are inflated in channel-level to generate \(v_{a,2}\) and \(v_{b,2}\), respectively. Therefore, \(v_{a,2}\) and \(v_{b,2}\) have the size as that of \(v_{a,1}\) and \(v_{b,1}\). With \(v_{a,1}\), \(v_{b,1}\), \(v_{a,2}\), and \(v_{b,2}\), the following equation is utilized to compute the difference between two feature maps in pixel-level:  
\[ \begin{cases}d_{a}=|v_{a,2}-v_{b,1}|\\ d_{b}=|v_{b,2}-v_{a,1}|\end{cases}  \tag{1}\]  
Figure 2: The pipeline of our attention module for pixel-level similarity computation. The module receives a pair of feature maps \(\{f_{a},f_{b}\}\) containing the foregrounds’ features to generate feature maps \(v_{a,1}\), \(v_{a,2}\), \(v_{b,1}\), and \(v_{b,2}\) through the reshape and inflation operations. Next, a pixel-level matching step is leveraged to capture the common information from \(v_{a,1}\), \(v_{a,2}\), \(v_{b,1}\), and \(v_{b,2}\) and generate the feature map \(D\), which is used to fuse with the concatenated feature map \(V\) and calculate the high quality dense feature \(V^{\prime}\) in the features fusion step. Then, \(V^{\prime}\) is fed into the up-sampling step to refine the feature maps from the baseline. Finally, the segmentation result is outputted. \(d_{a}\) denotes the feature maps representing the pixel-level difference between \(v_{a,2}\) and \(v_{b,1}\); \(d_{b}\) stands for the feature maps representing the pixel-level difference between \(v_{b,2}\) and \(v_{a,1}\). In Figure 2, we can observe that the Eltwise layers are set after \(v_{a,1}\) and \(v_{b,1}\) to complete the subtraction operation for computing the difference. Followed by the Eltwise layers, the AbsVal layers are used to obtain the absolute values of the results from the Eltwise layers to keep the values of these results positive. \(d_{a}\) and \(d_{b}\) are outputted from the two AbsVal layers, respectively. In \(d_{a}\) and \(d_{b}\), the feature values representing the common feature between two objects are smaller than the ones representing the dissimilar feature since the result from the subtraction operation between two similar features is smaller than that between two dissimilar features. However, the convolutional kernel usually considers the high feature values as the features belonging to the foreground objects.