# Deep Pixel-Level Matching via Attention for Video Co-Segmentation
## 3 Our Framework
### Our Attention Module
To ensure that the values belonging to the common feature are higher than the ones belonging to the dissimilar feature, we generate feature maps \(d_{a}^{\prime}\) by subtracting \(d_{a}\) from \(v_{a,2}\) and \(d_{b}^{\prime}\) by subtracting \(d_{b}\) from \(v_{b,2}\). \(d_{a}^{\prime}\) and \(d_{b}^{\prime}\) are two feature maps that satisfy the requirement of the convolutional computation. In order to obtain a spatial attention map, \(d_{a}^{\prime}\) and \(d_{b}^{\prime}\) are fed into two softmax layers, respectively, and the results from these two softmax layers are concatenated in a concat layer with a rectified linear unit (ReLU) layer [19] to generate the concatenated spatial attention map \(D\); \(v_{a,2}\) and \(v_{b,2}\) are also concatenated in a concat layer to obtain the concatenated feature map \(V\). First, \(D\) and \(V\) have an additional operation using Equation (2) and the result is then concatenated with \(V\) for further refining to generate the dense attention feature \(V^{\prime}\). This dense attention feature \(V^{\prime}\) is outputted as the result of feature fusion. \[V^{\prime}=Concat(V,V+D) \tag{2}\]  
**Up-sampling for refining features**. After receiving the dense attention feature \(V^{\prime}\), following the idea in [20], we set a batch normalization (BN) layer [21] with a ReLU layer to normalize the feature values in \(V^{\prime}\) for reducing internal covariate shift so that the loss function can converge easily during the training step. The result from the BN layer is fed into a convolutional layer and a convolutional block, and the high quality feature maps containing the common information outputted from the convolutional block are used to refine the feature maps from our baseline (ResNet-101). The feature maps in which the channel-size is \(m_{1}\) before processing by the last convolutional layer in our baseline are exported to concatenate with the feature maps in which the channel-size is \(m_{2}\) from the convolutional block in the concat layer. Through the channel-level concatenating operation, which is the same operation of concatenating \(V\) and the result of \(D\)+\(V\) in Figure 2, this concat layer provides the feature maps in which the channel-size is \(m_{1}+m_{2}\) to the convolutional block that follows this concat layer for refining features. Followed by the convolutional block for feature refining, an Interp layer up-samples the size of the feature maps to \(N/8\) (\(N\) is the original size of the input frame), and another convolutional block with dropout layer also receives the results from the Interp layer for generating the feature maps, which are then fed into the last convolutional layer of our framework to obtain segmentation results. At last, the Interp layer that follows the last convolutional layer recovers the segmentation results into their original sizes. Two Interp layers in this step up-sample the feature maps via bilinear interpolation. **Tile layer for inflation**. The tile layer is a type of layer in the Caffe framework [22] for inflating the size of feature maps. In our attention module in Figure 2, we can observe that the \(n^{2}\times 1\times 1\) feature map reshaped from \(f_{a}\) is inflated by two tile layers to form \(v_{a,1}\). Taking this process as an example, we show the details in Figure 3 and explain how the tile layer works. In the \(n^{2}\times 1\times 1\) feature map, each channel only contains one element with one value (we assume that this value is \(x\)). Thus, an array (\(n\times 1\)) in which the elements' values are \(x\) is generated when the tile layer inflates each channel's element along the height (the step of inflating in height-level in Figure 3); then a matrix (\(n\times n\)) whose elements' values are \(x\) is generated when the tile layer inflates each channel's array along the width (the step of inflating in width-level in Figure 3).