# Complementary Segmentation of Primary Video Objects with Reversible Flows
## 3 Initialization with Complementary CNNs
### Deep Optimization with Complementary CNNs  
Given the empirical loss (2) and the complementary loss (3), we can derive two models, \(\phi_{F}(\cdot)\) and \(\phi_{B}(\cdot)\), for per-frame initialization of the foreground and background maps by solving the optimization problem of objective functions (1). Toward this end, we need to first determine the form of the models and the algorithm for optimizing their parameters. Considering the impressive capability of convolutional neural networks (CNN), we propose to solve the optimization problem in a deep learning paradigm. The architecture of the proposed CNN can be found in Figure 3, which starts from a shared trunk and ends up with two separate branches, i.e., a foreground branch and a background branch. The main configurations and details are shown in Table 1. For simplicity, only the foreground branch is illustrated in Table 1, as the background one adopts the same architecture. Note that this network simultaneously handles two complementary tasks as well as their relationships and is denoted as a complementary segmentation network (CSNet). The parameters of the shared trunk are initialized from the ResNet50 networks [64], which are used to extract low- to high-level features that are shared by foreground objects and background distractors. We remove the pooling layer and the fully connected layer after the RELU layer of res5c and introduce two pooling blocks (see Figure 3) to provide features from additional levels and reduce parameters. In order to integrate both the local and global contexts, we sum up different levels of feature outputs by layer Res3, Res4 and Res5 and two pooling blocks by appropriate up/down-sampling operations. After that, a residual block with a 3 \(\times\) 3 CONV layer and a 1 \(\times\) 1 CONV layer is used to post-process the integrated features as well as increase their nonlinearity. Finally, the shared trunk takes a 320 \(\times\) 320 image as the input and outputs a 40 \(\times\) 40 feature map with 512 channels. \begin{table}
\begin{tabular}{l l l} \hline
**Type Name** & **Patch Size/Stride/Pad/Dilation/Group** & **Output Size** \\ \hline Conv1\_pb1 & \(3\times 3/1/1/1/32\) & \(10\times 10\times 256\) \\ Conv2\_pb1 & \(1\times 1/1/0/1/1\) & \(10\times 10\times 2048\) \\ Pool1 & avg pool, \(2\times 2\), stride 2 & \(5\times 5\times 2048\) \\ Conv3\_pb1 & \(3\times 3/1/1/1/32\) & \(5\times 5\times 256\) \\ Conv4\_pb1 & \(1\times 1/1/0/1/1\) & \(5\times 5\times 2048\) \\ Conv1\_pb2 & \(3\times 3/1/1/1/32\) & \(5\times 5\times 256\) \\ Conv2\_pb2 & \(1\times 1/1/0/1/1\) & \(5\times 5\times 2048\) \\ Pool2 & avg pool, \(2\times 2\), stride 2 & \(3\times 3\times 2048\) \\ Conv3\_pb2 & \(3\times 3/1/1/1/32\) & \(3\times 3\times 256\) \\ Conv4\_pb2 & \(1\times 1/1/0/1/1\) & \(3\times 3\times 2048\) \\ Interp1 & bilinear upsampling & \(40\times 40\times 1024\) \\ Conv1 & \(1\times 1/1/0/1/1\) & \(40\times 40\times 512\) \\ Interp2 & bilinear upsampling & \(40\times 40\times 2048\) \\ Conv2 & \(1\times 1/1/0/1/1\) & \(40\times 40\times 512\) \\ Interp3 & bilinear upsampling & \(40\times 40\times 2048\) \\ \hline \end{tabular}
\end{table} 
Table 1: Main configurations for CSNet. Note that x and y are integers in the range [1; 5]. After the shared trunk, the features are fed into two separate branches that address two complementary tasks, i.e., foreground and background estimation. Note that the two branches share the architecture with the input, but produce complementary outputs. In each branch, the shared features pass through a sequential of convolution blocks. These blocks all consist of \(1\times 1\) and \(3\times 3\) CONVs, but with different dilations. As such, we concatenate the output of each block to constitute feature maps at \(40\times 40\) resolution with 1280 channels.