# Complementary Segmentation of Primary Video Objects with Reversible Flows
## 4 Efficient Temporal Propagation with Neighborhood Reversible Flow
### Temporal Propagation of Spacial Features  
The flow \(\{\mathbf{F}_{uv}\}\) depicts how superpixels in various frames are temporally correlated, which can be used to further propagate the spatial foregroundness and backgroundness. Typically, such temporal refinement can obtain impressive performance by solving a complex optimization problem with constraints like spatial compactness and temporal consistency. However, the time cost will also grow surprisingly high [20]. Considering the requirement of efficiency in many real-world applications, we propose to minimize an objective function that has an analytic solution. For a superpixel \(\mathcal{O}_{ui}\), its foregroundness \(x_{ui}\) and backgroundness \(y_{ui}\) can be initialized as  
\[x_{vj}=\frac{\sum_{p\in\mathcal{O}_{ui}}\mathbf{X}_{u}(p)}{|\mathcal{O}_{ui}|},\;y_{ui}=\frac{\sum_{p\in\mathcal{O}_{ui}}\mathbf{Y}_{u}(p)}{|\mathcal{O}_{ ui}|}, \tag{8}\]  
where \(p\) is a pixel with foregroundness \(\mathbf{X}_{u}(p)\) and backgroundness \(\mathbf{Y}_{u}(p)\). \(|\mathcal{O}_{ui}|\) is the area of \(\mathcal{O}_{ui}\). For the sake of simplification, we represent the foregroundness and backgroundness scores of all superpixels in the \(u\)th frame with column vectors \(\mathbf{x}_{u}\) and \(\mathbf{y}_{u}\), respectively. As a result, we can propagate such scores from \(\mathcal{I}_{v}\) to \(\mathcal{I}_{u}\) according to \(\mathbf{F}_{uv}\):  
\[\mathbf{x}_{u|v}=\mathbf{F}_{uv}\mathbf{x}_{v},\;\mathbf{y}_{u|v}=\mathbf{F}_{ uv}\mathbf{y}_{v},\;\;\forall v\in\mathbb{T}_{u}. \tag{9}\]  
After the propagation, the foregroundness vector \(\mathbf{\hat{x}}_{u}\) and backgroundness vector \(\mathbf{\hat{y}}_{u}\) can be refined by solving  
\[\mathbf{\hat{x}}_{u} =\arg\min_{\mathbf{x}}\;\|\mathbf{x}-\mathbf{x}_{u}\|_{2}^{2}+ \lambda_{c}\sum_{v\in\mathbb{T}_{u}}\|\mathbf{x}-\mathbf{x}_{u|v}\|_{2}^{2}, \tag{10}\] \[\mathbf{\hat{y}}_{u} =\arg\min_{\mathbf{y}}\;\|\mathbf{y}-\mathbf{y}_{u}\|_{2}^{2}+ \lambda_{c}\sum_{v\in\mathbb{T}_{u}}\|\mathbf{y}-\mathbf{y}_{u|v}\|_{2}^{2},\]  
where \(\lambda_{c}\) is a positive constant whose value is empirically set to 0.5. Note that we adopt only the \(\ell 2\) norm in (10) so as to efficiently compute an analytic solution  
\[\mathbf{\hat{x}}_{u} =\frac{1}{1+\lambda_{c}\cdot|\mathbb{T}_{u}|}\Bigg{(}\mathbf{x}_ {u}+\lambda_{c}\sum_{v\in\mathbb{T}_{u}}\mathbf{F}_{uv}\mathbf{x}_{v}\Bigg{)}, \tag{11}\] \[\mathbf{\hat{y}}_{u} =\frac{1}{1+\lambda_{c}\cdot|\mathbb{T}_{u}|}\Bigg{(}\mathbf{y}_ {u}+\lambda_{c}\sum_{v\in\mathbb{T}_{u}}\mathbf{F}_{uv}\mathbf{y}_{v}\Bigg{)}.\]By observing (9) and (11), we find that the propagation process is actually calculating the average foregroundness and backgroundness scores within a local temporal slice under the guidance of neighborhood reversible flow. After the temporal propagation, we turn superpixel-based scores into pixel-based ones according to  
\[\mathbf{M}_{u}(p)=\sum_{i=1}^{N_{u}}\delta(p\in\mathcal{O}_{ui})\cdot\hat{x}_{ui }\cdot(1-\hat{y}_{ui}), \tag{12}\]  
\[\min_{\mathbf{x}}\ \|\mathbf{x}-\mathbf{x}_{u}\|_{2}^{2}+\lambda_{c}\sum_{p\in \mathbb{T}_{u}}\|\mathbf{x}-\mathbf{F}_{uv}\mathbf{x}_{v}\|_{2}^{2} \tag{13}\]  
where \(\mathbf{M}_{u}\) is the importance map of \(\mathcal{I}_{u}\) that depict the presence of primary objects. \(\delta(p\in\mathcal{O}_{ui})\) is an indicator function, which equals 1 if \(p\in\mathcal{O}_{ui}\) and 0 otherwise. Finally, we calculate an adaptive threshold which equals the 20% of the maximal pixel importance to binarize each frame, and a morphological closing operation is then performed to fill in the black area in the segmented objects.