# Fast Coherent Video Style Transfer via Flow Errors Reduction
## 3 Methods
### A New Initialization for Gradient-Based Optimization Network  
Based on the observation mentioned in Section 3.1, the most important part of the initial generation is to create a reliable flow mask to reduce flow errors. To this end, we propose a mask generation method to deal with it. At the beginning, we start with the following items: original adjacent video frames \(f_{v}^{t-2}\), \(f_{v}^{t-1}\) and \(f_{v}^{t}\), per-frame stylized results \(f_{s}^{t-2}\), \(f_{s}^{t-1}\) and \(f_{s}^{t}\). Then we rescale the original video frames into multiple resolutions. For the MPI Sintel dataset [66], we consider two scales \(r\in\mathcal{R}\) where \(\mathcal{R}=\{\sigma,\frac{1}{2}\sigma\}\) denotes the set of resolutions and \(\sigma\) denotes the original video resolution. And we utilize optical flow methods (e.g., [63]) to compute the corresponding forward flow \(F_{r}^{t}\) and backward flow \(F_{s}^{t}\). At this time, we obtain a warped image \(w^{t}=\mathcal{W}(f_{s}^{t-1},F_{s}^{t})\) at the original video resolution by warping the previous per-frame stylized result \(f_{s}^{t-1}\) and flow \(F_{s}^{t}\). Next, we calculate multi-scale per-pixel flow masks which are given by a forward-backward  
Figure 7: Network architecture overview. During optimization, the network takes \(\widehat{x}_{init}^{t}\) obtained from initial generation, current per-frame stylized result \(f_{s}^{t}\), mask \(M^{t}\) and a warped image \(w^{t}\) as inputs, and gradually optimizes initial image \(\widehat{x}_{init}^{t}\) into \(\widehat{x}_{out}^{t}\) based on gradients computed from losses. consistency check. The values at points of flow masks tend to be 1 in traceable flow regions where both forward and backward direction estimation agrees. On the contrary, the values at positions of flow masks tend to be 0 at disagreeing points. Then, we rescale the flow masks into the original video resolution and compose them into one mask in a value maximum manner which remains the maximum values from those flow masks at each pixel location. This step is able to fix unexpected untraceable flow errors. For example, the untraceable flow errors (black regions) in the red rectangle are fixed by multi-scale fusion in Figure 8. In addition, we find out that copying the current per-frame stylized results into the corresponding untraceable flow regions may cause worse ghosting artefacts as the flow untraceable regions become much thinner than before multi-scale fusion. Hence, we propose an incremental mask \(M_{\theta}^{t=>t-1}\) which generates an incremental circle along with untraceable flow regions. Specifically, the points in the circle closer to untraceable regions has lower values. In this work, the circle width is set by default to 3 pixels and the gradient is 0.2. To further reduce traceable flow errors where large motions occur, we consider combining multiple incremental masks \(M_{\theta}^{t=>t-i}\), \(i\in\mathcal{T}\) where \(\mathcal{T}\) denotes the set of indices of adjacent video frames. We combine \(M_{\theta}^{t=>t-i}\) (\(i\in\mathcal{T}\)) in a value minimum manner to correct errors. In general, the fusion in a maximum manner reduces untraceable flow errors (black regions, see Figure 8) and the fusion in a minimum manner reduces traceable flow errors (white regions). in Section 5.1. Eventually, we obtain a flow mask \(M^{t}=\min(M_{\theta}^{t=>t-i}),(i\in\mathcal{T})\) which is used for initialization generation by composing the warped image \(w^{t}\) and per-frame stylized result \(f_{s}^{t}\). The generation of the initial image is shown in Figure 9. The proposed initial image is defined as:  
\[\widehat{x}_{init}^{t}=M^{t}\otimes w^{t}+(1-M^{t})\otimes f_{s}^{t} \tag{1}\]  
where \(\otimes\) denotes element-wise multiplication. \(M^{t}\) is a single channel mask. \(\widehat{x}_{init}^{1}\) is the first per-frame stylized result \(f_{s}^{1}\) when \(t=1\). Figure 8: The process of multi-scale mask fusion and incremental mask. The unexpected flow untraceable errors (see red rectangles) are fixed in this step.