# Fast Coherent Video Style Transfer via Flow Errors Reduction
## 3 Methods
### Motivation
Secondly, naively copying and pasting pixels from a warped image \(w^{t}\) into corresponding flow traceable regions degenerates image quality and produces blurriness artefacts. For example, in Figure 4, the copied and pasted results will accumulate degeneration errors and produce image blurriness artefacts in eye regions (red rectangles) over a long period. To address the _texture discontinuity problem_, we propose a set of new mask techniques which include multi-scale mask fusion, incremental mask and multi-frame mask fusion. The reason behind the proposed mask design choices is to reduce the errors of untraceable (shown in black colour in mask images) and traceable flow regions (shown in white colour in mask images) in the mask obtained from the optical flow method (e.g., [63]). We discovered that the flow errors mostly occur at the boundaries of objects between adjacent frames as the optical flow method fails to accurately detect them. For example, the errors in untraceable flow regions are mainly caused by the low sensitivity of still objects in two adjacent frames, and the errors of traceable flow regions are mainly caused by the lack of  
Figure 4: Image blurriness artefacts. Images in the upper rows are original video frames. The blurriness artefacts become more obvious with time step. Figure 3: Texture discontinuity problem. Naively combining \(w^{t}\) and \(f_{s}^{t}\) via \(M^{t}\) into flow regions causes texture discontinuity problem. For example, in the green rectangle, the gray colours preserved from \(w^{t}\) lose the consistency of texture context (in red colours) which look like noise artefacts. long temporal information (e.g., considering multiple consecutive frames at the same time). To reduce the errors in untraceable flow regions, we propose a multi-scale mask fusion technique which will only preserve untraceable flow regions at all scale levels and remove the untraceable ones in few levels. In this way, the sensitivity of the optical flow method is leveraged to still objects. However, this multi-scale scheme could also make the untraceable flow regions become much thinner than before, especially at the boundaries of moving objects, which introduces ghosting artefacts. To solve this problem, an incremental mask is proposed to make the boundaries thick again. To reduce the errors of flow traceable regions, we propose fusing the mask results from multiple adjacent frames, which only preserves the traceable flow regions (white colour in mask images) in these consecutive frames. By this means, the proposed mask techniques are able to handle large motions in multiple consecutive frames as the flow errors are effectively mitigated. In the end, we obtain a mask with much fewer errors and compose the warped image \(w^{t}\) and per-frame stylized result \(f^{t}_{s}\) with it. The composition image will be the new initialization for the gradient-based optimization method to preserve consistency. To reduce the _image blurriness artefacts_, we adopt Perceptual Losses from [10] and Pixel Loss as the Sharpness Losses to update the pixel values iteratively. The aforementioned techniques and losses only preserve two-frames' coherency. To ensure coherency at entire video level, we introduce both multi-frame RGB-level and Feature-level Coherent Losses which contribute lower average stability errors than a single one of them independently [64]. In addition, we adopt the recurrent convolutional network strategy [65] so that our network takes the current stabilized warped frame \(w^{t}=\mathcal{W}(\widehat{x}_{out}^{t-1},F^{t})\) and the current per-frame stylized frame \(f^{t}_{s}\) as inputs, and then produces a stabilized output \(\widehat{x}_{out}^{t}\). During the optimization process, we enforce Coherent Losses and Sharpness Losses to ensure coherency and image quality between the generated image \(\widehat{x}^{t}\) and the previous output image \(\widehat{x}_{out}^{t-1}\). In this manner, our method propagates all the flow traceable points as far as possible during the entire video style transfer process.