# Fast Coherent Video Style Transfer via Flow Errors Reduction
## 3 Methods
### Motivation  
For artistic video stylization, the unexpected flickering problem hinders satisfactory results when still image style transfer methods (e.g., [10, 13]) are applied to process frames independently. As shown in Figure 1, the adjacent per-frame stylized results exhibit some colour and texture incoherence (e.g., middle columns in zoom-ins). To preserve the coherency for video style transfer, this work starts with a straightforward idea. To simplify, we start with two consecutive original video frames \(f_{v}^{t-1}\) and \(f_{v}^{t}\), and their corresponding per-frame stylized results \(f_{s}^{t-1}\) and \(f_{s}^{t}\), and their corresponding optical flow \(F^{t}\) from \(f_{v}^{t}\) to \(f_{v}^{t-1}\); then, we produce a warped image \(w^{t}=\mathcal{W}(f_{s}^{t-1},F^{t})\) by warping \(f_{s}^{t-1}\) with \(F^{t}\), and a mask \(M^{t}\) containing per-pixel flow traceable (e.g., values tend to be 1) and untraceable regions (e.g., values tend to be 0). (A warp operation with optical flow means that pixels in the image will be moved to the locations in the next frame by multiplying pixel-level motion vectors. The flow contains these pixel-level motion vectors.) To obtain a stable consecutive stylized result, a straightforward idea which came up was to compose the warped image \(w^{t}\) and \(f_{s}^{t}\) in the traceable and untraceable flow regions, respectively. In this way, the we are capable of preserving the coherency of the composition result as much as possible. However, there is one prerequisite that the pasted contents at untraceable regions must have the exact original content details especially in occlusion regions. Otherwise, a heavy image compensation/optimization process is needed during video stabilization. Fortunately, artistic style transfer methods for still images (e.g., [10]) satisfy this prerequisite as they may change the textures or colours in the occluded regions but they indeed do not damage the consistency of the original content's details. For example, in Figure 2, compared to the original frame \(f_{v}^{t}\), the content structures (in red and orange rectangles) of the composition result preserve the consistency better than the warped image \(w^{t}\), as the warped image \(w^{t}\) has unexpected content structures in the red box and duplicated ones in the orange box. Hence, this straightforward idea is worth carefully investigating to obtain stabilized video outputs. Figure 1: Flickering artefacts in video style transfer. The first row shows two original consecutive video frames (**left**) and the style image (**right**). The second row shows the temporal incoherence artefact by [PERSON] et al. [10]. The green and purple rectangles indicate the different appearances (texture patterns and colours) between these two stylized outputs, which exhibit flickering artefacts. The third row shows the stable results produced by our method, where the outputs preserve consistent texture appearances. Figure 2: Prerequisite. We test the straightforward idea by recomposing pasted stylized content in untraceable flow regions (please see zoom-in rectangles), which shows that the composition result preserves the consistency of content structures better than the warped image \(w^{t}\), as the warped image \(w^{t}\) has unexpected content structures in the red box and duplicated ones in the orange box. \(\otimes\) denotes the warp operation, which warps \(f_{s}^{t-1}\) into \(w^{t}\) with \(F^{t}\), and \(\oplus\) denotes element-wise addition in this paper. Please zoom in to see the details. Based on this observation, we investigate this idea, and find out that naively applying this straightforward composition may produce two new issues. Firstly, it clearly may produce discontinuous transferred textures in the composition result. For example, in Figure 3, the errors caused by the optical flow method lead to unexpected flow errors in the mask, which directly cause discontinuous textures or colours.