SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The disentanglement of motion and appearance, by the four appearance subnets, happens here in the phase 2 training. In this phase, we keep the 3D CNN encoder fixed, so the features which are input to the appearance subnets have entangled motion and appearance information. We train the network to minimize the VGG-19 feature loss for each frame, and hence only remember appearance information in the frames. When trained like that, the appearance subnets only gate the appearance information from the 3D encoder features as only that is required to minimize the loss, which is applied independently to each frame and, thus, has no dependence on motion. The evidence for such disentanglement is indirect, as when we do not do this, we get motion artifacts which we explain by the static nature of the style clip (constructed by repeating the style image), but when we do this, those motion artifacts disappear indicating that the features and the AdaIN3D based statistics transfer, only affected the appearance and not the motion.
3.3.4 Third phase: Entangle subnetwork training
In the third phase, we keep the decoder and the entangle subnetwork parameters trainable, while keeping the parameters of all appearance subnetworks as well as the encoder fixed. We initialize the decoder from scratch for this phase as the input features are different from the previously trained decoders. We use reconstruction loss for this phase. The network architecture is shown in Figure 2c. The aim of this phase of training is to learn to re-induce motion information into the appearance features.
3.3.5 Fourth and final phase: 3D decoder training
In the final phase, the full network is used, and only the decoder parameters are trained. We use the decoder trained in the third phase as the initialization for the decoder in this phase. All the other modules, i.e. encoder, all four appearance subnetworks, and entangle subnetwork, are kept fixed. The aim is to learn a final decoder which is able to do stylization. The combination of losses gives the full loss used for this phase, Eq. 7, 8, 9, 10,
$$
\mathcal {L} = \sum_ {i} \lambda_ {i} \mathcal {L} _ {i}, \text {w h e r e ,} i \in \{\text {c o n t e n t , s t y l e , t e m p o r a l , i n t r a} \}, \tag {11}
$$
i.e. this phase learns to do stylization while keeping intra-clip consistency, preserving overall content, and transferring style. This phase completes our training.
We now describe the experimental setup, and the qualitative and quantitative results comparing the proposed method with existing state-of-the-art approaches.
Dataset. A video dataset was not available in the existing style transfer literature for video stylization. Hence, we propose a new dataset curated from existing public benchmark datasets, to train for video stylization task. We will make the links and all details of the dataset public for use by the community.
Since, most of the challenge in video stylization stems from motion in the videos, we explicitly construct the dataset such that the clips have a high amount of motion, and are not just static repeated images. On analysing some videos downloaded we saw that some videos were simply repetition of a single static image, and were thus not appropriate.
We downloaded 45,574 videos from the first 100,000 URLs in the Sports1M dataset [14] (as many links were now defunct, and some downloads failed). We then generated 16 frame clips with a high amount of motion with the following steps. First, we randomly sampled a video from the 45K videos dataset, and used deep learning based shot detection by Souček et al. [23] to split the video into shots. We then randomly sample a 16 frame clip and computed the Farneback optical flow [5] of the frames of the sampled clip. We then selected the clips which have an average optical flow value above a threshold to be in the proposed dataset. Overall, we used 10,000 content clips generated from the above procedure as part of the dataset for training. In addition to the content clips, we use images from the WikiArt [20] as the style images, as has been done by the image stylization methods as well.
Input. We set clip size  $K$  to 16. While training we extract random  $128 \times 128$  size same patch from consecutive frames of a video clip in case of content clip. For, style clip we extract a random  $128 \times 128$  patch from the style image and repeat it 16 times which is equal to the length of the content clip. During testing, we pass the video in chunks of successive 16 frames clips.
Parameters. In the first phase of training, we train the encoder and decoder for 10k iterations. We used Adam optimizer with a learning rate of  $10^{-4}$  with a decay rate of  $5 \times 10^{-5}$  after every iteration for this phase of training. In the second phase of training, we train the decoder, appearance subnets for 5k iterations. We used Adam optimizer with a learning rate of  $10^{-4}$  with a decay rate of  $5 \times 10^{-5}$  after every iteration for this phase of training. In the third phase of training, we train the decoder and entangle subnet for 100k iterations. We used Adam optimizer with a learning rate of  $10^{-4}$  with a decay rate of  $5 \times 10^{-5}$  after every iteration for this phase of training. In the final phase training, we train the decoder for 40k iterations with content, style loss where values of  $\lambda_{c}, \lambda_{s}$  is 1, 2 respectively. After this, we train the decoder with content, style, temporal losses for 160k iterations where values of  $\lambda_{content}, \lambda_{style}, \lambda_{temporal}$  being 1, 2, 10 respectively. And finally, we fine-tune the decoder for 40k iterations with content, style, temporal, intra-clip loss where values of  $\lambda_{content}, \lambda_{style}, \lambda_{temporal}, \lambda_{intra}$  being 1, 2, 10, 10 respectively.
We discuss the takeaways from the ablation experiments we performed. It is difficult to show the video related effects with images. Kindly, see the video results in the supplementary materials corresponding to these discussions.
Disentangling appearance and motion using appearance subnetworks. We applied AdaIN3D operation directly to the encoded tensor of the content and style clips by the C3D network without using any appearance (disentanglement) subnetworks. However, we observed jerky motion in the generated videos, which followed a repeated pattern across the clips. Upon using appearance subnetworks, such motion artifacts were not observed anymore. The supporting videos are in supplementary material folder Videos_with_naive_extension_of_2D_stylization.
Temporal loss. When trained without temporal loss, the stylized video has two kinds of artifacts. First, color consistency across subsequent frames suffers as we transfer style with appearance subnetworks which remove motion component. Second, there is some loss of motion as well. Both of these artifacts are not observed, or are very mild once we use temporal loss. Supporting videos are in supplementary material folder Videos Without-temporal.
Intra-clip loss. We observe even after applying the temporal loss, there are some color inconsistencies leading to flashing like artifacts. However, when we use intra-clip loss such inconsistencies are addressed to a large extent. Supporting videos are in supplementary material folder Videos Without Intra.
Figure 3: First column is the style image and subsequent columns are a pair of frames generated using different approaches, in the following order: AdaIN[11], MCCNet[3], AdaAttn[18] and Ours.
Figure 4: Effect on content of different methods. First column is the style image, and the rows in the remaining columns contain frames and zoom-ins of original content clip, AdaAttN [18] stylized clip and our method. Notice how AdaAttN (middle) modifies the content significantly, while the proposed method (right) tends to keep the content relatively preserved.
4.3 Qualitative results
Figure 3 shows some qualitative results of the proposed method compared to existing methods. We observe that the proposed method focuses more on low level texture of the style image while AdaAttn [18] tends to flatten out the textured regions consistently. Figure 4 shows that AdaAttn also tends to modify the content, while the proposed method gives better textured outputs closer to the original content. Please see supplementary material for video results. Overall the results produced by the proposed method are aesthetically pleasing, and can be used by video creatives as one of the filters available.
4.3.1 User Study
We considered AdaIN [11], MAST [4] and two state-of-the-art methods, MCCNet [3] and AdaAttN [18], for comparison with our method. We randomly picked 15 content clips and 15 style images and considered random 20 stylized videos out of 225 possible combinations. We generated these stylized videos using all the compared methods. We then presented the stylized video of other methods along with that of our method side by side, but in random order, to the participants. We collected 900 votes from
Method	Content	Style	Overall
AdaIN	2.3	16.3	4.3
MAST	1.0	15.7	11.0
MCCNet	12.3	31.3	27.7
AdaAttN	41.0	20.0	28.3
Ours	43.3	16.7	28.7
Table 1: User study preferences(%) Bold, red, blue represent rank 1, 2 and 3 respectively.
Method	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	Mean
AdaIN	0.95	3.46	0.56	4.88	0.64	1.06	0.66	3.4	3.68	1.72	1.77	6.22	0.32	2.85	0.13	0.34	1.31	5.64	0.85	0.45	2.04
MAST	1.89	3.59	1.6	4.77	1.17	1.53	0.79	4.14	3.6	2.17	1.56	6.28	1.06	3.45	0.44	0.74	0.97	5.08	1.11	0.89	2.34
MCCNet	0.59	2.21	0.3	3.06	0.43	0.85	0.44	2.29	2.45	1.07	1.51	5.8	0.21	3.16	0.09	0.15	0.91	4.44	0.57	0.21	1.53
SANET	0.88	3.01	0.95	4.21	0.78	1.45	0.62	3.94	3.16	1.99	1.6	4.42	0.5	2.96	0.23	0.63	1.11	4.73	0.89	0.55	1.93
AdaAttN	0.39	1.96	0.39	2.94	0.39	0.64	0.29	2.25	1.77	0.98	0.83	5.51	0.19	2.91	0.07	0.08	0.34	2.69	0.36	0.24	1.26
Ours	0.65	3.98	1.36	4.57	0.32	0.65	0.85	2.79	2.92	1.64	0.76	3.58	0.41	2.01	0.15	0.2	0.69	4.29	0.55	0.35	1.63
Ours+temporal	0.37	1.89	0.31	2.45	0.27	0.43	0.44	2.17	1.97	0.94	0.59	1.96	0.22	1.58	0.08	0.15	0.42	3.25	0.41	0.21	1.00
Ours+temporal+intra	0.41	2.18	0.31	2.59	0.33	0.5	0.44	1.7	2.6	1.06	0.66	1.89	0.21	1.92	0.07	0.13	0.53	3.93	0.51	0.28	1.11
Table 2: Optical flow evaluation 20 test pairs of content videos and test images. **Bold**, red and blue color represents the rank 1, 2 and 3 minimum error respectively.
various participants for content preservation, degree of stylization and overall preference. The results in Table 1 show that our method leads for content preservation closely followed by AdaAttN [18]. In the case of the degree of stylization, MCCNet [3] leads us by a margin, but we argue that there is always a trade off between style transfer and content preservation, and so we can observe that MCCNet [3] performs poorly in the case of content preservation. Considering the overall preference in which participants have considered both content and style, our method performs better than all the other methods, proving our method's effectiveness as a successful alternate style transfer method.
We computed Farneback optical flow [5] for the content video and the stylized video. We report the mean of the absolute difference between the flows of content video to that of corresponding flow in stylized video. This is an approximate measure to ensure that the stylization method is preserving the motion in the original clip. All the methods are expected to distort the motion in some ways while performing stylization, and the degree of such distortion will depend on the input video. We observe that the proposed method achieves competitive flow errors.
4.5 Compute time
Since the proposed method is based on 3D CNNs, it is computationally heavier than the 2D CNN methods. To get a real world computation time estimate, we processed a video with 144 frames with resolutions of  $640 \times 360$  pixels. AdaAttN took 14 seconds and 8GB of GPU memory, while our proposed method took 60 seconds and 16GB of GPU memory on a machine with Intel Core i9-10900X processor and Nvidia RTX A4000 GPU.
We proposed a novel architecture and training method for the task of video stylization given a reference style image. The method uses 3D CNNs exclusively, and is the first to report stylization with them. It achieves stylization by disentangling appearance information from the 3D CNN based encoder's features, stylizing them, and then re-combining them with the motion information before finally decoding to the desired stylized output video. The training is done in phases with different phases ensuring different capabilities for the subnetworks in the architecture. We also proposed a video dataset containing videos with high degree of motion for training video stylization networks. We showed qualitatively and quantitatively that the proposed method produces results which are competitive to the existing methods and produce different and aesthetically pleasing results. We hope the proposed method will result in another filter in video creative's toolbox.
[1] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[2] Dongdong Chen, Jing Liao, Lu Yuan, Nenghai Yu, and Gang Hua. Coherent online video style transfer. In IEEE International Conference on Computer Vision, 2017.
[3] Yingying Deng, Fan Tang, Weiming Dong, Haibin Huang, Chongyang Ma, and Changsheng Xu. Arbitrary video style transfer via multi-channel correlation. In AAAI Conference on Artificial Intelligence, 2021.
[4] Yingying Deng, Fan Tang, Weiming Dong, Wen Sun, Feiyue Huang, and Changsheng Xu. Arbitrary style transfer via multi-adaptation network. In 28th ACM International Conference on Multimedia, 2020.
[27] Wenjing Wang, Shuai Yang, Jizheng Xu, and Jiaying Liu. Consistent video style transfer via relaxation and regularization. IEEE Transactions on Image Processing, 29, 2020.
[28] Yexun Zhang, Ya Zhang, and Wenbin Cai. Separating style and content for generalized style transfer. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.