ViSt3D: Video Stylization with 3D CNN
Authors: Ayush Pande
Gaurav Sharma
Venue: NeurIPS 2023 poster (2023)
Keywords: Video style transfer

Abstract: Visual stylization has been a very popular research area in recent times. While image stylization has seen a rapid advancement in the recent past, video stylization, while being more challenging, is relatively less explored. The immediate method of stylizing videos by stylizing each frame independently has been tried with some success. To the best of our knowledge, we present the first approach to video stylization using 3D CNN directly, building upon insights from 2D image stylization. Stylizing video is highly challenging, as the appearance and video motion, which includes both camera and subject motions, are inherently entangled in the representations learnt by a 3D CNN. Hence, a naive extension of 2D CNN stylization methods to 3D CNN does not work. To perform stylization with 3D CNN, we propose to explicitly disentangle motion and appearance, stylize the appearance part, and then add back the motion component and decode the final stylized video. In addition, we propose a dataset, curated from existing datasets, to train video stylization networks. We also provide an independently collected test set to study the generalization of video stylization methods. We provide results on this test dataset comparing the proposed method with 2D stylization methods applied frame by frame. We show successful stylization with 3D CNN for the first time, and obtain better stylization in terms of texture cf.\ the existing 2D methods.

ViSt3D: Video Stylization with 3D CNN
Ayush Pande
Gauray Sharma TensorTour IIT Kanpur
IIT Kanpur ayushp@cse. .ittiseeeeiitte........e..ec weitkiit..... in
gaurav@tensortour.co com
Abstract
Visual stylization has been very popular research area in recent times. While image stylization has seen rapid advancement in the recent past, video stylization, while being more challenging, is relatively less explored. The immediate method of stylizing videos by stylizing each frame independently has been tried with some success. To the best of our knowledge, video stylization using 3D CNN directly, building upon insights from 2D image stylization. Stylizing video highly challenging, the appearance and video motion, which includes both camera and subject motions, are inherently entangled in the representations learnt by 3D CNN. Hence, naive extension of 2D CNN stylization methods to 3D CNN does not work. To perform stylization with 3D CNN, we propose explicitly disentangle motion and appearance, stylize the appearance part, and then add back the motion component and decode the final stylized video. In addition, we propose a dataset, curated from existing datasets, train video stylization networks. We also provide an independently collected test set to study the generalization of video stylization methods. We provide results on this test dataset comparing the proposed method with 2D stylization methods applied frame by frame. We show successful stylization with 3D CNN for the first time, and obtain better stylization in terms of texture ef. the existing 2D methods. Project page: htts://ayush202..go http:///w.giiiiioooooo///s//////
present the first approach
Introduction
Image stylization is the task of transforming input content image into stylized image with the style taken from another input style image. has been very popular research topic as well as a widely used image editing tool in the recent past. Pictures taken by many users have been transformed into the styles of Picasso and Monet. While image stylization with 2D CNNs has been hugely popular [4, 9, 11, 16]. stylization of videos has been relatively less explored. Some recent works image stylization also show results on video stylization by doing frame by frame image stylization, e.g. [4, 11, 18), and have achieved some success. 3D CNN architectures designed for video processing, to the best of our knowledge, have not yet been made amenable for video stylization at all. Since the 3D convolutions used in the 3D CNNs, inherently combine the spatial and temporal information, the two factors of variations are deeply entangled in the resulting representations. This makes video stylization with 3D CNNs extremely challenging the style is usually for the appearance, i.e. indicated by a single image, and modifying the video representation to incorporate style in the content, but not in motion a very challenging task. Naive extension of 2D stylization, e.g. directly transferring feature statistics from style to content features using AdaIN [11] operation with 3D CNNs intermediate features. leads to unusable video output, with intermittent slow motion and jerky transitions due to the style input being static (see supplementary material for representative results).
We address the challenging task of video stylization using 3D CNNs directly. We propose architecture and training method which takes a content video clip and a style image, and produces an aesthetically pleasing stylized video clip. We do so by disentangling the appearance and motion
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

features from the encoder network and applying AdaIN operation extended for 3D CNN features, coupled with feature map reconstruction loss and style statisties losses. We show how to directly extend AdaIN operation to work with 3D CNN based encoders and decoders, which we call AdaIN3D, and train the network successfully despite the challenges with highly entangled appearance and motion information. All the sub-networks used in the proposed architecture are based on 3D CNN. Since a dataset was not available for the task of studying video stylization, we also propose a large-scale dataset with 10, 000 content clips curated from public benchmark Sports IM [14], paired with the train set of style images from WikiArt dataset [20] which are used to train 2D stylization methods. We show qualitatively that the proposed method produces better results than frame by frame stylization using state-of-the-ee 2D image stylization methods. We also give quantitative results based on optical flow errors, comparing the results of the proposed method to that of the state-of-the thttherrtt 2D stylization methods. Our contributions can be summarized as follows.
We propose novel method for video stylization using 3D CNN, and we are the first to report that stylization with 3D CNN is possible.
We propose novel approach to disentangle the appearance and motion features, which are deeply entangled in representations learnt by 31) CNN networks like C3D [25].
We curated the video dataset from the public Sports dataset [14] by filtering videos with motion automatically and use it for training our video stylization network.
We use existing loss functions and design new ones to encourage learning of artifact free video stylization.
We provide exhaustive comparisons with existing methods, detailing the differences.
Overall, since visual stylization is effectively a subjective creative filter, we hope add more tool to the video creatives toolbox.
2 Related works
Image stylization. Gatys et al. [8] proposed an approach to generate textures using CNN and optimization based technique. They used the loss function over Gram matrices on feature maps for maintaining feature map statistics and hence generating required textures. In their next work, Gatys al. [9] used similar optimization based approach for stylizing content image with the texture of style image. Such optimization based approach is quite slow, and the training has to be performed individually for every new pair of images. The next generation of image stylization approaches then turned to feedforward CNNs, e.g. [13, 26J. for speed and avoiding retraining on every test style image. However, they still could not be used with arbitrary style images on the fly, i.e. one network needs to be trained for each style image. Finally, the current generation of image stylization methods emerged that were designed for universal image stylization [11, 17, 28], i.e. the network is trained once, and then is used for transferring style from an arbitrary style image, fed along with the content image as input at test time. AdaIN [11] was a very successful step towards universal style transfer because of the simplicity of the network training and parameter free transfer of content into the style space using the AdalN operation. Among other works, Li al. [17] used whitening and color transforms for style transfer. However, such transformations produce unsatisfactory results, potentially mooing/modiffyin content in the output image. SANet [19] tried to introduce more style into the network but traded off the content from the original content image.
Video stylization. All the image stylization methods can be applied to videos by processing the videos frame by frame. Video stylization when done with image stylization methods such as AdaIN [11], WCT [17] etc., leads to stylized videos which suffer from various temporal and spatial inconsistencies. Chen et al. [2] and Gao et al. [7] hence proposed temporal consistency constraints to address these. Methods for video stylization include MCCNet [3] and AdaAttn [18]. MCCNet [3] mainly focused on video stylization, while AdaAttn [18] addressed image stylization using attention and extended the method for videos as well. Both methods lead to stylized video with some degree of
style transfer, but increasing the style transfer introduces artifacts quickly. We compare with both of these in our experiments. Wang et al. [27] added optical flow to improve video stylization with 2D CNNs, and here we also use optical flow-based loss but with 3D CNN directly. We observed that there is noticeable flickering in some examples generated by Wang et al. [27]. Also, it smooths out
2

---

Appearance subnet
Content loss Style
1_1
AdaIN3D
Temporal
Appearance subnet
Skip connections
Intra-clip loss
2_1
AdaIN3D
Content clip
C3D encoder
3D decoder
Appearance subreet
3_1
AdaIN3D
Entangle
Subnet
Stylized clip
Appearance
Style clip
AdaIN3D
subnet
Figure 1: Full architecture for the proposed network for video stylization. The network consists of 3D CNN based encoder and decoder networks, along with two other critical parts: (i) appearance subnets and (ii) Entangle layer. The appearance subnets are trained to disentangle appearance and motion components from the full 3D CNN feature and extract out the appearance part only. The entangle layer is trained to do the opposite, i.e. given the (stylized) appearance features as well as the full 3D CNN features, it re-introduces the motion component in the stylized features to enable them to be decoded into a stylized video by the decoder. The contribution of this paper is to report successful video stylization with 3D CNN directly, cf. 2D CNNs of existing works, using the above network, and a multi-phase training method to train the individual sub-networks proposed.
the low-level textural details, while the proposed method does not have heavy flickering, and also maintains the low-level textural details.
Stylization with 2D and 3D CNNs. All the above methods mainly work with 2D CNNs, e.g. [10, 15, 21, 24], for encoding the content and style images, and propose methods for transferring style by matching statisties of the intermediate feature maps. Videos are inherently 3D, and 3D CNNs, e.g. [1, 25], have been proposed for learning with videos. There has been no work on video stylization with 3D CNN, and in this paper we report the first successful network and training method to do stylization with 3D CNN. Other than breaking new ground, we also show that the proposed method produces richer texture the stylized video, and since stylization is contextual and subjective, is at the very least, another useful tool for video creatives.
Approach
We now present the proposed network architecture and training method for doing video stylization with 3D CNNs. Since we need to perform the challenging task of disentangling the motion and appearance components, the method is trained multiple phases. In the following, we first describe the full architecture used at inference time in Section 3.1, which takes the content video clip as well as style clip (the style image repeated K times for clips size K), giving details of each subnetwork. Then, we explain the losses and the multi-phase training method in Section 3.3.
3.1 Overall network
The task addressed here is to do video stylization with a reference style image. The network takes as input the content clip (which needs to be stylized) and the reference style image, and outputs the corresponding stylized video.
Figure shows the full network used for stylizing input video clip, with an input style image. Along with the input video clip, the network takes style clip as input. The style clip is constructed by repeating the style image K number of times, where K is the clip size for the input to the 3D CNN encoder. Both the content and style clips are first encoded with 3D CNN encoder. The outputs of the encoder, i.e. final layer feature maps and intermediate layer feature maps, are then fed to appearance subnetworks, which are responsible for extracting the appearance features only, disentangling them from the motion features. We will detail the method for achieving such disentanglement in the following sections. Once the outputs from appearance subnetworks

---

available for both content and style clips, we perform the AdaIN3D (extension of AdaIN[1 operation to 3D CNN features, described in Section 3.2.3 below) operation for transferring feature statistics from the style clip features to the content clip features. Finally the intermediate layer features are directly fed to the decoder, which also a 3D CNN. The final layer features after the AdaIN31) operation are then aggregated with the final layer feature maps from the 3D CNN encoder. This step for re-inducing the motion features into the appearance features which were disentangled by the appearance subnetworks. These aggregated features are then fed into the Entangle Subnet, the output of this subnet is passed as input to the decoder which produces the desired stylized output clip. Effectively, these steps can be seen as, (i) disentangle the appearance features from the full features which contains appearance and motion both, (ii) stylize the appearance features only, and finally (iii) re-induce the motion component into the stylized appearance features and decode them to obtain the stylized clip.
3.2 Details of network components
Now, we will explain each component of the full architecture shown in the Figure 1. Please refer to the supplementary material for the complete code-level details of the appearance, Entangle, and the 3D Decoder subnetworks.
3.2.1 Normalized encoder
Our encoder is a state-of-the-art 3D CNN, i.e. C3D [25], which was originally proposed for the task of human action classification. It takes an input clip with fixed number of K frames, and then successively forward passes them through several 3D convolution and normalization layers. In the proposed network, this network is used twice with weight sharing, taking as input the content clip and the style clip respectively. The style image repeated K times make static video clip to input to the network, as using 2D CNN to transfer feature statistics onto 3D CNN based content clip features for stylization did not give good results in initial experiments, as might be expected.
As we do feature statistics transfer between the content and the style feature vectors using AdaIN3D operation, we require the features of the layers of the C3D network to be of similar scale. This has been reported and solved before by doing network normalization by Gatys et al. [8]. Hence, we normalize the weights of our 3D encoder network using the UCF-101 dataset [22] following the method proposed by Gatys et al. [8].
3.2.2 Appearance subnets
In initial experiments, we attempted to do stylization by performing AdaIN3D operation between the features of content clip and style clip, as obtained from the C3D encoder. However, noticed "slow motion" artifacts in the stylized video. We attribute those to the fact that motion and appearance are entangled in the C3D features, as observed and qualitatively visualized in the original C3D paper [25]. When we did stylization with a static style video (made by repeating style image), it led to inducing the static motion in the stylized clip. Hence, to do stylization with 3D CNN, we need to disentangle the appearance part from the features and stylize only those with the style image, and then recombine them with motion and decode them to the output stylized clip.
Hence, to extract the appearance component of the features from the C3D features, we use appearance subnetworks, which are themselves sequences of 3D convolution followed by ReLU layers. We use four appearance subnetworks which work with four different intermediate levels of the C3D decoder. We learn these networks in different training phase, which we describe below when detail the training procedure in Section 3.3.
3.2.3
AdaIN3D
The AdaIN operation proposed in AdaIN [11] was designed to work on features of 2D CNN. We propose simple extension, for it to work with 3D features, as follows.
(x-n((x) m1xx)
x
AdaIN3D(z, y) = =1))
+ m(v).
(1)
((x))

---

H W
Uno(2)
Unothws
(2)
THW
T LEΣ (neehhw (nc(x))² +
(x)
THW
t=
where z represents the content clip and y represents the style clip, and t, h, w denote batch size, feature channels, temporal, height and width dimensions respectively. The mean and standard deviations of the style features in space and temporal dimensions are transferred to that of the content features.
3.2.4 Entangle subnetwork
order to recover both motion and appearance details the stylized video, we require an additional subnetwork to re-induce motion information to the appearance subnetwork features. To achieve this, we use an entanglement network which comprises single pair of 3D Conv and ReLU layers. The input the entangle layer is weighted combination of the appearance subnet 4's output and the final output of the C3D encoder, i.c.,
concat(0.7s.., 0.3
(4)
where, is the appearance feature map output of appearance subnet 4 and zm is the feature map output of the relu4_1 layer of the C3D network.
3.2.5 Decoder
The decoder is a 3D CNN used to generate the desired stylized video clip. comprises Conv, ReLU, ReflectionPad3d and Upsample layers. The input of the decoder is the output of the Entangle Subnet, and the outputs of the Apperance Subnets are added to the decoder as skip connections by concatenation along the channel dimension of the decoder's appropriate intermediate feature maps.
3.3 Training
We propose multi-phase training for learning the different subnetworks of the proposed network. In each phase, we train one more subnetworks while keeping the other subnetworks fixed. We first detail the different losses used, and then explain the different phases of the training.
3.3.1 Losses used for training
We use different combinations of losses during different phases of training. In this section we give details of the losses, and specify which losses are used which phases the respective sections describing the training phases below.
The first loss we the standard & reconstruction loss given by,
Ereconstraction |0, Iall2
(5)
where O, and I. are the output reconstructed and input content clips respectively. This loss minimizes the la distance between the input and the reconstructed content clip in the RGB pixel space.
Second loss we use the & loss on the output feature maps of the intermediate layers of the VGG-19 network, i.e.,
Cappearance
Σ (l,)(0,) - (T2))l1
(6)
where, d; is the feature map of layer of the ImageNet pretrained VGG-19 network. The intermediate layers of VGG-19 used, indexed by i, are relu1. 1, relu2_1, relu3_1 and relu4_1.
Most of the state-of-the-an style transfer methods [4, 11, 18] used features from the VGG-19 19 network pretrained on the ImageNet dataset. Since the VGG-19 features of individual frames only contain the appearance information, use loss based on VGG-19 features to help disentangle appearance

---

Appeasroe Subeet
Dreeder
Decoder
S.bmet
3D
Conercc
Retemmudeedd do
dibeet
up
(a) First phase: Autoencoder training
(b) Second phase: Appearance subnetworks training
Suowe
Reoorst uotten Ioee
Encoder
SLowd
Delodaa
Comnet do
Reconenacddd di
SL.We444
(c) Third phase: Entangle subnetwork training
Figure 2: The networks and settings for the different phases of training. The light gray subnetworks fixed, while the different dark gray subnetworks trained during the respective phases.
features from motion features in the C3D encoder. Further, we use content and style losses similar to AdalN [11]. The content loss is given by
Leoment = | (O) (12)))l2, = relu4_1,
(7)
where O and Ig is the output stylized and input content clip respectively and the style loss is given by
Sayle Σ EIlp(á((O)) - ((()(())))) + (lo():(O)) - (()((1)))
(8)
where indexes over the relui_1, relu2_1, relu3_1, relu4_1 layers of the VGG-19 network, I, is the input style clip, and jd-) and (-) are the mean and standard deviation the feature maps.
We also use temporal loss, which is inspired by Gao et al. [6]. and is given by
1 Ciemporal ΣΣ D. I(0, W.(OO-1)) (Je, W W,(
where j E (R, G, B) is each of the RGB channels of the image, and Y is the relative luminance channel. 0,-1 and O2 are the previous and the current stylized output frame, Ie,t-1 and Ie, are the previous and the current content input frame, D, H x W where H, W the height and width, respectively, of the input/output frames, and W, is the forward optical flow function, which is computed using FlowNet2.0 [12]. The purpose of using this loss is to maintain similar optical flow characteristics in each channel of the output frames, as that the luminance channel the input frame, and achieve motion consistency between frames.
Further, we propose an intra-clip loss given by,
Cintra ΣΣ (I(()((0)) - I t=
((00))))((((((0.00)) ((2.(0:-1))llz + ΣΣ
(10)
where, indexes over the layers of the VGG-19 network as above, O is the stylized clip, t represents the frame number in the stylized clip, uu()) and (()) are the mean and standard deviation of the feature map (-). This loss encourages the feature statistics to be similar between successive frames of the stylized clip. By ensuring that, we mitigate the effect of the frames being stylized differently. even when the style image is the same, and avoid jerky changes in color and intensity.
3.3.2 First phase: Autoencoder training
The first phase of training is to train the C3D encoder and decoder for video reconstruction task. The original C3D network was proposed for a discrete classification task, while the task here is of dense

---

prediction of stylized output video. To have a suitable initialization for the encoder and decoder, we pretrain them with reconstruction loss, Eq. 5, on our training dataset of videos, with the network shown in Figure 2a.
3.3.3 Second phase: Appearance subnetworks training
In the second phase, we keep parameters of all appearance subnetworks as well the decoder trainable, while keeping the parameters of the encoder fixed. The network architecture is shown in Figure 2b. We pass the features from different intermediate layers of the encoder to the corresponding appearance subnetworks for extracting out the multi-scale appearance information from the entangled feature maps of the encoder. The output of the appearance subnetwork 4 passed as the input the decoder while outputs of other appearance subnetworks are passed as skip connections, concatenated channel-wise, to the decoder. We initialize the decoder with the weights learned from the first phase for the second phase. We use ImageNet pretrained VGG-19 intermediate feature maps reconstruction loss, Eq. 6, during this phase. This ensures that the appearance subnetworks together with the decoder capture the appearance part of the input video clip, by reconstructing the VGG-19 feature maps. Since this loss is applied independently for each frame of the input clip, mainly captures the appearance information and discounts the motion information, leading to disentanglement of the appearance and motion features by the appearance subnetworks.
The disentanglement of motion and appearance, by the four appearance subnets, happens here in the phase training. In this phase, we keep the 3D CNN encoder fixed, so the features which are input to the appearance subnets have entangled motion and appearance information. We train the network to minimize the VGG-19 feature loss for each frame, and hence only remember appearance information in the frames. When trained like that, the appearance subnets only gate the appearance information from the 3D encoder features as only that required to minimize the loss, which is applied independently to each frame and, thus, has no dependence on motion. The evidence for such disentanglement is indirect, as when we do not do this, we get motion artifacts which we explain by the static nature of the style clip (constructed by repeating the style image), but when we do this, those motion artifacts disappear indicating that the features and the AdaIN3D based statistics transfer, only affected the appearance and not the motion.
3.3.4 Third phase: Entangle subnetwork training
In the third phase, we keep the decoder and the entangle subnetwork parameters trainable, while keeping the parameters of all appearance subnetworks well the encoder fixed. We initialize the decoder from scratch for this phase as the input features are different from the previously trained decoders. We reconstruction loss for this phase. The network architecture is shown in Figure 2c. The aim of this phase of training is to learn to re-induce motion information into the appearance features.
3.3.5 Fourth and final phase: 3D decoder training
In the final phase, the full network used, and only the decoder parameters are trained. We use the decoder trained in the third phase as the initialization for the decoder in this phase. All the other modules, i.c. encoder, all four appearance subnetworks, and entangle subnetwork, are kept fixed. The aim is to learn final decoder which is able to do stylization. The combination of losses gives the full loss used for this phase, Eq. 7, 8, 9, 10,
L Mr where, E {content, style, temporal, intra),
(11)
i.c this phase learns to do stylization while keeping intra-clip consistency, preserving overall content, and transferring style. This phase completes our training.
4 Experiments
We now describe the experimental setup, and the qualitative and quantitative results comparing the proposed method with existing state-of-the-art approaches.
7

---

4.1 Implementation Details
Dataset. A video dataset was available in the existing style transfer literature for video stylization. Hence, we propose a new dataset curated from existing public benchmark datasets, to train for video stylization task. We will make the links and all details of the dataset public for use by the community.
Since, most of the challenge in video stylization stems from motion in the videos, we explicitly construct the dataset such that the clips have a high amount of motion, and are not just static repeated images. On analysing some videos downloaded we saw that some videos were simply repetition of a single static image, and were thus not appropriate.
We downloaded 45, 574 videos from the first 100, 000 URLs in the Sports IM dataset [14] (as many links were now defunct, and some downloads failed). We then generated 16 frame clips with a high amount of motion with the following steps. First, we randomly sampled a video from the 45K videos dataset, and used deep learning based shot detection by Souček et al. [23] to split the video into shots. We then randomly sample a 16 frame clip and computed the Farneback optical flow [5] of the frames of the sampled clip. We then selected the clips which have average optical flow value above threshold to be in the proposed dataset. Overall, we used 10, 000 content clips generated from the above procedure part of the dataset for training. In addition to the content clips, we use images from the WikiArt [20] as the style images, as has been done by the image stylization methods as well.
Input. We set clip size K to 16. While training we extract random 128 x 128 size same patch from consecutive frames of a video clip in case of content clip. For, style clip we extract a random 128 x 128 patch from the style image and repeat it 16 times which is equal to the length of the content clip. During testing, we pass the video in chunks of successive 16 frames clips.
Parameters. In the first phase of training, we train the encoder and decoder for 10k iterations. We used Adam optimizer with learning rate of 10-4 with a decay rate of 5 x 10-5 after every iteration for this phase of training. In the second phase of training, we train the decoder, appearance subnets for 5k iterations. We used Adam optimizer with a learning rate of 10-4 with a decay rate of x 10-5 after every iteration for this phase of training. In the third phase of training, we train the decoder and entangle subnet for 100k iterations. We used Adam optimizer with a learning rate of 10 with a decay rate of 5 x 10-5 after every iteration for this phase of training. In the final phase training, we train the decoder for 40k iterations with content, style loss where values of Xe, A, is 1, respectively. After this, we train the decoder with content, style, temporal losses for 160k iterations where values of Acontens Xayles Asemponal being 1, 2, 10 respectively. And finally, we fine-tune the decoder for 40k iterations with content, style, temporal, intra-clip loss where values of Acomtent, Xayley Atemporal, Xintra being 1, 2, 10, 10 respectively.
4.2 Ablation Studies
We discuss the takeaways from the ablation experiments we performed. It is difficult to show the video related effects with images. Kindly, see the video results in the supplementary materials corresponding to these discussions.
Disentangling appearance and motion using appearance subnetworks. We applied AdaIN3D operation directly to the encoded tensor of the content and style clips by the C3D network without using any appearance (disentanglementt) subnetworks. However, we observed jerky motion in the generated videos, which followed a repeated pattern across the clips. Upon using appearance subnetworks, such motion artifacts were not observed anymore. The supporting videos are in supplementary material folder Videos_wite wwttwwhatthteeeaaeeeeeeeeeeeeeee www.iaaevieesstiieeaaeee www.eeiieiinn....
2D 2DD_stylizatii
Temporal loss. When trained without temporal loss, the stylized video has two kinds of artifacts. First, color consistency across subsequent frames suffers as we transfer style with appearance subnetworks which remove motion component. Second, there is some loss of motion as well. Both of these artifacts are not observed, or are very mild once we use temporal loss. Supporting videos are in supplementary material folder Videos wi sesisssitttuuu.. wwwchhuutttttprrraeeeeeeeeee
Videos_withooee
Intra-clip loss. We observe even after applying the temporal loss, there are some color incon- sistencies leading to flashing like artifacts. However, when we use intra-clip loss such inconsis- tencies are addressed to a large extent. Supporting videos are in supplementary material folder Videos_ withttt.... thout_intra..
8

---

BARE Move
Figure 3: First column the style image and subsequent columns are pair frames generated using different approaches, in the following order: AdaIN(11], MCCNet(3). AdaAttn[18] and Ours.
Figure 4: Effect content of different methods. First column is the style image, and the rows in the remaining columns contain frames and zoom-ins of original content clip, AdaAttN [18] stylized clip and our method. Notice how AdaAuN (middle) modifies the content significantly, while the proposed method (right) tends to keep the content relatively preserved.
4.3 Qualitative results
Figure shows some qualitative results of the proposed method compared to existing methods. We observe that the proposed method focuses more on low level texture of the style image while AdaAttn [18] tends flatten out the textured regions consistently. Figure shows that AdaAttn also tends to modify the content, while the proposed method gives better textured outputs closer to the original content. Please supplementary material for vidco results. Overall the results produced by the proposed method are aesthetically pleasing, and can be used by video creatives as one of the filters available.
4.3.1 User Study
Method
Content
Style
Overall
We considered AdaIN [11]. MAST [4] and two state-of- the-art methods, MCCNet [3] and AdaAuNN [18], for com- parison with our method. We randomly picked 15 content clips and style images and considered random 20 styl- ized videos out of 225 possible combinations. We gener- ated these stylized videos using all the compared methods. We then presented the stylized video of other methods along with that of our method side by side, but in random order, to the participants. We collected 900 votes from
AdaIN
2.3
16.3
4.3
MAST
1.0
15.7
11.0
MCCNet
12.3
31.3
27.7
AdaAuN
41.0
20.0
28.3
Ours
43.3
16.7
28.7
Table
User study preferences (%). Bold, red, blue represent rank 1, 2 and respectively.

---

013
1.20
SANET
4.11
1.93
AbAiN
1.96
2.94
1.77
291
0,77
0.36
0.24
1.26
3356
1.36
152
200
0.55
0:35
100
Oumetempera
0.37
1.89
0.31
2.88
0.27
044
0.44
2.17
1,90
0.94
199
0.22
1.58
0.08
0.42
3.35
0.41
0.21
1.00
Dumaeempealleints
0:41
2.1
0.31
2.50
0.39
0.44
1.7
266
1.06
0.66
1,89
0.21
1.90
0,77
013
0.53
1.91
0.51
0.78
1:11
Table 2: Optical flow evaluation 20 test pairs of content videos and test images. Bold, red and blue
color represents the rank 1, 2 and 3 minimum error respectively.
various participants for content preservation, degree of stylization and overall preference. The results in Table show that our method leads for content preservation closely followed by AdaAttN [18]. In the case of the degree of stylization, MCCNet [3] leads us by a margin, but we argue that there always a trade off between style transfer and content preservation, and so we can observe that MCCNet [3] performs poorly in the case of content preservation. Considering the overall preference in which participants have considered both content and style, our method performs better than all the other methods, proving our method's 's effecctiveness successful alternate style transfer method.
4.4 Quantitative Results
We computed Farneback optical flow [5] for the content video and the stylized video. We report the mean of the absolute difference between the flows of content video to that of corresponding flow in stylized video. This is an approximate measure to ensure that the stylization method is preserving the motion in the original clip. All the methods are expected to distort the motion in some ways while performing stylization, and the degree of such distortion will depend on the input video. We observe that the proposed method achieves competitive flow errors.
4.5 Compute time
Since the proposed method is based on 3D CNNs, it is computationally heavier than the 2D CNN methods. To get a real world computation time estimate, we processed a video with 144 frames with resolutions of 640x360 pixels. AdaAuN took 14 seconds and 8GB of GPU memory, while our proposed method took 60 seconds and 16GB of GPU memory on a machine with Intel Core 19-10900X processor and Nvidia RTX A4000 GPU.
5 Conclusion
We proposed a novel architecture and training method for the task of video stylization given a reference style image. The method uses 3D CNNs exclusively, and is the first to report stylization with them. It achieves stylization by disentangling appearance information from the 3D CNN based encoder's features, stylizing them, and then re-combining them with the motion information before finally decoding to the desired stylized output video. The training is done in phases with different phases ensuring different capabilities for the subnetworks in the architecture. We also proposed a video dataset containing videos with high degree of motion for training video stylization networks. We showed qualitatively and quantitatively that the proposed method produces results which are competitive to the existing methods and produce different and aesthetically pleasing results. We hope the proposed method will result in another filter in video creative's 's toolbox.
References
[1] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition'? new model and the kinetics dataset. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[2] Dongdong Chen, Jing Liao, Lu Yuan, Nenghai Yu, and Gang Hua. Coherent online video style transfer. In IEEE International Conference on Computer Vision, 2017.
[3] Yingying Deng, Fan Tang, Weiming Dong, Haibin Huang, Chongyang Ma, and Changsheng Xu. Arbitrary video style transfer via multi-channel correlation. In AAAI Conference on Artificial Intelligence, 2021.
[4] Yingying Deng. Fan Tang. Weiming Dong, Wen Sun, Feiyue Huang, and Changsheng Xu. Arbitrary style transfer via adaptation network. In 28th ACM International Conference on Multimedia, 2020.
10

---

[5] Gunnar Farnebäck. Two-frame motion estimation based on polynomial expansion. In Scandinavian Conference on Image Analysis, 2003.
[6] Chang Gao, Derun Gu, Fangjun Zhang, and Yizhou Yu. Reconet: Real- time coherent video style transfer network. In Asian Conference on Computer Vision, 2018.
[7] Wei Gao, Yijun Li, Yihang Yin, and Ming-Hsuan Yang. Fast video multi-style transfer. In JEEE/CVF Winter Conference Applicaations of Computer Vision, 2020.
[8] Leon Gatys, Alexander S Ecker. and Matthias Bethge. Texture synthesis using convolutional neural networks. In Advances in Neural Information Processing Systems, 2015.
[9] Leon A Gatys, Alexander S Feker, and Matthias Bethge. Image style transfer using convolutional neural networks. In JEEE Conference on Computer Vision and Pattern Recognition, 2016.
[10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[11] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In IEEE International Conference on Computer Vision, 2017.
[12] Eddy Ilg. Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[13] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision, 2016.
[14] Andrej Karpathy, George Toderici, Sanketh Shelly, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolutional neural networks. IEEE Conference on Computer Vision and Pattern Recognition, 2014.
[15] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, 2012.
[16] Xueting Li, Sifei Liu, Jan Kautz, and Ming-Hsuan Yang. Learning linear transformations for fast arbitrary style transfer. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[17] Yijun Li, Chen Fang, Jimei Yang. Zhaowen Wang, Xin La, and HinggHsuann/ Yang. Universal style transfer via feature transforms. Advances in Neural Information Processing Systems, 30, 2017.
[18] Songhua Liu, Tianwei Lin, Dongliang He, Fu Li, Meiling Wang. Xin Li, Zhengxing Sun, Qian Li, and Errui Ding. Adaattn: Revisit attention mechanism in arbitrary neural style transfer. In IEEE/CVF International Conference on Computer Vision, 2021.
[19] Dae Young Park and Kwang Hee Lee. Arbitrary style transfer with style-attentional networks. In JEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
[20] Fred Phillips and Brandy Mackintosh. Wiki gallery, inc.: case for critical thinking. Issues in Accounting Education, 26(3):593-60,, 2011.
[21] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In International Conference on Learning Representations, 2015.
[22] Khurram Scomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 12 12,.0440, 2012.
[23] Tomáš Souček and Jakub Lokoč. Transnet v2: an effective deep network architecture for fast shot transition detection. arXiv preprint arXiv:2008.0488. 2020.
[24] Christian Szegedy, Wei Liu, Yangging Jia, Pierre Sermanet. Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In IEEE Conference on Computer Vision and Pattern Recognition, 2015.
[25] Du Tran, Lubomir Bourdey, Rob Fergus, Lorenzo Torresani, and Manohar Paluri, Learning spatiotemporal features with 3d convolutional networks. In IEEE International Conference on Computer Vision, 2015.
[26] Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, and Victor S Lempitsky. Texture networks: Feed- forward synthesis of textures and stylized images. In International Conference on Machine Learning. 2016.
11

---

[27] Wenjing Wang, Shuai Yang, Jizheng Xu, and Jiaying Liu. Consistent video style transfer via relaxation and regularization. IEEE Transactions Image Processing, 29, 2020.
[28] Yexun Zhang. Ya Zhang, and Wenbin Cai. Separating style and content for generalized style transfer. In IEEE Conference Computer Vision and Pottern Recognition, 2018.
12


Reviews:
[{"review_id": "yzLoge1lbU", "reviewer": "Reviewer_Syan", "summary": "This paper proposes ViSt3D, which utilizes 3D CNN (C3D) as the encoder backbone for video style transfer. However, the motion and appearance information in C3D is intrinsically entangled. To address this problem, ViSt3D aims to separate these two features with appearance subnets and AdaIN3d. Results on sports1M datasets are used to demonstrate the performance of the proposed method.", "questions": "The authors are encouraged to address the weaknesses mentioned above, especially the motivations and contributions for using 3D CNN. If a 2D CNN is enough to address the stylization, it would be unnecessary to further design a disentangled module to solve a problem that is originally not there.", "limitations": "The authors discussed the limitations of the increase in both inference time and memory consumption. Although it may not be appropriate to directly compare the time and memory requirements of our approach with other methods based on 2D CNNs, it would be still helpful to give related statistics for a better understanding of the proposed method.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- As claimed, this paper is the first to use 3D CNN for video stylization.\n\n- The temporal and intra-loss improve the stylization stability as shown in the supplementary videos.\n\n-  Quantitative evaluations of optical flow demonstrate the effectiveness of the proposed method.", "weaknesses": "- The usage of 3D CNN is not well motivated. The authors should put more emphasis on the motivations and advantages of using 3D CNN for stylization. This also makes the technical contributions a bit weak. And as mentioned in the limitations 3D CNN also brings extra computational cost compared with 2D CNN-based methods.\n\n- In the paper, the style clip is formed by repeating a static image multiple times. Thus, I do not understand why you need a C3D to extract the style feature -- a 2D CNN could be sufficient. I suggest the authors conduct related experiments and provide thorough discussions.\n\n- In Equation (10), why not use the difference of warped features to measure the intra-clip consistency as (9)? I think using the mean and standard deviation will hurt the results when the scene changes or there is a large motion.\n\n- The model needs to be trained with four cascaded phases. This makes the model less elegant and may be hard to train and reproduce.\n\n- Comparison with SOTAs should be more comprehensive. Only the optical flow metric is not enough to evaluate the style transfer results (for example, directly outputting the content video will result in a relatively small error).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments, we answer your questions and concerns as follows.\n\n\n### 3D CNN vs 2D CNN, solved problem, motivation\n\nStylization is not a deterministic problem, in the sense that there is no single correct stylization for a (input content, target style) pair. It is akin to a high level filter which transfers characteristics from an image to another image or video. 3D CNN was a natural choice when working with videos. They had never been used for the task, and we expected to obtain different stylization characteristics upon using them, as eventually we did and reported in the paper. 2D CNNs have also been used with some success for the task, but none of the methods (2D or 3D CNN based) could be THE final solution. 3D CNNs do take more computational resources but they produce competitive and useful results as we reported in the paper and reinforce in the user study provided in the other response. Hence we believe our contribution of successfully doing stylization with 3D CNNs for the first time, is valuable for the research community as well as practitioners. The simple motivation of using 3D CNNs was to determine if stylization can be done with 3D CNN at all (which surely in hindsight, given the paper, looks obvious now, but was a hard problem to solve) and to derive a novel and different stylization which is useful.\n\n### Why 3D CNN for style clip and not 2D CNN\nStylization requires transfer of feature statistics from style image to the content video/image. Just as for image stylization using one model for encoding the content image (e.g. ResNet) and another for encoding the style image (e.g. InceptionNet) is not expected to lead to sensible results, as the statistics for intermediate features are expected to be very different to be successfully transferred, in the present case using two different networks for encoding content and style assets is not expected to work as well. We did initial experiments with 2D CNN encoding for style images, as a sanity check, but as expected it failed completely, i.e. the outputs did not converge to reasonable videos. Hence we moved on to encoding the style image with 3D CNN, by making a static clip, which was the naive extension to AdaIN with 3D CNNs for content and style both. We provided those results in the supplementary materials folder Videos_with_naive_extension_of_2D_stylization.\n\nPlease note similar repetition (as used for single style image), albeit for features, was used to bootstrap/initialize the 3D CNN parameters in the one of the 3D CNN for video classification papers, and so is not a new concept and has been used successfully in the past.\n\n### Intra-clip loss, temporal loss, high motion, scene change\nThe intra-clip loss in Eq.10 is to match the global color appearance of successive frames. While the temporal loss in Eq.9 is to preserve pixel level motion characteristics after considering the appropriate warping of frames. Even in case of large motion, since Eq.10 works with global averaged values, it successfully performs its job. We had provided qualitative results without intra-clip loss (folder Videos_without_intra) and also without temporal loss (folder Videos_without_temporal), in the supplementary material. Without temporal loss, we observe strong jittering and without intra clip loss we observe a subtle flashing, as the average color of the frames drifts, and then resets on a clip boundary. Adding the intra clip loss fixes this flashing.\n\nLarge sudden motions are challenging for all stylization methods, we had provided qualitative comparisons in supplementary (folder Comparative Analysis, result 2 has a large motion clip from Spiderman movie, and result 4 has a clip from Avengers movie). The proposed method does competitively or better than existing methods. \n\nNone of the video stylization methods address shot changes where the scene completely changes. The stylization is done on videos which have been separated by performing shot detection first as we explain in our dataset creation section (l265).\n\n\n### Model/training with four phases, less elegant, not reproducible\nWe agree that the current version of the model and training are quite involved. The current paper demonstrates that stylization can be done with 3D CNNs. In the work we are currently doing, we are shrinking the models in size and experimenting with end-to-end training with promising initial results. As a first result, we believe the current method is interesting for the community. We had already given the code of each subnetwork in the supplementary PDF, and each stage of training is a well understood training procedure for the community (auto-encoder, perceptual and reconstruction loss minimization), so we believe that the results can be reproduced by a reasonably skilled student researcher.\n\n### Comparison to SOTA, optical flow not good/sufficient metric\nWe agree to this point, but we highlight that evaluating a subjective task is always hard, and some proxy metrics are generally used. Stylization, like some other tasks, perhaps (semantic) edge detection, image captioning etc., do not have one true answer. So using proxy metrics to evaluate aspects of the output have been used by the community which we also follow. In addition, as another reviewer’s question’s response, we also did a user study in the present rebuttal, and we request you to refer to that as well. In particular among responses rating for preservation of content and transfer of style, we see a trade-off. The methods which preserve the content (in extreme case output the same video as you mention) lag in transferring style. Overall the proposed method strikes a balance as good as or better than the existing methods, while being a completely novel way of doing stylization.\n\n### Time and memory cf. 2D CNN\nWe processed 144 frames of 640x360. AdaAttN took 14, 8GB of GPU memory, while proposed method took 60s, 16GB of GPU memory on a machine w/ Core i9-10900X & A4000 GPU.\n"}, {"review_id": "r0se56WACW", "reviewer": "Reviewer_9xQH", "summary": "The paper studies the task of video stylization. The paper aims to stylize the video using 3D CNN and AdaIn3D. To perform the stylization, the authors propose to disentangle motion and appearance first, and then stylize the appearance part using AdaIn 3D. The results show state-of-the-art results compared to the baseline methods.", "questions": "Please see my questions regarding the details of the network in Weakness.", "limitations": "The authors have discussed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper proposes a novel framework using 3D CNN to perform the stylization. To achieve this, multiple designs are proposed. Such as the appearance subnet, and Entangle network.\n\n2) It is reasonable to train the model in different stages to enforce the corresponding networks to learn their functionalities.\n\n3) The results are significantly better compared to the baseline methods.\n\n4) It is non-trivial to train a model with many sophisticated designs and achieve plausible results.", "weaknesses": "1) It is unclear why four appearance subnets are needed. If the number of appearance subnets is cut down, how the performance is affected?\n\n2) Why the Entangle Subnet is necessary? What if the feature maps fed into the decoder are the weighted average of the output of the appearance network and C3D encoder?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments, we answer your questions and concerns as follows.\n\n### Unclear why four appearance subnets needed, affect on performance if the number of appearance subnets is reduced\n\nThe different appearance subnets work at different scales. The C3D architecture can be seen as four sequential logical blocks, and we used one appearance subnet for the output features of each of those blocks. This was inspired by skip connections architecture in many pixel to pixel networks, e.g. UNet for segmentation etc.\n\nIf we remove the appearance subnets, we would observe a deterioration in the stylization quality along with undesirable artifacts. In C3D features qualitatively it has been shown that appearance contributes more to the lower features (initial blocks) while motion contributes more to the higher features. If we remove the lower layer appearance subnet we would observe distorted appearance in the output while if we remove the upper layer, motion artifacts will corrupt the output.\n\n### Why is Entangle subnet needed, decoder features could be weighted average of appearance net and C3D encoder features\n\nOther than needing a simple projection layer to match different feature dimensions, a weighted average as you suggest could be a valid candidate for feature fusion. The Entangle network we use is very lightweight, i.e. just one 3D conv and one ReLU layer, and it effectively does a non-linear combination cf. a linear weighted average, ensuring the fusion is more effective.\n"}, {"review_id": "PJ4diFgfIe", "reviewer": "Reviewer_fJLx", "summary": "Image stylization is more popular than video stylization, research on video stylization is few due to its challenging. This manuscript first applied 3DCNN to video stylization task, it first explicitly disentangles motion and appearance, and then stylizes the appearance part, and then adds back the motion to decode the final stylized video. The method is trained in multiple phases. Experimental results show the superiority of the proposed method with existing methods. Hwever, there are still some problems in this manuscript.", "questions": "1.\tHow to disentangle motion and appearance by 4 appearance subnets is not clear in your description.\n2.\tIn line 190, ‘O and I are the output and input clips respectively’, and in line 213, ‘O is the stylized clip’. It is confusing whether they are content or style clip? \n3.\tCombined with the previous question, in Fig.2, it can be found that in all three phases, only content clip is used, where is the style clip? How to add style information to appearance and preserve motion?\n4.\tIn line 134, ‘Our encoder is a recent state-of-the-art 3D CNN, i.e. C3D [23]’, while C3D was proposed in 2015, ‘recent’ may be not suitable.\n5.\tThere are some fonts are inconsistency, e.g., line 172 ‘Appearance Subnet 4’, line 202 ‘relu1_1, relu2_1, relu3_1, relu4_1”.\n", "limitations": "1.\tThe authors point out that some flashing still occurs in challenging edge cases.\n2.\tIn Table 1, it can be found that the proposed method did not achieve minimum error on all videos, which means it cannot preserve the motion in the original clip very well.\n3.\tFrom the video results in supplementary material, it can be seen that the degree of stylization by the proposed method is not obvious, and the results still have some artifacts.\n4.\tIt is suggested to reorganize section 3 to introduce your method more clearly. \n", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Stylization has been a very popular research area, however video stylization is more challenging than image stylization. Most existing methods directly apply image stylization methods to videos by processing the videos frame by frame, results are not well. This manuscript proposes a novel method based on 3DCNN and achieves better results than existing methods. Moreover, a large-scale dataset with 10, 000 content clips curated from public benchmark Sports1M is built for this task.", "weaknesses": "The task is interesting and the proposed method has some improvement, but paper writing is a little weak. The logic of the article is confusing, especially in Section 3 APPROACH. The description and details of network structure is not clear, it's not easy to understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments, we answer your questions and concerns as follows.\n\n### Disentangle motion and appearance by 4 appearance subnets\n\nThe disentanglement of motion and appearance, by the 4 appearance subnets, happens by the phase 2 training. In Phase two we keep the 3D CNN encoder fixed, so the features which are input to the appearance subnets have entangled motion and appearance information. We train the network to minimize the VGG-19 feature loss for each frame, and hence only to remember appearance information in the frames. When trained like that the appearance subnets only gate the appearance information from the 3D encoder features as only that is required to minimize the loss, which is applied independently to each frame and has no dependence on motion. The evidence for such disentanglement is indirect, as when we do not do this, we get motion artifacts which we explain by the static nature of the style clip (constructed by repeating the style image), but when we do this those motion artifacts disappear indicating that that the features and the AdaIN3D based statistics transfer only affected the appearance and not the motion.\n\n\n\n### In line 190, ‘O and I are the output and input clips respectively’, and in line 213, ‘O is the stylized clip’. It is confusing whether they are content or style clips?\n\nWe apologize for the confusion. \n\nThe losses as functions of two variables, $I$ and $O$ are correct and what those variables are depends on the training phase.\n\nTo clarify, in L190 Eq.5 the reconstruction loss applies to phase 1 training of the autoencoder, so the two inputs to the loss function are the input clip, and the output reconstructed clip.\n\nWhile in L213, Eq.10 the intra clip loss applies to phase 4 of the training of full stylization, so the inputs to the loss function are the frames of the output stylized clip.\n\nNonetheless, to avoid confusion, we will change $O$ in L190 to $O_r$, as it is the reconstructed output clip from the auto-encoder. \n\n\n### How is style clip used and how is appearance and motion preserved while doing stylization?\n\nStyle clip is only used in the final phase of training shown in Fig.1. Style information is added using AdaIN3D operation, which generates a combined feature by taking content, and style features as input. The AdaIN3D operation introduces style to disentangled appearance features only, which are then combined with motion features. The combined features are then decoded and the final losses (content, style, temporal and intra-clip, Eq11) make sure that the output is close in appearance to the content clip, while having style of the style clip, and at the same time not having jittering and flashing artifacts.\n\n\n### C3D proposed in 2015, ‘recent’ may be not suitable.\n\nWe will remove recent from the text mentioned.\n\n### Font inconsistency, e.g., line 172 ‘Appearance Subnet 4’, line 202 ‘relu1_1, relu2_1, relu3_1, relu4_1”.\n\nWe wanted to highlight that those words indicate certain important layers’ features in the VGG-19 networks that are used. We will clarify this in the notations section.\n"}, {"review_id": "6n2PtMbqIU", "reviewer": "Reviewer_qZ5E", "summary": "This paper proposes a 3D CNN based video stylization method which explicitly disentangles motion and appearance and adopts multi-phrase training. Experiments show that the method achieves high quality results.", "questions": "1. Why are only three previous methods included in the qualitative and quantitative comparison? There have been quantities of style transfer papers every year and many of them are about video stylization.\n2. Why is user-study not included in the experiments?\n3. What is the advantage compared to adding optical flow motion compensation to single frame style transfer method?\n", "limitations": "This paper lacks several important parts:\n1. The discussion and analysis of each part of the method are not included in the paper\n2. User study is missing in experiments\n3. More previous methods should be included in comparison\n4. It would be better if more results of diverse scenarios could be shown. ", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed 3D CNN based framework and multi-phrase training is novel and effective, the paper also expand the AdaIN in time dimension to AdaIN3D. The quality of stylization results is also good.", "weaknesses": "I think the main weakness lies in the experiments.\n1. In ablation study, the authors only showed and described the results but didn't not analyze the reason\n2. In the qualitative results, this paper compares with only a few previous methods, while some related works should also be discussed. for example, \nKotovenko D, Sanakoyeu A, Lang S, et al. Content and style disentanglement for artistic style transfer[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2019: 4422-4431.\nKotovenko D, Sanakoyeu A, Ma P, et al. A content transformation block for image style transfer[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 10032-10041. \nBoth papers showed high-quality video stylization results\n3. For tasks where the quality is largely depended on subjective assessment, user study is a common criterion. For this paper, I feel the comparison is not sufficient and a user study is needed\n4. In quantitative comparison, the proposed method doesn‘t obviously outperform previous method. \n5. Lack a comparison to single image stylization with optical flow motion compensation\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments, we answer your questions and concerns as follows.\n\n### Why only three methods compared\n\nVideo stylization is relatively less popular cf. image stylization. We reported the recent methods which are leading in this task to the best of our knowledge. We have further included quantitative results from two more papers in the comparisons table (final table provided in global comments), but as expected they perform below the leading works we had already cited and compared with. In the absence of specific citations, we are unable to give any better answer.\n\n### Ablation study, results not analyzed\n\nThe ablation studies basically support the motivations given in our approach section. Due to lack of space we could not reiterate them, we will do so in camera ready. We will also point out specifically that there are supporting videos in the supplementary material (showing before after results) originally uploaded with the paper.\n\nTo summarize again: \n\n- First ablation shows that when the appearance and motion components in the features are not disentangled the results have jerky motion artifacts as the static style clip induces zero movement in the content clip. Once we do the disentanglement this is resolved. The supporting \nvideos are in supplementary material folder Videos_with_naive_extension_of_2D_stylization\n\n- Second ablation shows that temporal loss is needed to make sure that in videos pixel motion consistency is maintained. Without the loss we get jittery movements in videos, but adding the temporal loss resolves this. Supporting videos are in supplementary material folder Videos_without_temporal.\n\n- Third ablation shows that without intra clip loss, there is a slight drift in colors as the stylization is not sufficiently tied for the different frames of the clip. But with intra clip loss that is resolved. Supporting videos are in supplementary material folder Videos_without_intra.\n\n### User study\n\nNot all previous works have reported a user study. Based on your kind suggestions we did a user study during the rebuttal period and are reporting it in the global response which we request you to kindly refer to. In summary, the proposed method is competitive wrt existing methods in terms of user preferences as well, while being a first demonstration of stylization using 3D CNNs.\n\n### Comparison with methods\n\nIn the short period of time we could not do a comparison to the methods specifically mentioned in the review. Their code or pre trained models for inference were not available.\n\nHowever, we were able to compare with SANet and MAST quantitatively, and also used these in the user study (so compared qualitatively).\n\nWe iterate again however, the task of stylization is not a well defined task with a single true output for a specific (content, style) pair. We have demonstrated that the proposed method is competitive to the existing methods in terms of quantitative metrics, as well as subjective qualitative results and a user study.\n\n### Advantage cf. adding optical flow to image stylization methods\n\nWe also found another paper which does specifically this [A], i.e. adds optical flow to 2D CNN based image stylization methods. We have added a video with comparison to some of their results. We observe that in the first example, there is a large amount of flickering in the first example generated with [A], while in the second example the method smoothes out the texture of the pebbles on the beach, while the proposed method does not have heavy flickering, and it also maintains the texture of the pebbles.\n\nThe two approaches give different results which may be used by practitioners in different use cases.\n\n[A] Wang et al., Consistent Video Style Transfer via Relaxation and Regularization, Trans. Image Processing, 2020\n\n\n### More previous methods in comparison\n\nWe had compared with AdaAttn which had itself compared to many methods so we hoped demonstrating better results wrt AdaAttn would be sufficient. However, at your suggestion we have included two more methods in the comparison table, and have given the updated table in the global response. Kindly refer to that please.\n\n\n### More results in diverse scenarios\n\nDue to space limitation we couldn’t include more visual results in the main paper PDF.  But, we had already included a very variety of results in the supplementary which we summarize here: in the folder \"Comparative Analysis” we had given 4 video, each containing 3 (input video, target style image) pairs for our method as well as AdaIn, MCCNet and AdaAttn. These covered indoors, outdoors, high motion, animation and sports. In addition, in the folder ``Our results” we provided 6 result videos of a one input video each (4 short videos and 2 longer videos) stylized with three different style images, these also cover a variety of scenes and styles. We request you to kindly refer to the supplementary results videos. In addition we used 20 videos for the user study with 1 style each, which the users found reasonable as well (user study reported in global author response).\n\n\n"}]

Author rebuttal:
We thank the reviewers and ACs for their valuable time and constructive comments on the paper.

The reviewers have raised many valid points and concerns which we have answered to the best of our abilities and hope that the reviewers will find them satisfactory.

We would like to reiterate that video stylization with 3D CNNs was not attempted before, and while we do not think any method can claim to be the absolute best for the task, as the task is subjective, the method we propose gives distinctive stylization when compared to existing methods. On the request of the reviewers we also did a user study where we found that our method is competitive to the existing methods in terms of the trade-offs between stylization and content preservation as evaluated by human subjects. We are looking forward to any questions or queries the reviewers might have given our responses.

### Details and discussion of user study

We selected 20 random stylized videos from a combination of 15 content videos, 15 style images. We considered four other leading style transfer methods for comparison, namely AdaIN, MAST, MCCNet, AdaAttN. 

We took a total of 900 votes from 15 users for this user study. We asked the users to vote separately for three preferences: Style, Content and Overall, i.e. which of the five presented options (i) transfers the style best, (ii) preserves the content best and (iii) is preferable overall. 

The results of the user study are given in the attached PDF.

We observe that MCCNet performs best by a big margin in Style preference (31.3% MCCNet vs 20% AdaAttn and 16.7% proposed). While, our method leads in Content preference, closely followed by AdaAttN (43.3% vs 41%). In the case of Overall preference our method leads, and is closely followed by both AdaAttN, MCCNet (28.7%, 28.3%, 27.7% resp.). MCCNet tends to heavily stylize the output and distort the content, while the proposed method as well as AdaAttN maintain the content better. However, the styling as we observe in numerous qualitative examples in the supplementary as well, is quite different for these top performing methods. Hence we conclude that the proposed method is a competitive stylization method useful for end users.


### Discussion on the Optical flow metric table

As a response to a comment by reviewer qz5E, we could add two more methods for quantitative evaluation in the rebuttal period, i.e. SANET, MAST. We are giving the results here (Table in attached PDF) as they might be interesting for other reviewers too. Optical flow metric is one of the possible metrics for the task, it helps to make sure that the stylization method is preserving the motion in the original clip and there are no obvious/drastic failures. All the style transfer methods are expected to distort the content in some ways while performing stylization. Considering the average optical flow error our method has the least mean optical flow error and thus we can conclude our method is at par wrt the leading methods for the task. Our method also achieves mostly rank 1 and 2 among the methods.


Meta-review: This paper presents a method to perform video stylization with a network that uses 3D convolutions as well as a host of other design choices to obtain temporally consistent results. While reviewers were initially skeptical about the degree of novelty of this idea, during the discussion phase the authors managed to convince reviewers that simply using 2D CNNs for the task introduces artifacts. As a result, the reviewer consensus shifted to accept. Reviewers also appreciated that this work introduces a large scale dataset that will encourage follow up research in this domain.
Decision: Accept (poster)