SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ayush Pande
IIT Kanpur
ayushp@cse.iitk.ac.in
Gaurav Sharma
TensorTour & IIT Kanpur
gaurav@tensortour.com
Visual stylization has been a very popular research area in recent times. While image stylization has seen a rapid advancement in the recent past, video stylization, while being more challenging, is relatively less explored. The immediate method of stylizing videos by stylizing each frame independently has been tried with some success. To the best of our knowledge, we present the first approach to video stylization using 3D CNN directly, building upon insights from 2D image stylization. Stylizing video is highly challenging, as the appearance and video motion, which includes both camera and subject motions, are inherently entangled in the representations learnt by a 3D CNN. Hence, a naive extension of 2D CNN stylization methods to 3D CNN does not work. To perform stylization with 3D CNN, we propose to explicitly disentangle motion and appearance, stylize the appearance part, and then add back the motion component and decode the final stylized video. In addition, we propose a dataset, curated from existing datasets, to train video stylization networks. We also provide an independently collected test set to study the generalization of video stylization methods. We provide results on this test dataset comparing the proposed method with 2D stylization methods applied frame by frame. We show successful stylization with 3D CNN for the first time, and obtain better stylization in terms of texture cf. the existing 2D methods.
Project page: https://ayush202.github.io/projects/ViSt3D.html
Image stylization is the task of transforming an input content image into a stylized image with the style taken from another input style image. It has been a very popular research topic as well as a widely used image editing tool in the recent past. Pictures taken by many users have been transformed into the styles of Picasso and Monet. While image stylization with 2D CNNs has been hugely popular [4, 9, 11, 16], stylization of videos has been relatively less explored. Some recent works on image stylization also show results on video stylization by doing frame by frame image stylization, e.g. [4, 11, 18], and have achieved some success. 3D CNN architectures designed for video processing, to the best of our knowledge, have not yet been made amenable for video stylization at all. Since the 3D convolutions used in the 3D CNNs, inherently combine the spatial and temporal information, the two factors of variations are deeply entangled in the resulting representations. This makes video stylization with 3D CNNs extremely challenging as the style is usually for the appearance, i.e. indicated by a single image, and modifying the video representation to incorporate style in the content, but not in motion is a very challenging task. Naive extension of 2D stylization, e.g. directly transferring feature statistics from style to content features using AdaIN [11] operation with 3D CNNs intermediate features, leads to unusable video output, with intermittent slow motion and jerky transitions due to the style input being static (see supplementary material for representative results).
We address the challenging task of video stylization using 3D CNNs directly. We propose an architecture and training method which takes a content video clip and a style image, and produces an aesthetically pleasing stylized video clip. We do so by disentangling the appearance and motion
features from the encoder network and applying AdaIN operation extended for 3D CNN features, coupled with feature map reconstruction loss and style statistics losses. We show how to directly extend AdaIN operation to work with 3D CNN based encoders and decoders, which we call AdaIN3D, and train the network successfully despite the challenges with highly entangled appearance and motion information. All the sub-networks used in the proposed architecture are based on 3D CNN. Since a dataset was not available for the task of studying video stylization, we also propose a large-scale dataset with 10,000 content clips curated from public benchmark Sports1M [14], paired with the train set of style images from WikiArt dataset [20] which are used to train 2D stylization methods. We show qualitatively that the proposed method produces better results than frame by frame stylization using state-of-the-art 2D image stylization methods. We also give quantitative results based on optical flow errors, comparing the results of the proposed method to that of the state-of-the-art 2D stylization methods. Our contributions can be summarized as follows.
- We propose a novel method for video stylization using 3D CNN, and we are the first to report that stylization with 3D CNN is possible.
- We propose a novel approach to disentangle the appearance and motion features, which are deeply entangled in representations learnt by 3D CNN networks like C3D [25].
- We curated the video dataset from the public Sports1M dataset [14] by filtering videos with motion automatically and use it for training our video stylization network.
- We use existing loss functions and design new ones to encourage learning of artifact free video stylization.
- We provide exhaustive comparisons with existing methods, detailing the differences.
Overall, since visual stylization is effectively a subjective creative filter, we hope to add one more tool to the video creatives toolbox.
Image stylization. Gatys et al. [8] proposed an approach to generate textures using CNN and optimization based technique. They used the loss function over Gram matrices on feature maps for maintaining feature map statistics and hence generating required textures. In their next work, Gatys et al. [9] used a similar optimization based approach for stylizing content image with the texture of a style image. Such optimization based approach is quite slow, and the training has to be performed individually for every new pair of images. The next generation of image stylization approaches then turned to feedforward CNNs, e.g. [13, 26], for speed and avoiding retraining on every test style image. However, they still could not be used with arbitrary style images on the fly, i.e. one network needs to be trained for each style image. Finally, the current generation of image stylization methods emerged that were designed for universal image stylization [11, 17, 28], i.e. the network is trained once, and then is used for transferring style from an arbitrary style image, fed along with the content image as input at test time. AdaIN [11] was a very successful step towards universal style transfer because of the simplicity of the network training and parameter free transfer of content into the style space using the AdaIN operation. Among other works, Li et al. [17] used whitening and color transforms for style transfer. However, such transformations produce unsatisfactory results, potentially loosing/ modifying content in the output image. SANet [19] tried to introduce more style into the network but traded off the content from the original content image.
Video stylization. All the image stylization methods can be applied to videos by processing the videos frame by frame. Video stylization when done with image stylization methods such as AdaIN [11], WCT [17] etc., leads to stylized videos which suffer from various temporal and spatial inconsistencies. Chen et al. [2] and Gao et al. [7] hence proposed temporal consistency constraints to address these. Methods for video stylization include MCCNet [3] and AdaAttn [18]. MCCNet [3] mainly focused on video stylization, while AdaAttn [18] addressed image stylization using attention and extended the method for videos as well. Both methods lead to stylized video with some degree of style transfer, but increasing the style transfer introduces artifacts quickly. We compare with both of these in our experiments. Wang et al. [27] added optical flow to improve video stylization with 2D CNNs, and here we also use optical flow-based loss but with 3D CNN directly. We observed that there is noticeable flickering in some examples generated by Wang et al. [27]. Also, it smoothes out
Figure 1: Full architecture for the proposed network for video stylization. The network consists of 3D CNN based encoder and decoder networks, along with two other critical parts: (i) appearance subnets and (ii) Entangle layer. The appearance subnets are trained to disentangle appearance and motion components from the full 3D CNN feature and extract out the appearance part only. The entangle layer is trained to do the opposite, i.e. given the (stylized) appearance features as well as the full 3D CNN features, it re-introduces the motion component in the stylized features to enable them to be decoded into a stylized video by the decoder. The contribution of this paper is to report successful video stylization with 3D CNN directly, cf. 2D CNNs of existing works, using the above network, and a multi-phase training method to train the individual sub-networks proposed.
the low-level textural details, while the proposed method does not have heavy flickering, and it also maintains the low-level textural details.
Stylization with 2D and 3D CNNs. All the above methods mainly work with 2D CNNs, e.g. [10, 15, 21, 24], for encoding the content and style images, and propose methods for transferring style by matching statistics of the intermediate feature maps. Videos are inherently 3D, and 3D CNNs, e.g. [1, 25], have been proposed for learning with videos. There has been no work on video stylization with 3D CNN, and in this paper we report the first successful network and training method to do stylization with 3D CNN. Other than breaking new ground, we also show that the proposed method produces richer texture in the stylized video, and since stylization is contextual and subjective, it is at the very least, another useful tool for video creatives.
We now present the proposed network architecture and training method for doing video stylization with 3D CNNs. Since we need to perform the challenging task of disentangling the motion and appearance components, the method is trained in multiple phases. In the following, we first describe the full architecture used at inference time in Section 3.1, which takes the content video clip as well as the style clip (the style image repeated  $K$  times for clips of size  $K$ ), giving details of each subnetwork. Then, we explain the losses and the multi-phase training method in Section 3.3.
3.1 Overall network
The task addressed here is to do video stylization with a reference style image. The network takes as input the content clip (which needs to be stylized) and the reference style image, and outputs the corresponding stylized video.
Figure 1 shows the full network used for stylizing input video clip, with an input style image. Along with the input video clip, the network takes a style clip as input. The style clip is constructed by repeating the style image  $K$  number of times, where  $K$  is the clip size for the input to the 3D CNN encoder. Both the content and style clips are first encoded with a 3D CNN encoder. The outputs of the encoder, i.e. final layer feature maps and the intermediate layer feature maps, are then fed to appearance subnetworks, which are responsible for extracting the appearance features only, disentangling them from the motion features. We will detail the method for achieving such disentanglement in the following sections. Once the outputs from appearance subnetworks are
available for both content and style clips, we perform the AdaIN3D (extension of AdaIN[11] operation to 3D CNN features, described in Section 3.2.3 below) operation for transferring feature statistics from the style clip features to the content clip features. Finally the intermediate layer features are directly fed to the decoder, which is also a 3D CNN. The final layer features after the AdaIN3D operation are then aggregated with the final layer feature maps from the 3D CNN encoder. This step is for re-inducing the motion features into the appearance features which were disentangled by the appearance subnetworks. These aggregated features are then fed into the Entangle Subnet, the output of this subnet is passed as input to the decoder which produces the desired stylized output clip. Effectively, these steps can be seen as, (i) disentangle the appearance features from the full features which contains appearance and motion both, (ii) stylize the appearance features only, and finally (iii) re-induce the motion component into the stylized appearance features and decode them to obtain the stylized clip.
3.2 Details of network components
Now, we will explain each component of the full architecture shown in the Figure 1. Please refer to the supplementary material for the complete code-level details of the appearance, Entangle, and the 3D Decoder subnetworks.
3.2.1 Normalized encoder
Our encoder is a state-of-the-art 3D CNN, i.e. C3D [25], which was originally proposed for the task of human action classification. It takes an input clip with a fixed number of  $K$  frames, and then successively forward passes them through several 3D convolution and normalization layers. In the proposed network, this network is used twice with weight sharing, taking as input the content clip and the style clip respectively. The style image is repeated  $K$  times to make a static video clip to input to the network, as using a 2D CNN to transfer feature statistics onto 3D CNN based content clip features for stylization did not give good results in initial experiments, as might be expected.
As we do feature statistics transfer between the content and the style feature vectors using AdaIN3D operation, we require the features of the layers of the C3D network to be of similar scale. This has been reported and solved before by doing network normalization by Gatys et al. [8]. Hence, we normalize the weights of our 3D encoder network using the UCF-101 dataset [22] following the method proposed by Gatys et al. [8].
3.2.2 Appearance subnets
In initial experiments, we attempted to do stylization by performing AdaIN3D operation between the features of content clip and style clip, as obtained from the C3D encoder. However, we noticed "slow motion" artifacts in the stylized video. We attribute those to the fact that motion and appearance are entangled in the C3D features, as observed and qualitatively visualized in the original C3D paper [25]. When we did stylization with a static style video (made by repeating style image), it led to inducing the static motion in the stylized clip. Hence, to do stylization with 3D CNN, we need to disentangle the appearance part from the features and stylize only those with the style image, and then recombine them with motion and decode them to the output stylized clip.
Hence, to extract the appearance component of the features from the C3D features, we use appearance subnetworks, which are themselves sequences of 3D convolution followed by ReLU layers. We use four appearance subnetworks which work with four different intermediate levels of the C3D decoder. We learn these networks in a different training phase, which we describe below when we detail the training procedure in Section 3.3.
3.2.3 AdaIN3D
The AdaIN operation proposed in AdaIN [11] was designed to work on features of 2D CNN. We propose a simple extension, for it to work with 3D features, as follows.
$$
\operatorname {A d a I N 3 D} (x, y) = \sigma (y) \left(\frac {x - \mu (x)}{\sigma (x)}\right) + \mu (y), \tag {1}
$$
$$
\mu_ {n c} (x) = \frac {1}{T H W} \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \sum_ {w = 1} ^ {W} x _ {n c t h w}, \tag {2}
$$
$$
\sigma_ {n c} (x) = \sqrt {\frac {1}{T H W} \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \sum_ {w = 1} ^ {W} \left(x _ {n c t h w} - \mu_ {n c} (x)\right) ^ {2} + \epsilon}, \tag {3}
$$
where  $x$  represents the content clip and  $y$  represents the style clip, and  $n, c, t, h, w$  denote batch size, feature channels, temporal, height and width dimensions respectively. The mean and standard deviations of the style features in space and temporal dimensions are transferred to that of the content features.
3.2.4 Entangle subnetwork
In order to recover both motion and appearance details in the stylized video, we require an additional subnetwork to re-induce motion information to the appearance subnetwork features. To achieve this, we use an entanglement network which comprises of single pair of 3D Conv and ReLU layers. The input to the entangle layer is a weighted combination of the appearance subnet 4's output and the final output of the C3D encoder, i.e.,
$$
x _ {e} = \operatorname {c o n c a t} \left(0. 7 x _ {a}, 0. 3 x _ {m}\right), \tag {4}
$$
where,  $x_{a}$  is the appearance feature map output of appearance subnet 4 and  $x_{m}$  is the feature map output of the relu4_1 layer of the C3D network.
3.2.5 Decoder
The decoder is a 3D CNN used to generate the desired stylized video clip. It comprises of 3D Conv, ReLU, ReflectionPad3d and Upsample layers. The input of the decoder is the output of the Entangle Subnet, and the outputs of the Appearance Subnets are added to the decoder as skip connections by concatenation along the channel dimension of the decoder's appropriate intermediate feature maps.
We propose a multi-phase training for learning the different subnetworks of the proposed network. In each phase, we train one or more subnetworks while keeping the other subnetworks fixed. We first detail the different losses used, and then explain the different phases of the training.
3.3.1 Losses used for training
We use different combinations of losses during different phases of training. In this section we give details of the losses, and specify which losses are used in which phases in the respective sections describing the training phases below.
The first loss we use is the standard  $\ell_2$  reconstruction loss given by,
$$
\mathcal {L} _ {\text {r e c o n s t r u c t i o n}} = \left\| O _ {r} - I _ {c} \right\| _ {2} \tag {5}
$$
where  $O_{r}$  and  $I_{c}$  are the output reconstructed and input content clips respectively. This loss minimizes the  $\ell_{2}$  distance between the input and the reconstructed content clip in the RGB pixel space.
Second loss we use is the  $\ell_2$  loss on the output feature maps of the intermediate layers of the VGG-19 network, i.e.,
$$
\mathcal {L} _ {\text {a p p e a r a n c e}} = \sum_ {i} \| \phi_ {i} \left(O _ {r}\right) - \phi_ {i} \left(I _ {c}\right) \| _ {2} \tag {6}
$$
where,  $\phi_{i}$  is the feature map of layer  $i$  of the ImageNet pretrained VGG-19 network. The intermediate layers of VGG-19 used, indexed by  $i$ , are relu1_1, relu2_1, relu3_1 and relu4_1.
Most of the state-of-the-art style transfer methods [4, 11, 18] used features from the VGG-19 network pretrained on the ImageNet dataset. Since the VGG-19 features of individual frames only contain the appearance information, we use loss based on VGG-19 features to help disentangle appearance
(a) First phase: Autoencoder training
(b) Second phase: Appearance subnetworks training
(c) Third phase: Entangle subnetwork training
Figure 2: The networks and settings for the different phases of training. The light gray subnetworks are fixed, while the different dark gray subnetworks are trained during the respective phases.
features from motion features in the C3D encoder. Further, we use content and style losses similar to AdaIN [11]. The content loss is given by
$$
\mathcal {L} _ {\text {c o n t e n t}} = \left\| \phi_ {i} (O) - \phi_ {i} \left(I _ {c}\right) \right\| _ {2}, \quad i = \operatorname {r e l u 4} _ {-} 1, \tag {7}
$$
where  $O$  and  $I_{c}$  is the output stylized and input content clip respectively and the style loss is given by
$$
\mathcal {L} _ {\text {s t y l e}} = \sum_ {i} \| \mu \left(\phi_ {i} (O)\right) - \mu \left(\phi_ {i} \left(I _ {s}\right)\right) \| _ {2} + \| \sigma \left(\phi_ {i} (O)\right) - \sigma \left(\phi_ {i} \left(I _ {s}\right)\right) \| _ {2}, \tag {8}
$$
where  $i$  indexes over the relu1_1, relu2_1, relu3_1, relu4_1 layers of the VGG-19 network,  $I_{s}$  is the input style clip, and  $\mu (\cdot)$  and  $\sigma (\cdot)$  are the mean and standard deviation of the feature maps.
We also use temporal loss, which is inspired by Gao et al. [6], and is given by
$$
\mathcal {L} _ {\text {t e m p o r a l}} = \sum_ {t = 2} ^ {L} \sum_ {j} \frac {1}{D _ {t}} \| \left(O _ {t} - W _ {t} \left(O _ {t - 1}\right)\right) _ {j} - \left(I _ {c, t} - W _ {t} \left(I _ {c, t - 1}\right)\right) _ {Y} \| ^ {2}, \tag {9}
$$
where  $j \in \{R, G, B\}$  is each of the RGB channels of the image, and  $Y$  is the relative luminance channel.  $O_{t-1}$  and  $O_t$  are the previous and the current stylized output frame,  $I_{c,t-1}$  and  $I_{c,t}$  are the previous and the current content input frame,  $D_t = H \times W$  where  $H, W$  are the height and width, respectively, of the input/output frames, and  $W_t$  is the forward optical flow function, which is computed using FlowNet2.0 [12]. The purpose of using this loss is to maintain similar optical flow characteristics in each channel of the output frames, as that in the luminance channel of the input frame, and achieve motion consistency between frames.
Further, we propose an intra-clip loss given by,
$$
\mathcal {L} _ {\text {i n t r a}} = \sum_ {i} \sum_ {t = 2} ^ {L} \left| \left| \mu \left(\phi_ {i} \left(O _ {t}\right)\right) - \mu \left(\phi_ {i} \left(O _ {t - 1}\right)\right) \right| \right| _ {2} + \sum_ {i} \sum_ {t = 2} ^ {L} \left| \left| \sigma \left(\phi_ {i} \left(O _ {t}\right) - \sigma \left(\phi_ {i} \left(O _ {t - 1}\right)\right) \right| \right| _ {2}, \right. \tag {10}
$$
where,  $i$  indexes over the layers of the VGG-19 network as above,  $O$  is the stylized clip,  $t$  represents the frame number in the stylized clip,  $\mu (\cdot)$  and  $\sigma (\cdot)$  are the mean and standard deviation of the feature map  $\phi_i(\cdot)$ . This loss encourages the feature statistics to be similar between successive frames of the stylized clip. By ensuring that, we mitigate the effect of the frames being stylized differently, even when the style image is the same, and avoid jerky changes in color and intensity.
3.3.2 First phase: Autoencoder training
The first phase of training is to train the C3D encoder and decoder for video reconstruction task. The original C3D network was proposed for a discrete classification task, while the task here is of dense
prediction of stylized output video. To have a suitable initialization for the encoder and decoder, we pretrain them with reconstruction loss, Eq. 5, on our training dataset of videos, with the network shown in Figure 2a.
3.3.3 Second phase: Appearance subnetworks training
In the second phase, we keep parameters of all appearance subnetworks as well as the decoder trainable, while keeping the parameters of the encoder fixed. The network architecture is shown in Figure 2b. We pass the features from different intermediate layers of the encoder to the corresponding appearance subnetworks for extracting out the multi-scale appearance information from the entangled feature maps of the encoder. The output of the appearance subnetwork 4 is passed as the input to the decoder while outputs of other appearance subnetworks are passed as skip connections, concatenated channel-wise, to the decoder. We initialize the decoder with the weights learned from the first phase for the second phase. We use ImageNet pretrained VGG-19 intermediate feature maps reconstruction loss, Eq. 6, during this phase. This ensures that the appearance subnetworks together with the decoder capture the appearance part of the input video clip, by reconstructing the VGG-19 feature maps. Since this loss is applied independently for each frame of the input clip, it mainly captures the appearance information and discounts the motion information, leading to disentanglement of the appearance and motion features by the appearance subnetworks.