ColoristaNet for Photorealistic Video Style Transfer
Authors: Xiaowen Qiu
Ruize Xu
Boan He
Weifeng Ge
Venue: Submitted to ICLR 2023 (2023)
Keywords: Photorealistic Video Style Transfer
ColoristaNet
Style Removal-and-Reconstruction
Decoupled Instance Normalziation

Abstract: Photorealistic style transfer aims to transfer the artistic style of an image onto an input image or video while keeping photorealism. In this paper, we think it's the summary statistics matching scheme in existing algorithms that leads to unrealistic stylization. To avoid employing the popular Gram loss, we propose a self-supervised style transfer framework, which contains a style removal part and a style restoration part. The style removal network removes the original image styles, and the style restoration network recovers image styles in a supervised manner. Meanwhile, to address the problems in current feature transformation methods, we propose decoupled instance normalization to decompose feature transformation into style whitening and restylization. It works quite well in ColoristaNet and can transfer image styles efficiently while keeping photorealism. To ensure temporal coherency, we also incorporate optical flow methods and ConvLSTM to embed contextual information. Experiments demonstrates that ColoristaNet can achieve better stylization effects when compared with state-of-the-art algorithms.

Under review as a conference paper at ICLR 2023
COLORISTANET FOR PHOTOREALISTIC VIDEO STYLE TRANSFER
Anonymous authors Paper under double-blind review
ABSTRACT
Photorealistic style transfer aims to transfer the artistic style of an image onto an input image or video while keeping photorealism. In this paper, we think it's the summary statistics matching scheme in existing algorithms that leads to unrealistic stylization. To avoid employing the popular Gram loss, we propose a self-supervised style transfer framework, which contains style removal part and style restoration part. The style removal network removes the original image styles, and the style restoration network recovers image styles in supervised manner. Meanwhile, to address the problems in current feature transformation methods, we propose docouple instance normalization to decompose feature transformation into style whitening and restylization. works quite well in ColoristaNet and can transfer image styles efficiently while keeping photorealism. To ensure temporal coherency. we also incorporate optical flow methods and ConvLSTM to embed contextual information. Experiments demonstrates that ColoristaNet can achieve better stylization effects when compared with state-of-the arr al algorithms.
INTRODUCTION
Nowadays rapid development of video-capture devices has made videos become a mainstream information carrier (Hansen 2004). People usually post videos accompanied with different color styles on social media (Kopf' et 2012 Xu et al. (014)) to share daily life, express different emotions, and get more exposures Yan et al.) 2016 Zabaleta Bertalmío 202I). Thus, photorealistic video style transfer or automatic color styliza lizatintion becomes popular in many mobile devices. Different from artistic style transfer (Gatys et al., 2016 Huang Belongie 2017), photorealistic video style transfer or automatic color stylization needs to replace color styles in original videos with one or multiple reference images and keep the outputs maintain "photorealism". The photorealism style transfer refers to that stylization results should look like real photos taken from cameras without any spatial distortions or unrealistic artifacts. Moreover, algorithms need to run in realtime.
Several popular algorithms have been proposed to conduct photorealistic style transfer for single image. DeepPhoto (Luan et al., (017) incorporated semantic segmentation masks to guide style transfer and utilized photorealism regularization term reduce spatial distortions. PhotoWCT (Li eet al.] 2018) exploited whitening and coloring transforms (WCT' (Li et al.) 2017ch) to conduct arbitrary style transfer and used photorealistic smoothing to remove spatially inconsistent stylization. WCT2 YYoo et al. 2019) proposed a wavelet corrected transfer based on WCT to preserve structural information while stylizing images the same time. PhotoNAS (An al., 2020 proposed a neural architecture search framework for photorealistic style transfer and achieved impressive results.
Although these algorithms can conduct style transfers many scenarios, their stylization results still contain unpleasant artifacts or look unreal, and some algorithms need additional supports. In Figure (a), given a content image which contains tree in autumn and a style reference, previous state-of- the-art algorithm WCT (Yoo et al. 2019) will generate synthesized images with obvious structural artifacts. Besides, these algorithms conduct style transfer by matching the summary statistics of content features with style references completely, which will lead to unrealistic stylization in Figure (b). For photorealistic style transfer in videos, there are only very few existing algorithms that only perform style transfer with constraints. MVStylizer (Li et al. 2020) need good stylization initilaization at the first frame and Xia's 's's method (Xia et al.) 2021) incorporates additional semantic masks for each frame in videos. These problems limit these methods usage in many real applications.

---

Under review as a conference paper at ICLR 2023
[a.3] Content Image
(a.2) Style Image
(b.1) Content Image
1b.21 Style Image
[c.1]
Video Stylization Example
Encoder
Style
Censtriints stpliaatin
istokiztionn semontic mosks and et.
[a.3) WCT
(c.2) Stylization Framowork
(a) Structural Artifacts
Unrealistic Stylization
[c) Style Transfer with Constraints
Figure 1: Illustration of unsolved problems in photorealistic style transfer. From left right: (a) Previous state-of-the-eee algorithm WCT2 (Yoo et al., (2019) generates stylization results with obvious structural artifacts. (b) The stylization result produced by WCT YYoo et al. (211)) looks painterly and slightly unreal. (c) Video stylization algorithms need additional inputs, such as good stylization
initialization (Li al. 2020 or semantic masks (Xia al. 202Ib, to guide style transfer.
In this paper, we aim to solve the problems listed above in photorealistic video style transfer. Different from previous algorithms which match summary statistics of content images to that of style references through whitening and coloring transformation (Li et al. 2018), adaptive instance normalization (An al., 2020) and the Gram loss (Luan al., 2017), we propose style removal and restoration framework in a sulf-supervised manner to conduct arbitrary style transfer while keeping photorealism. Our motivation that during photorealistic style transfer, we can remove the style of image content without destroying image structures, we can recover its original style by using the content image both as style reference and stylization target. According to our experiences, artifacts produced by PCotoWCT (Li et al. [2018). WCT (Yoo et al. (011)). and PhotoNAS (An et al. (2220) come from two parts: (1) the Gram loss; (2) whitening and coloring transformation (WCT (Li et al. 2017ch). In method, we avoid using the Gram loss and train networks with the content loss only (Gatys et al.) 2016). We improve the summary statistics matching scheme with decoupled instance normalization which can remove original image styles and add new styles for inputs without hurting image structures. Meanwhile, decoupled instance normalization does not match styles of reference images completely and avoid unrealistic stylization in Figure (b). To keep temporal consistency in videos, we exploit optical flow estimation (Teed & Deng 2020) and ConvLSTM (Shi al. 2015a) conduct consecutively style transfer. We summarize our contributions as follows:
In this paper, we propose a novel photorealistic video style transfer network called ColoristaNet, which can conduct color style transfer in videos without introducing painterly spatial distortions and inconsistent flickering artifacts. We put many videos in the supplementary material to compare with other state-of-the-ee algorithms.
We propose decoupled instance normalization which works together with CTnVLSTM Shi et al. 2015a) to implement structure-preservinn preserving and temporally consistent feature transformation. The decoupled instance normalization decomposes style transfer into feature whitening and stylization, which can avoid unrealistic style transfer.
ColoristaNet can adapt color styles in videos consecutively with multiple different style references and runs faster than most of recent algorithms. Qualitative results and user study show that our method outperforms other of algorithms in making balance between good stylization results and photorealism. Besides, we also conduct extensive ablation studies whose results demonstrate the effectiveness of different modules and designs in ColoristaNet clearly.
2

---

Under review as a conference paper at ICLR 2023
2 PRELIMINARIES AND MOTIVATIONS
Content Image
Coloristaliet
Style Removal Result
Coloristallet
Content Image
ColoristaNiee
Restoration Result
Stilization Result
Image
(a) Training Pipeline ColoristaNet
Test Pipeline ColoristaNet
The training and test pipeline of ColoristaNet. During training, ColoristaNet is firstly exploited to replace the color style of content image with style reference. Then, another Col- oristaNet restores the style of the synthesized image by using the original content input style reference. During testing, the style restoration ColoristaNet can conduct style transfer efficiently.
Figure 2:
Neural style transfer algorithms (aayys et al. 2016 Li et al. 2017b) have achieved great success in creating artistic images of high perceptual quality. Using neural representations to separate and recombine content and style of arbitrary images is widely investigated and adopted by researchers (Li et al. 2217aa Zhu et al. 201 JJonnonn et al. 2016 [edig al. 2017). In Gatys' paper, the Gram matrix consists of the correlations between different filter responses and describe the overall image style, and features in deeper layers is thought capturing the high-level content in term of objects and their arrangement. Then style transfer problems can be solved by matching summary statistics of content inputs to that of style references. However, although such framework works quite well for artistic style transfer, it is not suitable for photorealistic style transfer. Because matching the summary statistics of content images with arbitrary style references will generate unpleasant artifacts distortions. Photorealistic style transfer algorithms, such as DeepPhoto (Luan et al. 2017). PCT (Li et al. 2018). WCT2 YYoo et al. 2019)) and PhotoNAS (An et 2020), focus on eliminating artifacts distortions with additional smoothing term or other regularization terms. As shown in Figure] structural artifacts and unrealistic stylization is hard to be avoided.
In this paper, we hold an assumption that previous methods, it's the summary statisties matching scheme in learning objectives and feature transformation modules that lead to structural artifacts or unrealistic stylization. That means the Gram loss (Gatys et al. (016). AdaIN (Huang & Belongie 2017]] and WCT (Li et al. 2018) are problematic photorealistic style transfer. To address these issues, we propose ColoristaNer with: (I) a self-supervised style transfer framework that avoids employing the Gram loss during training; and (2) novel feature transformation module to substitute AdaIN or WCT to perform summary statistics matching. For the ceff-superviee style transfer framework, as shown in Figure 2 if we can remove the style of an image without hurting its structure, the style restoration problem becomes a fully supervised one. The reason why our idea works is because in the photorealistic setting, image structures are shared and unchanged during style transfer. The benefits of our self-supervised learning scheme come from two folds: (I) We avoid employing the Gram loss or other regularization loss functions that will result structural artifacts or blur effects; (2) Our learning targets are real photos which can ensure that the stylization results make a good balance between stylization and photorealism. We discuss more details in Appendix B.
To address the problems brought by AdaIN and WCT, ColoristaNet incorporates novel feature transformation module called decoupled instance normalization (DecoupleIN) to match the styles of content images with that of reference images. DecoupledIN is inspired by AdaIN, and decompose the feature transformation into a style whitening step and a restylization step. This decomposition avoids forcing the feature statistics of content images to match that of style images directly, and show impressive results. In addition, as we conduct video style transfer, we need to keep temporal coherency in consecutive frames. So employ optical flow methods to estimate pixel locations in a next frame and propagate style information through ConVLSTM (Shi et al.) 2015a) as shown in We give more explanations about the design of ColoristaNet in Appendix B.2
Figure 3
3

---

Under review as a conference paper at ICLR 2023
3 METHOD
Ios Is
Tes Is,
Ics Is
Iea Is
Ics Is.
VegNet
VggNett
VagNiot
VagNNtt
VggNet
DacoupiedIN
DecoupiedIN
DacouplaciNN
DacoupiediN
DecouplddNN
Decodert
Decodart
Docodert
Decodart
Decodar1
Content Loss!
Content Loss
Content Loss
Content Loss
Content Loss
Jc Ics
Ic
Te Ice
Ic
VagNet
VggNett
VggNNtt
VegNet
VggNiet
Flow Est
Flow Est
Flow Est
Flow Est
DecoupiecIN 2
DecoplediN 2
DecxupladIN 2
DecouplediN
DacoupledIN
CTMLLSTM
ConVLSTTV
ConVLSTIM
ConVLSTM
CanVLSTM
Decoder2
warpping
Decoder2
warpping
Decoder2
warpping
Decoder2
warpping
Decoder
Time Step
Time Step
Time Step 3
Time Step
Time Step 5
Figure 3: Illustration of the training pipeline of ColoristaNet. There are five content frames and five style references for a video clip. For each frame, it firstly passes through style transfer network to conduct style removal and then go through another style transfer network for style restoration. In style restoration, features from different time steps are connected with a Convl. ConvL.TT unit. A flow estimation network (RAFT (Teed & Deng, 2020) with fixed parameters) predicts optical flow between two adjacent frames to warp the hidden states of ConvLSTM for movement compensation. Note that parameters of style removal and restoration networks at different time steps are shared.
3.1 OVERVIEW OF THE PROPOSED METHOO
In this section, we introduce the training pipeline of ColoristaNet. For the test pipeline, we give more details in Appendix A.2 As discussed in the previous sections, we exploits a style removal ColoristaNet and a style restoration ColoristaNet to conduct end-to-end training (as shown in Figure 3) Given a video clip with five image frames, we send them together with randomly selected style references to style transfer network to conduct style removal. Then these style removal results used as content inputs and the original inputs are used as style images to perform style restoration with another style transfer network. Both style transfer networks in similar structure except that the style restoration incorporates RAFT (Teed & Deng 2020) and ConvLSTM (Shi et al. (2015a) to keep temporal coherency. For the learning objective, they are trained with two content loss respectively, and their learning targets are original video frames. During the test, the second style transfer network exploited conduct style transfer without additional constraints.
3.2 STYLE TRANSFER NETWORK
Figure 3 shows the architecture of two style transfer networks roughly. A style transfer network consists of VGG-19 encoder (Sioonyan & Zisserman (2014), four decoupled instance normalization modules, four ConvLSTY units Shi et al. (2015b), and decoder to generate final output. Note that, in style removal, ConvLSTM units are removed, since it doesn't need context information. Given an input image pair (Ic,, Is,) ), feature maps at "convl_1" "cone2_11" VGG-19 network Pegg with frozen parameters are extracted: These multiscale features pass through four decoupled instance normalization modules (shown in Figure map feature statistics of the content image to match that of its style reference, In style restoration, RAFT is exploited to estimate pixel locations in adjacent frames and ConvL.STM is responsible to incorporate contextual information.
"conv3__" and "conv4_1" of
4

---

Under review as a conference paper at ICLR 2023
Then, U-net --nnneeeoneeergee et al. (2015) style decoder fuses information across different scales and generate stylization results. We give very detailed structure configurations in Appendix A.2
3.3 DECOUPLED INSTANCE NORMALIZATION
Matching feature statisties through feature trans- formation has been proven powerful in both artistic style transfer and photorealistic style transfer (AdaIN (Huang & Belongie (2117) and WCT (Li et al., 2017c)). But directly apply- ing AdaIN will hurt some subtle image details, and WCT often leads unrealistic stylization. Here we propose decoupled instance normal- ization (DecoupledIN) that decomposes the fea- ture transformation into feature whitening and stylization. Figure 4 shows the DecoupledIN module. Given a content input fc..: and style input f at i-th layer, we remove the style of fc.,t Equation |
Nomalization
Conv
Conv
mean
std
Conv
(a) Decoupled Instance Normalization
firstly, and then send the whitened result fc., and style input fs.,i into 3 x 3 convolutional layer with 2c filters conduct AdaIN (c is the number of input feature channel). Finally, we reduce the feature chan- nels of stylized features It, to be the same with inputs. The overall process of DecoupledIN can be described with the following equations:
Content Style
AdaiN
DecoupleIN
(b) Visual Comparison AdaIN and DecoupledIN
Figure 4: Conceptual illustration of decoupled in- stance normalization and visual comparison with AdaIN (Huang Belongic, (017).
Whitening: -Cc,, -p
a (fcin)
Transform i' Conv (l Conv (fse.),
(I)
SC.. (JCrN)
Stylization It.; = a (S%..)
+ 1 (f
(FC).)
where and calculate the mean and standard deviation for each feature channel respectively. Figure 4 indicates that style transfer through DecoupledIN will generate better stylization results without hurting image details when compared with using AdaIN. We attribute this that directly changing the content feature statistics will make some neurons work out of their working range and remove detailed structures. When we separate feature whitening from stylization, transformation more smoothly and get better results. We disucss DecoupledIN and conduct more ablations in Appendix B.2 investigate our assumptions. Experiments indicates that more whitening will lead to better stylization effects.
conduct feature
3.4 LOss
Unlike previous works which employed multiple loss functions including content loss, temporal loss, Gram loss, and etc, simply employ content loss to constrain the structure of the stylization results to be the same as content inputs (for style removal) and guide ColoristaNet to generate photorealistic videos like real world videos (for style restoration). Given a stylization result Ig and an input image Ic, we use the feature maps at conv4_1 1 layer of VGG-19 to calculate the content loss. The two ColoristaNets trained end-to-end without sharing parameters. For video clip with N frames, the learning objective becomes:
C Σ Ccontent (IG,,, + XCcontent (I.2,)
(2)
where Io., and IG2, are generated images of style removal and restoration networks respectively. In experiments, set À and get impressive results.
5

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTS
4.1 EXPERIMENTAL DETAILS
We conduct extensive experiments indicates the effectiveness of the proposed method. Due the page length limit, we put implementation details into the appendix, including datasets, evaluation protocols and training and test settings (in Appendix [A). We also put more results and discussions in the appendix part to compare with late-of-theearr methods and investigate the effectiivnness of different designs and modules in ColoristaNet.
4.2
QUALITATIVE COMPARISON
Figure 5: Multiple color stylization a set of consecutive frames with ColoristaNet. Stylization targets are visualized in the right up corner of first, fourth, and seventh columns. From left right, color styles of input frames changed smoothly without any painterly distortions or flickering artifacts. More stylization videos are packed the supplementary material.
Photorealism. In photorealistic style transfer, the most important principle is to change color styles of images without resulting distortions artifacts. Meanwhile, photorealism means that stylization results should looks like taken from cameras. Figureo compares ColoristaNet with state-of state-of-the -oe------the-- art methods in terms of photorealism. When zooming to check details of local image patches, no obvious unpleasant artifacts in results produced by ColoristaNet.
Coherency, For video stylization, maintaining temporal coherency vital in many real applications. In figure from left to right are stylization results at different time steps. There are three different style images large style variations for each video. Gaussian smoothing function is exploited to smooth the stylization vectors among frames with different style references. Stylized video frames change smoothly without any flickers. Nor there is any structural inconsistency between stylized video frames. This proves that ColoristaNet is able to produce temporally coherent videos even when there are multiple style references. Please refer to our supplementary material for videos generated by ColoristaNet (with single style reference and multiple style reference).

---

Under review as a conference paper at ICLR 2023
(a) Content and Style
(b) WCT2
(c) PhotoNAS
(d) Ours
Figure Visual comparison with popular algorithms including WCT2 Yoo al. 2019), Pho- toNAS (An et al., 2020) and our ColoristaNet. Images in the first column and their top right corners are content images and their style counterparts. Each row contains stylization results rendered by different styles. While ColoristaNet generates photorealistic results, other methods either damage image structure or produce over-stylization.
4.3 QUANTITATIVE COMPARISON
Quantitative Metrics. Following PhotoNAS (An et al. 2020), we evaluate the stylization results with SSIM, LPIPS (Zhang et al. 2018), content loss (Gatys et al. 2016) and Gram loss (Gatys et al. 2016). SSIM. LPIPS (Zhang et al. 2018), content loss (Gatys al. 2016) employed to measure the streture similarity between two images, and gram loss is used to calculate the style distance two images. We randomly select 100 video/style image pairs evaluate the performance of state-of-art
7

---

Under review as a conference paper at ICLR 2023
algorithms. We use official codes and models provided by PhotoWCT (Li al. 2018). WCT (Yoo
et al. 2019) and PhotoNAS (An et al. 2020 to generate stylization results. In Table I the proposed ColoristaNet achieves better scores in SSIM, LPIPS, the content loss, and comparable gram loss. It
means our ColoristaNet has a stronger ability in preserving structural details.
Table 1:
Quantitative comparison with state-of-art photorealistic style transfer algorithms. Higher SSIM scores mean test images are more similar to input contents with fine details. Lower LPIPS scores mean higher perceptual similarities between stylization results and content images.
Method
WCT2
PbatoNAS
SSIM
0.548
0.555
0,737
0.785
LPIPS}
0.464
0,391
0.326
0.223
Content Loss..
11.035
7.256
4.351
2.427
Gram Loss.
0.00025
0.00032
0.00028
0.00026
Table 2: User study results.
Table 3: Computing-timee comparison.
Method
PhotoWCT(full)
WCT
PhotoNAS
Ours
Image Size
PhatoWCT(full)
WCT-
PhotoNAS
Ours
Photorealism 2.20
2.77
2.67
3.57
600x 360
0.408s
2.13s
0.124s
0.068s
Stylization
2.03
2.70
2.17
3.40
x54
0.495s
2.15s
0.175s
0.111s
Coherency
2,00
3.37
3.27
3.70
x280x720
0.811s
4.267s
0.383s
0.225s
Overall quality
3.03
1920x
1.430s
4.362s
connected with this project to evaluate the quality of the stylization results. The testers are asked to take the quality of details in the image and the stylization effects into consideration during their evaluation. Images are rated on a scale of 1-5, where higher scores stand for better stylization results. In total, we collected 900 responses (30 videos x 30 users) for each kind of method. As shown in Table 2 our approach perform best for photorealism. stylization effects, temporal coherency and overall quality.
User Study. We recruited 30 testers who
Inference Speed. To demonstrate the efficiency of our method, we compare the inference speed of the different models. We use GeForce RTX 3090 GPU to test all state-of-the-art methods. We randomly select different videos, each which contains running time of each method. Furthermore, we also test the speed of these algorithms in different resolutions. As shown in Table our ColoristaNet much faster than other methods.
frames, and compute the average
Content Style
without RAFT or ConvLSTM
ColoristaNet
Time Step
Time Step
Time step
Time step
Time Step 2 without ConvLSTM
Time Step 3
(a) without RAFT
(b)
Figure Investigation of the effectiveness of RAFT and ConyLSTM in ColoristaNet.
4.4 ABLATION STUDY
Whether optical flow estimation and ConvLSTM necessary? To check whether RAFT together with ConvLSTM are necessary for video style transfer, we remove them to test the performance of ColoristaNet. From Figure7 we can find that when remove RAFT optical flow estimation,
8

---

Under review as a conference paper at ICLR 2023
ColoristaNet will generate noticeable artifacts. When remove all ConvLSTM units, some images details disappeared. It shows both RAFT and ConvLSTM are necessary.
(a) Content and Style
(b) AdaiN
(c) DecoupledIN
(d) Content and Style
(e) AdaIN
(f) DecoupledIN
Figure 8: Visual comparison of the results produced by AdaIN and DecoupledIN.
Whether D is important to get good results? To verify decoupledIN's ability in preserving subtle image details, we replace all decoupledIN modules in ColoristaNet with AdaIN. As shown Figure 8. AdaIN powerful in generating synthesised images with good stylization effects and keeping photorealism. However, if we zoom in to see more details, we find that AdaIN will overwrite some image details and produce images that seem be "overexposed".
(a) Content & Style
(b) ColoristaNet without "conv3 1" and "conv4_1"
(c) ColoristaNet without "conv4 branch
(d) ColoristaNet
branches
Figure 9: Results of ablation on the multi-scale feature fusion scheme of ColoristaNet. (d) is complete multi-scale feature fusion scheme, (c) removes the feature transformation the "conv4 1" stage, (b) further removes the feature transformation at the "conv3. 1" stage on the basis of (c).
Whether multi-scale features are useful the decoder? We remove feature maps produced by "conu4 1" and "conu3 1" stages and compare their results with ColoristaNet in Figure 9 From left to right, we list the stylization results produced by ColoristaNet with two, three, and four different feature scales. These results demonstrate that multi-scale features can generate better stylization results. More results in Appendix i.4diidicssss more feature scales will lead to less artifacts.
CONCLUSIONS, LIMITATIONS AND FUTURE WORKS
In this paper, we propose ColoristaNet, photorealistic video style transfer network, along with a samoval-and-restorer training pipeline. ColoristaNet learns color stylization self-supervised manner and generates stylization results looking as if taken from cameras. Two important components of ColoristaNet are decoupled instance normalization and ConvLSTM units that can implement arbitrary style transfer while preserving salient image structure and temporal stylization coherency. Experiments show that results of ColoristaNet have strong visual quality and high artistic value.
Limitation. However, since ColoristaNet makes a balance between stylization results and photore- alism, it doesn 't completely embed styles of reference images into input videos like that other methods. According to our experiments, in many scenarios, enforcing stylization results to have the exactly same style of references will make images looks like paintings not real photos. Besides, we exploit RAFF to compute optical flow and many other modules with heavy computational cost to conduct style transfer. This makes ColoristaNet unable to run in realtime on single NVIDIA GeForce RTX 3090 GPU, which deserves thorough analysis in the future. In the future, will try to work on this problem to alleviate these limitations. Our techniques can used in many social media platforms and we haven't found obvious negative societal impact of ColoristaNet.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Jie An, Haoyi Xiong, Jun Huan, and Jiebo Luo. Ultrafast photorealistic style transfer via neural architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 10443-10450, 2020.
Alexander G. Anderson, Cory P. Berg, Daniel P. Mossing, and Bruno A. Olshausen. Deepmovie: Using optical flow and deep neural networks to stylize movies. arXiv: Computer Vision and Patter Recognition, 2016.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamners Advances in neural information processing systems, 33:1 33:1877-1901, 202. 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings the IEEE/CVF International Conference on Computer Vision, pp. 9650-9660, 2021.
Dongdong Chen, Jing Liao, Lu Yuan, Nenghai Yu, and Gang Hua. Coherent online video style transfer. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1105-1114, 2017.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Xinghao Chen, Yiman Zhang, Yunhe Wang, Han Shu, Chunjing Xu, and Chang Xu. Optical flow distillation: Towards efficient and stable video style transfer. In European Conference Computer Vision, pp. 614-630. Springer, 2020b.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 04800, 2018.
Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas. Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 1766, 2015.
Wei Gao, Yijun Li, Yihang Yin, and Ming- King-Hsuan Yang. Fast video multi-style transfer. workshop applications of computer vision, 2020.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks, In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2414 2423, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
Mark Boris Nicola Hansen. New philosophy for new media. MIT press, 2004.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In Proceedings of the IEEE international conference computer vision, pp. 1501-1510, 2017.
Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. arxiv: Computer Vision and Pattern Recognition, 2016.
10

---

Under review as a conference paper at ICLR 2023
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pp. 694-711. Springer, 2016.
Stephan Kopf, Stefan Wilk, and Wolfgang Effelsberg. Bringing videos to social media. In 2012 IEEE International Conference on Multimedia and Expo, pp. 681-686. IEEE, 2012.
Hildegard Kuchne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In 2011 International conference on computer vision, pp. 2556-2563. IEEE, 2011.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2017.
Ang Li, Chunpeng Wu, Yiran Chen, and Bin Ni. Mvstylizer: an efficient edge-assisted video photore- alistic style transfer system for mobile phones. In Proceedings of the Twenty-Firss International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing, pp. 31-40, 2020.
Shaohua Li, Xinxing Xu, Liqiang Nie, and Tat-Seng Chua. Laplacian-steerel neural style transfer. In Proceedings of the 25th ACM international conference on Multimedia, pp. 1716 1724, 201 7a.
Yanghao Li, Naiyan Wang, Jiaying Liu, and Xiaodi Hou. Demystifying neural style transfer. arXiv 2017b.
preprint /
Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming- Hsuan Yang. Universal style transfer via feature transforms. Advances in neural information processing systems, 30, 2017c
Yijun Li, Ming- Yu Liu, Xueting Li, Ming- Hsuan Yang. and Jan Kautz. A closed-fome solution to photorealistic image stylization. In Proceedings of the European Conference Computer Vision (ECCV), pp. 453-468, 2018.
Fujun Luan, Sylvain Paris, Eli Shechtman, and Kavita Bala. Deep photo style transfer. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4990-4998, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett (eds.). Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019. URL /tt.pppppp/ppppe......rs........
9015- 1015-pyyorch-an--11 hrgprptceee...aaaaaa.eeeeeeeeeee litiaarrrrnnnn.bbrr...... librar. pdf
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234 241. Springer, 2015.
Xingjian Shi, Zhourong Chen, Hao Wang, Dit- Yan Yeung, Wai kin Wong, and Wang chun Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting. arXiv: Computer Vision and Pattern Recognition, 2015a.
Xingjian Shi, Zhourong Chen, Hao Wang, Dit- Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting. Advances in neural information processing systems, 28, 2015b.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Zachary Teed and Jia Deng. Raft: Recurrent all-s field transforms for optical flow. In European conference on computer vision, pp. 402-419. Springer, 2020.
11

---

Under review as a conference paper at ICLR 2023
VideoNet. Videvonet. https://wee net.
Xide Xia, Tianfan Xue, Wei-sheng Lai, Zheng Sun, Abby Chang. Brian Kulis, and Jiawen Chen. Real-time localized photorealistic video style transfer. In Proceedings of the IEEE/CVE Winter Conference on Applications Computer Vision, pp. 1089-1098, 2021.
Jie Xu, Mihaela Van Der Schaar, Jiangchuan Liu, and Haitao Li. Forecasting popularity of videos using social media. IEEE Journal of Selected Topics in Signal Processing, 9(2):300-343, 2014.
Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang. Youtube-vor vos: large-scale video object segmentation benchmark. arXiv preprint arXiv: 1809,0330. 2018.
Zhicheng Yan, Hao Zhang, Baoyuan Wang, Sylvain Paris, and Yizhou Yu. Automatic photo adjust- ment using deep neural networks. ACM Transactions on Graphics (TOG), 35(2):1-15, 2016.
Jaejun Yoo, Youngjung Uh, Sanghyuk Chun, Byeongkyu Kang, and Jung- Woo Ha. Photorealistic style transfer via wavelet transforms. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9036-9045, 2019.
YouTube. Youtube. https:/ www. youtube.com.
Itziar Zabaleta and Marcelo Bertalmío. Photorealistic style transfer for video. Signal Processing: Image Communication, 95: 16240, 2021.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017a.
Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. Advances in neural information processing systems, 30, 2017b.
12

---

Under review as a conference paper at ICLR 2023
Appendix
A IMPLEMENTATION
A.1 SETTINGS
Dataset. ColoristaNet is trained by videos collected from HMDB Kuehne et al. 2011 and Youtube VOS (Xu et al., 2018) as content inputs and images collected from Internet as style images. We have around 6000 videos and around 24000 images in our training set. During test, we download videos from Youtube YYu' Tube),,,, Videvo (VideoNet) and other websites, and select style images that can generate pleasant stylization results.
Evaluation. To check the color stylization ability of ColoristaNet, we conduct style transfer on various high-definition videos with different style images as shown in Figure 5 We compare with photorealistic image style transfer algorithms,, such as WCT (Yoo et al. 2019). PhotoNAS (An al. 2020p. We directly conduct style transfer frame by frame using their official codes. We can't compare with photorealistic video style transfer algorithms MVStylizer (Li et al. (222)) and Xia et al. (Xia et al. 2021) and other methods, since their source codes are not released. We conduct both quantitative comparison and user study to evaluate different algorithms.
Training. All experiments are implemented with PyTorch (Paszke et al. 2019). and run on two NVIDIA GeForce RTX 3090 GPUs with 24 GB RAM. Parameters of VGG-19 (Simonyan & Zis serman 2014) and RAFT (Teed & Deng 2020 are initialized with pretrained weights and are fixed during training We use the SGD optimizer with momentum 0.9 and basic Icarning rate le-5 to optimize parameters ColoristaNet in 80 epochs. We schedule our learning rate a bit different from common practise. The learning rate the first epoch is set to 0.01, and then is decreased to le-5 in the next five epochs. After that we apply cosine decay for the rest epochs. Images are cropped and resized into the resolution 128 128 during training.
Io
Ja
Top
To
10.
Siyl
Is
Is.
Is.
VigNuat
Veghle
VggNti
Viggtwi
Viggha
Flow
DacoppddNN
DacoupiddN
DecoppaellN
DecnuplediN
DecouptentN
ConwLSTM
ConVLSTM
STM
Decoder 2
warpsing
Decoder
warppina
Docoder 2
Docoder
Decoder 2
Time Step
Time
Time Step
Time N-1
Time Step N
Figure 10: Inference with style transfer network. For video clip, each frame is paired with style reference to pass VGG-19 feature extractor, decoupled instance normalization, ConvLSTM and the decoder to generate the final output. There may be multiple style references the same time. ColoristaNet can conduct arbitrary style on videos of any length according to users' preference.
A.2 CONFIGURATIONS OF STYLE TRANSFER NETWORK
13

---

Under review as a conference paper at ICLR 2023
Veg canu?
condl
Corn,
Sicl Conw,
Seš Comy
Daocupadd
Dascup welk
Decoupledh
DecoupledN
GeL
CeLU
GeLU
Conv,
Gonw,
Cond.
CoM
Gomt STM
Cond5STM
GILD
GeLU
Ses Coru,
Interplation
(c) Downsample
Basch Norm
Block
(b) Upsample Block
Add
Conailbcck
Conoutlics
2kd Com, 32
Oxeilooo
Condltck
Conilina, 26a
3x3 Conk,
Oratkoc b4
Ceneteck, 13
Conastos, 251
Cont 512
Batch Norm
(x) Detailed
Style
Concatenation
Figure 11: Detailed illustration of style transfer network. There are five shared blocks in style trans- fer network: (b) Upsample block, (c) Downsample Block, (d) Convolution Block. (e) Concatenation Block, and (f) Output Block. Structures of style transfer networks in style removal and restoration slightly different. In style removal network, there ConvLSTMM (Shi et al. 20155) units.
Figure 11 gives detailed architectures of ColoristaNet. The style transfer network has a U-net (Ron- neberger et al., 2015) style encoder-decoder structure. During training, networks shared by different time steps. The number of convolution filters in each block is denoted with c. "Conv Block, 64" stands for the filter number convolution block is 64 and the kernel size of convolutional layers is 3 x 3. Other blocks have the similar notations. The style removal network and style restoration network have a similar architecture, except that there are no ConvLSTM (Shi et al.) 20155) units across style removal networks.
B COLORISTANET: DETAILS, ADDITIONS AND ABLATIONS
To generate photorealistic stylization results which look like taken from cameras, ColoristaNet exploits set of training strategies and micro designs to avoid structural distortions and painterly artifacts, including self-supervised learning, decoupled instance normalization, flow estimation network (RAFT) (Teed & Deng 2020), ConVLSTM (Shi et al. 2015a), multi-scale feature learning and etc. Here, we give more details, additional experiments and ablations on the designs and choices of these different modules.
B. SELF-SUPERVISED LEARNING IN COLORISTANET
Self-supervised learning obtains supervisory signals from unlabeled data itself and thus leverages underlying structure and common representation in data, which has achieved great success in natural language processing (Devlin al. 2018 Brown et 2020) and computer vision (Chen al. 2020a Caron et al. 202T He et al. 22022). In unpaired image-to-image translation, CycleGAN (Zhu et al. 2017a) introduces consecutive amage-to-image translations in cycles to ensure content consistencies in images with the help of generative adversarial networks (Goodfellow et al. 2014). In this paper, we exploit the supervipe learning strategy for photorealistic style transfer. Our motivation here is that if the style of an image can be replaced without hurting subtle structures arbitrarily, its style can be recovered by using itself as the style reference. Such an assumption holds photorealistic style transfer because the underlying image structure remains unchanged during color style transfer. So in our training pipeline, we apply two ColoristaNets that are responsible for style removal and style restoration respectively. When styles of images removed, the style transfer restoration task becomes a fully supervised one. As shown Figure [2] such a strategy has been proven to be the key of the success of NolorsitaNet.
14

---

Under review as a conference paper at ICLR 2023
(a) Content and Style
(b) Without self self-superviiion
(c) Style removal net
(d) Style restoration net
Figure 12: Stylization results comparison of different style transfer networks: (a) Content and style images, (b) A single ColoristaNet simply trained with content loss (without the removal-restoration pipeline). (c) The style removal ColoristaNet in the proposed training pipeline, (d) The style removal ColoristaNet in the proposed training pipeline.
Style Removal. During style removal, style transfer network overwrites styles of images with given style references through decoupled instance normalization at different feature resolutions. Conducting style transfer without resulting any distortions or artifacts the most important principle in style removal. We implement style removal with decoupled instance normalization and give detailed introduction and analysis in Appendix
B.2, Mcanwhile, to ensure image structures to be full preserved, we simply exploit the content loss (aayys al. 2016) to enforce the structure consistency. As is known that style loss (Gatys et al.) (016)) can conduct mixed transfers of both texture and color, will result in painterly distortions of image structures. Figure [22 visualizes stylization results of the style removal network. It can be found that image structures of content images are preserved and original image styles are partly removed. Its stylization effects are unsatisfactory and look just like paintings.
Style Restoration. Following the assumption described above, the style restoration network just takes the style removal result in as the content input and use the original image as the style reference to conduct style transfer. Thus, the stylization results can be expected to be the same with the original content image. In this way, the content image, the style reference and the stylization result can be defined clearly. We use the style transfer network in the same architecture with that in the style removal part to conduct supervised style transfer. Again, no style loss is exploited enforce good stylization. Surprisingly, we can find that such a strategy can help to generate good stylization results without obvious distortions or artifacts. We attribute this to that the linear feature transformations in DecoupledIN and the self-supervised learning framework without style loss. Besides, we find that training the style removal and restoration networks jointly will make the training much more stable and converge faster. To validate the effectiveness of the two stage style transfer framework, we conduct an ablation study to check whether the self-supervised learning strategy is necessary. We train ColoristaNet with a randomly selected style reference perform parameter learning without self-supervision. Figure [2 shows the stylization results produced by ColoristaNet without self- supervision, the style removal and restoration ColoristaNets respectively. It can be found that without the sule-super learning strategy, ColoristaNet can not transfer the style of a reference image to the target successfully.
15

---

Under review as a conference paper at ICLR 2023
(a) Content and Slyle
AdalN
(c) DecoupledIN
Figure 13: Visual comparison of the results produced by AdalN and DecoupledIN.
B.2 DECOUPLE INSTANCE NORMALIZATION
Transferring styles of images arbitrarily through feature transformations has been widely accepted in style transfer (Huang & Belongie 2017: Li et al. 2017c 2018: Yoo et al. (019). In (Huang & Belongie| 2017), Huang et al. proposed adaptive instance normalization (AdalN) to aligns the mean and variance of the content features with those of the style features to achieve arbitrarily artistic style transfer without iterative optimization process. el el al. (Li et al., (017c) conducted universal style transfer through the whitening and coloring transforms (WCT) to match feature covariance of the content image to a given style image. PhotoWCT (Li et al. 2018) brought the idea of conducting style transfer through feature transformations from WCT (Li et al. 2017c) to perform photorealistic style transfer. However, the feature transformation in WCT is nonlinear, and thus lead to distortions in image structures obviously. PhotoWCT (Li et al. 2018) exploits photorealistic smoothing term to ensure local consistency in pixel intensities. But in many cases, there are obvious artifacts produced by WCT that can be removed by the smoothing term. Besides, the smoothing term also make images blurry and lost some subtle local contrast. Yoo et al. (Yoo et al.] 2019) proposed a wavelet corrected transfer based on whitening and coloring transforms (WCT²) to avoid introducing additional masks and unfavorable blurry artifacts. Although these methods are powerful and can conduct arbitrary style transfer, they can't 't achieve photorealistic stylization results. According to our analysis, the feature whitening and coloring transforms in WCT (Li et al.) 2017c) can match the feature correlation of content images to that of style references exactly, but will change the local feature contrast which maintains image structures. AdaIN has very nice properties that won't It damage local feature contrast but can not match the feature statistics of style references compactly and thus lead to inferior stylization effects.
In this paper, we aim to achieve photorcalistic stylization while still make synthesized images look like taken from cameras. As shown in Figure 13 AdaIN is powerful to generate synthesized images with good stylization effects and keeping photorealistic. However, if we zoom in to see more details, we find that AdaIN will overwrite some image details and produce images that seem to be "overexposed". We attribute this to that AdaIN aligns feature statistics of content images with their style references in a shared feature space, which may weaken the local structural details. To alleviate this issue and achieve better stylization results, the decoupled instance normalization aims decompose the feature transformation into style whitening step and a restylization step. Thus, we need to insert convolutional layer after the style whitening operation. The motivation of designing the 3x3 convolutional layer with 2c filters is to expand the feature channels to recover the missing conduct
information during the whitening step. Then in the subsequent stylization step, style transfer in higher dimension to benefit from the rich information provided by more feature channels. At last, fuse the transformation with simple convolution operation. From Figure L3 and Figure 16) we can find that DecoupledIN can preserve more image details and achieve better stylization results.
Multiple Style Whitening in DecoupledIN. The structure of DecoupledIN modules with more whitening operations demonstrated in figure 14 and figure 1.5. From Figure 16 can find
16

---

Under review as a conference paper at ICLR 2023
Nomalization
3x3 Conv
Nomalization
3x3 Conv
3x3 Conv
std
fs
3x3 Conv
Figure 14: Visualisation of the module structure of DecoupledIN with two style whitening.
fe
Nomalization
3x3 Conv
Nomalization
3x3 Conv
3x3 Conv
mean
std
3x3 Conv
Figure 15: Visualisation of the module structure of DecoupledIN with three style whitening.
(a) Content Style
(b)
(c) DecoupledIN
(d) Whitening
(e) Whitening
Figure 16: Visual comparison of AdaIN and DecoupledIN with different style whitening strategics. From left right: (a)content and style images, (b) AdaIN, (c) DecoupledIN with style whitening, (d) DecoupledIN with two style whitening, and (e) DecoupledIN with three style whitening. The stylization effects is enhanced from left to right gradually.
17

---

Under review as a conference paper at ICLR 2023
that althrough DecoupledIN gets better stylization effects compared with AdaIN, the style of the original content input is remained and combined with the style reference to generate mixed stylization results. To remove the original image style more completely, DecoupledIN and visulaize the results in Figure [6] We conduct the style whitening in D by 1, and 3 times respectively. When looking at these image carefully, whitening will lead to better stylization effects. Since the influence of different feature transformations hard to be distinguished, please zoom in to check the color variations carefully.
conduct multiple style whitening in
find that more style
B.3 FLOW ESTIMATION AND CONVLSTM FOR TEMPORAL CONSISTENCY MAINTENANCE
Optical flow estimation methods (Dosovitskiy et al. 2015 Ig et al., 2016 Teed & Deng. 2020) and ConVLSTM (Shi al. 2015a have been exploited in many video style transfer literature Chen et al., 2017 2020b Anderson et al. 2016 Gao et al. 2020) to ensure temporal coherency. In ColoristaNet, exploit RAFT (Teed & Deng 2020) to track pixels across different frames and use ConvLSTM to pass the contextual information to adjacent frames. With RAFT and ConvLSTM, we implement temporal consistent feature transformations and thus can conduct photorealistic video style transfer with high temporal coherency. While as it has been shown in the supplementary videos, simply transferring styles of each frame in videos with image-based photorealistic style transfer algorithms can still generate stylized videos in good quality. It's because that consecutive frames in videos are temporally coherent and we use a Gaussian smoothing function smooth the style vectors during switches between different style references. However, if we zoom in to check the details of other state-of- -f-t-e--thherr art methods, such as PhotoWCT (Li et al. 2018), WCT2 (Yoo et al. 2019) and PhotoNAS (An et al., 2020). we can find that there are many structural distortions and flickering artifacts. Since these previous methods can't 't ensure the photorealism
most of their stylization results, the temporal inconsistency problem are not in the first place to be solved.
Without RAFT.
To check whether RAFT can maintain temporal consisteney during style transfer, conduct ablation study by removing RAFT from ColoristaNet and still use ConvLSTM to fuse information across different frames. Figure 17 shows that without RAFT, ColoristaNet generates many stylization results with obvious artifacts. It's because without the guidance of optical flow, the information propagation between adjacent frames becomes incorrect. In fact, in some cases, RAFT failed to generate accurate correspondences between image pixels and thus lead to bad stylization effects.
Without COnVLSTM. To check whether ConvI.STM can incorporate contextual information among adjacent frames, we remove the ConvLSTM units to test the performance of ColoristaNet. Since ConvLSTM aims to propagate information across different frames, if there no COnvLSTM, RAFT should be removed too. That means we just conduct vidco style transfer with basee algorithms as that of PCotoWCT (Li al., 2018). WCT2 (Yoo et al., 2019) and PhotoNAS (An al. 2020). Figure 18 shows two different kinds of scenarios that ConvLSTM is important for ColoristaNet. In the first case, if there is no ConvLSTM module, there will have some flickering artifacts. And in the second case, ColoristaNet with ConvLSTM can ensure style consistency when there are multiple styles to be transferred.
B.4 EMPLOYMENT OF MULTI-SCALE FEATURES
Since deep convolutional neural networks have multi-scale hierarchical structures and can encode images into features with different semantic levels, it's popular to conduct style transfer with the benefits of multi-scale features (Li et al. 2018 Yoo et al., 2019). PhotoWCT (Li et al.) 2018) passed the lost spatial information to the decoder in hicrarchical manner to facilitate reconstructing fine details for photorealistic image synthesis. WCT2 (Yoo et al. (211)) inherited the U-net style structures from PhotoWCT and achieved impressive results. PhotoNAS (An et al. 2020) searched the best architecture for photorealistic style transfer and found that the multi- stylization strategy is important for style transfer in high quality. We adopt the multi-scale architecture as previous methods to conduct style transfer. We conduct ablations on the multi-scale feature fusion scheme of ColoristaNet. In the style transfer network, there are feature transformations at four different resolutions. Denote the feature transformations in ColoristaNet from lower resolutions to higher resolutions with "conel 1", "conv2 1", "conv3 1" and "conv4 1" respectively. We remove the feature transformation at the "conu4 1" stage at first, and remove other feature transformations
18

---

Under review as a conference paper at ICLR 2023
Content & Style
without flow estimation
ColoristaNet
Content & Style
without flow estimation
ColoristaNet
Content & Style
without flow estimation
ColoristaNet
Time Step
Time Step
Time Step 3
Time Step
Time Step
Figure 17: Investigation of the effectiveness of flow estimation network in ColoristaNet. We test ColoristaNet in three different scenarios see that without flow estimation network (RAFT (Teed & Deng 2020)), ColoristaNet will generate many artifacts.
19

---

Under review as a conference paper at ICLR 2023
Content
Style
without LSTM
ColoristaNet
Content Style
without LSTM
ColoristaNet
Time Step 1
Time Step 2
Time Step 3
Time Step
Time Step
Figure 18: Investigation of the effectiveness of ConvLSTM ColoristaNet.
20

---

Under review as a conference paper at ICLR 2023
(a) Content Style
(b) ColoristaNet without "conv3 1" and "conv4_ branches
(c) ColoristaNet without the "conv4 1 branch
(d) ColoristaNet
Figure 19: Results of ablation the multi-scale feature fusion scheme of ColoristaNet. (d) is complete multi-scale feature fusion scheme, (c) removes the feature transformation the "conv4_1" stage, (b) further removes the feature transformation at the "conv3_1 __ stage the basis of (c).
consecutively to check the stylization results. As shown In Figure 19 from left right, we list the stylization results produced by ColoristaNet with two, three, and four different feature scales. It can be found that with more different scales, we get better stylization results with less artifacts.
C EXPERIMENTAL RESULTS WITH FURTHER ANALYSIS
C.I COMPARISON WITH STATE-OF-THE-ARE METHODS
Previous state-of-fe the-hhheat art photorealistic style transfer algorithms, such MVStylizer (Li al. 2020
and Xia's video style transfer (Xia et al.) 2021). don't 't make their codes publicly available, so we can just compare with image-based photorealistic style transfer algorithms. We compare with four state-of-the-art algorithms including PhotoWCT (Li et al. 2018), WCT (Yoo et al. 2019) and PhotoNAS (An al. 2020). We simply conduct style transfer on image frames independently and put the stylization results together to generate stylized videos. To avoid drastic style change when there are multiple style references, Gaussian smoothing function exploited smooth the stylization vectors among frames with different style references. The Gaussian kernel size is 20. That means for every 20 video frames, their corresponding stylization vectors are smoothed. Figure 20 Figure 21 Figure 27 Figure 28 and Figure 29 give the visual comparison with state-of-thh-arr algorithms. All previous state-of- sate--o-the-arr artifacts. While ColoristaNet generates videos that look like taken from cameras without any blur, distortions and artifacts. Although ColoristaNet does not transfer the colors style references to the targets completely, the stylization results still very competitive.
the-hrtt art algorithms will produce observable blur, structural distortions and flickering
21

---

Under review as a conference paper at ICLR 2023
(a) Content and Style
(b) WCT
(c) PhotoNAS
(d) Ours
Figure 20: Visual comparison with popular algorithms including WCT2 Yoo et al. 2019). Pho-
tCWCT (Li al. 2018) PhotoNAS (An et 2020) and our ColoristaNet.
22

---

Under review as a conference paper at ICLR 2023
Content Style
WCT2
PhotoWCT
PhotoNAS
ColoristaNet
Figure 21: Additional Visual comparison with popular algorithms including WCT2 Yoo et al. 2019 (nn et al. 2020 and our ColoristaNet.
PhotoWCT (Li et al. 2018 PhotoNAS
Contradictions between Stylization and Photorealism. Separating styles images from their contents is very difficult since there is no concrete definition of the content and style of an image. In Gatys' paper (Gatys et al. 2016), they stated features in higher layer of deep convolutional neural networks capture the high-levi contents interms of objects and their arrangements and can be thought as the content representation. However, as stated in Lapstyle (Li et al. (017a), the low-level features of the content image is substituted by that of style references and thus will generate unpleasing artifacts. Photorealistic style transfer is some kind of color transfer that need to get the color styles of references image while keeps image content unchanged. In fact, it's hard to achieve balance between stylization and photorealism. If one want to get stylization result that matches the style of another image exactly, the color style and the local textures need to be modified. Modifications in local textures will lead to painterly artifacts easily as shown in Figure 26
C.2 COMPUTATIONAL COST ANALYSIS
In Table 4 list the computational cost of different sections of ColoristaNet. There three main components in ColoristaNet, including VGG-19 feature backbone (Simonyan & Zisserman 2014), RAFT (Teed & Deng 2020) and the decoder. There 3.5M, 1M and 72.2M parameters in them respectively. The VGG-19 feature backbone has only 3.5M parameters because we only use feature maps at the first four convolutional stages. There are relative less parameters in these convolutional layers. RAFT has IM parameters and the inference speed is very slow compared with other components. The decoder has 72.2M parameters because there four of CSnVLSTM units, DecoupledIN modules and many other convolutional operations. It can be found that RAFT takes the most time in NolorsitaNet. But when there is RAFT, ColoristaNet will generate obvious artifacts in many cases shown in Figure 17 So how to replace RAFT with some other faster methods that can track pixels across different frames an important problem for future study.
C.3 STYLIZATION EFFECTS CONTROL
ColoristaNet achieves a balance between good stylization effects and photorealism. We try to improve and control the stylization effects by incorporating style loss during training, adding
23

---

Under review as a conference paper at ICLR 2023
Modules
Parameters (M)
Inference Speed (ms)
VGG-19 backbone
3.5
5
RAFT
209
the decoder
72.2
14
Table 4: Comparison of efficiency between the VGG-19 feature backbone, RAFT and the decoder.
stylization factor in DecoupledIN, and exploiting several DecoupledIN consecutively. We visualize the stylization results in Figure 22 and give some analysis in the following subsections.
Incorporation the Style Loss. We add stylization loss in both style removal and restoration networks. We have try different weights for the style loss, but these training don't 't converge.
Stylization Factor. Since DecoupledIN can conduct style transfer by matching feature statistics of content images to that of style references, We add stylization factor À E [0, 1] in the matching process as follows
Whitening: Scc., fcas-14 (fcra)
a (Sc..))
Transform: SC Conv (Fc,s) Conv (Jsa)):
Reweighting Varnew, Meannew Xo (J%..) + X) (Je.)), Au (S%..) +11 (11 1)) (FC.))
Sč.. - (FCcs) o (f'...
Stylization Ir,i Varnew
Meannews
where μ and a calculate the mean and standard deviation for each feature channel respectively, and Var and Meannew are the reweighted mean and standard deviation vectors. During the training, we add the stylization factor in the style restoration network, and get the À value randomly using the sampling strategy based uniform distribution. During the inference, set À from 0.2 to 1.0. Figure 22 visualizes the stylization results. can be found that ColoristaNet can control the stylization effects explicitly.
Multiple Decoupled Instance Normalization. In some cases, the style of the original content input is still contained in stylization results produced by ColoristaNet. Here aim to get stronger stylization results by matching the style of reference images much more closely through adding more DecoupledIN modules consecutively. Figure 23 indicates that when we apply more DecoupledIN modules consecutively in ColoristaNet, the stylization effects will become stronger.
D FAILURE CASES AND FURTHER DISCUSSIONS
To examine the robustness of ColoristaNet, we conduct experiments a plenty of videos to find the failure cases of ColoristaNet. After huge amount of experiments, we find some failure cases and summarize them into three categories, including (1) temporal inconsistency in small regions; (2) under-stylization; (3) over-stylization (caused by extra whitening steps). As in figure 24 we can see that in the original frames, the background color in the red boxes remain unchanged, while in the stylized frames, the coloring is shifting between bright green and dark green. Such inconsistency caused by properties of DecoupledIN. As the foreground object moves dramatically, the mean and standard deviation of content feature channels will change, which will influence stylizations of both the moving object and the remaining stationary pixels. As such inconsistency can fixed by ConvL LSM combined with flow estimation in most cases, they become identifiable when flow estimation fails. Among several hundreds of test videos we have examined, we only spotted subtle temporal inconsistencies like Figure 24 As discussed in Appendix B.3 we attribute this to that the input content videos are temporally coherent so in most cases the temporal inconsistency is not obvious. Compared with sfate-of-the- -he--rt methods, (in (igure[5), This is expected since we prioritize photorealism. the future, we aim to enhance the stylization performance of our model while still maintaining high level of photorealism. When
methods sometimes suffer under-sty stylizait
24

---

Under review as a conference paper at ICLR 2023
(a) Content and Style
(b) X 0.2
(c) A
(d) A= 0.6
(e) A=
x0)x10
Figure 22: Illustration of the gradual change in stylisation effect when different style control factors À are taken. ColoristaNet can control the stylization effect through the stylization factor explicitly.
25

---

Under review as a conference paper at ICLR 2023
(a) Content and Style
(b) AdaiN
(c) DecoupledIN
(d) DecoupledIN x
(e) DecoupledIN
Figure 23: Visualising the results of using AdaIN, decoupledIN, two consecutive decoupledIN and three consecutive decoupledIN.
(a) Frame 32
(b) Frame 40
(c) Frame 128
(d) Frame 132
Figure 24: Illustration of failure cases produced by ColoristaNet: Temporal Inconsistency.
(a) Content and Style
(b) WCT2
(c) PhotoNAS
(d) Ours
Figure 25: Illustration of failure cases produced by ColoristaNet: Under-stylization.
26

---

Under review as a conference paper at ICLR 2023
(a) Content and Style
(b) Normalize once
(c) Normalize twice (d) Normalize three times
Figure 26: Illustration of failure cases produced by ColoristaNet: Over-stylization.
apply multiple whitening steps in our DecoupledIN module, it is possible that too much information erased, which makes the stylized video look misty (in Figure 26). This is trade off for stronger stylization effects.
27

---

Under review as a conference paper at ICLR 2023
(a) Content and Style
(b) WCT?
(c) PhotoNAS
(d) Ours
Figure 27: Additional comparison between ColoristaNet and state-of-the-an methods.
28

---

Under review as a conference paper at ICLR 2023
(a) Content and Style
(b) WCT*
(c) PhotoNAS
(d) Ours
Figure 28: Additional comparison between ColoristaNet and state-of-the-ard methods.
29

---

Under review as a conference paper at ICLR 2023
(a) Content and Style
[b) WCT2
(c) PhotoNAS
(d) Ours
Figure 29: Additional comparison between ColoristaNet and state-of-the -he-trrtrtt methods.
30


Reviews:
[{"review_id": "zk1nvKMBalq", "reviewer": "Reviewer_AmBd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a novel method for photorealistic video style transfer that conducts color style transfer in videos without undesirable painterly spatial distortions and temporally inconsistent flickering artifacts. The proposed style removal-and-restoration framework, namely ColoristaNet, is capable of learning stylization in a self-supervised fashion. The authors propose to leverage docoupled instance normalization and ConvSLTM units to achieve arbitrary style transfer while keeping the photorealism and temporal conherency. Through a number of comparisons and ablation study, the authors demonstrate ColoristaNet's superior quality to previous state-of-the-art photorealistic style transfer algorithms.", "summary_of_the_review": "I am leaning towards acceptance. The paper is a technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.", "strength_and_weaknesses": "Strengths:\n\nThe paper is well written and the exposition is clear.\n- Inspired by AdaIN that applied adaptive affine transformations, the authors propose decoupled instance normalization (DecoupledIN) that uses linear transforms in both feature whitening and stylization, capable of conducting photorealistic style transfer effectively.\n- To maintain temporal consistency and avoid flickering artifacts, a novel architecture employing optical flow estimation and contextual information with ConvLSTM units is introduced to capture temporal dependencies.\n- Sufficient amount of qualitative and quantitative comparisons with prior work and ablation studies are conducted to show the efficiency and quality improvement of the proposed photorealistic style transfer framework.\n\nWeaknesses:\n\n- It seems there is no explicit control, i.e., hyperparameters, to balance the content and style in the stylization output. It is known that more stylized results typically tend to present painterly distortions instead of maintaining the salient image structure. Still, it would be nice to present such extensions or simply show a figure to display results with different level of stylization.\n- Show some failure cases. For example, what happen when optical flow estimation is inaccurate? What happen when content video and style exemplar does not share any similarity in high-level contents?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TH_JWOBvHdP", "reviewer": "Reviewer_K1SY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper thinks that the summary statistics matching scheme in existing algorithms leads to unrealistic stylization and it proposes a self-supervised style transfer framework to keep photorealism. This frame work contains a style removal part and style restoration part. Besides, it proposes decoupled instance normalization to decompose feature transformation into style whitening and restylization. And it conducts extensive experiments to validate the stylization effecrtiveness of the proposed algorithm.", "summary_of_the_review": "This work proposes a self-supervised style transfer framework to keep photorealism and decoupled instance normalization to decompose feature transformation into style whitening and restylization. Overall, the paper conducts a lot of experiments to research, which is great. But the designs of the framework and the algorithms lack novelty. And it also lacks the fair comparison with new state-of-the-art methods. ", "strength_and_weaknesses": "Strength:\n1. This work notice that a lot of previous algorithms will cause unpleasant artifacts or unrealistic results. And it propose a framework which remove style for content image at first and then restore style with the guidance of given style image.\n2. This paper conduct extensive experiments, and the visual resutls looks great.\n\nWeakness:\n1. From my point of view, almost all components in this work are exsiting models or algorithms proposed by previous work, which makes the novelty of this paper weak.\n2. The pipeline in Figure.3 looks complex and ugly. In fact, you can only draw two time step, the repeated others are meaningless for the illusration.\n3. the decoupled instantce normalization seems too simple as a main contribution.\n4. The paper compares their results with works before 2020, but there are a lot of works[1,2,3] for image style transfering in 2021 and 2022. For fair comparison, I think the results compared with latest methods should be given.\n5. There are some problems in writting of this paper, such as 'docouple' in abstract.\n\n[1] Deng, Yingying, et al. \"StyTr2: Image Style Transfer with Transformers.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n[2] Kim, Sunwoo, Soohyun Kim, and Seungryong Kim. \"Deep translation prior: Test-time training for photorealistic style transfer.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 1. 2022.\n[3] Wang, Pei, Yijun Li, and Nuno Vasconcelos. \"Rethinking and improving the robustness of image style transfer.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DfBCtIOVhln", "reviewer": "Reviewer_fibA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates video color style transfer problem, and proposes a self-supervised style transfer framework. The framework first stylizes the image (style removal) and then reconstruct the original image with the stylized image’s content features and the original image’s style features (style restoration). By doing so, the structure can be better preserved. This paper also proposes decoupled AdaIN for feature fusion and use ConvLSTM and optical flow for temporal consistency.", "summary_of_the_review": "This paper proposes an interesting idea of style restoration framework for style transfer, which improve the structure preservation. However, the baselines this paper compare do not well matches the topic of this paper. More appropriate baselines could be compared to make the evaluation fairer and more convincing. And the proposed decoupled AdaIN is less novel.\n\n", "strength_and_weaknesses": "**Strengths:**\n+ The idea of transforming a style transfer task to a style restoration task is interesting to me. This makes the unsupervised task becomes a self-supervised task, which makes the output’s structure less distorted. \n\n**Weaknesses:**\n+ I think the choice of the baseline methods for comparison is not very appropriate. The baselines in this paper (photoWCT, WCT2 and photoNAS) are focusing on improve the photorealism and (or) speed of the style tranfer method of WCT. WCT is known to have strong style effects and high computational cost. Therefore, the starting point of the baselines are to maintain the strong style effects while eliminate structure distortion. But this paper, to me, is more like a pure color transfer method (filter style transfer), and is based on a modified AdaIN, which is known to have less distortion and fast speed. Therefore, in speed comparison in Table 1, the proposed method achieves the fast speed since other baselines all use WCT operation. I recommend some baselines [1-3] that better match the topic of this paper (In [3], the backbone CNNMRF can be replaced with AdaIN). \n+ Another problem is the contribution of decoupled AdaIN. First, in Eq. (1), the whitening is to make the feature maps uncorrelated. However, this paper seems only to normalize the feature maps channel-wisely as in AdaIN rather than adjusting the inter-channel covariance. Therefore, why it is called whitening? Second, the decoupled AdaIN seems only add two conv before the addition and multiplication. The novelty is limited. From the visual results, I can hardly tell the difference between AdaIN and the proposed decoupled AdaIN in Fig .4, Fig. 8 and Fig. 13, which weakens the second claimed contribution.\n+ For the video part, the optical flow and LSTM are also common components in video style transfer.\n\n**Some small issues:**\n+ ’’conv1_1’’ could be revised to ``conv1_1''\n+ Fig. 6 is referenced before Fig. 5 \n\n[1] 2020 ECCV Filter Style Transfer between Photos\n\n[2] 2020 ECCV Joint Bilateral Learning for Real-time Universal Photorealistic Style Transfer\n\n[3] 2017 BMVC Photorealistic Style Transfer with Screened Poisson Equation\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject