Style Spectroscope: Improve Interpretability and Controllability through Fourier Analysis
Authors: Zhiyu Jin
Xuli Shen
Bin Li
Xiangyang Xue
Venue: Submitted to ICLR 2023 (2023)

Abstract: Universal style transfer (UST) infuses styles from arbitrary reference images into content images. Existing methods, while enjoying many practical successes, are unable of explaining experimental observations, including different performances of UST algorithms in preserving the spatial structure of content images. In addition, methods are limited to cumbersome global controls on stylization, so that they require additional spatial masks for desired stylization. In this work, we provide a systematic Fourier analysis on a general framework for UST. We present an equivalent form of the framework in the frequency domain. The form implies that existing algorithms treat all frequency components and pixels of feature maps equally, except for the zero-frequency component. We connect Fourier amplitude and phase with Gram matrices and a content reconstruction loss in style transfer, respectively. Based on such equivalence and connections, we can thus interpret different structure preservation behaviors between algorithms with Fourier phase. Given the interpretations we have, we propose two manipulations in practice for structure preservation and desired stylization. Both qualitative and quantitative experiments demonstrate the competitive performance of our method against the state-of-the-art methods. We also conduct experiments to demonstrate (1) the abovementioned equivalence, (2) the interpretability based on Fourier amplitude and phase and (3) the controllability associated with frequency components.

Under review as a conference paper at ICLR 2023
STYLE SPECTROSCOPE: IMPROVE INTERPRETABILITY AND CONTROLLABILITY THROUGH FOURIER ANALY-
SIS
Anonymous authors Paper under double-bling blind review
ABSTRACT
Universal style transfer (UST) infuses styles from arbitrary reference images into content images. Existing methods, while enjoying many practical successes, are unable of explaining experimental observations, including different performances of UST algorithms in preserving the spatial structure of content images. In ad- dition, methods are limited to cumbersome global controls on stylization, so that they require additional spatial masks for desired stylization. In this work, pro- vide a systematic Fourier analysis on general framework for UST. We present equivalent form of the framework in the frequency domain. The form implies that existing algorithms treat all frequency components and pixels of feature maps equally, except for the zero-frequency component. We connect Fourier amplitude and phase with Gram matrices and content reconstruction loss in style transfer, respectively. Based on such equivalence and connections, we can thus interpret different structure preservation behaviors between algorithms with Fourier phase. Given the interpretations we have, we propose two manipulations in practice for structure preservation and desired stylization. Both qualitative and quantitative experiments demonstrate the competitive performance of our method against the of-te--f-th-attt methods. We also conduct experiments demonstrate (1) the abovementioned equivalence, (2) the interpretability based on Fourier amplitude and phase and (3) the controllability associated with frequency components.
INTRODUCTION
Style transfer deals with the problem of synthesizing an image which has the style characteristics from style image and the content representation from content image. The seminal work (Gatys et al. 2016) uses Gram matrices of feature maps to model style characteristics and optimizes re- construction losses between the reference images and stylized images iteratively. For the purpose of gaining vivid visual styles and less computation cost, more trained feed-forward networks are proposed (Wang et al., 2020 Li & Wand, 2016; Johnson al.) [2016] [Sheng et al., 2018 Li al. Sty] Dumoulin et al. 2017). Recent works focus on arbitrary style transfer (Park & Lee
2017b
2019
Chen al. 2021 a Chandran al. 2021). artistic style (Chen al. 202 Liu al.
2021 Chen et al. 202 lc). These works capture limited types of style and cannot well generalize to
unseen style images (Hong et al. 2021).
To obtain the generalization ability for arbitrary style images, many methods are proposed for the task of universal style transfer (UST). Essentially, the main challenge UST to properly extract the style characteristics from style images and transfer them onto content images without any prior knowledge of target style. The representative methods of UST consider various notions of style characteristics. For example, AdaIN (Huang & Belongie, 2017) aligns the channel-wise means and variances of feature maps between content images and style images, and WCT (Li et al.) 22117a) further matches up the covariance matrices of feature maps by means of whitening and coloring processes, leading to more expressive colors and intensive stylization.
While these approaches and their derivative works show impressive performances styliza- tion, they behave differently in preserving the structure of content images. It is observed that the operations performed by AdaIN can do better in structure preservation of content images while those

---

Under review as a conference paper at ICLR 2023
of WCT might introduce structural artifacts and distortions. Many follow-up works focus on alle- viating the problem of WCT (Li et al. 2018 Chiu & Gurari 2022 Yoo et al. 2019), but seldom analytically and systematically explain what makes the difference. In the field of UST, need an analytical theory to bridge algorithms with experimental phenomena for better interpretability. potentially leading to better stylization controls. To this end, we resort to apply Fourier transform for deep analysis, aiming to find new equivalence in frequency domain and bring new interpretations and practical manipulations to existing style transfer methods.
In this work, we first revisit a framework by (Li et al. 2017a) which unifies several well-known UST methods. Based on the framework, we derive an equivalent form for in the frequency domain, which has the same simplicity with its original form in the spatial domain. Accordingly, the derived result demonstrates that these UST methods perform a uniform transformation in the frequency domain except for the origin. Furthermore, these UST methods transform frequency components (excluding the vero-frequency component) and spatial pixels of feature maps an identical manner. Thus, these UST methods perform manipulations on the whole frequency domain instead of specific subsets of frequencies (either high frequencies or low frequencies).
Secondly, through the lens of the Fourier transform, we further explore the relation of Fourier phase and amplitude with key notions in style transfer, and then we present new interpretations based on hand, we prove that content reconstruction loss between two
the equivalence have. On feature maps reaches a local minimum when they have identical Fourier phase, which implies that Fourier phase of feature maps contributes to the structure of stylized results. On the other hand, prove that the Fourier amplitude of feature maps determines the diagonals of their Gram matrices, which implies that Fourier amplitude contributes Next, We demonstrate that AdaIN does preserve the Fourier phase of feature maps while WCT does not, and we interpret the different behaviors between the UST methods in structure preservation as a consequence of their different treatment with the Fourier phase of feature maps.
the intensity information of stylized images.
Thirdly, based on the connection we establish between style transfer and Fourier transfer, we propose two manipulations on the frequency components of feature maps: 1) a phase replacement operation to keep phase of feature maps unchanged during stylization for better structure preservation. 2) feature combination operation to assign different weights to different frequency components of fea- ture maps for desired stylization. We then conduct extensive experiments to validate their efficacy.
The contributions of this paper are summarized as follows:
Equivalence We present theoretically equivalent form for several state-of-the-art UST methods the frequency domain and reveal their effects on frequencies. We conduct corresponding experiments to validate the equivalence.
Interpretability We connect Fourier amplitude and phase with key notions in style transfer and present new interpretations on different behaviors of UST methods. The interpretations are validated by experiments.
Controllability We propose two manipulations for structure preservation and desired styl- ization. We have experimental validation for their efficacy and controllability.
PRELIMINARIES
2.1 FOURIER TRANSFORM
The Fourier transform has been widely used for the analysis of the frequency components in signals, including images and feature maps in the shallow lavers of neural networks. Given an image F e xC the discrete Fourier transform (DFT) (Jenkins & Desai 1986) decomposes it into unique representation F E CCXHXW in the frequency domain as follows:
H-1W-1
Fu,
Σ h=0
Σ
F 52 (o }²=
where (h, w) and (u, v) are the indices on the spatial dimensions and the frequency dimensions, respectively. Since images and feature maps consist of multiple channels, we here apply the Fourier
2

---

Under review as a conference paper at ICLR 2023
transform upon each channel separately and omit the explicit notation of channels. Each frequency component Fu, can be decomposed into amplitude |Fu,el and phase LFu
Fu,nl = - (Ru..) (Iu.,)? LF,, attan22nn Ru,.);
(2)
where Ru and Zu are the real part and the imaginary part of the complex frequency component Fu,e, respectively. Intuitively, cluding the contrast or the difference between the brightest and darkest peaks of images, and phase crucially determines the spatial content of images (Gonzalez & Woods 2008).
for images, amplitude carries much of intensity information, in-
2.2 A UNIFIED FRAMEWORK FOR UNIVERSAL STYLE TRANSFER
To better demonstrate the connection between style transfer and the Fourier transform, a unified framework of different style transfer methods is preferred serve as bridge. Given a content image _e and a style image 1*, we denote the feature maps of _e and 1 as F E RCXH"*X x and F E RC x W" respectively, where C denotes the number of channels, Hº (H*) the height and We (W*) the width. For a majority of UST methods, their goal is to transform the content image feature maps Fe into stylized feature maps Frs,, whose first-order and second-order statistics are aligned with those of the style image feature maps Fe, Accordingly, their methods mainly depend on the corresponding channel-wise mean vectors μe, ,u* E IRC and the covariance matrices Σ E" E RCxC of F and FS, respectively.
A framework is proposed in (Lu et al. 2019) for unifying several well-known methods under the same umbrella. Specifically, each pixel Fi of F is first centralized by subtracting the mean vector n where h and w are indices spatial dimensions. Then the framework linearly transforms F with the transformation matrix T E RCCC and re-centers Fi by adding the mean vector μr8 of the style. Each pixel FRee E R of stylized feature maps can be represented as follows:
(3) where the transformation matrix T has multiple forms based on variety of configurations of differ- ent methods. We here demonstrate the relation between the unified framework and several methods in details.
Fh = T (Fh..u ~") + n²,
AdaIN Adaptive Instance Normalization (AdaIN) (Huang & Belongie 2017). the trans- formation matrix T = diag (2*) /diag (2"), where diag (2) denotes the diagonal ma- trix of given matrix Y and / denotes the element wise division. Because of the character- isties of diagonal matrices, only the means and variances within each single feature channel of Fes are matched up to those of F*, iignoring the correlation between channels.
2. WCT Instead of shifting single set of intra-channel statisties, (Li et al. 2017a) pro- poses a Whitening and Coloring Transform (WCT) that focuses further on the align- ment of covariance matrices. Similar with AdaIN, the transformation matrix for WCT is 7 = ( (E') "leading to well-aligned second-order statistics.
3. LiT While WCT generates stylized images more expressively, it is still computa- tionally expensive because of the high dimensions of feature maps in neural networks. (Li al. 2019) proposes LinearWCT use wiightee neural networks to model the linear transformation T by optimizing the Gram loss, known as widely- style reconstruction objective function:
T arg min
pesTTpTT P*FSTT 1|:
T
where Pes pespesT is the Gram matrix for pes and || [| denotes the squared Frobenius norm of the differences between given matrices.
4. OPT Similarly, (Lu et al. 2019) proposes OptimalWCT to derive a closed-form solution for T without the help of optimization process:
T= (29)~ (E))))))))))))) + (29)-
(5)
Their method reaches the theoretical local minimum for the content loss L. ges [| which is widely-used in style transfer (Huang & Belongie 2017 Gatys et al. 2016 Lu al. 2019) for structure preservation of content images.
F
3

---

Under review as a conference paper at ICLR 2023
3 METHOD
In this section, we first show an equivalent form of the framework in the frequency domain. this way, all the methods based the framework in Section 2.2 can be interpreted effecting on the frequency domain. We further connect amplitude and phase with existing concepts in the context of style transfer, and explain why WCT might preserve the structure of content images. Finally, we propose two operations for better structure preservation and desired stylization.
3.1 THE EQUIVALENT FORM OF THE FRAMEWORK IN THE FREQUENCY DOMAIN
We theoretically analyze the unified framework from the angle of 2-D DFT. We denote the DFT CCCH xw where C is the set of complex numbers. According to the unified we can derive each complex frequency component Fa.e as:
of FeE as Fes framework in Eq.
13
Hº
W
# W
F
F
Σ
[
1 p= 525
h=0
h=0
W
T
Σ
e 125 (o #
+1pe Tu
e (u
h=0
==
h=0
w==
H'
u uu=va v= 0;
HOWo
if u = == 0; else
F a,
else
0,
WH"",,,
if - 0;
H'W", HoWo TFc 0,01
F6.0.
if u = U= 0;
TF 6,00
else
else
(6) where u and are indices upon the frequency dimensions, and F and F are the DFTs of Fe and
Fs, respectively. According to the Fourier transform in Eq. 1 Fo,o = - -1 Σ FA (=W)) Fō.o when u = e = 0. Therefore, in the frequency domain, style transfer methods based on the unified framework are simple linear transformations on J except for the zero-frequency component Fo,0, which is replaced with the re-scaled zero- aero-frequency component of f
HWp. Thus, we have n,v = H = ( (HWw))
From (6. we find that each individual frequency component (excluding the gero-frequency com- ponent) has an identical linear transformation with pixels on the feature maps. In this way, there is no entanglement between different frequencies in the process of style transfer. Thus, it is feasible to treat and manipulate each frequency component of Fes as an individual for practical usage. There- fore, we justify the claim that mainstream methods in Section 2.2 for UST are not sole transfer on specific subsets of frequencies (either high frequencies or low frequencies), but essentially on the whole frequency domain.
3.2 CONNECTIONS AND INTERPRETATIONS AMPLITUDE AND PHISE
To better bridge style transfer with the Fourier transform, we connect phase and amplitude with a reconstruction loss and the Gram matrix in style transfer, respectively.
Phase and the content loss We here demonstrate the relation between phase and the content loss, which widely serves as construction loss for optimizing the differences of spatial arrangement between stylized images _ and content images I Given their feature maps Fes, F e xxC Fr,, corresponding DFTs Jess _F, Fourier amplitude | |FO] and Fourier phase LFOs LF" the content loss between Fes and Fe can be derived as:
C ΣΣΣ k=0 10=0000mm o=0
C. =| - F
ΣΣΣDE
(F
FC FR,h..)
Fos Flu F FE.a,
HW
1=0 A=0
H W
ΣΣΣ F 12 + 1F 12_ 2) ( 1i 1o08 (FFbse FFra,,))
HW
1=0
v==
4

---

Under review as a conference paper at ICLR 2023
where the second equality is held by the Parseval's theorem and k, (h, w) and (u, v) are indices channels, spatial dimensions and frequency dimensions, respectively. When F is optimized for the content loss, since [FRs k.u.el and [ k.u. non-negative numbers, the content loss Ce reaches local minimum when 2, L.F... for all (ki u, v). Furthermore, whenever 2F,,u.v gets closer to u. the content loss decreases, demonstrating the crucial role of phase of feature maps in determining the spatial information of corresponding decoded images. Therefore, we can interpret the structure preservation abilities of methods from the perspective Fourier phase. Furthermore, can manipulate Fourier phase for better performances in structure preservation.
Interpretations on structure preservation Based on the equivalent form in Eq. 6 and the re- lation between Fourier phase and the content loss,
give interpretations to different behaviors of methods in structure preservation. Concerning AdaIN and WCT as instances of the equivalent framework in the frequency domain, we have Fa.e TF when (u, v) (0, Note that for AdaIN, the transformation matrix T diag (E*) /diag (E") E RCxC is a real diagonal matrix, which has the same scaling upon the real part and the imaginary part of FFs. As result, AdaIN can preserve the phase in each feature channel and keep the content loss of feature maps local
minimum. While WCT provides non-diagonal matrix T (2*)² (>=))==== for transformation, the information between different channels is consequently entangled, the phase of each channel is dis- turbed and the content loss after the process of WCT is likely to increase much more than the one after the process of AdaIN. Therefore, WCT needs more efforts to preserve the spatial information of content images, resulting in its less appealing performances in structure preservation.
Amplitude and Gram matrices We theoretically demonstrate the connection between the Fourier amplitude of feature maps and their Gram matrices. Given feature maps F RCC corresponding Fourier amplitude [F] and Fourier phase LF of their DFT F, the pixels of the Gram matrix FF can be derived as:
H
W
H w
(FFT)
Σ
Σ
F Fer 022a.1
01
HW
h=0 w=
u=0 4=0v=0 v=
H
F F
Formull / cos (F
LF LFzzuw))
HW
01
where Cı, C2 are indices the channels, (*) represents complex conjugate and the second equality is held by the Parseval's 's theorem. Since (FFT) ),,e2 is real number, we omit the imaginary part in the final step. In special case where e₁ equals c2. (FF*).. cc.0z equals HW This indicates that the sum of the square of amplitude components directly determines the diagonals of Gram matrices. Each elements on the diagonal of Gram matrices represents infra-chamn second- order statistics of feature maps, measuring the intensity information in each channel. Therefore, if we only manipulate the Fourier phase of the DFTs of feature maps and keep the Fourier amplitude unchanged, it can expected that the intensity presentations of the corresponding decoded images are roughly the same. For detailed proof, see the Supplementary Materials.
11 Eve 1=
3.3 MANIPULATIONS STYLIZED FEATURE MAPS IN THE FREQUENCY DOMAIN
The equivalent form in Eq. (6) and abovementioned connections enable further manipulations for better structure preservation desired stylization. We propose two simple operations upon fre- quency components of feature maps, which we call phase replacement and frequency combination.
Phase replacement Given the DFT of the content feature maps F and the DFT of the stylized feature maps Jess, we calculate the phase of F denoted as LF E [0, 2T) Cx x and the
amplitude of Jes, denoted as | E R CC* where R denotes the set of non-negative real numbers. We then reconstruct Jeee
F
Fll @ cos LF. 0eo + F| @ sin L
0,07
where and sin element-wiss operators on vectors (e.g., cos ó [cos の¹ ..... cos dC @ is the element-wise multiplication and j is the imaginary unit. Based on the connections established in Section 3.2 when we replace the phase of Jee with /Fe, the content loss between Fess and Fe
5

---

Under review as a conference paper at ICLR 2023
is reduced and in this way, the structure of content images is more preserved in Fes. In addition, since the amplitude of F is not changed, the diagonal of prases" stays unchanged and so does the basic intensity information of the stylized results. A similar amplitude transferring method is proposed in (Yang& & Soatto 2020) for semantic segmentation, which shares resembling views phase and amplitude with ours.
Frequency combination To accommodate different requirements from users, appropriate con- trol on the stylization is needed for practical usage. Plenty of works for style transfer use linear combination of content feature maps F and stylized feature maps Fee as shown in Eq. (T0):
= (1 (1.
(10)
where a is the weight for controlling on the stylization. In this way, all the global characteristics of images (e.g., the sharp edges of trees and the smooth background of sky) are combined uniformly. While in most cases, users are expecting for customized global changes on images (e.g.. having the details of trees less stylized but keeping the sky moderately stylized). Since high frequencies determine the details and low frequencies determine the overview of images, we can accommodate the customized needs of users with combination of frequencies in different proportions.
Given the DFT of content feature maps F and the DFT of stylized feature maps F we first rear- ranges their frequency components with the zero-ffreueee zero-frequency components in the center point (uo, vo), following a common technique in digital image processing. In this way, the frequency components close to (Mo, 2b) are low-frequency components whereas the rest of components represent high fre- quencies. Next, we combine Fos and F using a weighting function a R - [o, 1]:
0. = a(u, U) + [1 - a(,,v))FF.
(11)
where a serves as the stylization weighting function dependent on the indices (u, v). For example, if users want have the details less stylized, higher frequencies of F need to be less weighted, and accordingly lower value of a can be set for (u, v) indexing higher frequencies. In practice, the function a is set be controlled by a hyper-parament a:
(u ) + (v 01)
a(u, v) = exp
(12)
where a represents the degree for combining the low frequencies of Fes. When a gets larger, the value of (u, v) increases for every v). In this way, more low frequencies of Jes (indexed by (u, v) close to (No, vo) ) are gradually kept.
4 EXPERIMENTS
In this section, we first introduce method specification and implementation details. Then compare our method with the state-of-the-art style transfer methods in terms of visual effect, struc- ture preservation and computing time. Moreover, conduct experiments validate the equiva- lence presented in Eq. (6), the interpretations on Fourier amplitude and phase introduced in Section 3.2 and the efficacy of manipulations proposed in Section (3.3 More qualitative results and imple- mentation details are available in the Supplementary Material..
Method specification. Based on the equivalence and connections mentioned above, the proposed method first performs selected UST algorithm in the frequency domain. In practice, we choose to implement our method in conjunction with WCT because WCT produces expressive stylization in spite of introduced distortions. To deal with the distortions, we adopt phase replacement (PR) to substitute the Fourier phase of stylized feature maps with that of content feature maps. Since PR can optimize the content loss to local minimum according to Eq. 7 the structure of content images is preserved. Finally, the proposed method uses inverse discrete Fourier transform to reverse the frequency components back to the spatial domain.
Implementation details. We adopt a part of the VGG-19 network (Simonyan & Zisserman, 2015) (from the layer conv1_1 to the layer comv4_1) our encoder. The weights of our encoder are borrowed from SaageeNe-pretrainee weights, following existing style transfer methods. We train decoder for image reconstruction by minimizing the La2 reconstruction loss. During the

---

Under review as a conference paper at ICLR 2023
Content
Style
AdaIN
AdaIN w/ ge8
WCT
WCT w/ Jo
Optimal WCT
Linear WCT
Ours
Qualitative comparison the state-of-the-art UST algorithms. AdaIN w/ Jeee and WCT w/ Jae are implemented following Eq. in the frequency domain, which perform equivalent stylization with the original methods.
Figure I:
Table 1: The SSIM scores, the Gram loss and the time cost (seconds) for different UST methods.
Method
AdaIN
WCT
LinearWCT
OptimalWCT
AvatarNet
SANet
Self-Contained
Ours
SSIM
0.307
0,234
0.250
0.329
0.310
0.276
0.403
SSIM PR)
0.307
0.251
0.427
0.263
0.438
Time
0.0218
0.3167
0.0038
0.6247
3.027
0.0052
0.0513
0.098
Time (w/ PR)
0.0226
0.3293
0.0045
0.6321
0.1
inference stage, we apply our method to feature maps in each layer of the decoder. We choose MS- COCO dataset (Lin al. (2044) and WikiArt dataset (Nichol (016)) as our content dataset and style dataset, respectively. Our decoder is also trained on the content dataset, whose training images are first resized into 512x512 and randomly cropped into a size of 256x256. We all the experiments on single NVIDIA Tesla V100.
4.1 PERFORMANCE COMPARISON
Qualitative comparison In Figure we show some visualization results the qualitative com- parison between the proposed methods and the state-of-the-art UST methods (i.e., AdaIN (Huang

---

Under review as a conference paper at ICLR 2023
(4) Costent image
(b) Style image
Stylized image
(0) Amplinude (b) plase (c)
(e) Amplitude of (1)
and phose
(m)
Content
(b) Style
Stylized image
(4) Pluse of (b) amplitude of (c)
Phose of (e) and amplinade (c)
Figure 2: Synthesized results with replaced Fourier amplitude or phase.
Belongie 2017,. WCT (Li al. 2017a), LinearWCT (Li et al. 2019), OptimalWCT (Lu et al, 2019) PArkt(Park & Lee 2019), AvatarNet(Sheng al. 2018) and Self-Containee (Chen et al, 20200. We observe that AdaIN roughly preserves the structure images, but often produces unap- pealing patterns the edges (e.g., 2nd 3rd and 5th columns). WCT and OptimalWCT can produce intensive but distorted artistic style and yield images less similar with content images in structure (e.g., 2nd 4th, and 5th columns). LinearWCT roughly preserves the spatial structure of content images, but the stylization is less intensive (e.g., 1st 2nd, and 4th columns). Comparatively, the proposed method performs well in the contrast and intensity of stylized results (e.g., the misty sky and the red lips). The proposed method also well preserves the spatial structure of content images, including the details (e.g. the contours of architectures and hair) and the overview (e.g. the light and shadow of cloudy sky and the human face) of images.
Quantitative comparison In addition to the comparison on visual effect, we conduct quantita- tive comparison. Moreover, we implement PR on each method and present corresponding quantita- tive scores. The Structural Similarity Index (SSIM) between the stylized images and corresponding content images is adopted the metric for evaluating structure preservation. In Table [ our method achieves the highest SSIM score and PR improves SSIM scores of all methods except for AdaIN, since AdaIN does not change the Fourier phase of feature maps during stylization. The improved SSIM validates our interpretations on structure preservation behaviors between alogrithms. Regard- ing the computing time, while method needs to utilize Fourier transform, it still has competitive time cost compared with other methods. is noteworthy that our method has less additional compu- tational cost if PR adopted, since has already performed style transfer the frequency domain.
4.2 EQUIVALENCE, INTERPRETATION AND MANIPULATIONS
Validation of equivalence For AdaIN and WCT. we implement them in the frequency domain based on Eq. (6). shown from the 3rd to the 6th row in Figure I It can be observed that these two implemented UST algorithms in the frequency domain produce the same visual effect with original algorithms. This observation validates the proposed equivalence.
Interpretations on amplitude and phase To validate the roles of amplitude and phase, we re- place the Fourier amplitude or phase of stylized feature maps in each layer during the stylization and present the results in Figure can be observed that feature maps with the same Fourier phase produce images with highly similar spatial arrangements. This observation matches up with interpretations on phase, provided by its connection with the content loss in Eq. 7 On the other hand, feature maps with same Fourier amplitude produces images with highly similar contrast and intensities colors. This observation aligns with our interpretations on amplitude, supported by its connection with Gram matrices in Eq. (25)
Stylization manipulations First, we empirically display the effect of PR in Section 3.3 for image stylization, whose results shown in Figure It can be observed that for results without PR, the
8

---

realistic (e.g., the buildings and the sailboat more Under review as a conference paper at ICLR 2023
Content
Cootent
Style
Ours w/o PR
Ours w/ PR
Stylo
Ours PR
Ours w/ PR
Figure 3: Results on the efficacy of phase replacement (abbr. PR).
(a) Content
(b) Style
(c) Linear
(d) Ours
Figure 4: Comparison between (c) linear combination and (d) frequency combination. It is worth noting that the proposed method helps the background remain stylized (e.g., the sky more sketch- stylized than (c)) while renders the details realistic than (c)). Note that the proposed method achieves this performance without any spatial masks identify where stylize. The hyper-parammttr for results in (d) is 0.99.
details (e.g., edges of the eyes and the nose) and the overview (e.g., the sky and the sea) become messier and more distorted, yielding unappealing distortions. The reason might be that PR can preserve the phase of both high frequencies and low frequencies, which are responsible for the spatial arrangement of the details and overview of images, respectively.
Second, to demonstrate the manipulations of frequency combination (FC) in Section 3.3 we present example in Figure 4 We choose the weighting function in Eq. (12) and adjust the hyper- parameter a for stylization controls. In Figure 4 with different value of a, FC can have the details less stylized (e.g., the colorful buildings in the 4th column) while keeping the background moder- ately stylized (e.g. the sky with sketch style in the 4th column). Furthermore, FC can be customized for various purposes and the linear combination in Eq. (TO) can be viewed by setting the weighting function auu, v) as a simple scalar. Therefore, the controllability of FC is better than that of linear combination.
instance of FC
5 CONCLUSION
In this paper, we apply Fourier analysis to a unified framework of UST algorithms. We present the equivalent form of the framework and reveal the connections between the concepts Fourier transform with those of style transfer. We give interpretations on the different performances between UST methods in structure preservation. We also present two operations for structure preservation and desired stylization. Extensive experiments between the framework and proposed form, (2) the interpretability prompted by Fourier analysis upon style transfer and (3) the controllability through manipulations frequency components.
conducted to demonstrate (1) the equivalence

---

Under review as a conference paper at ICLR 2023
REFERENCES
Prashanth Chandran, Gaspard Zoss, Paulo Gotardo, Markus Gross, and Derek Bradley. Adaptive convolutions for structure-aware aware style transfer. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7972-7981, June 2021.
Haibo Chen, lei zhao, Zhizhong Wang, Huiming Zhang, Zhiwen Zuo, Wei Xing Ailin Li, and Dongming Lu. Artistic style transfer with internal-external learning and contrastive learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Ad- vances in Neural Information Processing Systems. volume 34, pp. 26561-26573. Curran Asso- ciates, Inc., 2021a. URL htttpps::/prrceeedngg. newrips.. cc wwtpp/pppppp////ee/ee.e ( 7bb.aa099999999696655655 Paper. pdf
Haibo Chen, Lei Zhao, Zhizhong Wang, Huiming Zhang, Zhiwen Zuo, Ailin Li, Wei Xing, and Dongming Lu. Dualast: Dual style-learning networks for artistic style transfer. In Proceedings of the Conference on Computer Vision and Puttern Recognition (CVPR), pp. 872-881, June 2021b.
Haibo Chen, Lei Zhao, Huiming Zhang, Zhizhong Wang, Zhiwen Zuo, Ailin Li, Wei Xing, and Dongming Lu. Diverse image style transfer via invertible cross-space mapping. In Proceedings of the International Conference on Computer Vision (ICCV), pp. 14880-14889, October 2021c
Hung- Yu Chen, I-Sheng Fang, Chia-Ming Cheng, and Wei-Chen Chiu. Self-contained stylization via steganography for reverse and serial style transfer. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), March 2020.
Tai-Yin Chiu and Danna Gurari. Photowet 2
Compact autoencoder for photorealistic style transfer resulting from blockwise training and skip connections of high-frequency residuals. pp. 2978- 2987, 01 2022. doi: 10.1
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. Learned representation for artistic style. In International Conference on Learning Representations, 2017.
Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 2414-2423, 2016. doi: 10.1109/CPR.2201.2222
Rafael C. Gonzalez and Richard E. Woods. Digital image processing, pp. 286- 306. Prentice Hall, Upper Saddle River, N.J., 2008 ISBN 9780131687288
013168728X 9780135052679
013505267X.
URI. http:/ /ttt:::/wwww www. amazon. com/
Digital-
3rd- Edition dp/ /pp/611188788
Kibeom Hong. Seogkyu Jeon, Huan Yang. Jianlong Fu, and Hyeran Byun. Domain-aware universal style transfer. In Proceedings of the International Conference Computer Vision (ICCV), pp. 14609-14617, October 2021.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance nor- malization. In International Conference on Computer Vision (ICCV), pp. 1510-1519, 2017. doi: 10. N0.1109//CCV.2017..16
William Frost Jenkins and Mita D. Desai. The discrete frequency fourier transform. IEEE Transac- tions on Circuits and Systems, 33:732-734, 1986.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. European conference on computer vision. In Perceptual losses for real-time style transfer and super-resolution, pp. 694-711, 2016.
Chuan Li and Michael Wand. Combining markov random fields and convolutional neural networks for image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2479-2486 2016.
Xueting Li, Sifei Liu, Jan Kautz, and Ming- Hsuan Yang. Learning linear transformations for fast im- age and video style transfer. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5804-3812 1,, 2019.
doi: 10. 1
10

---

Under review as a conference paper at ICLR 2023
Yijun Li, Chen Fang, Jimei Yang. Zhaowen Wang, Xin Lu, and Ming Hsuan Yang. Universal style transfer via feature transforms. In Advances neural information processing systems, pp. 386-396, 2017a.
Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang. Xin Lu, and Ming- Baangn Yang. Diversified tex- ture synthesis with feed-forward networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017b.
Yijun Li, Ming-Yi Yu Liu, Xueting Li, Ming-Hsuai Yang, and Jan Kautz. A closed-form solution to photorealistic image stylization. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018
Tsung- Yi Lin, Michael Maire. Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pictro Perona, Deva Ramanan, C. Lawrence Zimicck and Piotr Dollár. Microsoft coco: Common objects in context, 2014. URL.tttps://aaxxv http://///.a...o.//////ooo org/ abs/ 111//4105..00 0312
Xiao-Chang Liu, Yong-Liang Yang, and Peter Hall, Learning to warp for style transfer. Proceed- ings of the Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3702-3711, June 2021.
Ming Lu, Hao Zhao, Anbang Yao, Yurong Chen, Feng Xu, and Li Zhang. closed-form solution to universal style transfer. In International Conference Computer Vision (ICCV), pp. 5951-5960, 2019.
Nichol. Painter by numbers. volume 34, 2016. URL htEps://www.kggle..o bynntter-----uubrre
www
Dae Young Park and Kwang Hee Lee. Arbitrary style transfer with style-attentional networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5880-5888, 2019.
Lu Sheng, Ziyi Lin, Jing Shao, and Xiaogang Wang. Avatar-net: Multi-scale zero-shot style transfer by feature decoration, 2018. htl.:tttsa///ziiv.oo////////////
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
Zhizhong Wang, Lei Zhao, Haibo Chen, Lihong Qiu, Qihang Mo, Sihuan Lin, Wei Xing, and Dong- ming Lu. Diversified arbitrary style transfer via deep feature perturbation. In Proceedings of the IEEE International Conference Computer Vision, pp. 7789-7798, 2020.
Yanchao Yang and Stefano Soatto. Fda: Fourier domain adaptation for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2020.
Jaejun Yoo, Youngjung Uh, Sanghyuk Chun, Byeongkyu Kang, and Woo Ha. Photorealistic style transfer via wavelet transforms. In International Conference on Computer Vision (ICCV), pp. 2019. doi: 10. 10.11090 10.100//CCV2221900091..
A APPENDIX
In Supplementary Materials, we first provide the proofs of the main conclusions in Section 3, then we introduce the model architectures, experimental details and computing infrastructure, and finally we show the additional experimental results on our method for the performance comparison and the validation of interpretability and controllability.
B PROOFS
B. PROOFS FOR THE EQUIVALENT FORM OF THE FRAMEWORK IN SECTION 3.1
Given the content image Ie and the style image 1s, we denote the feature maps of I as F e CC*H x and the feature maps of 1 as ps E CxCH xw" where C' is the number of channels,
11

---

Under review as a conference paper at ICLR 2023
Hº (H*) is the height and W (W*) is the width. Universal style transfer (UST) algorithms trans- form F into the stylized feature maps F and synthesize stylized images. Based on the unified framework (Lu et al. 2019), each pixel of F is calculated as:
F T (F - -²) p="
(13)
where T E RCCC is a transformation matrix, (h, w) are indices on the spatial dimensions and , 14 RC are the channel-wise mean for fe and Fs, respectively. We denote the 2-D discrete Fourier transform (DFT) of F as F E CCx H" xw where C is the set of complex numbers. Based on the Fourier transform and Eq. equation 13
we derive each complex frequency component
Fag as:
H
W
H W Σ
F
F e
12 AeetuuAetee tu
[T(F
-0 to 11)) + 1e 52
h=0
w=0
h=0
==
H W1
-- W
T
Σ
R 12 AC to we) + (15 Txy)
Σ
Σ
52 A tr
h=0
h=0
the first
second
(14)
where (u, v) are indices on the frequency dimensions and j is the imaginary unit. Note that the first term of Fes, equals: 0,0
H--1W W
T
Σ
Σ
22 + TF.
(15)
h=0 w=0
Especially, when == 0, TFe a,o equals:
3"-1W"-1
TF = T Σ
Σ
F e = 0 wo,,.
(16)
h=0
w=
And the second term of Fas equals:
1--1w-11
H-W Ty²), if
(1r - Tn
Σ Σ
-52r(u +
(17)
10
else
Thus, F can be derived as:
(T(H'W wC) T(H + W= if
Jes
TF if== 0:
else
(18)
H-Wo
H*W* HoW>
Fo.0- if
O:
TF else
T
else
where F and F are the DFTs of F and F", respectively. is worth noting that Fi,0 =
DH' -1 Fi H* Therefore, we have 4.v H&WS (#"W*) Fö.o when
0.
B.2 PROOFS FOR THE CONNECTION BETWEEN PHASE AND THE CONTENT LOSS IN SECTION 3.2
Given matrices X,Y E # and their 2-D DFTs X,Y #xw have the Parseval's theorem in the context of DFT that:
H-1W-1
#- W-
ΣΣ
XN X Y
Σ
(19)
HW
A=0
where (+) represents complex conjugate and u, v are indices on the frequency dimensions. Espe- cially, when X - Y and X,Y IR HWW we can have:
8-1W 4-1W-1
H 1-IW-1
Σ
Σ
(Xr,w)
Σ
Ku,el?
(20)
HW
h=0
w=0
4=0
v=0
12

---

Under review as a conference paper at ICLR 2023
We then denote the feature maps of the stylized image and the content image as F F RCxH*XX HxW their corresponding DFTs as F ,r, Fourier amplitude as [F01) and Fourier phase as LFes LFi, respectively. Note that for any F Fka,00 and F k.u. we have:
Fk,. k, LF kNN.v + F, sin 2F,,u, F Fk,u,o |Fk, LF,,u, + -,,,, sin 2F
(21)
where le and (u, v) are indices channels and frequency dimensions, respectively. Note that for complex numbers M = a + jb and V = + jd with a, b, d E IR, we have:
M ./2² (a c)²
d)².
(22)
Based on the Eq. equation 20 Eq. equation 21 and Eq. equation 22 Pre and Fe can be represented as:
the content loss Le between
C-1 H-
C-1 H
C.
FR. FE
F k,u,
Fr. Fk
HIW
0= h=0
w=0
k=0 u=0
e=0
C-18-1W-1
ΣΣΣ
[IFF cOs k,a,0
FFR COS LFF L.
HW K=
k,u, sin 2F,,,, 1Fk.u,ol sin LFF kFk,,,))
LFCs
C-18-1W-1
HW ΣΣΣ
Rju, + F
2 2 kyaw || cos LFk, k,w COS LFr
0=0
2 k.u, 1...lFFF sin FFss
R.u, sin LFF
1 C-IH-IW-
HW EE
cos Ff 11 ++ +IFR,
LF,, ].
(23)
is worth noting that cos(a - 3) = cos a cos sin a sin 3 for any a, B E (-
B.3 PROOFS FOR THE CONNECTION BETWEEN AMPLITUDE AND GRAM MATRICES IN SECTION 3.2
According to Eq. equation 19 when X, Y E HxW Yn,w
Yi, we have:
-1
ΣΣ
X Vn,w
X Nu
(24)
HW
h=0
which serves as the basis for our following proofs. Given feature maps F E RC their corresponding DFTs F, Fourier amplitude [F] and Fourier phase LF, the pixel of the Gram matrix FF can be derived as:
H-1 W-1
H H-1 W-1
(FFT)
Σ
Σ
Σ
Fer.a.FFF og.ar
HW
hm w==
u=0 v=0
H-
W-1
cos (LFo LF FFogmw.) [ moll
HW
(25)
4=0
u=0
sin (FFo LF [Fozme))
H-1W-1
HW ΣΣ
cos (e,,, LF Fog.u,))
where Cı, 02 are indices on the channels. Considering that (FFT) e1re2 is real number, the imagi-
nary part HW Σ#=1 2x=0 sequently omitted the final step.
WW-1 F II Fog sin (LFa LF FF))..)) equals zero and is con-
13

---

Under review as a conference paper at ICLR 2023
C IMPLEMENTATION DETAILS
C.I METHOD SPECIFICATION
The encoder of the proposed method part of the VGG-19 network (Simonyan & Zisserman 2013) (from the layer conv1_1 to the layer comv4_1). The decoder is generative network to perform the stylization and transform the stylized feature maps into images. The decoder performs both whitening and coloring transform (WCT) (Li et al.) (017a) and the proposed phase replacement operation in the frequency domain only for the inference stage. The detailed network architecture is described as follows:
The encoder:
x 1 Conv,
X 1 Reflection Padding, 3 Conv, 64 ReLU
- X 1 Reflection Padding, x 3 Conv, 64 ReLU
2 X 2 Max pool, stride
stride 2: 1 x Reflection Padding, 3 x 3 Conv, 128 ReLU I Reflection Padding,
Conv, 128 ReLU
2 X 2 Max pool, stride 2: X Reflection Padding, 3 x Conv, 256 ReLU
1 X 1 Reflection Padding, 3 3 Conv, 256 ReLU
1 X 1 Reflection Padding, 3 Cony, 256 ReLUU
X 1 Reflection Padding, 3 Conv, 256 ReLU
2 x 2 Max pool, stride 2: X Reflection Padding, 3 x Conv, 512 ReLU
The decoder:
Universal style transfer module; Phase replacement
X 1 Reflection Padding, 3 Conv, 256 ReLU
2x nearest-neighhor upsample;
Reflection Padding, 3 x Conv, 256 ReLU
X Reflection Padding, 3 Conv, 256 ReLU
1 X 1 Reflection Padding, 3 3 Conv, 256 ReLU
Universal style transfer module; Phase replacement
X Reflection Padding, 3 Conv, 128 ReLU
2x neirest- upsample; 1 x Reflection Padding, 3 x 3 Conv, 128 ReLU
Universal style transfer module; Phase replacement
x Reflection Padding, 3 Conv, 64 ReLU
2x nearest-neighhor upsample; x Reflection Padding, 3 x 3 Cony, 64 ReLU
Universal style transfer module; Phase replacement
x 1 Reflection Padding, Cony, 3
C.2 EXPERIMENTAL DETAILS
The encoder of the proposed method pre-trained by the ImageNet dataset (?). The decoder of the proposed method is trained for image reconstruction by minimizing the L2 reconstruction loss. The corresponding training dataset is MS-COCO dataset (Lin et al.) 2014). whose images are first resized into 512x512 and randomly cropped into size of 256 255 for training. During the learning stage, the batch size 64 and the number of training steps is 140,000. The initial learning rate is 10-4 and is decayed with factor 0.5 for every 20,000 steps. For the stylization, we choose MS-COCO dataset (Lin al. 2014) and WikiArt dataset (Nichol 2016) as our content dataset and style dataset, respectively. For the quantitative comparison, randomly select 15 content images and 25 style images to synthesize 375 stylized images for each method.
C.3 COMPUTING INFRASTRUCTURE
Experiments are conducted on a server with Intel Xeon Silver 4214 CPUs and a single NVIDIA Tesla V100 GPU. The code is developed based the PyTorch framework (?) with version 1.8.1.
14

---

Under review as a conference paper at ICLR 2023
Table 2: The SSIM scores for different methods the architectures of AdaIN and WCT.
Method Architecture
AdaIN
WCT
AdaIN
0.307
0.315
WCT
0.274
0.234
WCT w/ PR
0.280
0.251
D INVOLVED EXISTING ASSETS
Existing assets involved in this work include: 1) the codes and models weights of AdaIN (Huang
Belongie 2017), WCT (Li et al. 2017a), LinearWCT (Li et 2019), Avatar-Net (Sheng al. 2018), OptimalWe WCT (Lu et al. 2019). SANet (Park & 2019). and Self- Ceff-Contained (Chen et al.
2020). and 2) the images of MS- COCO dataset (Lin et al. 2014) and WikiArt dataset (Nichol 2016).
Their URLs and licenses are as follows:
AdaIN: http://sihha.ccmmmmoooooo/yyyyyh MIT License.
WCT: http://gghhhbboommmmiiiii/yyyyct Unable to find its license.
LinearWCT: http://ggthbbooooo/niiiiieeeoone cense.
BSD 2-Clause Li-
Avatar-Net: http://ghhhbb.comm//uuasssee/een Unable to find its license.
OCt http:///ttttllccccooommomooo///t SetahhoitiiiamlSyeelleeeteee Trannsfr. Unable to find its license.
MS MIS-COCO dataset: http:///cooddaasccoo//////bbbbba Unable to find its license.
WikiArt dataset: http:///www.aagggecccooo///mmmem
Unable to find license.
SANet: http://ggtthbbbmmmm/tttSessshnnn
Delf-Contained: http://stttbbbocommmSssfggnnngft StnyiiineedSSylllzttioo
Stylization
E ADDITIONAL EXPERIMENTAL RESULTS
In this section, we present more experimental results in this work, including (1) more performance
comparison against the state-of-the thr-ttr-rrt style transfer methods in Figure 6 (2) more validation the
interpretations of Fourier phase and amplitude in Figure 7 and Figure and (3) more experiments
on the efficacy of proposed manipulations in Table 2 Figure and Figure 10
Note that the structure preservation abilities of different methods are not only related with their styl- ization transformations feature maps, but also related with their architectures and model weights. To better understand the differences between AdaIN and WCT, we implement different methods on their architectures and present corresponding quantitative scores. As shown in Table 2 WCT is improved by phase replacement (PR) while AdaIN achieves the highest SSIM scores. The reason might be that WCT has more influence on the Fourier amplitude of feature maps compared with AdaIN, According to Eq. equation 23 when the Fourier phase is well preserved by AdaIN and PR, the content loss only depends on the differences of Fourier amplitude, which means that changing Fourier amplitude is likely to harm the SSIM score. Since WCT has lower SSIM scores compared with AdaIN, more likely to influence more on the Fourier amplitude. This also explains why WCT can produce much more intensive and expressive stylization, considering the connection be- tween Fourier amplitude and the intensities of stylized images.
REFERENCES
Prashanth Chandran, Gaspard Zoss, Paulo Gotardo, Markus Gross, and Derek Bradley. Adaptive convolutions for structure-aware style transfer. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7972-7981, June 2021.
15

---

Under review as a conference paper at ICLR 2023
Content
Style
AdaIN
WCT
Optimal WCT
Linear WCT
Avatar Net
SANet
Self-Contained
Figure 5: Qualitative comparison on the state-of-the-art UST algorithms. Compared with the pro- posed method, AvatarNet and SANet sometimes generates less intensive colors (e.g., in 2nd and 4th columns) and produce unexpected style pattern (e.g., eyes in the 5th column), while Self- Coff-Contained sometimes introduces unexpected color (e.g., in the 4thh column)
Haibo Chen, lei zhao, Zhizhong Wang, Huiming Zhang, Zhiwen Zuo, Wei Xing Ailin Li, and Dongming Lu. Artistic style transfer with internal-external learning and contrastive learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Ad- vances Neural Information Processing Systems, volume 34, pp. 26561-26573. Curran Asso- ciates, Inc., 2021a. URL http:ttts/s:/ccdiiiiiee wtwpeeeiiiin....iuiiieeeeee newrinsuss.... cc wttpp//ppep/////11/ee.... LE534699111e3eebb099ee2bbbbb3333
Paper. pdf
Haibo Chen, Lei Zhao, Zhizhong Wang, Huiming Zhang, Zhiwen Zuo, Ailin Li. Wei Xing, and Dongming Lu. Dualast: Dual style-learning networks for artistic style transfer. In Proceedings of the Conference Computer Vision and Pattern Recognition (CVPR), pp. $72-881, June 2021b.
Haibo Chen, Lei Zhao, Huiming Zhang, Zhizhong Wang. Zhiwen Zuo, Ailin Li. Wei Xing, and Dongming Lu. Diverse image style transfer via invertible cross-space mapping. In Proceedings of the International Conference Computer Vision (ICCV), pp. 18880-14889 October 2021c
16

---

Under review as a conference paper at ICLR 2023
Optimal WCT
Lincar WCT
Avatar
Content
Style
AdaIN
WCT
Net
Figure 6: Qualitative comparison on the state-of-the-ae UST algorithms. On one hand, compared with AdaIN and LinearWCT, the proposed method produces more intensive colors (e.g., 3rd 4th 6th and 9th rows). On the other hand, the proposed method does better in the structure preservation of images than the rest of methods (e.g., 1st 2nd 5th, 7th and 8th rows).
Hung- Yu Chen, I-Sheng Fang, Chia-Ming Cheng, and Wei-Chen Chiu. Self-contained stylization via steganography for reverse and serial style transfer. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision (WACV), March 2020.
Tai- Yin Chiu and Danna Gurari. Photowet Compact autoencoder for photorcalistic style transfer resulting from blockwise training and skip connections of high-frequency residuals. pp. 2978-
17

---

Under review as a conference paper at ICLR 2023
(a) Content image
(b) Style image
Stylized image
(d) Amplitude of (b) and phase of (c)
Amplitude of and phase of (c)
Figure 7: Additional synthesized results with the same Fourier phase and replaced Fourier ampli- tude. We replace the Fourier amplitude of stylized feature maps in each layer and use the proposed method to synthesize stylized images. Note that feature maps with the same phase produce images with highly similar spatial arrangements (e.g., the triangular structure of the architecture and the human face), even in some details (e.g., the light the human check). Therefore, Fourier phase is validated to play an important role on the spatial arrangements of images.
(a) Content image
Style image
Stylized image
(d) Phase and amplitude of (c)
(e) Phase of and amplitude of
Figure 8: Additional synthesized results with the same Fourier amplitude and replaced Fourier phase. We replace the Fourier phase of stylized feature maps in each layer and use the proposed method (without phase replacement) synthesize stylized images. Note that feature maps with the same amplitude produce images with highly similar contrast and intensities, (e.g., the same differ- ences between the brightest pixels and the darkest pixels of (c), (d) and (c)). Therefore, we validate the connection between Fourier amplitude and the intensities of synthesized images.
2987, 01 2022. doi: 10.1100NAACV145522222.22
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. Learned representation for artistic style. In International Conference on Learning Representations, 2017.
Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks, In Proceedings the IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 2414-2423, 2016. doi: 10. 00.1000222..
Rafael C. Gonzalez and Richard E. Woods. 306.
Digital image processing, pp. 286-
Prentice Hall, Upper Saddle River, N.J., 2008. ISBN 9780131687288
18

---

Under review as a conference paper at ICLR 2023
WCT PR
OptimalWCT
Ours
Content
Style
WCT
OptimalWCT
PR
Ours
PR
Figure 9: Results on the efficacy of phase replacement (abbr. PR). We implement PR on different methods and show the visual difference. It can be observed that PR improves the visual effect of all methods in the spatial arrangement of not only the details (e.g., it improves the unpleasant human eyes and vague contours of architectures ), but also overview (e.g. it improves the uneven light and shade) of images.
URL http:/ www amazon. com/ Diçital- Secaa--maae--a SwobseeProccssing---e wtw.eeees....oooaaaaaa.oooooooe. dp/ 013168728X
013168728X 97801 9780135052679 013505267X
Kibeom Hong, Seogkyu Jeon, Huan Yang, Jianlong Fu, and Hyeran Byun. Domain-aware universal style transfer. Proceedings of the International Conference on Computer Vision (ICCV), pp- 14609-14617, October 2021.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance nor- malization. In International Conference on Computer Vision (ICCV), pp. 1510-1519, 2017. doi: 10.1 10.1109/CCV.2217..16.
William Frost Jenkins and Mita D. Desai. The discrete frequency fourier transform. IEEE Transac- tions Circuits and Systems, 33:732-734, 1986.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. European conference on computer vision. In Perceptual losses for real-time style transfer and super-resolution, pp. 694-711, 2016.
Chuan Li and Michael Wand. Combining markov random fields and convolutional neural networks for image synthesis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2479 2299222862222 2016.
Xueting Li, Sifci Liu, Jan Kautz, and Ming-Hsuan Yang. Learning linear transformations for fast im- age and video style transfer. Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3804-881181111 3812, 2019. doi: 10.1 1119/CCPR.2019.003999
Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang. Universal style transfer via feature transforms. In Advances neural information processing systems, pp- 386-396, 2017a
Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang. Diversified tex- ture synthesis with feed-forward networks. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2017b.
Yijun Li, Ming-Yi Yu Liu, Xueting Li, Ming- Hsuan Yang, and Jan Kautz. A closed-form solution to photorealistic image stylization. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
Tsung- Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zimicck and Piotr Dollár. Microsoft coco: Common objects in context, 2014.
wrw.vvrrr//aaae abs 1405 0312
19

---

Under review as a conference paper at ICLR 2023
(a)
(
(a)
(b)
Figure 10: Comparison between (a) linear combination and (b) frequency combination. Images the first column are stylized images and images in the last column are content images. The proposed frequency method maintain the low frequencies of stylized images and focus the high frequencies of content images. On one hand, the proposed method can fix distorted details by introducing the high frequencies of content images, since these distortion frequencies of stylized images (e.g., the difference between the first column and the second column of (b)). On the other hand, the proposed method can enhance the details while keeping the overview stylized (e.g., the difference between the fourth column of (a) and (b)). Note that the proposed method achieves this performance without any spatial masks identify where stylize.
caused by the high
Xiao-Chang Liu, Yong-Liang Yang, and Peter Hall, Learning to warp for style transfer. Proceed- ings of the Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3702-3711, June 2021.
Ming Lu, Hao Zhao, Anbang Yao, Yurong Chen, Feng Xu, and Li Zhang. closed-fomm solution to universal style transfer. In International Conference Computer Vision (ICCV), pp. 5951-5960,
20

---

Under review as a conference paper at ICLR 2023
2019.
K Nichol. Painter by numbers. volume 34, 2016. URL https://www.c. http://wwwwwgglcccccoc com/c/ painter-by- numbers
Dac Young Park and Kwang Hee Lee. Arbitrary style transfer with style-attentional networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5880-5888, 2019,
Lu Sheng, Ziyi Lin, Jing Shao, and Xiaogang Wang. Avatar-net: Multi-scale zero-shot style transfer by feature decoration, 2018. htth:::::// hattpp////axxiv.oo///////////// 03857
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
Zhizhong Wang, Lei Zhao, Haibo Chen, Lihong Qiu, Qihang Mo, Sihuan Lin, Wei Xing, and Dong- ming Lu. Diversified arbitrary style transfer via deep feature perturbation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 7789-7798 2020.
Yanchao Yang and Stefano Soatto. Fda: Fourier domain adaptation for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Jaejun Yoo, Youngjung Uh, Sanghyuk Chun, Byeongkyu Kang, and Jung-Woo Woo Ha. Photorealistic style transfer via wavelet transforms. In International Conference on Computer Vision (ICCV), pp. 9035-9044, 2019. doi: 10.1 10.1109/CCV22219.00091..
21


Reviews:
[{"review_id": "oR91tRvHOy", "reviewer": "Reviewer_niHJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper analyzes the global style transfer method from the perspective of Fourier analysis. Specifically, the formula of style transfer is converted into Fourier transform under the umbrella of global statistics. Thus the relationships between Fourier phase (Fourier amplitude) and content loss (Gram matrix) are achieved. In this way, the authors design the phase replacement operation to improve the preservation of content structure and the frequency combination operation for content & style feature interpolations with different frequency components.", "summary_of_the_review": "The reviewer has reviewed this paper in NeurIPS 2022, and it was rejected by all reviewers. It is surprising that the authors did not address any of my main comments (e.g. the technical novelty issue, trivial experimental improvement and lack key evaluation results), but directly recycle it to ICLR 2023 by just doing some minor revisions. But as I mentioned during NeurIPS review, some main concerns need to be addressed before acceptance. And I think I can do nothing if the authors decide to ignore my (and other reviewers') suggestions. Thus I will also copy my previous review from NeurIPS 2022 console and paste it here. I strongly encourage the authors to incorporate the suggestions from all the NeurIPS reviewers into their revisions.", "strength_and_weaknesses": "Strengths:\n\n+ Apply frequency decomposition for the holistic statistics-based style transfer method.\n+ Building the relationship between content loss (&& Gram matrix) and amplitude (&& phase) of Fourier transform.\n\n\nWeakness:\n+ The designed method in this paper is completely based on the holistic style statistics hypothesis, which limits its performance. The authors propose two operations to enhance the content structure of stylized results and interpolate between $F_{cs}$ and $F_c$ respectively. However, it is still at the cost of balancing the style information and content structures. See point 2 for details. So I would say this contribution to the style transfer task seems to be limited.\n+ From Fig. 1, the results of this method are weaker than those of WCT regarding the style textures; for example, the color of the content image (e.g. green regions) are preserved on the $3^{rd}$ and $5^{th}$ columns. Therefore, does it mean that the phase replacement operation plays a negative effect on the preservation of style features? This can also be reflected in Fig. 2. I assume that this is connected to the fact that Eq. (8) contains a cosine term with different channel features, and the phase replacement operation will weaken this relationship, thereby damaging the style characteristics in the results.\n+ Patch based style transfer methods (e.g. SANet) and methods combining patch alignment and holistic statistics matching (e.g. AvatarNet), also aim for the issues of spatial structure distortion, and thus can maintain a good content structure. The authors should make comparisons with such methods.\n+ This submission mainly depends on the qualitative comparisons, which might be very subjective. I suggest to use some widely used quantitative metrics, like perceptual loss, to evaluate the capability in capturing style patterns. At least, the authors should perform some user studies over randomly generated results.\n+ As far as I know, the running speed of linear WCT is comparable to that of AdaIN. However, in Tab. 1, the running speed of linear WCT is four times more than that of WCT, and even fourteen times more than that of AdaIN, which is inconsistent with other papers.\n+ How to enhance the stylistic features in synthesized results is the core issue of style transfer. It would be better, if the authors can improve it from the perspective of Fourier transform. For instance, the authors can weigh the $F_{cs}$ and $F_c$ with different frequency characteristics for trade-off. I look forward to more interesting frequency-based manipulation strategy for stylized images.\n+ There is still no user study result.\n+ How to do the quantitative experiments is also unclear. For example, how to select the input content && style images? How to select the synthesized images to compute metrics? Can the authors release their test data for later reproduction? What are the resolutions of test images?\n+ The selected representative images look too similar to LinearWCT. And its running speed is also unconvincing. I ran many experiments with this method before and it is even faster than AdaIN. Considering the limited visual performance, I would say the empirical improvement is very trivial.\n+ Some other experiment problems: e.g. the compared methods are a bit out-of-date; More state-of-the-art methods should be compared with, like MCCNet [1] and StyTr2 [2].\n\n[1] Arbitrary Video Style Transfer via Multi-Channel Correlation, AAAI 2021.\n\n[2] StyTr2:Unbiased Image Style Transfer with Transformers, CVPR 2022.\n\n====== My last NeurIPS comments ======\n\nThank the authors for their further feedback with details! After reading the response, however, I would like to preserve my opinions.\n\nWith the updated running time, it can be seen that LinearWCT is ~25 faster than the proposed one and having comparable (even better) results. It is very hard to convince me to give a positive rate.\n\nThus, I encourage the authors to improve their paper by incorporating the suggestions from all the reviewers into their revisions.\n\n\n====== Other comments ======\n\nHere I do not plan to append more comments from other NeurIPS reviewers now, but I do agree with some of their feedbacks including concerns about the technical novelty, limited experimental improvements and lack key evaluation results. Especially for the technical novelty and limited experimental improvements, all the NeurIPS reviewers have serious concerns. Thus I think these issues should be addressed before acceptance.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "SsbgfTwvya", "reviewer": "Reviewer_pCyN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on style transfer. Different from previous works, this work shows the equivalent form of UST methods in the frequency domain. Moreover, the proposed method revisits existing UST methods in the frequency domain, showing the effects of Fourier amplitude and phase. With these experiments, this work proposes two manipulations for better style transfer. Some experiments are conducted to show the advantages of the proposed manipulations over previous UST methods.   ", "summary_of_the_review": "The paper has some novelty and experiment flaws, which are the most important factors in my rating. In addition,  the advantage of the proposed method is not much obvious. The authors can provide comments based on the points I mentioned in the weaknesses. After that, I will consider my final decision. \n\n", "strength_and_weaknesses": "Strength\n\n+Interesting view\nThis paper revisits style transfer from the view of the Fourier transform. Although the Fourier has been used in low-level vision tasks such as image restoration, it has not been used in style transfer. \n\n+New insight\nThe paper provides new insights into the effect of amplitude and phase of Fourier transfer on the performance of style transfer.\n\n+Solid proof.\nThis work provides theoretical proof of the equivalent form of UST methods in the frequency domain.\n\nWeakness\n\n-Limited improvement\nFrom the results, even using the proposed method, the performance of style transfer is still unsatisfactory. There are artifacts in the results. Thus, it is not clear the effectiveness of the proposed method.\n\n-Insufficient experiment\n1) For the style transfer tasks, more visual results should be included, especially in complex and challenging cases.\n2) More results works should be included for comparison. It seems only several previous UST methods are compared. It is difficult to see the advantages of the proposed method.\n3) The paper uses only SSIM as an evaluation metric. Why not use other metrics?\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Czy5_REoYYJ", "reviewer": "Reviewer_vcHP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Based on Fourier analysis for existing universal style transfer (UST) approaches, this paper demonstrates that existing methods handle all frequency components identically, with the exception of the zero-frequency component. They also show that the whitening and coloring transform (WCT) may affect phase information whereas AdaIN does not. This backs up the empirical finding that AdaIN retains its structural integrity better than WCT. They suggest replacing the phase information for better structure preservation based on this research. ", "summary_of_the_review": "Overall, the idea of the paper is interesting, but further research is needed.", "strength_and_weaknesses": "Pros\n* This paper provides a new interpretation based on the frequency analysis. \n* The paper is written clearly and idea is reasonable. \n\nCons\n* The frequency combination (FC) does not seem to give good stylization effect. It is well-known that the phase information contains the structural information of an image. In style transfer, one may want to have strong strokes or textures. However, since the proposed method replace the phase information directly, this is hard to be done by design. To me, this method seems to work better for photorealistic stylization, not artistic.\n\n* Even the metrics shown in Table 1 is using SSIM, which is the metric for checking structural preservation, not perceptual quality. If this method is meant for artistic style transfer, please measure some perceptual metrics such as GRAM difference to the results of the original optimization method and user study. That said, please perform user studies asking various aesthetic aspects using randomly generated results. \n* Missing comparison to relevant work. The problems of spatial structure distortion are also addressed by SANet and AvatarNet, which are reported to preserve content structure pretty well with nice stylization effect. The writers ought to contrast their approaches with similar ones.\n* Overall, the results seem to be very weak. For example, the authors say that the FC gives better controllability and stylization by manipulating them in frequency domain (eq. 12). However, in my opinion, this gives more unnatural effect at the edges (e.g., boundaries of buildings, Figure 10) compared to the linear combination. Where to apply styles in terms of frequency difference seems not to be a good choice since content of an image is a more entangled entity that cannot be detached merely by frequencies. \n\nMinor\nLine 58, Next, We -> Next, we\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7mYzQJ6j2E", "reviewer": "Reviewer_eNXa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a systematic Fourier analysis on the universal style transfer framework. It reveals the relationship between Fourier amplitude / phase with Gram matrices / a content reconstruction loss. Based on the analysis, this paper proposes a new structure preservation method by phase swapping and perform non-uniform style transfer based on frequency. ", "summary_of_the_review": "This paper provides a novel view of UST framework in frequency domains and validate the analysis with comprehensive experiment as well as algorithms of structure preservation and non-uniform style transfer. The analysis is interesting to me and may enlighten the follow-up style transfer researches. ", "strength_and_weaknesses": "**Strengths:**\n+ The analysis of current universal style transfer framework in Fourier frequency domain is interesting. And based on the analysis, the authors leverage the characteristics of the frequency domain for structure preservation and non-uniform style transfer. I think the idea is interesting and insightful. The usage also proves the application value. \n+ The analysis is not limited to a simple style model but a general UST framework. \n\n**Weaknesses:**\n+ Comparing to the phase part, relating amplitude to Gram matrices is less convincing. Since amplitude is only related to the diagonals of Gram matrices, which is more similar to the channel-wise second order statistics of feature maps in the style loss of AdaIN than the Gram -based style loss. The amplitude didn’t explain the key inter-channel correlations in the Gram matrices. Compared to AdaIN, WCT also uses inter-channel correlations to model image styles. Therefore, I think it will be more appropriate to use other word than Gram matrices here. \n\n**Some small issues:**\n+ Some typos. For example, ``Next, We demonstrate``, \n+ In Table I, why LinearWCT has better scores in Time but the scores of AdaIN is bold? \n+ Figure 4, (b) is the style transfer results rather than the style image. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject