Completeness and Coherence Learning for Fast Arbitrary Style Transfer
Authors: Zhijie Wu
Chunjin Song
Guanxiong Chen
Sheng Guo
Weilin Huang
Venue: Accepted by TMLR ()

Abstract: Style transfer methods put a premium on two objectives: (1) completeness which encourages the encoding of a complete set of style patterns; (2) coherence which discourages the production of spurious artifacts not found in input styles. While existing methods pursue the two objectives either partially or implicitly, we present the Completeness and Coherence Network (CCNet) which jointly learns completeness and coherence components and rejects their incompatibility, both in an explicit manner. Specifically, we develop an attention mechanism integrated with bi-directional softmax operations for explicit imposition of the two objectives and for their collaborative modelling. We also propose CCLoss as a quantitative measure for evaluating the quality of a stylized image in terms of completeness and coherence. Through an empirical evaluation, we demonstrate that compared with existing methods, our method strikes a better tradeoff between computation costs, generalization ability and stylization quality.

Published in Transactions on Machine Learning Research (09/2022)
Completeness and Coherence Learning for Fast Arbitrary Style Transfer
Zhijie Wu
shijienn@cs.uke ubc.ca
Deportment of Computer Science University of British Columbia
Chunjin Song
chunjine@cs.uube.. ubc.ce
Deportment of Computer Science University of British Columbia
Guanxiong Chen
grchen@cs. wwwpnnc.sss...b.... ubc.ca
Department of Computer Science University of British Columbia
Sheng Guo MYbank, Ant Group
guosheng1001 @gmail com
Weilin Huang
weilin h&hoommall com
Alibaba Group
Reviewed on OpenReview: https: // // openreview. net/ forum? id 4N6T6Rop6k
Abstract
Style transfer methods put a premium on two objectives: (1) completeness which encour- ages the encoding of complete set of style patterns; (2) coherence which discourages the production of spurious artifacts not found in input styles. While existing methods pursue the two objectives either partially implicitly, present the Completeness and Coherence Network (CCNet) which jointly learns completeness and coherence components and rejects their incompatibility, both in an explicit manner. Specifically, we develop an attention mechanism integrated with bi-directional softmax operations for explicit imposition of the two objectives and for their collaborative modelling. We also propose CCLoss as quanti- tative measure for evaluating the quality of a stylized image terms completeness and coherence. Through an empirical evaluation, we demonstrate that compared with existing methods, method strikes better tradeoff between computation costs, generalization ability and stylization quality.
1 Introduction
Regarding neural style transfer problems (Jing et al. 2020bb. Gatys et al. (016 has demonstrated that a pretrained VGG network (Simonyan & Zisserman 2015) can produce features that entail content structures
and style patterns in their correlation. But the optimization-basee method (Gatys et al. 2016) is prohibitively
slow and its range of application is fairly limited. Since then a great number of efforts (Johnson et al.) 2016
Chen et al. 2017 Sanakoyeu et al.) 22088 Gu et al. 2018 Chen et al. 2021 Deng el. al. 2022) have been dedicated to striking the balance between speed, generalization capability and stylization quality. To achiove fast and arbitrary style transfer, Statistics-bees methods adjust the holistic statistics of a content image
1x:: Corresponding author

---

Published in Transactions on Machine Learning Research (09/2022)
Content
Style
Full Coherence
Durs
Full Completeness
(a)
(b)
Figure 1: (a) Given a content-style image pair, the full coherence learning of input styles produces concrete but incomplete patterns while the full completeness learning presents rich textures with white distorted artifacts. (b) Instead, CCNet can capture notable holistic styles comprehensively and fine-grained details laithfully by jointly capturing completeness and coherence.
in accordance to that of a style image (Huang & Belongie 2017 Li et al. 2017 Sheng et al. 2018 Zhang
et al. 2019 Li et al. 2019 Wu et al. 2020). These methods can capture a diverse set of styles (indicating completeness) but often fail to synthesize style characteristics faithfully, as they often introduce uncxpectod distorted patterns. Patch-based methods on the other hand swaps local feature patches (Chen & Schmidi,
2016
Park & Lee 2019 Although these methods do not suffer much from pattern distortions, and excel synthesizing fine-grained style details (indicating coherence), they often cannot synthesize a complete set of input styles, and they repeat undesirable style patterns such that blureed contents are produced.
The aforementioned two lines of work dedicates to either completeness or coherence. Others have sought to achieve both objectives at once: Gu al. (2018); ; Sheng et al. (2018); Park & Lee (2019); Wu et al. (2021); (2022) have already attempted to unify statistics- based and patch-based methods. Yet their solutions are either partial or indirect: without explicit, direct formulation of coherence comploteness, fully exploiting the two transfer priors is impossible. Sceing the shortcomings of our predecessors, we formally define completeness and coherence in the context of style transfer: for an input style feature F, and initial stylized result Fes, completeness requires as many patches in F, as possible to be preserved in Fex; coherence requires all patches in Fcs to be from F, (Eq. [1. In fact, our formulation of the two transfer priors resonate with symmetric Chamfer matching (Sinakov et al. 2008 Fan et al. 2017
Liu et al.] (2021): : Deng al. (2022): ; Zhang
Yang et al. 2018). Our explicit definition allows us to 1) explicitly impose completeness and coherence as stylization objectives that lead to unbiased results; 2) parse the relation between completeness and coherence features for compatibility learning: 3) identify the optimal trade-off between the two objectives by tuning their relative weights. To this end, we introduce paich-based similarity measure (Simakov et al. 2008 to both forward propagation during inference and back propagation during training.
Hence we devise the Completeness and Coherence Network (CCNet), a dual-branch style transfer framework with feature diffusson networks inside leverage our formulation of completeness and coherence. The gist of the CCNet the Non-local Diffusive Attention Module, which further stylizes Fax by completeness and coherence while preserving its spatial structures: based on the semantic correlation between Fos and F, (Sheng et al. 2018), update each patch in Fes by its closest patch in F, for coherence modeling. Similarly, we propagate each patch in F, to its closest patch in Frs for completeness modeling. This joint analysis paradigm aims to learn the interrelation of these objectives, as we assume that compatibility between completeness and coherence feature leads to better stylization.
Turning to our implementation- -ling at the heart of the Non-local Diffusive Attention Module a bi- directional softmax operator (Fig. 5): each direction of the operator situated in one branch, for CCNet to learn of the two objectives at low computational cost, before the two branches collaborate for joint
2

---

Published in Transactions on Machine Learning Research (09/2022)
Content
SANet PLoss
SANet+ CCLoss
CCNet PLoss
CCNet CCLoss
Style
Figure 2: We show that both CONet (for feed-forward inference) and CCLoss (for backward training) are effective by comparing them with SANet and perceptual loss. Compared with perceptual loss, CCLoss significantly reduces distorted patterns in the background and halation around edges (check the patches joined by the red curve). Compared with SANet, CCNet renders a more diverse set of styles, as we can see from the color variation in the sky (check the patches joined by the blue curve). Neither CCNet (for forward propagation) nor CCLoss (for back propapropagttion) can lead to satisfactory performance on its own. Check the appendix for more results.
analysis. Finally, a learnable decoder reconstructs a stylized image from learned features. Inspired by prior work in which symmetric Chamfer matching was used to formulate losses (Simakov et. al. 2008 Fan et. al., 2017 Yang et al 2018 propose the completeness and coherence loss (CCLoss) measure to train our network. Unlike the holistic perceptual loss (PLoss) (Johnson et al. 2016).
patch based similarity CCLoss
allows us to explicitly balance between completeness and coherence. Its patch-wise nature also helps with
representing detailed style patterns (Fig. [ We also apply the identity loss (Park & Lee 2019 to speed up training and to maintain content structure without compromising style richness.
To the best of our knowledge, we are the first to explicitly capture completeness and coherence and analyze them altogether in patch-based manner. Our contributions are follow:
We explicitly introduce completeness and coherence to style transfer, and we present a novel framework (CCNet) and loss function (CCLoss) to fully exploit the two objectives via joint analysis.
We design an improved softmax operator (Fig. 5 for fast completeness and coherence modeling, and we investigate the impacts of our architectural design choices via a comprehensive ablation study.
We demonstrate that CCNet better at capturing diverse and coherent style patterns than existing methods, and that it achieves better stylization quality without incurring heavey computation burdens.
2
Related Work
Style transfer. Here we only review the most relevant methods for neural arbitrary style transfer, and we refer readers to Jing et al. (222bb) for comprehensive survey.
Gatys et al. 2016 is a pioneer work that achieved impressive stylization results. Nevertheless it was built upon time-consuming optimization method; moreover, it attends only to completeness, but does not address style pattern distortions at all (Li & Wand 2016a Chen & Schmiddt, 2016 Sheng et al., 2018 Since then, Chen & Schmidtt (2016) realized fast arbitrary style transfer with a patch-swap operation for intermediate learned features, but the method cannot parse the complete set of style information. Several
later approaches (Huang & Belongie 2017 Li et al. 2017 Sheng et al.) 2018 Song et al.] 2019 Li et al.
2019 Jing et al. 2020a) propose to replace the statistics of content images with those of style images. Taking advantage of the strong representation power of second-order statistics, they (Li et al. 2017 Song et al, 2019) are able to render a rich set of style elements, hence achicving completeness. But these methods often produce spurious artifacts and distort both spatial layouts and style patterns in their renderings, thus being incoherent with respect to their inputs. SANet Park Lee 2019) can produce the most coherent results w.r.t. input styles, since it directly combines content features with their closest patches from style features. But it often biases too much towards contents because the matched style patches strictly follow local content structures. Therefore it often fails to render complete representation of input styles.
3

---

Published in Transactions on Machine Learning Research (09/2022)
F
L
VGG
CCNet
Cokerence
Joint Analysis Attention
Atlention
Eush
Fis
one
Dec
Ps.
F
VGG
CCNel
F
10
F
Completeness Attention
Com
Joint Analyais
Attention
Cr
(a)
(b)
Figure 3: (a) Architecture of the entire style transfer pipeline. A VGG-based encoder transforms content image le and source image L, to Fe and F. for i E {4,5} respectively. Each feature map is then fused by a CCNet module. The output from each CCNet is then summed up and passed to a trainable decoder to produce images Tes- (b) Architecture of a CCNet module. Content feature Pg and source feature F, are passed diffusive attention modules (Fig. 4 to model completeness and coherence. Then Feoh and Foom are passed to Joint-Analysis Attention Module to compute skip connections for better compatibility between Feoh and Feom Finally, the stylization feature Fee is produced via an element wise multiplication.
While the aforementioned work can achieve either completeness coherence but not both, later work (Gu let al. 2018 Sheng et al.) 2018 Liu et al.) 2021 Deng et al.) 2022) aim to model both completeness and coherence, but only indirectly without individually formulating completeness and coherence features. Specif- ically, Gu et al. (2018) proposes a feature reshuffle module to boost style richness by reducing the chances to search similar style patches. [Sheng et al. (0118) matches normalized feature maps before adjusting their statistics to mitigate conflicts belween content structures and target styles. Liu et al. (2021) boosts its ability to synthesize details by adaptively performing attentive normalization
per-point basis. Other similar statc-of-tho-art methods include internal-external contrastive learning (Chen et al., 2021 paramctric style composition (Wu el al. 2021); adversarial learning (Xu et al.
2021 vision transformer (Deng et al.
2022) and domain enhancement based style projector (Zhang et al.
2022
For arbitrary stylization, our approach follows the local patch alignment paradigm, which is closely relatod to patch-based methods (Li & Wand 2016b Chen & Schmidti, 2016; [Sheng al., 2018). But unlike these methods, we align images by explicitly imposing the completeness and coherence of input styles and jointly analyzing them. This transfer strategy can simultaneously preserve the richness and fine-grained details of style patterns better than prior. While other methods have yet to achieve this, the CCLoss allows us to balance completeness and coherence by adjusting their relative weights. As an ablated form of CCNet, SANct (Park& & Lee 2019 only takes coherence into account; see Fig. 2 and Sec. 3.2 for detailed discussions.
Symmetric Chamfer matching. Chamfer matching (Barrow al. 1977 has a multitude of applications in computer vision and graphies (Borgefors) 1988 Thayananthan al. 2003 Ma et al., 2010 Wu et al., 2019). Simakov et al. (2008) first introduced symmetric Chamfer matching to the field of visual summary by explicitly delining completeness and coherence. In the context of deep learning, several work (Pan el al. 2017 Achlioptas et al.) 2018: Yang et al.) 2018) have applied similar ideas to reconstruct 3D point clouds. Since then, symmetric Chamfer distance has been extensively adapted to mcasure the synthesized quality of 3D shapes. But to the best of our knowledge, we are the first to realize style transfer with explicit and full completeness and coherence modeling. Unlike existing methods, we perform completeness and coherence modeling in both feed-forward style inference and backward network training to maximize their reprosentation potentials. Moreover, we aim to jointly analyze these two objectives at inference. We present evaluation of these design components in Fig. Fig. and in the ablation study in Sec. 4
3
Method
Fig. 3 illustrates the overall architecture our style transfer pipeline. Our discussion follows a bottom-up
approach: we start off with detailing the gist of our CCNet-tthl Non-local Diffusive Attention Module in Sec.
3.1
The module leverages affinity matrix (Jiang et al. 2018 Wang al. 2018 Zhu al.) 2019 which allows
the output style feature to encapsulate richer style patterns and more fine-grained details. Then in Sec. 3.2
4

---

Published in Transactions on Machine Learning Research (09/2022)
Fc
F
Fc, Ps: normalized Fc, Fs
Ho, Wo.C)
1x1 Conv
[B. Ho.HoWe.Oo C)
Acom (Acok)
Matrix multiplication
Column-wise
(Row-wise)
Ps
Fs
Softmax
F Fuom (F (Fooh) He. Wo,C)
Hg C)
1 1 Conv
(B, HeWe, HyWa)
(gyC)
Fs
Conv
(..))W
Figure 4: The Non-local Diffusive Attention Module as a fundamental building block of the CCNet in Fig, 3 (b). Given Fe with shape (B, He. We. and F, with shape (B, H,, We, C) as imput content & style features, use diffusion network to produce completeness (coherence) feature Feom (Fcoh)-
Style Patches
H&W
HW Style Patches
Puchos H&y
HAW
Acoh
Content Feature Fe
Style Feature Fs
Acm
Figure 5: Here detail the five steps to compute Feom from Fig, 4 1) First, we divide content feature F. and source feature F, into 9 patches. 2) Then compute 9 x 9 affinity matrix A to measure affinity between content patches from F, and style patches from F.. 3) Next, we apply the softmax function along each column of A to produce the completeness matrix Acom- Each of its element. A'd tells how much style information from the j-th style patch should propagate to the i-th content patch. 4) So for each style patch, we can identify its "nearest content patch" with the largest value max; A (see the blue cells) ensure every style patch can be included in the stylized content feature, hence enriching stylized patterns, 5) Finally, for completeness transfer, diffuse each source style patch to its closest content patch through From Acam F. to encourage more diverse results. Here 5." is dot product. Similarly, we compute the coherence matrix (Acoh) by computing softmax along each row of A, identify the largest values (orange cells), look up for the nearest style patch for cach content patch and finally produce Fooh- We visualize the closest content patch (w.r.t) a style patch in teal and a closest style patch in red.
we discuss the Completeness- Completeness-Cohheeenee Network (CCNet) module with four attention modules instantiated inside. While the Coherence (Completeness) Attention Module focuses solely on its own objective, each Joint Analysis Attention Module processos the learned coherence and completeness feature collaboratively. In Sec. we discuss our overall architecture, with emphasis on the CCLoss which is for balancing the effects of completeness and coherence while preserving details. Finally we discuss implementation details Sec. 3.4 For more discussions on our model design, please refer to the appendix.
3.3
3.1 The Non-local Diffusive Attention Module
Inspired by Fan et al. (2017); Simakov al. (2008), we assume a stylized feature F... visually coherent and complete style feature F, if many possible patches of Fes are preserved in F,, and vice versa. Namely, for each stylized content patch in Fes, we search for its most similar patch in F, and evaluate their
5

---

Published in Transactions on Machine Learning Research (09/2022)
distance, and vice-versa.
d( Fes: Fo)= min Q) -
E
Σ
min D(Q,P) PCF..
(1)
QCP.
PCF
QCF.
where P and Q denote patches in Fos and Fs respectively. The first term in Eq. measures deviation from coherence; the second term measures deviation from completeness. Neither completeness nor coherence on its own is enough to produce good style transfer.
We use F. to denote initial stylized feature, and we use it alongside F. to compute final stylized feature Fes. Similar to Simakov et al. (2008), we propagate F. to Fe in patches to optimize the similarity in Eq. [ Specifically, for each patch Q C F., we find its closest patch P C F. and apply the patch Q to update / such that all style patches in Fg are incorporated into Fes, leading to more diverse stylized results. Looking at the other way round: for each patch P C Fes we find the closest patch Q C Fs and apply the patch Q to update P' for coherence, leading to more adaptive results. We then implement this symmetric nearest neighbor scarch with softmax opcration along each axis of an affinity matrix (check Fig. for dctails). This feature update procedure encourages to employ the non-local diffusion architecture (Jiang et al. 2018 Wang
et al. 2018) in Fig. A to achieve this motivation. We also create instances of the non-local diffusive module to jointly analyze completeness and coherence for compatibility learning. Although overlapping patches with larger sizes are better at capturing coarse scales and modeling relationships within a local region, we set the patch size to order to balance visual performance and computation cost.
3.2 The Completeness and Coherence Network (CCNet)
CCNet (Fig. 3 computes the stylized feature Fes by spatially rearranging F, with Fe. To do this, we create multiple instances of the Non-local Diffusive Attention Module: Coherence and Completeness Attention for modelling completeness and coherence, followed by two Joint Analysis Allention modules.
Following Sheng et al. 20188) Park & Lee 2019 first normalize Fe and Fg to remove their style informa- tion so that the later style diffusion can be realized based on the content structures of input images, yiclding F. and F., Turning out attention to the Completeness (Coherence) Attention Module: Let softmar; be a softmax operation along the 1thh axis and the starting index of axes be O. Following our discussion in Sec. 3.1 we compute features for coherence and completeness az
Feoh = softmarz2 ( (Fc) . ..g0h (F.)²) (Fs):
(2)
Foom so ttma (V (Fc) v.g (F.)²) v.co (F.).
(3)
Here (ygoh, veehh weeh teeom v sleom ()gom and 9." represent learnable convolutions and dot-product similarity individually; see Fig. 5 for more details.
Next, we feed Fooh and Feom into the Joint Analysis Attention Module (Fig. (b)). We instantiate Joint Analysis Attention for two reasons: 1) We aim to improve pixel-level compatibility between Fooh and Feomi 2) We aim to combine the information at different positions of an image to capture the long-range dependen- cies between pixels. To these ends, we feed Fooh and Foom to another shared diffusion block which allows us to rearrange feature vectors of Feoh (From) to fit Foom (Feoh) well; see the appendix for more details. Thus we compute two residual features as:
Froh (attmaazyaaaFFaa. (Fcom) (aom))).oFFo (o(FFoh) vin ( (Fooh). From = so coftmazz (Fcor))) (bg (Fccm) )) v/ (Fcom),
(4)
(5)
where (Wh, Vn, 3 denotes learnable convolution parameters and denotes dot-product similarity. Then we update Froh and From as Faoh Pecsh + Feah: From From + f.om- And we fuse Froht and Feom together as Fes Fooh O From to further facilitate their compatibility in a channel-wise manner, where @ indicates an element-wise multiplication. Finally, we merge F. into Fes to better preserve the input content structures as Fese = F..+t. F..
Both sicoh (Fa) ((Fe)---ooo (goo((F.)) (F.)T and weom (Pa)- wgom (F.)T measure the affinities between F. and F. whose shape is [batch size, resolution Fe, resalution of Fe]; check details the appendix.
6

---

Published in Transactions on Machine Learning Research (09/2022)
Discussion on SANet. Both our CCNet and SANet use the non-local network (Wagg et al. 2018 as fundamental building block, since both of our networks measure feature correlations and then diffuse target style features to content features. But our approach differs from that of SANet in the follow aspects:
As mentioned in Sec. 2 our motivation differs from SANet's. Unlike SANet which matches semantically nearest style features onto content features, we take complete styles into account. Our approach allows address coherence and completeness more comprehensively than existing patch- and statistics- based methods, shown in Fig. and 6
As far model architecture, we can think of SANet an ablated form of CCNet: if we keep the Coherence Attention Module only, and strip away the Completeness Attention Module well as the Joint Module
then we have SANet. In Fig. 2 and [8]
we show that CCNet with minor architectural changes from SANet
yields remarkable visual improvement.
SANet trained with perceptual loss [Johnson et al. 2016:; which generates style distortions and in- troduces blurr halation around the odges in Fig- 2 Our CCLoss yiclds much neater contours and less distorted style patterns via coherence modeling and patch-wise computation.
3.3 Style Transfer Pipeline with CCNets
Aside from putting in the CCNet modules, in style transfer pipeline retain the most of SANet's architecture, so that when benchmarking for completeness and coherence we can attribute any performance improvement directly to CCNet: see Fig, 2 and Fig, 6 In Fig. 3 (a), our entire pipeline consists of three parts: one VGG-based encoder (denoted as E), a symmetric decoder and two CCNet modules. The VGG- based encoder takes a content image Le and a style image 1g as imputs, and produces content feature map Ft and style feature map Fú, where stands for the map being produced by the relu_ i layer of the encoder. We instantiate two CCNet modules in our framework to caplure the style patterns at multiple scales with modest computational cost. Each CCNet takes the content and style feature map from single layer (relu 4 or relu_5) 5) as inputs and synthesizes the stylized features as:
F1 eac CCN F.), F CCNNtt(F F.).
(6)
Then Fesa and &ec are fused as:
esc =
F ese + u( F
(7)
where the u(-) stands for upsampling. Finally, a trained decoder maps Fise back to stylized image Lee which possesses the content structures from Ie and the style patterns from I..
Loss function. Our loss function for jointly training the two CCNets and the decoder consists of two parts:
Liotal Lidd + Lec-
Lia is the identity loss PPar & Lee 2019) which speeds up training and maintains the content structure without losing richness of styles. Following Park & Lee (019)). we define the identity loss as Lia Mabba + Li and both Lia and Lid are computed as:
1 TTe Tell2 1|T.- T2l12,
(9)
L3 =
E (loc) - (3ool)1 + ([E'( - - (1.)ll,
(10)
where Toc (or Tax) denotes reconstruction results from stylizing two identical content images (or style images) and E-)-) (-) indicates the features outputted by the layer relu_i i of the VGG encoder.
Lee the Completeness and Coherence Loss (CCLoss), which we use to facilitate the fine style patterns and control relative effects of completeness and coherence. Following Fq. T we set patch size to and employ the
7

---

Published in Transactions on Machine Learning Research (09/2022)
cosine distance our affinity measure. Then have Lee Xcom Lee
Lec
AechLeeh
Lee plE)"(la)
E
min D(p".q") (L)
min p¹ p'EEa(la))
D(p'.g")
(11)
(EE
L Leom ('EE"(L))
E
min (la)
D(q p +
min D(g .p"), p (lea)
(12)
('EE"(I)
where (p',9'), E (4,5) are achieved via the encoder (-(-) for transfer results and input style images individually, The distance between p' and q¹ is:
(p',9")
D(P ( =1 I|e'll |1
(13)
Here we only calculate the CCLoss for feature maps produced by the relu and relu 5 layer of the encoder, corresponding to the imput to each CCNet.
3.4 Implementation Details
We use 80, 000 images from MS-COCO (Lin al.) 2014) and 80, 000 images from WikiArt (Nichol 2016) the content and style dataset respectively for training. We initialize the encoder with pro-trained VGG network (Simonyan & Zisserman 2015) and freeze it during training. As far as the decoder, we take the same setting from Huang & Belongie (2017). We also apply the Adam optimizer (Kingma & Ba 2015) with
batch size set to four image pairs, and learning rate set to le-4 for 200K iterations. During training, first
we resize the smaller dimension of each image to 512 but keep the initial ratio. Then we randomly crop a
region of size 256 x 256. But testing an input image can be of any size. Throughout our experiments, we set M Vid Xea and Xcoh respectively 50, I, 300 and 5.
4 Experimental Results
First we compare our approach with several state-of-the-art methods qualitatively and quantitatively. Note that all of our results on the baselines are obtained from publicly available, pre-trained models under their default settings. Then we show results from our ablation study in which we investigate the impact of several design decisions. At last, showcase two runtime applications of our method to demonstrate the flexibility of our model. In the appendix we show more light version of the Non-local Diffusive Attention Module, and we provide more details on our experiments. We will release our source code upon publication.
Qualitative comparison. We present qualitative results from our method and the baselines in Fig. 0 and Fig. 7 While evaluating artistic style transfer still an open problem within the vision community [Li al. 2017 Zhang et al. 2022 we focus on matters the community deems be the most pressing: style distortions (Shngg et al. 2018). clear outlines of prominent objects (Zhang et al. 2019), full style modeling Li et al.
2017] and reduction of repetitive patterns (Park & Lee 2019
2016)) achieves arbitrary style transfer with a slow optimization method and often generates unstable results with distorted spatial layouts and style decoration (e.g. 1s and 4th row). While AdaAttn [Liu et al. 2021 can preserve content structures well, it fails to capture some prominent colors (e.g. the conspicuous oranges 4thh row) and target textures (e.g- the block-wise patterns with black outlines in the 2nd row) in most samples. It only adjusts colors for some but not all regions and produces spurious blob-like artifacts in the background (e.g. the 1st and 3rd row). WCT Li et al. 2017) enhances its style representation ability by matching the covariance matrices of style features. However, it cannot produce fine-grained styles (e.g. the plume-like textures in 3rd row and block-wise appearances in the 5th row) and often blurs the content structures with distorted patterns (e.g. the 2nd row). AvatarNet (Sheng et al. (018)) introduces a feature decorator to facilitate the adaptive style patterns, but still blurs the fine ingredients (c.g. the plume-like textures in the 3rd row and brush strokes in the 4th and 6th row). It can not keep semantic structures (e.g. the 2nd and 4th row) as well and its background is overlaid with unseen colors (e.g. the 3rd row). For SANet. (Park & Leel 2019; its style feature alignment biases towards content structures
In Fig. Gatys et al.
8

---

Published in Transactions on Machine Learning Research (09/2022)
Content
Garys
AdaAttn
WCT
AvatarNet
SANet
Ours
Style
Figure 6: Comparison with prior methods. As mentioned in Sec. we compare our method with the prior ones how well we tackle the challenges that the style transfer community deems to be critical: style distortions (Sheng et al. 2018), blurry object outlines (Zhang et al.
2019 Park & Lee 2019), repetitive patterns (Zhang et al. 2019) and lack of style richness (Li et al.] 2017). Our method is better at reflecting full style elements (e.g- color distribution in the sky of 4th row), introducing faithful textures (e.g. plume- like textures in 3rd row), generating neat contours (c.g. 1st, 2nd 3rd and 4th rows) and reducing distorted patterns in background (e.g. 1st and 3rd rows). Looking into visual details, we see CCNet preserves richer fine aue-grained textures (e.g. the block-wise appearances with different colors in 5th row and the brushstroke- wise patterns in last row). Neither SANet nor AdaAttn can perceive fine-grained textures (5th row); they also repeat the eye pattern over the entire image (as red arrows shown in last row) which is not desirable. SANet. also introduces blurry edges with halation and generates unseen cluiters in background in the 5th row. Also, AvatarNet and WCT misrepresent the imput block-wise and brushstroke- wise style details and Gatys et al. (2016) fails to capture the holistic color distribution. More detailed evaluation is included in the appendix; zoom in for better visualizations.
too much. Therefore it cannot always parse complete set of style patterns (e.g. colors in the sky of thh

---

Published in Transactions on Machine Learning Research (09/2022)
Content
ArtFlow
MCCNet
StyTr2
CAST
Ours
Style
Comparisons with more state-of-the-art methods in terms of representing details. As discussed in Fig. G again our method outperforms the prior at preserving holistic color distributions and variations (1st row), as well as al synthesizing brushstrokes (1 row) and block-wise textures (2nd row).
Figure 7:
row) and often repeats undesirable patterns in smooth regions (e.g. distorted artifacts in the background of the 3rd and 5th row). Moroover, SANet tends to blur out edges (e.g. the 4th and 5th row). Looking closely at the details in the last row, we both SANet and AdaAttn copies the eye over the entire content image (highlighted by the red arrows).
In parallel, in Fig. 7 ArtFlow AAn et al. 2021 | and CAST (Zhang et al. 2022 consistently produce unnatural artifacts over smooth background while IEST
Chen et al. 2021 and Sty (Deng el al. 2022 are weak in discerning significant textures (e.g. the colorful brushstrokes on 1st row and blob-wise patterns on 2nd row).
All methods including MCCNet (Deng et al., 2021 present unseen halations.
Our method generalizes well to multitude of styles, from holistic color distribution (e.g. skies with diverse colors in the 4th row) to local brush strokes (e.g. the last row) and detailed textures (e.g. block-wise patterns in the 2nd and 5th row). Therefore we have demonstrated that our method can catch more style ingredients while faithfully preserving details from target styles. Meanwhile, the CCLoss allows us to significantly reduce various artifacts (e.g. the 3rd row) and render sharp contours in all cases. We can reach the same conclusion in Fig. 7 for ArtFlow, IEST, MCCNet, StyTrr and CAST.
Quantitative comparison. Following Li et al. (0177) our method against baselines quantitatively with the perceptual style loss. We randomly pick 20 content images and 30 style images from our test set to synthesize 600 stylized images, and list losses in the first row of Tab. D We see 1) CCNet can already achieve a decent style loss even if optimized with the CCLoss only; 2) when trained with both CCLoss and style loss, CCNet yields a remarkably lower style loss than other methods: L, 0.4634, Loom = 0.3087 and Leok 0.3058. These results confirm that explicitly modelling completeness and coherence allows CCNet to capture style patterns more efficiently than the baselines.
: Song al. (2019); Deng et al. (2022), we assess
Also, following (Chen al. 2021 Zhang al. 2022 we use the Deception score to mcasure the closeness between synthesized images and images created by human artists. Specifically, we take the 30 style images reference, then for each method compute the percentage of stylized outcomes mistaken as hunan-created. We collect 3,000 responses from 30 participants and use the average votes the measurement. We also compute the CCLosses defined in Eq. I and Eq. 12 to quantitatively assess coherence and completeness. We see CCNet achieves the lowest scores on the three metrics, comprehensively demonstrating its superior performance over its predecessors.
In the 4th row of Tab. | we report the average running time of each method over 100 test images. Gatys et al. (2016) is the slowest due to its slow optimization regime. SANet achieves the fastest speed due to
10

---

Published in Transactions on Machine Learning Research (09/2022)
Content
Mesh
Mcon
Masnt
Mail
Style
Figure 8: Ablation study on completeness and coherence modeling as well as joint analysis. Molt denotes the unablated CCNet. Mooin refers to the CCNet with Joint Analysis Attention Modules removed; Mooh refers to the CCNet with the Completeness Attention Module removed; Moom refers the CCNet with the Coherence Attention Module removod.
Content 1:1
1:100
1:200
Style
Figure 9: Results obtained by gradually biasing stylization towards coherence. A larger coherence-over- completeness ratio allows the model to capture style patterns (e.g. the dominated textures in the sky and wall) more completely, but would lead to more distorted patterns (e.g. the undesirable gray in the sky and blurry textures on the wall).
its simplicity; WCT and AvatarNet take more running time since they both require the SVD operation. CCNet's running time is fairly competitive when compared with the baselines, and in fact comparable to that of SANet (Pak & Lee 2019).
We then conduct a user study to assess the visual appeal of images rendered by all methods. First, we randomly select 25 content images and 30 style images from the test datasel, and create their complete combination synthesize 750 stylized images for each method. For each participant, 10 stylized images of CCNet and one of the other methods are displayed side-by-side in random order. Thus, cach participant needs to make 90 votes for 9 baseline methods and receive 3150 response in total. During user study, the participants were asked to choose the image that learns the most characteristics from the style image. Specifically, the participants were told that the preservation of significant style patterns was the primary evaluation point. Additionally, the assessment time is longer than 30 seconds for each question so that each participant can make careful decision. We list their preference scores in the last row of Tab. [ The Fig. 20 further illustrates the age and gender distributions of the 35 participants. It can be seen that, the ages and genders of the participants spread uniformly, which clearly indicates the validity and unbiasedness of our experimental settings.
Ablation study. To investigate the impact of completeness and coherence modelling to the performance of our method, we first. remove the Joint Analysis Atlention Modules from our pipeline, so the network would not seek for the best compatibility between completeness and coherence. From Fig. g we see the ablated model (Mooine) distorts textual details and produces weird black pattern the sky. Further, we remove the Completeness (Coherence) Attention Module to prevent our pipeline from explicitly imposing the completeness (or coherence) property. We see the model with only the Coherence Attention Module (Mook)
11

---

Published in Transactions on Machine Learning Research (09/2022)
Table 1: Following prior work, we assess the methods quantitatively with the perceptual style loss, stylization speed, deception score and user preference. We also CCLoss evaluate the completeness (Lcom) ) and coherence (Looh) of rendered images. We measure stylization speed by running time and report them in
seconds. The preferences represent the percentage of votes that deems a baseline's result is inferior to ours.
Loss
Gatys
AdaAttn
WCT
Avatar
SANet
ArtFlow
IEST
MCC
StyTr2
Ours
SLy)e(L,)
0.5751
1.1561
0.5620
1.1019
0.6215
1.9667
2.0925
1.0710
0.7334
1.0911
Trom +
0.3263
0.3393
0.3293
0.3194
0.3204
0.3478
0.3530
0.3356
0.3366
0.2979
Leoh
0.3332
0.3255
0.3397
0.3150
0.3222
0.3501
0.3376
0.3315
0.3306
0.2795
Speed
56.81
0.1083
0.4952
0.6772
0.0716
0.6139
0.0914
0.0758
0.7453
0.0908
Deception score
0.36
0.30
0.39
0.40
0.44
0.27
0.29
0.24
0.30
0.54
Preference 0.814
0.740
0.780
0.749
0.657
0.786
0.723
0.791
0.700
cannot render the diverse set of styles from the original style images (e.g. colors in mountainous areas). While the model with the Completeness Attention Module only (Meom) retains more salient patterns, it introduces spurious artifacts in the sky. Hence we can confirm completeness and coherence modelling as well the joint analysis are all essential to the foed-forward process.
We also demonstrate the crucial role of completeness and coherence modelling in training in Fig. 2 CCNet on its own cannot produce visual satisfactory results- see the result produced by CCNet trained on perceptual loss. But our objective function (CCLoss) improves the result notably- patterns become less distorted and edges got sharpened up. We then evaluate the effect of each term CCLoss by gradually increasing the ratio of coherence weight over completeness weight (Ace Accm).. Fig. g shows that raising the ratio leads full style modeling but introduces more distorted patterns. So we conclude that CCLoss allows us easily strike the balance between completeness and coherence by explicitly tuning the weights.
Runtime applications. We demonstrate the flexibility of CCNet by showcasing two runtime applications with a trained stylization model: 1) Adjusting the degree of stylization at runtime (Fig. 10h. We can strike a balance by interpolating between two feature maps, Fese and Fece as: Fese + aFFec + (1 1--a aa Va E [0, 1]. Feee represents the output feature given two identical content images as inputs. The network either replicates the content image if we set a 0, or produces a fully stylized image Ies if we set = 1. We show that changing a from 0.2 to leads to a smooth transition. 2) Segmented stylization of an image (Fig. [I.. Inspired by WCT (Li et al. 2017),
apply k masks M {M,, M2, ,M, to indicate the spatial correspondence between l image regions and desired styles (J, I2, ... Tx). Then we extract specifie content region, f = M2 O Ie, where @ denotes simple mask-out operation. Next, we achieve the specified stylized image
f. by leeding ft and 1 into CCNel as ft = CCNel(f 0, 1t) @ Mj. The final spatially-contrrlleed sty vliization Ice is produced by combining all P ....... 1&) as Ice = E.Po.
5 Conclusion, Limitations, and Societal Impact
We make the first attempt to explicitly define completeness and coherence for style transfer in patch-basod manner. We propose the Completeness and Coherence Network (CCNet) as well as the CCLoss to explicitly impose the two objectives and model their inter-dependency. CCLoss allows us to strike a balance between the two objectives and to generate vivid stylization details. Experimental results show that compared with prior work, our method renders a richer set of styles while preserving fine-grained details.
Content
Style
Figure 10: We can explicitly balance between content and style by tuning a.
12

---

Published in Transactions on Machine Learning Research (09/2022)
Figure 11: An example of spatially-controlled stylization via mask-out operations.
Limitations. One limitation of this work is that our CCNet is patch-based, thus is relatively weaker than sratistics-based alternatives in capturing and augmenting information at different scales. Another limitation is that the complexity of the computed affinity matrices is O making CCNet limited to a pair of 2048 x 1024 images on a TitanX GPU with 12GB memory. But those limitations can be addressed in future work. One can polentially draw inspiration from AvatarNet (Sheng al. 2018) or simply merge features with different patch sizes for multi-scale communication. Other interesting directions include approximating the affinity matrices (Zhu et al. 2019) and extending our main idca to other related fields, like image translations and texture synthesis.
Societal impact. Our work can lead to more efficient image editing pipelines. While such algorithms can empower artists to author more creative contents, they could be used maliciously, e.g. for creating fake portraits on social media. Nonetheless, recent work such as Wang et al. (2020) has demonstrated great potential detecting fake images produced by powerful generators (Karras al. 2019 2020 therefore can ameliorate the concern some extent.
References
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Lconidas Guibas. Learning representations and generative models for 3d point clouds. In ICML, 2018.
Jie An, Siyu Huang, Yibing Song, Dejing Dou, Wei Liu, and Jiebo Luo. Artflow: Unbiased image style transfer via reversible neural flows. In CVPR, 2021.
HG Barrow, JM Tenenbaum, RC Bolles, and HCr Wolf. Parametric correspondence and chamfer matching: Two new techniques for image matching. In Image Understanding Workshop, 1977.
G. Borgefors. Hierarchical chamfer matching: a parametric edge matching algorithm. JEEE TPAMI, 1988.
Dongdong Chen, Lu Yuan, Jing Liao, Nenghai Yu, and Gang Hua. Stylebank: An explicit representation for neural image style transfer. In CVPR, 2017.
Haibo Chen, Zhizhong Wang, Huiming Zhang, Zhiwen Zuo, Ailin Li, Wei Xing, Dongming Lu, et al. Artistic style transfer with internal-extents learning and contrastive learning. NewrIPS, 2021.
Tian Qi Chen and Mark Schmidt. Fast patch-based style transfer of arbitrary style. In NeurIPS Workshop, 2016.
Yingying Deng, Fan Tang, Weiming Dong, Haibin Huang, Chongyang Ma, and Changsheng Xu. Arbitrary vidco style transfer via multi-channel correlation. In AAAI, 2021.
Yingying Deng, Fan Tang, Xingjia Pan, Weiming Dong, Chongyang Ma, and Changsheng Xu. Stytr 2: Unbiased image style transfer with transformers. CVPR, 2022.
Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object reconstruction from single image. In CVPR, 2017.
Leon A Gatys, Alexander Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. Tn CVPR, 2016.
13

---

Published in Transactions on Machine Learning Research (09/2022)
Shuyang Gu, Congliang Chen, Jing Liao, and Lu Yuan. Arbitrary style transfer with deep feature reshuffle. In CVPR, 2018.
Xun Huang and Serge J. Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV, 2017.
Peng Jiang, Fanglin Gu, Yunhai Wang, Changhe Tu, and Baoquan Chen. Difnet: Semantic segmentation by diffusion networks. In NewrIPS, 2018.
Yongcheng Jing, Xiao Liu, Yukang Ding, Xinchao Wang, Errui Ding, Mingli Song, and Shilei Wen. Dynamic instance uormalization for arbilrary style transfer. In AAAT, 2020a.
Yongcheng Jing, Yezhou Yang, Zunlei Feng, Jingwen Ye, Yizhou Yu, and Mingli Song. Neural style transfer: A review. In TVCG, 2020b.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. ECCV, 2016.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
Tero Karras, Samuli Laine, Miik Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In CVPR, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Chuan Li and Michael Wand. Combining markov random fields and convolutional neural networks for image synthesis. In CVPR, 2016a.
Chuan Li and Michael Wand. Combining markov random fields and convolutional neural networks for image synthesis. In CVPR, 2016b.
Xucting Li, Sifci Liu, Jan Kautz, and Ming-Hswe Hsuan Yang. Learning lincar transformations for fast arbitrary style transfer. In CVPR, 2019.
Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang. Universal style transfer via feature transforms. In NewrIPS, 2017.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Songhua Liu, Tianwei Lin, Dongliang He, Fu Li, Meiling Wang, Xin Li, Zhengxing Sun, Qian Li, and Errui Ding, Adaattn: Revisit attention mechanism in arbitrary neural style transfer. In ICCV, 2021.
Tianyang Ma, Xingwei Yang, and Longin Jan Latecki. Boosting chamfer matching by learning chamfer distance normalization. In ECCV, 2010.
K Nichol. Painter by numbers, wikiart. htte::///ww..aagggeeee.////////r
2016.
Dae Young Park and Kwang Hee Lee. Arbitrary style transfer with style-attentional networks. In CVPR, 2019.
Artsiom Sanakoyeu, Dmytro Kotovenko, Sabine Lang, and Björn Ommer. A style-aware content loss for real-time hd style transfer. In ECCV, 2018.
Lu Sheng, Ziyi Lin, Jing Shao, and Xiaogang Wang. Avatar-net: Multi-scale zero-shot style transfer by feature decoration. In CVPR, 2018.
Denis Simakov, Yaron Caspi, Eli Shechtman, and Michal Irani. Summarizing visual data using bidirectional similarity. In CVPR, 2008.
14

---

Published in Transactions on Machine Learning Research (09/2022)
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. TCLR, 2015.
Chunjin Song, Zhijie Wu, Yang Zhou, Minglun Gong, and Hui Huang. Etnet: Error transition network for arbitrary style transfer. In NewrIPS, 2019.
Arasanathan Thayananthan, Bjoern Stenger, Philip HS Torr, and Roberto Cipolla. Shape context and chamfer matching in cluttered scenes. In CVPR, 2003.
Sheng- Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A Efros. Cnn-generated images are surprisingly easy to spot... for now. In CVPR, 2020.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018.
Xiaolei Wu, Zhihao Hu, Lu Sheng, and Dong Xu. Styleformer: Real-time arbitrary style transfer via parametric style composition. In TCCV, 2021.
Zhijic Wu, Xiang Wang, Di Lin, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. Sagnet: Structure-aware generative network for 3d-shape modeling. In SIGGRAPH, 2019.
Zhijic Wu, Chunjin Song, Yang Zhou, Minglun Gong, and Hui Huang. Efanct: Exchangeable feature
aligument network for arbitrary style transfer. In AAAI, 2020.
Wenju Xu, Chengjiagg Long, Ruisheng Wang, and Guanghui Wang. Drb-gan: A dynamic resblock generative adversarial network for arlistic style transfer. In ICCV, 2021.
Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian. Foldingnet: Point cloud auto-encoder via deep grid deformation. In CVPR, 2018.
Yulun Zhang, Chen Fang, Yilin Wang, Zhaowen Wang, Zhe Lin, Yun Fu, and Jimei Yang. Multimodal style transfer via graph cuts. In ICCV, 2019.
Yuxin Zhang, Fan Tang, Weiming Dong, Haibin Huang, Chongyang Ma, Tong- Yee Lee, and Changsheng Xu. Domain enhanced arbilrary image style transfer via contrastive learning. In ACM SIGGRA PH, 2022.
Zhen Zhu, Mengde Xu, Song Bai, Tengteng Huang, and Xiang Bai. Asymmotric non-local neural networks for semantic segmentation. In ICCV, 2019.
A Appendix: Overview
In Sec. B first of all we showcase more experimental results to demonstrate that modelling completeness and coherence allows our method to effectively render fine details. Then we present more results from our ablation study on the impact of joint analysis, completeness and coherence modeling, patch size as well as multi-scale embedding. Next, in Sec. [ we detail the implementation of our Non-local Diffusive Attention Module (Wang et al., 2018), and we present a lightweight version of this module in Sec. D Next, in Sec. D we present additional qualitative results on SANet (Park Lee, 2019) to compare with CCNet in terms of
synthesis quality and design paradigms. From Sec. F to Sec. H
a stylization matrix and samples
of stylized high-resolution images to demonstrate that our method generalizes well. We also include
discussion on the applicability of various contents in Sec. H Note that the MS-COCO (Lin et al.) 2014) and WikiArt (Nichol 2016) dataset in our experiments are public and under a Creative Commons Attribution 4.0 License, which permits us to distribute, remix, tweak, and build upon them.
B More Experimental Results
In this section, we present more evaluation results about completeness and coherence tradcoffs. We also show more results from our ablation study experiments to verify the joint analysis paradigm, the completeness and coherence modeling, as well multi-scale embedding.
15

---

Published in Transactions on Machine Learning Research (09/2022)
Content
Patch 1
Patch = 3
Style
Figure 12: Ablation study on the impact of patch size.
Comparison with previous methods on synthesizing details. In order to demonstrate our capability in generating detailed style patterns, attach close-up views of figures from the main text: in Fig. [3] show that the CCNet outperforms prior methods reducing repetitive patterns, lowering distorted clutters (e.g. 1st row) and rendering clear contours (e.g. 3rd row). The CCNet also excels at synthesizing detailed ures (e.g. the patterns in 2nd row) and improving style diversity (e.g. the brush strokes in the sky of 1st and 2nd rows).
Trading off between completeness and coherence. Our CCLoss plays a pivotal role in improving the synthesis of details and trading off between completeness and coherence. We can explicitly balance between completeness and coherence by adjusting the ratio between the coherence and completeness term (Acoh Acom) in CCLoss: from left. to right in Fig. 14 ((a) (f)) are the stylized images generated by six ablated models obtained by gradually increasing the ratio. As the ratio becomes larger, the rendered output manifests more complex style patterns (e.g. the rich color distribution in the mountainous region). But larger ratio also introduces incoherence: see the light gray clutter in the sky of the 1st row.
The effects of patch size. Fig. 12 illustrates the influences of patch size on stylization. As mentioned in Sec. 3.1. a larger patch size allows us to more effectively capture coarse-scale structures and model local information, so that we can preserve clearer contours: see the edges within the box containing the mountain and the grassland. Morcover, a model with larger patches can produce more adaptive style details, such the fine-grained lextures in the enlarged region. However, increasing patch size leads to significantly higher computational cost. We set patch size to for experiments so that performance while incurring only moderate computational cost.
achieve satisfactory rendering
Quantitative results from our ablation study. Aside from the qualitative results in Sec. we show quantitative results on our ablated models in Tab. 2 We see that the computed ablation scores back our claims from the main text. These results further validate the effectiveness of completeness and coherence learning, our proposed joint analysis paradigm and CCLoss.
Joint analysis. We use a lightweight version of the Non-local Diffusive Attention Module (Fig. 16] to show that our joint analysis paradigm with non-local blocks is indoed effective. Similar the canonical structure in Fig. 16 (a), we feed Fooh and From into two convolutional layers respectively. Then we fuse the completeness and the coherence feature via element-wise multiplication. The fused feature then further
16

---

Published in Transactions on Machine Learning Research (09/2022)
Content
Gatys
AdaAttn
WCT
AvatarNet
SANet
Ours
Style
Figure 13: More comparison with existing methods on synthesizing details.
Table 2: Following Tab. D we include more quantitative results as a part of our ablation study. We randomly
select ten content images from MS-COCO (Ln et al. 2014) and ten style images from WikiArt (Nichol 2016
to produce 100 stylized images and averaged their metrics. Here M refers to the SANet framework (Fig. 2 which contains only the Coherence Attention Module of the CCNet. We use McCLoss and Mytoss to refer respectively to the model trained with CCLoss and the perceptual loss. We use each of {Mai, Mi:100, M }
to represent the CCNet trained by setting completeness and coherence ratio (Aeok, :Acom to 1 1, 1 100 and 1: 200 respectively. We carry the notation {Mook, Moom, Mjoint) over here from Fig. to demonstrate
quantitatively the evenesiveness of each attention module in CCNet. Note that McCLoss equals to Meah here.
Loss
Mooh
Mcom
M Mjoint
M1.1
M1.100
Mi.200
Mptoes
Mploss
McCLoss
Mfuli
Style(L.)
192.52
164.29
179.01
604.32
170.12
168,99
180.36
162.78
192.52
169.58
Loom
29.36
28.74
28.76
30.67
27.82
27.65
29.03
27.59
29.36
27.93
Looh
27.01
27.99
27.87
26.86
27.25
27.97
27.74
28.21
27.01
27.19
processed by sigmoid operator, becoming element-wise weight that weighs how much information of F Foom should be used to compute the residual feature F.om- Finally, F From is also applied to update From as in the default joint analysis. The element-wise mulliplication allows the model to select. compatible features between completeness and coherence, so we can employ matthed data to coherence to update the information from completeness, making From to be more consistent with Fook- By feeding the Fooh and From into the shared non-local block in the opposite direction, we also can enforce F Foo be more similar From shown in Eq. 4 of the main text.
17

---

Published in Transactions on Machine Learning Research (09/2022)
Content
(a)
(b)
(c)
(d)
(e)
(n)
Style
Figure 14: Ablation study for the completeness and coherence term (Acoh and Acem) ).
Content
Style
w/o Non-local
Full model
Figure 15: Ablation study for joint analysis.
In Fig. 15 we see the ablated model fails to preserve the texture consistency within smooth region (c.g. black artifacts within the lower region) due to the lack of long-range dependency modeling, In contrast, the full model is better at presenting the large-scale patterns and reducing repeated artifacts in the background. Note that addition clement-wise multiplication, we also attempted other fusion operations, such clement-wise addition or computing dot product, but achieved only subtle visual differences on the ablated models.
18

---

Published in Transactions on Machine Learning Research (09/2022)
Modelling completeness and coherence. In Fig. 18 showcase more results from ablation study the effectiveness of completeness and coherence well as on the joint analysis between them. The model with only Coherence Attention preserves the content structures well and yields faithful style details, but without completeness modelling it fails to distinguish the overall style distributions. The model with only Completeness Allention can introduce more complex style variations to the results, but often generates unseen light black colors the background regions (e.g. the sky). The model without joint analysis distorts the style patterns, because the model fails filter out completeness and coherence features that are incompatible. Compared with the ablated models, our full model is better parsing the complete style components with vivid details.
Pr. rimaared Ri. Ps
Masta my Rplaat
Elementsiee mull multplccoonn
1x1 Caoe
Amx Meai
1x1 Coes
(i Moy
Pass 1 (
Chlamnsdte
Styeck
Raom (PoN))
Pan (Pael
Pon (f mn))
(R.C Rella
(a)
(b)
Figure 16: (a) The architecture of our Non-local Diffusive Attention Module. (b) The architecture of an ablated version of (a) which jointly analyzes completeness and coherence but only locally. By replacing the matrix multiplication and softmax operation with clement-wise multiplication (denoted as @) and sig- moid operation ion respectively, we ensure the model only considers the completeness and coherence information a pair of corresponding pixels, thus impossible to capture long-range dependencies within an entire image. Please refer to paragraph Joint analysis in Sec. and Sec. C for details.
Acc
Con
FG
Pc
Came
Syfimu
CCUNt
F's
Fs
Fos
Stltnor
Cont
Fs
Ass
F's
1x1 Cont
SeBasax
Coee
(a)
(b)
Figure 17: (a) Architecture of the lightweight Non-local Diffusive Attention Module, (b) Architecture of our CCUnit which captures completeness and coherence at once. Please refer Soc. for details.
Multi-scale embedding. Prior work has used mutli-scale modelling extensively improve stylization performance by enriching local and global patterns (Sheng et al., 2018 Park & Lee 2019 Here would like to determine if our multi-scale approach is effective. So we remove the branch for relu_4 and relu respectively. As shown in Fig. 19 while relu_4 can produce features that entail local color distribution and preserve spatial layouts, it fails to synthesize the eircular pattern. But relu. 5 is able to render the circular patterns because its larger receptive field. But relu__ 5's content structures are severely distorted and details in the patterns are blurred out. By integrating these two scales into our model, we can capture richer salient. style patterns and maintain the content structures simultaneously, yielding better stylization results.
Stability for quantitative metrics. In order to measure the data stability of computed metries, we report the error bars in Fig. 21 for the Tay Tromy Look and preference scores used in Tab. 1 For Lzg, Trom
19

---

Published in Transactions on Machine Learning Research (09/2022)
Content
w/o completeness
w/o coherence
w/o joint analysis
Ours
Style
Figure 18: Ablation study on modelling completeness and coherence.
and Leoh, the mean and standard deviation in each bar are computed based on the images produced by a specific method. For example, we compute the mean and standard deviation of the 600 stylized images' Leom for Galys et al. and report the result. the first column of Fig. 21 (a). We also list the 90% confidence intervals in Tab, 3 to directly indicate the representative level of results. As for the preference score, however, each baseline's standard deviation computed over the votes from each participant (Fig. 21 (d) and votes corresponding to each method (Fig. 21 (e)). It is seen that, the produced scores quite stable, showing the strong representation capability of the selected image set.
Content
Style
Relu_5
Relu
Combined Result
Figure 19: Ablation study multi-scale embedding.
Female
55-45
43-33
19%)
(16%)
(2956)
9 (26%%
(b)
Figure 20: In the user study, the gender (a) and age (b) of the participants distribute uniformly, showing the unbiasedness of the conducted evaluation. See the user study part in Sec. for more detailed discussion.
Lastly, in Fig. 22 we show stylization results with more inputs on the following baselines: ArtFlow An et al. 2021). IEST Chen et al.] 2021), MCCNet (Deng et al., (021), Sty (Deng et al. (2022). CAST (Zhang et al. 2022). Still our method outperforms the rest capturing the complete style without leaving spurious artifacts, further proving empirical advantages.
20

---

Published in Transactions on Machine Learning Research (09/2022)
THe
18 Lon
Leah
Style (la)
Study (method)
(4) User Study (particiann)
Figure 21: The error bars for Leom; Leohs Lg and preference scores used Tab. D
Note that (d) presents
the average votes computed for each method while (e) presents each participant 's vote percentages which prefer our results.
Table 3: Confidence intervals for L,. Leom and Leoh to show the quantitative result stability (part 1).
Loss
Galys
AdaAttn
WCT
Avatar
SANet
Ours
Style(Ls) 4 0.5751 0.0365
1.1561 + 0.0748
0.5620 + 0.0214
1.1019
0.0479
0.6215 = 0.0377
1.0911 + 0.0258
Loone
0.3263 + 0.0015
0.3393 + 0.0015
0.3293 + 0.0017
0.3194 + 0.0015
0.3204 = 0.0016
0.2979 + 0.0016
Leak +
0.3332 0.0014
0.3255 + 0.0017
0.3397 + 0.0017
0.3150 0.0015
0.3222 + 0.0016
0.2795 + 0.0017
Table 4: Confidence intervals for Lg, Loom and Leoh to show the quantitative result stability (part 2).
Loss
ArtFlow
TEST
MCC
StyTr2
Ours
Style(T..) 4
1.0967 + 0.0657
2.0925 + 0.1481
1.0710 + 0.0597
0.7334 + 0.0608
1.0911 + 0.0258
Teom +
0.3478 + 0.0014
0.3530 + 0.0017
0.3356
0.0017
0.3366 + 0.0018
0.2979 + 0.0016
Leoh
0.3501 + 0.0013
0.3376 = 0.0019
0.3315 + 0.0017
0.3306 = 0.0017
0.2795 = 0.0017
C More Discussion on the Non-local Diffusive Attention Module
As we have mentioned many times, the primary goal of our method is achieving completeness and coherence. To this end, we introduce a bi-directional patch-based similarity measure to quantify the visual relation- ships between one stylized result and an input style image. As mentioned in the main text, the similarity optimization procedures inspire us to update the content features by diffusing style information with two different affinity kernels. Specifically, we implement this motivation with the non-local blocks (Wang et al., 2018), as shown in Fig. 16 (a). And we set the patch size to 1 here.
For the diffusion modules in Fig. 16 (a), we take content feature F. and style feature F. as inputs whose shapes are B% x H. x Wo x C' and B x H, x W, x C' respectively. Here B, H, W and C' indicate batch size, height, width and channel dimension individually. We first feed the normalized content feature F, and style feature F, into the Attention blocks, followed with two different 1 convolutions. Then we compute the similarities between the normalized F. one pixel and the normalized F, at another pixel to make a content semantics-basse style diffusion (Sheng et al. 2018)) And the closest neighbor search for cach pixel in F, and Fg can be approximated by softmax performed in different axes. After the reshape operation, the shape of the similarity matrix is B x H.W, x H.W,, Specifically, let each row of the similarity matrix represent relationships between one pixel in F. and all the pixels in F.. Correspondingly, each column indicates the affinities between one pixel in F, and all the pixels in Fa; refer to Fig. 3 in the main text for details. When we search the closest feature vector in Fg for each feature vector of Fer we should perform the softmax operation for the similarity matrix along cach row. In the opposite direction, we should perform the softmax along cach column search the closest feature vector in F. for each feature vector of F... Finally, the stylized feature is realized by multiplying the similarity matrices with the input style feature F.. Note that the affinity matrix for completeness Acom shares the same size with the affinity matrix for coherence Acoh-
We can jointly analyze Fesh and From to improve their pixel-level compalibility in the same way. Specifically, we compute a residual feature Foon as a weighted sum of feature vectors in Fooh by simultaneously considering each feature vector of From and feature vectors at all positions of F Fooh- Then Fooh is similar to From at each of their positions. In other words, we rearrange feature vectors of Feoh, (Feom) to fit From (Feoh) together well; see Eq. 4 - of the main text for details. Then the refined feature Froh. = Feot + Froh (Feom Feom + From)
21

---

Published in Transactions on Machine Learning Research (09/2022)
Content
ARTFlow
IEST
MCCNet
StyTR-2
CAST
Ours
Style
Figure 22: Comparisons with the basclines using the same input images Fig.
will be more similar/compatible From (Feat) than Feoh (Feom). To verify this joint analysis scheme, we conduct another ablation study experiment; check out the paragraph on joint analysis in Sec. B for more information. Hence with the non-local block, it is effective for network to model the completeness and coherence and capture the rongerange dependency between pixels.
D Simplified Non-local Diffusive Attention Module
In order to further demonstrate the design flexibility of our completeness and coherence concepts, we derive lightweight version of the Non-local Diffusive Attention Module from its canonical counterpart (Fig 16 (a)
As shown in Fig. 17 (a), we simplify by unifying the two attention layers for completeness and coherence modeling and one attention layer for joint analysis into one simple block. Compared to the heavyweight architecture, the simplified one has its softmax layer replaced with learnable CCUnit to output the affinity matrix for further feature diffusion. Fig. 17 (b) shows the architecture of the CCUnit which captures completeness and coherence at once. Taking Fe, F, and Acs inputs, CCUnit feeds F, and F, into two x 1 conv layers respectively and compute the corresponding self-similarity matrices (denoted as & in purple). Then CCUnit applies softmax along each axis to compute the relative correlations bet one pixel and all the pixels in Fe (or F.) and output Acc and Ass. Specifically, the red arrow and green arrow indicate the th and 2th axes correspondingly. Still we apply softmax along one
learn completeness and along another learn coherence. The leared features are Acom and Acch- We further multiply Ace with Acom and Acoh with A.s (denoted 6 in gray) that we explicitly model the long-range dependencies
22

---

As mentioned, SANet does not impose the completeness and is strictly biased to the content features during stylization, which prevents it from considering the interactions between various style elements. Thus it might distort the local textures in smooth regions (c.g. in the sky and surface of the sca) and the dotailed variation (e.g. the roof on the top left corner does not preserve the texture arrangements). It also copies repeated patterns to results (e-g. the eyes spread over the whole image). Without explicit constraints on stylization Published in Transactions on Machine Learning Research (09/2022)
in input content-style image pairs and better preserve the content structures and coarse style patterns in inputs. Note that the matrix multiplication between Acc and Acom (Acoh and Ass) implicitly corresponds to the functionality of the joint analysis attention layer in main text. It is because joint analysis stage of CCNet aims to implicitly recover the repetitive information in content images for completeness modeling and enrich the style diversity style images for coherence modeling. Finally we add the two output matrices and obtain the affinity matrix to diffuse style features.
Note that the output affinity matrix of the simplified CCNet can be used to further reduce the computation cost by incorporating it into the multi-scale procedure. Specifically, the output affinity matrix from coarser scale enables the adaptation of style features at finer scales. Despite these extra bonus, they are beyond our focus to verify the elfectiveness of completeness and coherence.
Fig. 23 illustrates stylization results produced by the simple version of CCNet to further evaluate the effec- tiveness of completeness and coherence modeling. All input images can be found the main text. One can see that, our simplified CCNet can maintain clear contours of the prominent objects (e.g. outlines of all the buildings) and introduce faithful style details (e.g. brush strokes in 1st row and the block bise textures in 2nd and 3rd rows) concurrently. It consistently matches the advantages presented in our full model. In contrast, SANet highly biases towards content structures during aligning style features so that it repeats undesired patterns in smooth regions (e.g- gray distorted artifacts in the background of 1st and 3rd rows). It often introduces some unwanted halation around edges as well (e.g. 1st 2nd and 3rd rows) and sometimes suffers from incapability to capture dominanted style patterns (e.g. lack of block-wiss wise textures in the temple region of 3rd row).
E Comparing with SANet on Synthesizing Details
As for style transfer, we make first efforts to associate bi-directional Chamfer matching with non-local blocks (Wang et al. 2018). Compared other patch-based methods, this perspective introduces some unique advantages to method. Specifically, the completeness and coherence modcling enables us to (a) explicitly capture these two constraints, (b) minimize artifacts in results, (c) reduce chances to repeat similar patterns in smooth regions, (d) control the effects of completeness and coherence in stylizations, (e) measure the stylization ability via a newly developed metric. Actually, the Style-Attention Network (SANet) (Park] & Lee 2019) can be regarded as an ablated version of our framework, by only keeping the coherence branch but removing the joint analysis module and completeness branch; see more discussions Sec. 3.2 of the main text. In Fig. 24. we exhibit additional close-up views to further compare the synthesized details of SANct.
results, the style patterns in background are prone to be distorted (e.g. the black clutters in the sky).
In contrast, our model can better capture the notable colors and dominated textures (e.g. cyan and blue textures on the roof and in the sky). Generally speaking, the explicit completeness and coherence modeling enables us to seamlessly reassemble the style elements and work for different kinds of styles, from global hue to local strokes and detailed texture variations. Additionally, the completeness modeling may also help reduce the chances to repeat similar patterns to different regions; hence eyes in the style image only appear once in our results. The necessary content components are well perceived in the stylized images as well (c.g. the outlines of house and mountains).
F Style Interpolation Results
We can convexly combine the style patterns from multiple images {1 with weight {wx)k that K W/k such that the stylized feature are Fesc SK Ek Poee Note that, the superscript k indi-
23

---

Published in Transactions on Machine Learning Research (09/2022)
Content
SANet
Ours(Simple)
Ours(Full Model)
Style
Figure 23: Comparison between the simplified CCNet, the full CCNet and SANet. We show the input content and style images on the two sides. The simplified CCNet can produce clear contours of prominent objects (c.g. outlines of all the buildings) and yield faithful style details (e.g. brush strokes 1st row and the block-wise textures in 2nd and 3rd rows), demonstrating the same advantages possessed by our full model. SANet on the other hand biases highly towards content structures while aligning style features, so much so that it repeats spurious patterns in smooth regions (e.g. gray distorted artifacts in the sky of 1 and 3rd rows). It often introduces halation around edges as well (e.g. 1st 2nd and 3rd rows) and sometimes fail to capture dominant style patterns (e.g. lack of the block-wise textures in the temple region of 3rd row).
cates one style image hore, but not the feature from relu i layer. Finally, the interpolated feature maps are led into trained decoder to reconstruct the stylized images, as shown Fig. 25
G High-resolution Stylization
Here we present example of high-resolution stylization to demonstrate the applicability to images of large spatial resolutions. As we can see in Fig. 26 the result displays concrete multi-scale style patterns, from the color distributions to faithful style details. Morcover, the texture consistency within different smooth regions (e.g. the sky in the background) also kept well, clearly displaying the effectiveness of our method for large images. Note that, the usability large images can be further advanced via the more economical computation strategies for similarity kernels, such the asymmetric non-local networks (Zhu al. 2019
More Discussion
Additionally, one may argue that the patch- based diffusion strategy may constrain the application of our method to input pairs both having similar content structures. But as shown in Fig. 6 the proposed method can generalize pairs of very different contents by propagating the style information based on the similarities of content structures. It can be realized by performing the normalization before feeding the content. feature
24

---

Published in Transactions on Machine Learning Research (09/2022)
Style
Content
SANet
Ours
Content
SANet
Ours
Figure 24: The close-ups for detailed comparisons between SANet and our method.
Fe and style feature F. into the Completeness and Coherence Attention layers, which shares the same spirit with AvatarNet (Sheng et al.) 2018) and SANct (Park & Loc 2019). During the joint analysis process of
the CCNet, the long-range dependency modeling also helps to improve the applicability for different content structures shown in Fig. 15 Therefore due to the patch-based alignment nature, the proposed method still can present its advantages over the methods based holistic statistic matching, even when the input
image pair differ a lot from each other (Sheng et al. 2018 Park & Lee 2019
25

---

Published in Transactions on Machine Learning Research (09/2022)
Figure 25: Style interpolation using four different styles.
26

---

Published in Transactions on Machine Learning Research (09/2022)
Figure 26: An example of high-resolution stylization. The resolutions of both the content and the style image are 1024 x 1024.
27


Reviews:
[{"review_id": "yzG9DzHsqz", "reviewer": "Reviewer_aHVM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n+ Impressive results for fast arbitrary style transfer\n+ Loss function makes sense and is novel\n+ Network architecture (attention module) appears to be suitable\n+ Qualitative and quantitative demonstration that results are of high quality\n+ Paper is well written and easy to follow\n\n\n\n### Weaknesses\n\n- CCNet architecture could potentially be simplified\n- User study insufficiently described and reported\n- 4th claim not really well defined / supported\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a method for fast arbitrary style transfer that improves over previous method by providing both completeness and coherence of the resulting stylized image. This means that (a) all style patterns present in the style image are reproduced in the result (\"completeness\") and (b) the result does not contain any patterns or artifacts not present in the style (\"coherence\"). The authors claims are\n\n1. Novel loss function that captures both of these objectives\n2. Novel network architecture that can produce images that satisfy both objectives\n3. Superior performance of their approach to previous methods\n4. Optimal balance between running speed, generalization capability and stylization quality\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Overall I think this is a great paper. I think the authors could improve it by considering some of the following points. \n\n### Required\n\n1. To enable reproducibility, please describe the 100 images test set used for the quantitative evaluation in Table 1. How were the 100 images selected? Can other authors get access to the same set? How do you ensure that the set is large/representative enough to draw general conclusions about the quality of the methods? What are the error bars / confidence intervals of the metrics on this test set?\n\n1. Similar to above, the user study should be documented better. How were the images selected? Are they a representative, random sample or hand-picked by the authors? How many images (range, average) did each participant rate? What exactly were the instructions to the participants? Please also report error bars on the preference scores\n\n1. The fourth claim of \"optimal balance between running speed, generalization capability and stylization quality\" (p. 3, top) is not really well defined / supported in my opinion. What constitutes optimality here and why is your method optimal in that sense? If you think about the claim, you're saying that no other method can improve, since yours already achieves the optimal balance. I think this is a minor point that can be fixed by simply dropping this claim.\n\n\n### Recommended\n\nI leave it to the authors to judge whether they consider the following suggestions useful. I consider none of them mandatory:\n\n1. To what extent is the particular choice of attention modules in CCNet important? While I appreciate the ablation regarding the Joint Analysis Attention, it is possible that the reduced performance simply stems from the fact that removing it makes the network less expressive (shallower). I wonder whether one couldn't simply replace CCNet by a standard Transformer decoder where the encoding of the style image provides the context and the encoding of the content image is processed through multiple standard Transformer decoder layers.\n\n1. Related to above, the ablation results are very qualitative and rely on individual examples. While I generally trust the authors' judgement, I wonder whether you could make this point more quantitative by reporting the same quantities as in Table 1 rows 1–3 for the ablations.\n\n1. The method from Gatys et al. 2016 seems to underperform in this paper. It might be the particular choice of examples images (it's well known that it doesn't deal well with unstructured, uniform background), the selection of styles (a van Gogh or Picasso would be nice), the implementation (no URL given) or just my subjective impression. In my experience, the subjective difference in quality between Gatys and all available fast methods is quite substantial. This difference doesn't come across from the examples shown in the paper, and the user study also seems to suggest Gatys' method is the worst. These two observations make me think that you may not have run Gatys' method with the best settings. Note that if you're using either Leon Gatys' or Justin Johnson's code, you need the old, original VGG checkpoints and cannot use the pretrained ones form torchvision, since the latter have been trained differently and use different preprocessing.\n\n1. Eq (2) uses only a dot product followed by softmax, as is common in Transformer attention modules. However, this formulation is quite sensitive to the contrast of image patches. Since you use the cosine distance for the loss (Eq. 11+12), I wonder whether it would make sense to also use the cosine here?\n\n1. Eq. (7): It seems unnecessarily complicated to include features from both conv4 and conv5. Have you tried using only one of them?\n\n1. Order of Figs. 9 and 10 should be reversed so they appear in the same order as they appear in the text.\n\n1. Fig. 11: The text is missing a lot of important details. For instance, it is not clear where in the pipeline the masking is happening, how that affects the CCnet architecture and attention etc. Please expand the text to provide the necessary details or remove the figure. \n", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PVa3wRqeP2", "reviewer": "Reviewer_1FEZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Pros:**\n- To my knowledge, this is the first work that explicitly models and considers completeness and coherence in stylization.\n- The paper is well-written.\n- Both quantitative and qualitative results are provided.\n- The discussions on limitations and social impact are included.\n\nI’m almost satisfied with this submission, except for some minor concerns:\n\n**Cons:**\n- More details on the user study should be provided, e.g., what are the age and gender distributions of the participants?\n- It would be great if the authors could consider including more quantitative results such as for example, IS/FID scores.\n- The comparison methods are a little bit out-of-date. It would be great if the comparative results of more state-of-the-art methods can be added, like AdaAttN.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors explicitly model the concept of completeness and coherence in style transfer and accordingly propose a completeness and coherence network (CCNet) architecture along with a novel CCLoss to balance the completeness and coherence.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Please see the weakness section for the requested changes.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xNmJ4I6vwr", "reviewer": "Reviewer_Hm3y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The designs of Completeness and Coherence Network and loss are reasonable for the style transfer task. The usage of column&row softmax operation and attention operation is simple to be applied to other frameworks as well.\n\n- The discussion on SANet is good. It clearly shows the difference between this work and the baseline work and points out the main contributions of this work. \n\n- The limitations including are well discussed. It is claimed that the proposed method is not good at capturing and augmenting information at different scales. There is an ablation study on multi-scale embedding for features relu_4 and relu_5. Why not also involve relu_3?\n\n- Abundant and comprehensive experiments have been done to demonstrate the superiority of the proposed method in terms of completeness and coherence. In Figures 6 & 7, the results and comparisons are well illustrated. The analysis is comprehensive and convincing. In Figure 10, the coherence-completeness loss weight ratio experiment provides clear proof for the claimed effectiveness of coherence loss and completeness loss. More detailed comparisons (including architectures and running time) and ablation studies are also included in the supplementary materials. I would like to see more high-resolution results.\n\nWeaknesses\n\n- The intuition behind the column&row softmax operation is not clear in the beginning. But Figure 5 provides a good explanation. I suggest adding some reference to Figure 5 when talking about the column&row softmax operation.\n\n- In Section 3.2, it is mentioned that “we first normalize $F_S$ and $F_S$ to make a style diffusion based on content structures, yielding $\\bar{F}_C$ and $\\bar{F}_S$.“ It is confusing to me why the normalization operation can diffuse the style information? The $F_C$ and $F_S$ features seem to be normalized independently, and no statistics are exchanged here.\n\n- The symbol $E^i$ in Equation (10) is not described. Does it mean the VGG encoder? As it is claimed at the beginning of Section 3.3 “VGG-based encoder (denoted as E)”. I suggest describing the symbols more clearly somewhere near the equation.\n\n- The overall organization of this paper is easy to follow. The concepts of style coherence and completeness are not that concrete. I would expect some better illustrations similar to Figure 10 to provide a more concrete example for the readers at the beginning of this paper.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work explicitly introduces the concept of completeness and coherence to style transfer. It is claimed that existing methods pursue\nthe two objectives either partially or implicitly. This work achieves the two objectives explicitly with a novel network design (CCNet) and loss design (CCLoss). The explanation and analysis are easy to follow. Comprehensive experiments have been done to support the claims and the superiority of the proposed technics.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "There are some minor changes required from my side (discussed in the weaknesses ).\n\n- Please add a reference to Figure 5 when talking about the column&row softmax operation.\n- Please explain the style diffusion of $F_S$ and $F_S$ in Section 3.2.\n- Please add explanation of symbol $E^i$ in Equation (10).\n- Please add more concrete illustrations of style coherence and completeness at the beginning of this paper.", "broader_impact_concerns": "The societal impact has been well discussed in this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission explicitly formalises two different objectives of image style transfer: coherence and completeness, which are the concepts that were not directly formulated in the prior works. Based on this formalisation,  the authors introduce the DNN architecture specifically tailored for higher coherence/completeness as well as the loss function that explicitly enforces better coherence/completeness.

The advantage of the proposed style transfer method is demonstrated via the results of the user study and a number of qualitative examples.

Three reviewers unanimously recommended acceptance with only minor concerns on the paper organisation, need of clarifications, and insufficient description of the user study. These concerns were successfully addressed in the revision.
Decision: Accept (poster)