Title: A LEARNED REPRESENTATION FOR ARTISTIC STYLE

Abstract: The diversity of painting styles represents a rich visual vocabulary for the construction of an image. The degree to which one may learn and parsimoniously capture this visual vocabulary measures our understanding of the higher level features of paintings, if not images in general. In this work we investigate the construction of a single, scalable deep network that can parsimoniously capture the artistic style of a diversity of paintings. We demonstrate that such a network generalizes across a diversity of artistic styles by reducing a painting to a point in an embedding space. Importantly, this model permits a user to explore new painting styles by arbitrarily combining the styles learned from individual paintings. We hope that this work provides a useful step towards building rich models of paintings and offers a window on to the structure of the learned representation of artistic style.

1. INTRODUCTION:

A pastiche is an artistic work that imitates the style of another one. Computer vision and more recently machine learning have a history of trying to automate pastiche, that is, render an image in the style of another one. This task is called style transfer, and is closely related to the texture synthesis task. While the latter tries to capture the statistical relationship between the pixels of a source image which is assumed to have a stationary distribution at some scale, the former does so while also attempting to preserve some notion of content.

On the computer vision side, Efros & Leung (1999) and Wei & Levoy (2000) attempt to "grow" textures one pixel at a time using non-parametric sampling of pixels in an examplar image. Efros & Freeman (2001) and Liang et al. (2001) extend this idea to "growing" textures one patch at a time, and Efros & Freeman (2001) uses the approach to implement "texture transfer", i.e. transfering the texture of an object onto another one. Kwatra et al. (2005) approaches the texture synthesis problem from an energy minimization perspective, progressively refining the texture using an EMlike algorithm. Hertzmann et al. (2001) introduces the concept of "image analogies": given a pair of "unfiltered" and "filtered" versions of an examplar image, a target image is processed to create an analogous "filtered" result. More recently, Frigo et al. (2016) treats style transfer as a local texture transfer (using an adaptive patch partition) followed by a global color transfer, and Elad & Milanfar (2016) extends Kwatra's energy-based method into a style transfer algorithm by taking content similarity into account.

On the machine learning side, it has been shown that a trained classifier can be used as a feature extractor to drive texture synthesis and style transfer. Gatys et al. (2015a) uses the VGG-19 network (Simonyan & Zisserman, 2014) to extract features from a texture image and a synthesized texture. The two sets of features are compared and the synthesized texture is modified by gradient descent so that the two sets of features are as close as possible. Gatys et al. (2015b) extends this idea to style transfer by adding the constraint that the synthesized image also be close to a content image with respect to another set of features extracted by the trained VGG-19 classifier.

While very flexible, this algorithm is expensive to run due to the optimization loop being carried. Ulyanov et al. (2016a) , Li & Wand (2016) and Johnson et al. (2016) tackle this problem by introducing a feedforward style transfer network, which is trained to go from content to pastiche image in one pass. However, in doing so some of the flexibility of the original algorithm is lost: the style transfer network is tied to a single style, which means that separate networks have to be trained for every style being modeled. Subsequent work has brought some performance improvements to style transfer networks, e.g. with respect to color preservation (Gatys et al., 2016a) or style transfer quality (Ulyanov et al., 2016b) , but to our knowledge the problem of the single-purpose nature of style transfer networks remains untackled.

We think this is an important problem that, if solved, would have both scientific and practical importance. First, style transfer has already found use in mobile applications, for which on-device processing is contingent upon the models having a reasonable memory footprint. More broadly, building a separate network for each style ignores the fact that individual paintings share many common visual elements and a true model that captures artistic style would be able to exploit and learn from such regularities. Furthermore, the degree to which an artistic styling model might generalize across painting styles would directly measure our ability to build systems that parsimoniously capture the higher level features and statistics of photographs and images (Simoncelli & Olshausen, 2001) .

In this work, we show that a simple modification of the style transfer network, namely the introduction of conditional instance normalization, allows it to learn multiple styles (Figure 1a ).We demonstrate that this approach is flexible yet comparable to single-purpose style transfer networks, both qualitatively and in terms of convergence properties. This model reduces each style image into a point in an embedding space. Furthermore, this model provides a generic representation for artistic styles that seems flexible enough to capture new artistic styles much faster than a single-purpose net- work. Finally, we show that the embeddding space representation permits one to arbitrarily combine artistic styles in novel ways not previously observed (Figure 1b ).

2. STYLE TRANSFER WITH DEEP NETWORKS:

Style transfer can be defined as finding a pastiche image p whose content is similar to that of a content image c but whose style is similar to that of a style image s. This objective is by nature vaguely defined, because similarity in content and style are themselves vaguely defined.

The neural algorithm of artistic style proposes the following definitions:

• Two images are similar in content if their high-level features as extracted by a trained classifier are close in Euclidian distance. • Two images are similar in style if their low-level features as extracted by a trained classifier share the same statistics or, more concretely, if the difference between the features' Gram matrices has a small Frobenius norm.

The first point is motivated by the empirical observation that high-level features in classifiers tend to correspond to higher levels of abstractions (see Zeiler & Fergus (2014) for visualizations; see Johnson et al. (2016) for style transfer features). The second point is motivated by the observation that the artistic style of a painting may be interpreted as a visual texture (Gatys et al., 2015a) . A visual texture is conjectured to be spatially homogenous and consist of repeated structural motifs whose minimal sufficient statistics are captured by lower order statistical measurements (Julesz, 1962; Portilla & Simoncelli, 1999) .

In its original formulation, the neural algorithm of artistic style proceeds as follows: starting from some initialization of p (e.g. c, or some random initialization), the algorithm adapts p to minimize the loss function

EQUATION

where L s (p) is the style loss, L c (p) is the content loss and λ s , λ c are scaling hyperparameters. Given a set of "style layers" S and a set of "content layers" C, the style and content losses are themselves defined as

L s (p) = i∈S 1 U i || G(φ i (p)) -G(φ i (s)) || 2 F (2) L c (p) = j∈C 1 U j || φ j (p) -φ j (c) || 2 2 (3)

where φ l (x) are the classifier activations at layer l, U l is the total number of units at layer l and G(φ l (x)) is the Gram matrix associated with the layer l activations. In practice, we set λ c = 1.0 and and leave λ s as a free hyper-parameter.

In order to speed up the procedure outlined above, a feed-forward convolutional network, termed a style transfer network T , is introduced to learn the transformation (Johnson et al., 2016; Li & Wand, 2016; Ulyanov et al., 2016a) . It takes as input a content image c and outputs the pastiche image p directly (Figure 2 ). The network is trained on many content images (Deng et al., 2009) using the same loss function as above, i.e.

EQUATION

While feedforward style transfer networks solve the problem of speed at test-time, they also suffer from the fact that the network T is tied to one specific painting style. This means that a separate network T has to be trained for every style to be imitated. The real-world impact of this limitation is that it becomes prohibitive to implement a style transfer application on a memory-limited device, such as a smartphone.

2.1. N-STYLES FEEDFORWARD STYLE TRANSFER NETWORKS:

Our work stems from the intuition that many styles probably share some degree of computation, and that this sharing is thrown away by training N networks from scratch when building an Nstyles style transfer system. For instance, many impressionist paintings share similar paint strokes but differ in the color palette being used. In that case, it seems very wasteful to treat a set of N impressionist paintings as completely separate styles.

To take this into account, we propose to train a single conditional style transfer network T (c, s) for N styles. The conditional network is given both a content image and the identity of the style to apply and produces a pastiche corresponding to that style. While the idea is straightforward on paper, there remains the open question of how conditioning should be done. In exploring this question, we found a very surprising fact about the role of normalization in style transfer networks: to model a style, it is sufficient to specialize scaling and shifting parameters after normalization to each specific style. In other words, all convolutional weights of a style transfer network can be shared across many styles, and it is sufficient to tune parameters for an affine transformation after normalization for each style.

We call this approach conditional instance normalization. The goal of the procedure is transform a layer's activations x into a normalized activation z specific to painting style s. Building off the instance normalization technique proposed in Ulyanov et al. (2016b) , we augment the γ and β parameters so that they're N × C matrices, where N is the number of styles being modeled and C is the number of output feature maps. Conditioning on a style is achieved as follows:

EQUATION

where µ and σ are x's mean and standard deviation taken across spatial axes and γ s and β s are obtained by selecting the row corresponding to s in the γ and β matrices (Figure 3 ). One added benefit of this approach is that one can stylize a single image into N painting styles with a single feed forward pass of the network with a batch size of N . In constrast, a single-style network requires N feed forward passes to perform N style transfers (Johnson et al., 2016; Li & Wand, 2016; Ulyanov et al., 2016a) .

Because conditional instance normalization only acts on the scaling and shifting parameters, training a style transfer network on N styles requires fewer parameters than the naive approach of training N separate networks. In a typical network setup, the model consists of roughly 1.6M parameters, only around 3K (or 0.2%) of which specify individual artistic styles. In fact, because the size of γ and β grows linearly with respect to the number of feature maps in the network, this approach requires O(N × L) parameters, where L is the total number of feature maps in the network.

In addition, as is discussed in subsection 3.4, conditional instance normalization presents the advantage that integrating an N + 1 th style to the network is cheap because of the very small number of parameters to train. 3 EXPERIMENTAL RESULTS

3.1. METHODOLOGY:

Unless noted otherwise, all style transfer networks were trained using the hyperparameters outlined in the Appendix's Table 1 .

We used the same network architecture as in Johnson et al. (2016) , except for two key details: zero-padding is replaced with mirror-padding, and transposed convolutions (also sometimes called deconvolutions) are replaced with nearest-neighbor upsampling followed by a convolution. The use of mirror-padding avoids border patterns sometimes caused by zero-padding in SAME-padded convolutions, while the replacement for transposed convolutions avoids checkerboard patterning, as discussed in in Odena et al. (2016) . We find that with these two improvements training the network no longer requires a total variation loss that was previously employed to remove high frequency noise as proposed in Johnson et al. (2016) .

Our training procedure follows Johnson et al. (2016) . Briefly, we employ the ImageNet dataset (Deng et al., 2009) as a corpus of training content images. We train the N -style network with stochastic gradient descent using the Adam optimizer (Kingma & Ba, 2014) . Details of the model architecture are in the Appendix. A complete implementation of the model in TensorFlow (Abadi et al., 2016) as well as a pretrained model are available for download1 . The evaluation images used for this work were resized such that their smaller side has size 512. Their stylized versions were then center-cropped to 512x512 pixels for display.

3.2. TRAINING A SINGLE NETWORK ON N STYLES PRODUCES STYLIZATIONS COMPARABLE TO INDEPENDENTLY-TRAINED MODELS:

As a first test, we trained a 10-styles model on stylistically similar images, namely 10 impressionist paintings from Claude Monet. Figure 4 shows the result of applying the trained network on evaluation images for a subset of the styles, with the full results being displayed in the Appendix. The model captures different color palettes and textures. We emphasize that 99.8% of the parameters are shared across all styles in contrast to 0.2% of the parameters which are unique to each painting style.

To get a sense of what is being traded off by folding 10 styles into a single network, we trained a separate, single-style network on each style and compared them to the 10-styles network in terms of style transfer quality and training speed (Figure 5 ).

The left column compares the learning curves for style and content losses between the single-style networks and the 10-styles network. The losses were averaged over 32 random batches of content images. By visual inspection, we observe that the 10-styles network converges as quickly as the single-style networks in terms of style loss, but lags slightly behind in terms of content loss.

In order to quantify this observation, we compare the final losses for 10-styles and single-style models (center column). The 10-styles network's content loss is around 8.7 ± 3.9% higher than its single-style counterparts, while the difference in style losses (8.9 ± 16.5% lower) is insignificant. While the N -styles network suffers from a slight decrease in content loss convergence speed, this may not be a fair comparison, given that it takes N times more parameter updates to train N singlestyle networks separately than to train them with an N -styles network.

The right column shows a comparison between the pastiches produced by the 10-styles network and the ones produced by the single-style networks. We see that both results are qualitatively similar.

3.3. THE N-STYLES MODEL IS FLEXIBLE ENOUGH TO CAPTURE VERY DIFFERENT STYLES:

We evaluated the flexibility of the N -styles model by training a style transfer network on 32 works of art chosen for their diversity. Figure 1a shows the result of applying the trained network on evaluation images for a subset of the styles. Once again, the full results are displayed in the Appendix.

The model appears to be capable of modeling all 32 styles in spite of the tremendous variation in color palette and the spatial scale of the painting styles.

3.4. THE TRAINED NETWORK GENERALIZES ACROSS PAINTING STYLES:

Since all weights in the transformer network are shared between styles, one way to incorporate a new style to a trained network is to keep the trained weights fixed and learn a new set of γ and β parameters. To test the efficiency of this approach, we used it to incrementally incorporate Monet's Plum Trees in Blossom painting to the network trained on 32 varied styles. Figure 6 shows that doing so is much faster than training a new network from scratch (left) while yielding comparable pastiches: even after eight times fewer parameter updates than its single-style counterpart, the finetuned model produces comparable pastiches (right).

3.5. THE TRAINED NETWORK CAN ARBITRARILY COMBINE PAINTING STYLES:

The conditional instance normalization approach raises some interesting questions about style representation. In learning a different set of γ and β parameters for every style, we are in some sense learning an embedding of styles. 

4. DISCUSSION:

It seems surprising that such a small proportion of the network's parameters can have such an impact on the overall process of style transfer. A similar intuition has been observed in auto-regressive models of images (van den Oord et al., 2016b) and audio (van den Oord et al., 2016a) where the conditioning process is mediated by adjusting the biases for subsequent samples from the model. That said, in the case of art stylization when posed as a feedforward network, it could be that the specific network architecture is unable to take full advantage of its capacity. We see evidence for this behavior in that pruning the architecture leads to qualitatively similar results. Another interpretation could be that the convolutional weights of the style transfer network encode transformations that represent "elements of style". The scaling and shifting factors would then provide a way for each style to inhibit or enhance the expression of various elements of style to form a global identity of style. While this work does not attempt to verify this hypothesis, we think that this would constitute a very promising direction of research in understanding the computation behind style transfer networks as well as the representation of images in general.

Concurrent to this work, Gatys et al. (2016b) demonstrated exciting new methods for revising the loss to selectively adjust the spatial scale, color information and spatial localization of the artistic style information. These methods are complementary to the results in this paper and present an interesting direction for exploring how spatial and color information uniquely factor into artistic style representation.

The question of how predictive each style image is of its corresponding style representation is also of great interest. If it is the case that the style representation can easily be predicted from a style image, one could imagine building a transformer network which skips learning an individual conditional embedding and instead learn to produce a pastiche directly from a style and a content image, much like in the original neural algorithm of artistic style, but without any optimization loop at test time.

Finally, the learned style representation opens the door to generative models of style: by modeling enough paintings of a given artistic movement (e.g. impressionism), one could build a collection of style embeddings upon which a generative model could be trained. At test time, a style representation would be sampled from the generative model and used in conjunction with the style transfer network to produce a random pastiche of that artistic movement.

In summary, we demonstrated that conditional instance normalization constitutes a simple, efficient and scalable modification of style transfer networks that allows them to model multiple styles at the same time. A practical consequence of this approach is that a new painting style may be transmitted to and stored on a mobile device with a small number of parameters. We showed that despite its simplicity, the method is flexible enough to capture very different styles while having very little impact on training time and final performance of the trained network. Finally, we showed that the learned representation of style is useful in arbitrarily combining artistic styles. This work suggests the existence of a learned representation for artistic styles whose vocabulary is flexible enough to capture a diversity of the painted world. Claude Monet, Poppy Field (1873) .

Berthe Morisot, Under the Orange Tree (1889).

Giulio Romano (Giulio Pippi), Victory, Janus, Chronos and Gaea (about 1532-1534).

Wassily Kandinsky, White Zig Zags (1922).

:

https://github.com/tensorflow/magenta

For instance, https://github.com/jcjohnson/neural-style

Please see the code repository for real-time, interactive demonstration. A screen capture is available at https://www.youtube.com/watch?v=6ZHiARZmiUI.

Figures and Tables:

(a) With conditional instance normalization, a single style transfer network can capture 32 styles at the same time, five of which are shown here. All 32 styles in this single model are in the Appendix. Golden Gate Bridge photograph by Rich Niewiroski Jr. (b) The style representation learned via conditional instance normalization permits the arbitrary combination of artistic styles. Each pastiche in the sequence corresponds to a different step in interpolating between the γ and β values associated with two styles the model was trained on.

Figure 1: Pastiches produced by a style transfer network trained on 32 styles chosen for their variety.

Figure 2: Style transfer network training diagram(Johnson et al., 2016; Ulyanov et al., 2016a). A pastiche image is produced by feeding a content image through the style transfer network. The two images, along with a style image, are passed through a trained classifier, and the resulting intermediate representations are used to compute the content loss L c and style loss L s . The parameters of the classifier are kept fixed throughout training.

Figure3: Conditional instance normalization. The input activation x is normalized across both spatial dimensions and subsequently scaled and shifted using style-dependent parameter vectors γ s , β s where s indexes the style label.

Figure 4: A single style transfer network was trained to capture the style of 10 Monet paintings, five of which are shown here. All 10 styles in this single model are in the Appendix. Golden Gate Bridge photograph by Rich Niewiroski Jr.

Figure 5: The N -styles model exhibits learning dynamics comparable to individual models. (Left column) The N-styles model converges slightly slower in terms of content loss (top) and as fast in terms of style loss (bottom) than individual models. Training on a single Monet painting is represented by two curves with the same color. The dashed curve represents the N -styles model, and the full curves represent individual models. Emphasis has been added on the styles for Vetheuil (1902) (teal) and Water Lilies (purple) for visualization purposes; remaining colors correspond to other Monet paintings (see Appendix). (Center column) The N-styles model reaches a slightly higher final content loss than (top, 8.7 ± 3.9% increase) and a final style loss comparable to (bottom, 8.9 ± 16.5% decrease) individual models. (Right column) Pastiches produced by the N -styles network are qualitatively comparable to those produced by individual networks.

and β values to blend very distinct painting styles (Figure1b; Figure7, left column). Employing a single convex combination produces a smooth transition from one style to the other. Suppose (γ 1 , β 1 ) and (γ 2 , β 2 ) are the parameters corresponding to two different styles. We useγ = α × γ 1 + (1 -α) × γ 2 and β = α × β 1 + (1 -α) × β 2 tostylize an image. Employing convex combinations may be extended to an arbitrary number of styles 3 . Figure7(right column) shows the style loss from the transformer network for a given source image, with respect to the Bicentennial Print and Head of a Clown paintings, as we vary α from 0 to 1. As α increases, the style loss with respect to Bicentennial Print increases, which explains the smooth fading out of that style's artifact in the transformed image.

Grainstacks at Giverny; the Evening Sun (1888/1889). Claude Monet, Plum Trees in Blossom (1879).

Peer review (iclr2017):
This paper addresses the problem of efficient neural stylization. Instead of training a separate network for N different styles (as is done, e.g., in Johnson et al.), this paper extends the instance normalization work of Ulyanov et al. to train a single network and learn a smaller set “conditional instance normalization” parameters dependent on the desired output style. The conditional instance normalization applies a learnt affine transformation on normalized feature maps at each layer in the network. Qualitative results are shown. I have not worked in this area, but I’m generally aware of the main issues in transferring artistic style. The paper addresses a known challenge of incorporating different styles into the same net, which have a number of practical benefits. As far as I can tell the results look compelling. As I’m less confident in my expertise in this area, I’m happy to support another reviewer who is willing to champion this paper. My main comments are on the paper writing. As far as I understand, the main novelty of the approach starts in Section 2.2, and before that is review of prior art. If this is indeed the case, one suggestion is to remove the subsection heading for 2.1 so it’s grouped with the first part of Section 2, and to cite related work for the feedforward network (e.g., Johnson et al.) in the text and in Fig 2 so it’s clear. In fact, I’m wondering if Figs 2 and 3 can be combined somehow so that the contribution is clearer in the figures. I was at first confused by Eq (5) as x and z are not defined anywhere. Also, it may be helpful to write out everything explicitly as is done in the instance normalization paper. In Eq (4), perhaps you could write T_s to emphasize that there are separate networks for different styles. Fig 5 left: I’m assuming the different colors correspond to the different styles. If so, perhaps mention this in the caption. Also, this figure is hard to read. Maybe instead show single curves with error bars that are averages over the loss curves for N-styles and individual styles. Typos: Page 1: Shouldn’t it be “VGG-16” network (not *VGG-19”)? Page 2: “newtork” => “network”. Paragraph after Eq. (5): “much less” => “fewer”.