Title: A Learned Representation For Artistic Style
Authors: Vincent Dumoulin
Jonathon Shlens
Manjunath Kudlur
Venue: ICLR 2017
Keywords: Computer vision
Deep learning

Abstract: The diversity of painting styles represents a rich visual vocabulary for the construction of an image. The degree to which one may learn and parsimoniously capture this visual vocabulary measures our understanding of the higher level features of paintings, if not images in general. In this work we investigate the construction of a single, scalable deep network that can parsimoniously capture the artistic style of a diversity of paintings. We demonstrate that such a network generalizes across a diversity of artistic styles by reducing a painting to a point in an embedding space. Importantly, this model permits a user to explore new painting styles by arbitrarily combining the styles learned from individual paintings. We hope that this work provides a useful step towards building rich models of paintings and offers a window on to the structure of the learned representation of artistic style.

TL;DR: A deep neural network to learn and combine artistic styles.

Review discussion:
In Section 2.2 the authors write that the proposed approach allows one to “stylize a single image into N painting styles with a single feed forward pass of the network.” Can you elaborate on this point? If I’ve understood the model correctly, then when using a single network to stylize an image using two different styles, the computation can only be shared up to the first conditional instance normalization, at which point the style-specific scale and shift parameters will cause the intermediate representations to diverge;
therefore I don’t understand how a single forward pass can be used to stylize an image in multiple styles.
Thank you for the question. We apologize we were not terribly clear in the manuscript and we will amend the manuscript to make this clearer.
Our comment highlights how previous methods required N feed-forward passes of N separately trained networks in order to provide N stylizations for a given photograph. Our current model requires 1 feed-forward pass in which the batch size is N.
While there is no computational savings when styling a single image to N styles in our method, it is well known that larger mini-batches can be implemented much more efficiently on modern processors and GPUs.
Thus it is much faster to do N stylizations in our method, compared to the previous approaches, which have to run the less efficient single image inference on N independent networks.
If one requests N distinct styles to be applied to a single image in a given batch, then there will be a slight performance degradation because each item in the batch would use different parameters for the conditional instance normalization. Thankfully this is not a large issue because the conditional instance normalization is just a linear transformation and the vast majority of the computation (i.e. convolutions) is a computation that is shared and amortized across the batch.
We uploaded a new version to arxiv.org with these revisions (linked in openreview). We took the opportunity to provide some pointers to code we posted online, a YouTube video demonstrating a real-time interactive demo and added a brief section highlighting some recently posted, concurrent work on this topic:
https://arxiv.org/pdf/1610.07629v2.pdf
Dec 5 at 5:00pm PST:
FYI, we uploaded an updated version to arxiv.org with these revisions (linked in openreview). We took the opportunity to provide some pointers to code we posted online, a YouTube video demonstrating a real-time interactive demo and added a brief section highlighting some recently posted, concurrent work on this topic:
https://arxiv.org/pdf/1610.07629v2.pdf
Questions:
- In Fig. 6, as I understand, stylisations are compared between the fine-tuned network after 5k training iterations and a network trained from scratch after 40k iterations. What does the stylisation from the trained-from-scratch network after 5k iterations look like? Is it also comparable? It would be informative to include that comparison in the plot.
- The paper shows nicely that a large number of styles can be encoded in a condensed feature space of shift and scaling parameters for the feature maps. Interpolation results are shown where the style parameters are linearly interpolated between up to 4 styles. My questions is: Did the authors try other interpolation schemes than just linear interpolation?
For example to have different style parameters at different layers (eg. all layers apart from one have style A and one layer has style B), or different style parameters for different features in the same layer? In that way one might be able to visualise the specific contribution of single layers or units to the stylisation outcome.
- Finally, in the discussion of the paper it says that "it could be that the network architecture is overspecified for the task. We see evidence for this behavior in that pruning the architecture leads to qualitatively similar results".
Nevertheless, the stylisation results presented in this paper and with feed-forward networks in general are still substantially behind the quality that is achieved by the iterative procedure of the Gatys et al. algorithm (which is also not perfect).
Thus it sounds strange to me that the network is 'overspecified' (as in 'too expressive') for the task, because the it does not solve the task conclusively.
So shouldn't one further qualify that statement by adding that just for the given architecture of the network, adding more parameters does not improve stylisation performance?
Small comment:
- It would be nice to have a legend for the first plot in Fig. 5
Thank you for the comments and suggestions. We will be posting a revised version of the manuscript shortly to address these reviewer issues.
> - In Fig. 6, as I understand, stylisations are compared between the fine-tuned network after 5k
> training iterations and a network trained from scratch after 40k iterations. What does the
> stylisation from the trained-from-scratch network after 5k iterations look like? Is it also
> comparable? It would be informative to include that comparison in the plot.
After 5k iterations, training from scratch produces poorly stylized images.
Conversely, after 5k iterations, ‘fine-tuning’ produces images that are qualitatively similar to the final stylized image after 40k iterations. This is consistent with the performance in the loss curves. We have updated Figure 6 to additionally show sample stylized images that highlight this qualitative difference.
> - The paper shows nicely that a large number of styles can be encoded in a condensed
> feature space of shift and scaling parameters for the feature maps. Interpolation results are
> shown where the style parameters are linearly interpolated between up to 4 styles. My
> questions is: Did the authors try other interpolation schemes than just linear interpolation? For
> example to have different style parameters at different layers (eg. all layers apart from one
> have style A and one layer has style B), or different style parameters for different features in
> the same layer? In that way one might be able to visualise the specific contribution of single
> layers or units to the stylisation outcome.
We appreciate the author’s suggestion as we think this is an interesting direction to explore - particularly the suggestion of mixing interpolation across layers or units. We can qualitatively compare the rendered images but if the reviewer has any suggestions for how to quantitatively measure the effects of various layers or units, we would be very excited to explore.
Also, note that in Figure 7 (right) one sees that the style loss does not grow linearly in spite of the fact that we use a linear interpolation. Other colleagues have suggested trying to interpolate in a manner that provides linear, uniform spacing of the style loss, but linear interpolation is the only method that has been tried so far.
> - Finally, in the discussion of the paper it says that "it could be that the network architecture is
> overspecified for the task. We see evidence for this behavior in that pruning the architecture
> leads to qualitatively similar results".
Nevertheless, the stylisation results presented in this
> paper and with feed-forward networks in general are still substantially behind the quality that is
> achieved by the iterative procedure of the Gatys et al. algorithm (which is also not perfect).
> Thus it sounds strange to me that the network is 'overspecified' (as in 'too expressive') for the
> task, because the it does not solve the task conclusively.
So shouldn't one further qualify that
> statement by adding that just for the given architecture of the network, adding more
> parameters does not improve stylisation performance?
We agree with the reviewer’s point and have adjusted the text to the following: “That said, in the case of art stylization when posed as a feedforward network, it could be that the specific network architecture is unable to take full advantage of its capacity.”
> Small comment:
> - It would be nice to have a legend for the first plot in Fig. 5
We have amended the caption with the identities of the 2 paintings indicated in the dark curves (Vetheuil = teal and Water Lillies = purple) . The other light curves correspond to the other Monet paintings listed in the Appendix.
The link to the manuscript has been updated and a new PDF may be downloaded. Changes:
- Updated Figure 6 to show the stylized rendering based on 'fine-tuning' or 'from scratch' after a small number of training steps (i.e. 5k steps).
- Added more details about the training algorithm per the reviewer request.
- Updated caption to Figure 5 to provide the identities of the painting styles.
- Some minor text changes per the reviewer comments.
CONTRIBUTIONS
The authors propose a simple architectural modification (conditional instance normalization) for the task of feedforward neural style transfer that allows a single network to apply many different styles to input images. Experiments show that the proposed multi-style networks produce qualitatively similar images as single-style networks, train as fast as single-style networks, and achieve comparable losses as single-style networks.
In addition, the authors shows that new styles can be incrementally added to multi-style networks with minimal finetuning, and that convex combinations of per-style parameters can be used for feedforward style blending. The authors have released open-source code and pretrained models allowing others to replicate the experimental results.
NOVELTY
The problem setup is very similar to prior work on feedforward neural style transfer, but the paper is the first to my knowledge that uses a single network to apply different styles to input images; the proposed conditional instance normalization layer is also novel. This paper is also the first that demonstrates feedforward neural style blending;
though not described in published literature, optimization-based neural style blending had previously been demonstrated in https://github.com/jcjohnson/neural-style.
MISSING CITATION
The following paper was concurrent with Ulyanov et al (2016a) and Johnson et al in demonstrating feedforward neural style transfer, though it did not use the Gram-based formulation of Gatys et al:
Li and Wand, "Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks", ECCV 2016
CLARITY
The paper is very well written and easy to follow.
SIGNIFICANCE
Though simple, the proposed method is a significant addition to the growing field of neural style transfer. Its benefits are especially clear for mobile applications, which are often constrained in both disk space and bandwidth. Using the proposed method, only a single trained network needs to be transmitted and stored on the mobile device;
in addition the ability of the proposed method to incrementally learn new styles means that new styles can be added by transmitting only a small number of new style-specific parameters to a mobile device.
EVALUATION
Like many other papers on neural style transfer, the results are mostly qualitative. Following existing literature, the authors use style and content loss as a quantitative measure of quality, but these metrics are unfortunately not always well-correlated with the perceptual quality of the results. I find the results of this paper convincing, but I wish that this and other papers on this topic could find a way to evaluate their results more quantitatively.
SUMMARY
The problem and method are slightly incremental, but the several improvements over prior work make this paper a significant addition to the growing literature on neural style transfer. The paper is well-written and its experiments convincingly validate the benefits of the method.
Overall I believe the paper would be a valuable addition to the conference.
Pros
- Simple modification to feedforward neural style transfer with several improvements over prior work
- Strong qualitative results
- Well-written
- Open-source code has already been released
Cons
- Slightly incremental
- Somewhat lacking in quantitative evaluation, but not any more so than prior work on this topic
Thank you for your feedback. We like the idea of adding new styles to a mobile application by only sending over a small number of style-specific parameters. We will be adding it to our discussion and credit you for the suggestion in our next revision.
We will be adding a reference to Li and Wand alongside other feedforward style transfer citations. Thank you for pointing it out.
We also acknowledged Justin Johnson’s GitHub repository as prior work on optimization-based style blending in the paper. One interesting difference that we would like to point out is that our approach to style blending produces homogeneous pastiches, whereas the optimization-based approach produces heterogeneous pastiches with some areas exhibiting one set of style features and other areas exhibiting a different set of style features.
We agree that the style and content losses are not always well correlated with the perceptual quality of the results. In the paper, we use these metrics to back the statement that the optimization of the N-styles version of the feedforward style transfer network is not significantly impacted when compared with single-style networks. One potential pitfall is that the N-styles network could behave very differently than single-style networks despite having comparable losses, which we account for by qualitatively comparing pastiches. We welcome suggestions to improve the reliability of the quantitative evaluation.
The paper introduces an elegant method to train a single feed-forward style transfer network with a large number of styles. This is achieved by a global, style-dependent scale and shift parameter for each feature in the network.
Thus image style is encoded in a very condensed subset of the network parameters, with only two parameters per feature map.
This enables to easily incorporate new styles into an existing network by fine-tuning.
At the same time, the quality of the generated stylisations is comparable to existing feed-forward single-style transfer networks.
While this also means that the stylisation results in the paper are limited by the quality of current feed-forward methods, the proposed method seems general enough to be combined with future improvements in feed-forward style transfer.
Finally, the paper shows that having multiple styles encoded in one feature space allows to gradually interpolate between different styles to generate new mixtures of styles. This is comparable to interpolating between the Gram Matrices of different style images in the iterative style transfer algorithm by Gatys et al. and comes with similar limitations: Right now the parameters of the style feature space are hard to interpret and therefore there is little control over the stylisation outcome when moving in that feature space.
Here I see the most potential for improvement of the paper: The parameterisation of style in terms of scale and shift parameters of individual features seems like a promising basis to achieve interpretable style features. It would be a great addition to explore to what extend statements such as “The parameters of neuron N in layer L encodes e.g. the colour or brush-strokes of the styles” can be made. I agree that this is a potentially laborious endeavour, but even just qualitative statements of this kind that are demonstrated with the respective manipulations in the stylisation would be very interesting.
In conclusion, this is a good paper presenting an elegant and valuable contribution that will have considerable impact on the design of feed-forward stylisation networks.
Thank you for your comments. We agree with you that the interpretability of the learned style representation is an important area of investigation.
This is something we are currently working on. The results are very preliminary, and we do not feel like they should be included in the manuscript yet, but following another reviewer’s point we tried breaking the network into 3 sections (encoder, residual and decoder) and independently carrying out the style interpolation for the 3 sections. Broadly speaking we found the "encoder" did almost nothing; the "residual" performed most of the interesting spatial decomposition of the image and the "decoder" performed the colorization of the image.
Like was said above, more work is needed in this direction, but our goal would be to provide a more detailed analysis in the camera-ready version of the paper.
This paper addresses the problem of efficient neural stylization.
Instead of training a separate network for N different styles (as is done, e.g., in Johnson et al.), this paper extends the instance normalization work of Ulyanov et al. to train a single network and learn a smaller set “conditional instance normalization” parameters dependent on the desired output style.  The conditional instance normalization applies a learnt affine transformation on normalized feature maps at each layer in the network.  Qualitative results are shown.
I have not worked in this area, but I’m generally aware of the main issues in transferring artistic style.  The paper addresses a known challenge of incorporating different styles into the same net, which have a number of practical benefits.  As far as I can tell the results look compelling.  As I’m less confident in my expertise in this area, I’m happy to support another reviewer who is willing to champion this paper.
My main comments are on the paper writing.  As far as I understand, the main novelty of the approach starts in Section 2.2, and before that is review of prior art.  If this is indeed the case, one suggestion is to remove the subsection heading for 2.1 so it’s grouped with the first part of Section 2, and to cite related work for the feedforward network (e.g., Johnson et al.) in the text and in Fig 2 so it’s clear.
In fact, I’m wondering if Figs 2 and 3 can be combined somehow so that the contribution is clearer in the figures.
I was at first confused by Eq (5) as x and z are not defined anywhere.
Also, it may be helpful to write out everything explicitly as is done in the instance normalization paper.
In Eq (4), perhaps you could write T_s to emphasize that there are separate networks for different styles.
Fig 5 left: I’m assuming the different colors correspond to the different styles.  If so, perhaps mention this in the caption.
Also, this figure is hard to read.  Maybe instead show single curves with error bars that are averages over the loss curves for N-styles and individual styles.
Typos:
Page 1: Shouldn’t it be “VGG-16” network (not "VGG-19”)?
Page 2: “newtork” => “network”.
Paragraph after Eq. (5): “much less” => “fewer”.
Thank you for your comments. We found them very useful in improving the quality of the manuscript. Following your suggestions, we have made the following changes to the manuscript:
+ Subsection heading 2.1 was removed to group all prior art review in the same place.
+ Work related to feedforward networks is now cited in the prior art review section and in Fig 2.
+ We appreciate the reviewer’s suggestion to combine Figures 2 and 3. After some thought we decided to keep these figures separate because (a) the two figures convey distinct points and (b) Figure 2 recapitulates previous work while Figure 3 highlights our unique contribution. The latter point is quite important as we would not want to confuse a reader about what is unique to this paper. If the reviewer feels strongly otherwise, please do let us know.
+ For Equation 5, we have added additional text to clarify x and z.
+ For Equation 4, we felt that the equations were already quite heavy so rather then add a subscript s, we added explicit, additional text to highlight that the style transfer network is specific to a painting style.
+ In Fig 5 (left), an explanation for the curve colors has been added to the caption.
+ The “newtork” typo has been corrected in page 2.
+ “much less” has been replaced with “fewer” after Eq 5.
Regarding the mention of VGG-19 in page 1: This is intentional. Gatys et al. use VGG-19, whereas we use VGG-16 like Johnson et al.
The link to the manuscript has been updated and a new PDF may be downloaded. Changes addressed all of the reviewer comments.

Meta-review:
role: PC
data: comment: The reviewers (two of whom stated maximum confidence) are in consensus that this is a high-quality paper. It also attracted some public feedback which was also positive. The authors have already incorporated much of the feedback into their revised paper. This seems to be a clear accept in my opinion.

Decision: Accept (Poster)