Adjustable Real-time Style Transfer

Abstract: Artistic style transfer is the problem of synthesizing an image with content similar to a given image and style similar to another. Although recent feed-forward neural networks can generate stylized images in real-time, these models produce a single stylization given a pair of style/content images, and the user doesn't have control over the synthesized output. Moreover, the style transfer depends on the hyper-parameters of the model with varying ``optimum" for different input images. Therefore, if the stylized output is not appealing to the user, she/he has to try multiple models or retrain one with different hyper-parameters to get a favorite stylization. In this paper, we address these issues by proposing a novel method which allows adjustment of crucial hyper-parameters, after the training and in real-time, through a set of manually adjustable parameters. These parameters enable the user to modify the synthesized outputs from the same pair of style/content images, in search of a favorite stylized image. Our quantitative and qualitative experiments indicate how adjusting these parameters is comparable to retraining the model with different hyper-parameters. We also demonstrate how these parameters can be randomized to generate results which are diverse but still very similar in style and content.

Reviews and author responses:


We thank the reviewer for their helpful and comprehensive review. We updated the paper the address the main raised issues and clarify the questions. Please find the detailed answers below.

==============================

[Q] I'd highly encourage the authors to consolidate some of the early parts of the paper and greatly expand the technical section. For example, the first two paragraphs in section 4 can easily go into the introduction to avoid having to motivate the method twice.

[A]  Thank you for the great suggestion. We reformatted the paper by merging the suggested paragraph with the introduction as well as merging the background with proposed method. We also added more details regarding the technical aspects of our proposed method. This improves the structure of the paper and provides a more coherent and consistent story. The technical details of the paper are easier to read and understand.

=====================================

[Q] Figure reference bottom page 2 broken (??)
[A] Fixed in the updated version. Thanks for pointing out.


We thank the reviewer for their helpful and comprehensive review. We updated the paper the address the main raised issues and clarify the questions. Please find the detailed answers below.

==============================

[Q] I am not sure to understand why playing on instance normalization weights turns to be effective for learning an equivalent of a reduction of per-layer loss. I would have liked more motivation and intuition behind it.

[A] We thank the reviewer for asking clarification questions. We updated Section 3.2 (proposed method) of the paper to provide more clarification. In conditional instance normalization rather than learning the Gamma and Beta parameters of batch normalization layer, we generate them as a function of some input. Although these parameters are a tiny fraction of model’s parameters, they are quite effective in changing the outputs (check Perez 2018 and Dumoulin 2016 for more details). In other words, these parameters combined with non-linearities can manipulate feature maps in numerous ways including scaling, negating, filtering and selective thresholding. We have independent Gamma and Beta parameters for each feature map which gives the model an exponential number of ways (with respect to the number of feature maps) to affect the feature representation. Previous works (e.g. Dumoulin 2016) have shown that these parameters are very effective in generating models such as style transfer; where we can generate images with different styles by changing only these parameters. That was our main intuition for choosing conditional instance normalization for generating adjustable stylized images. 

========================

[Q] it is unclear how the specific approach of learning instance normalization weights extends beyond style transfer.

[A] Exploring this idea beyond style transfer is for sure one of our research projects. Instance normalization in particular has been used mostly for style transfer in previous works. However, in theory, it should be possible to condition any predictive or generative model on a few of its own hyperparameter (essentially any parameter that changes the generated output and is available at the training time), the same way that we conditioned our model on loss weights. This should allow for adjustment of these hyperparameters after the training (which comes at the cost of a more complex model and therefore more computation). There is already some qualitative evidence that the same method can be applied to other areas but unfortunately they are not comprehensive enough to be published as a scientific paper, yet :(, and requires more exploration. 

========================

[Q] I found Figure 7 important because it guarantees learning has the desired effect. However, there is a bit a lack of baseline/topline: how do the true losses decrease when their weights increase?

[A] We would like to thank the reviewer for raising this point which we %100 agree with. Unfortunately, adding a proper baseline requires training the base model for hundreds of times which was out of the time budget of the rebuttal, but this is surely an addition which we will consider for the final version of the paper. Nevertheless, it is worth mentioning that our main goal from this visualization was to demonstrate that the value of loss decreases, as expected, by adjusting the input parameters. That is why the y-axis is normalized. However, it sounds completely reasonable to draw the same graph for the base model by retraining it multiple times. 

========================




We thank the reviewer for their helpful and comprehensive review. We updated the paper the address the main raised issues and clarify the questions. Please find the detailed answers below.

==============================

[Q] The "adjust loss weights" are able to control the synthesized output, but does each of them have a particular meaning? For example, if I want to generate other outputs more realistic, colorful, or in other styles, is it possible for the users to (roughly) know which parameter should be adjust, and how? 
[Q] Following the above question, I suggest the authors to provide more qualitative results with different "loss weights", and give some detailed explanations.

[A]
We would like to thank the reviewer for asking clarification questions. We updated the caption of Figure 3 to make this more clear. In short, there is a one to one correspondence between each parameter and the weight of each loss term which means, adjusting each parameter is similar to adjusting the weight of corresponding loss term. Since we are using VGG layers for computing our loss function, lower level attributes corresponds to lower level VGG features (such as lines and curves) while higher level attributes correspond to higher level VGG features (such as shapes).This can also be explained by the increasing receptive field sizes as well as feature complexity along the processing hierarchy of VGG (check Gatys 2016 for more details).  Adjusting color and other more high level features requires defining a separate loss for each one of these features at the loss layer. 

We updated the caption of Figure 3 to make this more clear. More examples can be found in Figure 11. For more examples, we highly recommend visiting the project’s website at https://goo.gl/PVWQ9K which contains numerous results both interactive and non-interactive. 

=======================
[Q] Some typos and missing links could be corrected.
[A] Thanks for pointing out the problems. We fixed the missing links and some other typos in the updated version. 


The paper proposed a generative model for image style transfer in real time. In particular, comparing to the existing work, the proposed method is able to generate a series of transferred images instead of one, and more importantly, users can adjust different parameters without re-training the network to control over the synthesized output. The proposed method was evaluated on publicly-available datasets, and achieved convincing experimental results.

Pros:
- The problems that the paper tried to solve is interesting and important. I think the proposed model could make some contributions to related research communities.
- The idea of learning and incorporating "adjustable loss weights" is interesting and reasonable.
- The paper is well written and easy to understand.
- Experimental results appear to be promising.

Cons:
- The "adjust loss weights" are able to control the synthesized output, but does each of them have a particular meaning? For example, if I want to generate other outputs more realistic, colorful, or in other styles, is it possible for the users to (roughly) know which parameter should be adjust, and how?
- Following the above question, I suggest the authors to provide more qualitative results with different "loss weights", and give some detailed explanations.
- Some typos and missing links could be corrected.

I think the proposed approach is reasonable and and experimental results are convincing. I'm not an expert in this area, and I'm not sure about the method's novelty. If other reviewers find other existing work that proposed very similar ideas, then this would have an impact on my recommendation.

Strengths:
 + Interesting problem
 + Good results
Areas of improvement:
 - Structure of paper
 - Main technical section (4.1) very hard to understand.

The paper works on a very interesting and important problem. To the best of my knowledge there is no real time style transfer method that can adjust certain style parameters after it has been trained. The paper addresses this by parametrizing the normalization layers in the style transfer model, and training with a parametrized loss function. The results look compelling both in the paper, and the supplemental webpage.

The main area of improvement is the structure of the paper itself. The paper spends 4 pages on introduction, motivation and background. This unfortunately only leaves less than half a page for the main technical section. This main technical section is very dense, technical choices poorly motivated, and the section is impossible to understand in a single read (I had to read it 3-4 times). I'd highly encourage the authors to consolidate some of the early parts of the paper and greatly expand the technical section. For example, the first two paragraphs in section 4 can easily go into the introduction to avoid having to motivate the method twice. The main technical section should be motivated better. I had to constantly jump forth and back between background and technical section to understand where the technical changes are applied. It would help to have a more consistent story.


Minor:
 * Figure reference bottom page 2 broken (??)

The paper presents an approach for style transfer with controlable parameters. The controllable parameters correspond to the weights associated to "style losses" or ordinary style transfer models (distance between gram matrices of generated vs style image at specific layers of a network). The authors propose to learn a single architecture that takes these parameters as input to generate an image that resembles what would be generated by optimizing directly on these parameters. Examples of transfer and of the effect of these parameters are given. A quantitative evaluation shows that the effect of changing the parameters of the new network has the effect of reducing the loss at the desired layers.

Overall, I found the idea of learning controllable parameters interesting. The technical contribution is to show that learning the weights of instance normalization as a function of the hyperparameters actually is satisfactory. To be honest, I am not sure to understand why playing on instance normalization weights turns to be effective for learning an equivalent of a reduction of per-layer loss. I would have liked more motivation and intuition behind it.

The paper seems overall a bit incremental with respect to prior work on style transfer. While adjustable parameters could have application in more general tasks of image generation (for instance, in the line of work on disentangled representations), it is unclear how the specific approach of learning instance normalization weights extends beyond style transfer. As such, my feeling is that the paper is borderline.

other comments:
- I found Figure 7 important because it guarantees learning has the desired effect. However, there is a bit a lack of baseline/topline: how do the true losses decrease when their weights increase?

Meta-review: This paper offers an innovative approach to adjusting style transfer parameters.
The reviewers were consistent, and all recommend acceptance.  I concur. 


Decision: accept-poster