Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors introduce an internal-external learning framework for style transfer. In the proposed method, the classical VGG-based style loss, which encourages the stylized result to capture the low-level statistics of the style image, is combined with a GAN loss designed to capture the style priors from the style database. In addition, contrastive losses are designed to encourage the results to better capture the stylization-to-stylization relation. Experiments in the paper demonstrate the proposed method can generate visually more appealing results compared to previous style transfer methods.
main_review: 
Strength:
 + The paper is well written and easy to follow.
 + The idea of incorporating GAN loss to capture the style priors from a style database is interesting. The provided results seem to indicate that such external loss can help produce results that look more natural compared to previous works.
 + The evaluation processes for evaluating the stylization results and comparison with existing works are solid, with both visual results on a range of different content and a thorough user study.

Weaknesses:
 + I found the use of the term "internal learning" in this context a bit misleading. Internal learning has been used in the literature to refer to the process of optimizing the model parameters on a single image at **test time**. In this paper, the internal learning part seems to refer to the conventional style-transfer loss (which consists of VGG-based losses for style and content loss), which is used at training time and applied when trained on the whole dataset rather than for test-time optimization. I think it is fine to overload the term if it serves the purpose but clarification needs to be provided to avoid confusion. 
 + While I find it interesting to explore applying GAN loss to style transfer problem and I appreciate the improvement it brings to the results, I feel that more discussions and examples should be provided to analyze the effects of it. In particular, one challenge in incorporating GAN loss into this setting could happen when the provided style image has a unique style that does not appear in the style database. How would (and how should) the model behave in that case? I would appreciate more discussions on that regard along with more examples to analyze the benefit of incorporating GAN loss.
 + It's also not clear how much improvement was gained by the incorporation of the contrastive loss. Only three examples were provided (in Fig. 5) and the improvements from using contrastive loss in those examples do not seem obvious to me. More explanations on this would make the contributions of these additional losses better justified.

limitations_and_societal_impact: 
The limitation and societal impacts have been discussed in the paper (incorporated in the supplementary materials).
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: The paper proposes a neural style transfer algorithm, for transferring the artistic style of one image to another.   It is basically SANet [32] with some incremental novelties.

SANet is an encoder-decoder transformer architecture that accepts two images (content/desired style) and outputs a stylized version of the content by optimizing a Gatys-like loss function.

Two novelties are claimed over SANet.  

1) the use of ‘human external information’ by which the authors mean a corpus of real artworks in the desired style, to train a GAN discriminator added to the SANet.  It is argued that adding knowledge of multiple real artworks of the desired style in this way helps produce more plausible stylizations.

2) the use of contrastive learning to train their architecture.  The idea here is to ensure pair-wise consistency between style descriptors of generated artwork and the aforementioned corpus of artwork sharing that desired style.

main_review: The main contribution of this paper is extending SANet to draw upon external knowledge – i.e. in addition to taking in a single content/style image pair, a supporting corpus of images with similar content and similar style are provided.
The extensions are fairly straightforward..  the same exact SANet losses are used (ensuring Gatys-like content and style consistency using pre-trained VGG) – here referred to as ‘internal content/style’ losses – and an identity loss agains, as defined, in SANet.

The first technical tweak is adding a loss for the discriminator, which conducts the usual real/fake check on the output of the stylization vs. unpaired set of style images.  This is shown via a ‘deception score’ in Sec.4 - a kind of artistic Turing test user study  -  that the output artwork is considered more realistic, and qualitatively this is shown in the figures of the paper.  The idea is quite simple but seems to improve the output aesthetics.  The practical downside is the need to have a large corpus of images in the desired output style.

The second tweak is adding a contrastive loss and contrastive training methodology to the network – the formation of the batches is explained, but it is not explained how the training curriculum works in tandem with the GAN discriminator.  Is there some alternating training pattern similar to DCGAN?  How is this contrastive training integrated?   A key technical detail omitted from the implementation description is the batch size – it is mentioned that Figure 2 diagram 'uses a batch size b=8 for illustration' but what was used for the experiments reported? Presumably for contrastive learning to make any difference you need batch size in the hundreds or even thousands?  Again, is that practical given the number of style images required – and how would all of this fit into typical GPU VRAM it seems with this architecture it would be challenging.  The actual contrastive loss introduced has limited novelty it is just SimCLR temperature softmax approach with a header network, exactly as in the original SimCLR paper.

Overall I am not convinced by this paper.  The novelty is not significant, and the benefit of the external corpus of style images is not well demonstrated versus the practical disadvantage of requiring it.  The time cost of training such a network to perform a stylization, as well as the undescribed practical issues around large batch size, make me doubt this is a valuable approach.
limitations_and_societal_impact: Societal impact not discussed
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes two new additional losses for style transfer:
1. A contrastive loss to incorporate to bring the style of the stylized images with the same style image closer to each other, while pushing the stylized images from different style images further from each other. The same for content.
2. An adversarial loss to learn the general aesthetics of stylized images over a larger dataset.
The authors demonstrate that a combination of these losses plus the regular style-content loss can generate high quality, and surprisingly consistent stylized images, both qualitative and quantitatively as well as user studies.
main_review: ===== Writing

The paper is very well written but still can be improved.
- The motivation behind the new additions are clearly explained, although it can use more visual examples to demonstrate the shortcomings of the current approaches. Figure 1 can do this with a longer description as why the previous methods fail and how the proposed method shines. In fact, given the subjective nature of style transfer such descriptions are quite important. 
- The method is clearly described and is visualized in Figure 2 (texts can be slightly bigger for better readability in print). The reasoning behind the design decisions are clearly described and justified.
- The details of the experiments are noted, however, the training details can be expanded (in supplementary materials). The code is not included and including such details is crucial to attain reproducibility. 


===== Methodology

The proposed method is novel, clever and impactful. To the best of my knowledge, and as claimed by the authors, this is the first usage contrastive loss in style transfer and it is applied in a way which is intuitive and makes sense.  The paper tested their proposed method against multiple datasets and the majority of the SOTA models. For qualitative comparison, the authors conducted multiple user studies to capture the deception and preferences score which is pivotal for style transfer papers which lack good reliable metrics. 

However, I strongly suggest the authors to include a "non-cherry picked" section of their results to demonstrate the generalization capability of their model. The current set of results, look impressive but cherry-picked. This can be done as part of the supplementary materials or even better a website :) 

Another  missing section from the paper, is the training and inference complexity. I suspect by adding the proposed losses the training time substantially increases while the generation runtime stays the same. Training time is important mainly because the proposed model cannot be applied to learning-free style transfer methods.
limitations_and_societal_impact: The broader impact and limitation is missing from the paper and instead is in the supplementary materials. The authors enumerated the positive and negative impacts of their work however it is quite minimal and can be expanded.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper describes a method for feed-forward image stylization that combines several losses. The novel losses are a contrastive loss that optimizes the style embedding of the training image to match the embedding of a target style and content and to be distinct from other style and contents.  Other losses used during training include feature mean/variances and a GAN loss.

main_review: Overall, the visual quality of the results (Figure 3) looks substantially better than the baselines. The detailed structure and scene contours are preserved better than the baselines, and the regions are filled with sensible textures from the exemplars. I haven't attempted to survey the literature to judge if these are fair representations of the baselines

It's unclear to me how these improvements results from the contrastive losses. Better representation of image structure doesn't seem to encoded in the contrastive losses.

The writing of the paper is very difficult to follow and I'm not sure that it's reproducible. Section 3 never lists the inputs or outputs for training. It took a long time to figure out that this section describes training and not test time. How are the parameters of the h_s and h_c networks determined?  What feature space are the first two terms in Eq 4; in pixel space?  Eqs 3 and 4 use ||_||_2 notation; does this mean there are square-roots in these terms?

The motivation and explanation of the algorithm coudl also use a lot of improvement.  The Englihs is not very good. 
* L26: "existing style transfer methods often confine themselves to the internal style statistics of a single image". This is obviously false, e.g., see pix2pix, CycleGAN, CUTG. 
* L31: "... more consistent with human perception" This is a bold claim, either remove it provide citations to justify whatever assertion is being made here.  
* L50: "... this is the first work successfully leveraging the power of contrastive learning in style transfer": This is false; see CUT [34].
* L109: "Existing style transfer methods usually produce unsatisfying stylization results with disharmonious colors..."  So the claim here is that all previous methods are terrible?


The video style transfer results are so weak that I think they should be removed if the paper is accepted.  The frame rate of the results is so low that one cannot judge the temporal coherence of the provided videos. These are more like sequences of independent still images.  

The paper fails to compare to the state-of-the-art in video style transfer, or to cite it:

Ondřej Jamriška, Šárka Sochorová, Ondřej Texler, Michal Lukáč, Jakub Fišer, Jingwan Lu, Eli Shechtman, and Daniel Sýkora
Stylizing Video by Example    
In ACM Transactions on Graphics 38(4):107, 2019 (SIGGRAPH 2019, Los Angeles, USA, July 2019), ACM.

Ondřej Texler, David Futschik, Michal Kučera, Ondřej Jamriška, Šárka Sochorová, Menglei Chai, Sergey Tulyakov, and Daniel Sýkora
Interactive Video Stylization Using Few-Shot Patch-Based Training    
In ACM Transactions on Graphics 39(4):73, 2020 (SIGGRAPH, August 2020)

limitations_and_societal_impact: The paper doesn't discuss limitations.
review_title: null

Meta-review: The submission proposes an extension of SANet's artistic style transfer loss formulation which adds adversarial and contrastive terms to the loss. The adversarial loss term is computed using a discriminator trained to classify images as either belonging to the style dataset or to the distribution of stylized content images. The contrastive loss terms encourage the embeddings of stylized images which either share the same content or style image to be closer and vice versa.

The proposed approach is compared against SANet and other baselines through qualitative (showing stylizations) and quantitative (user study, LPIPS) means. Stylizations are compared for a few content/style image pairs to qualitatively assess the effect of the adversarial and contrastive loss terms.

Reviewers appreciate the writing quality and agree on the fact that the results presented in the submission represent an improvement over previous approaches, but there is disagreement on whether the analysis of the effect of individual loss components is sufficient, and on whether the limitations of the GAN loss are sufficiently discussed.

On the limitations of the GAN loss, reviewer JMx5 objects that its benefit may not be realized for style images that are very different from the training styles. The authors respond that the style images they evaluate on have not been seen during training. Reviewer zxcr points out in the reviewer discussion that the extent to which the effect of the GAN loss generalizes to held-out style images likely depends on whether they are in-distribution or out-of-distribution (the "distribution" here being the distribution of style images seen during training). This is a good point, and the authors' response here isn't entirely satisfactory: even if the model generalizes to in-distribution held-out style images, there's no guarantee that it would generalize to out-of-distribution style images, and that limitation should be discussed. In the discussion, Reviewer vsGU mentions that discussing the limitations of the proposed approach belongs to the main paper and not the Appendix. I agree with that.

There was also a debate on whether the distinction between style transfer and domain translation is artificial, which I think requires some nuance. Since domain translation approaches work by imitating the statistics of a population of domain-specific images, it does feel weird that the submission dismisses style transfer approaches as "neglecting the external style information reserved in the large-scale style dataset" without acknowledging domain translation approaches. There is a distinction between the two (the former targets a specific style image while the latter targets the common "look and feel" of a collection of style images), but I think the submission would benefit from being more generous in drawing connections between various families of approaches.

On the influence of the added loss terms, I would break the issue down into two questions:
- Are the current ablations convincing enough?
- If they are not, how much does that weaken the submission?

I agree with Reviewers JMx5 and zxcr that a qualitative assessment on three content/style image pairs is a very small number of data points to draw conclusions from. This issue could have been fixed by adding ablated models among the approaches evaluated in Section 4.3's user study.

On whether this is a serious issue, the risk here is that the effect of one of the two loss terms could be marginal, which would nullify one of the submission's two major contributions. The principle of incorporating global information from a corpus of styles into the training signal remains an interesting contribution nevertheless, but obviously the paper's impact would be more substantial if it better characterized the contributions of the two loss terms.

This submission is very much a borderline case. I read the paper attentively, and I agree that the idea of incorporating information from a corpus of styles rather than treating each style as separate is a new and interesting contribution which incorporates insights from domain translation. Even though the submission has its flaws, I recommend accepting it provided the following is addressed in the final manuscript:
- A discussion on the approach's limitations, and in particular those of its GAN loss, is incorporated into the main text.
- The relationship between the proposed approach and domain translation is better highlighted.
- Additional examples are shown in the Appendix to accompany Figure 5.
Decision: Accept (Poster)