Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces a styleGAN based stylization approach that allows the transfer of arbitrary style using a single model. The contribution includes: (1) the large-scale AAHQ dataset, (2) design of the style encoder using self-supervised representation learning, (3) the general idea of separating the latent code into a face code and a style code and using a weighted blending module to control how to blend the codes together to control the amount of style that get transferred into the result. 
main_review: I think the paper contains novel ideas and demonstrates compelling results. Though not the first paper to support the transfer of arbitrary style, compared to the previous feed-forward or optimization-based stylization approaches, the result quality has visibly improved. Therefore, I am in favor of accepting this paper but I do have the following suggestions and questions.

The results at a first glimpse is great. But with close examination, there are following issues. 
In the supplementary material, some results indicate that identity is sometimes not well-maintained. For example, in Figure 5 bottom right corner some results show visible gender change. In figure 7, the rightmost column, women are getting beard from the style reference. These reflect that style and identity is not completely disentangled during the training. Some of the identity aspect is carried over from the style reference. Figure 6 shows that, the stylization results sometimes have different gaze of facial expression from the input (for example, the middle two results in the rightmost column have the eyes closed). Figure 7 shows that style is sometimes not preserved very well. For example, the lip color in the results is different from the style reference in the rightmost two columns and the second column from the left. I think this reflects the fundamental limitation of this type of stylization approach which is that it lacks semantic correspondence between the style reference and the result. I think the main paper should discuss these limitations. 

It will be great to see ablation studies to analyze the effect of the style latent discriminator. What is the architecture of the style latent discriminator?  Several of the decision choices in the paper lack motivation and analysis. For example, the exact design of the two MLPs seems quite arbitrary. How do you decide how many layer to use?

In Line 224, the paper says “StarGANv2 … have some subtle artifacts and are not as natural as our results”. can you be more specific and point out the artifacts in StarGAN2 results? Also since the styles of StarGAN2 and the current paper are different, it is a bit hard to directly compare the results. 

I think the WBM paragraph in Section 3.2 can use some higher level description and motivation before diving into the equations. 

The supp mentions 128M images are used to train the style encoder. How did you obtain the images? They are augmented from the AAHQ dataset or is it a new dataset? Why do you need so many images ? Have you tried encoder training with less?  Later on, the supp mentions 1.6M images used to train the generator and discriminators. How did you obtain these images through augmenting FFHQ + AAHQ? 

Figure 2 and Figure 3 in the supp mention that the source images are sampled from FFHQ dataset and the reference image from AAHQ. Are these images held off from training or are they part of the training set?



limitations_and_societal_impact: No
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposes a method to synthesize faces with arbitrary artistic styles, where the style can be obtained from random samples or a reference style of a real face image. To achieve this, the authors introduce a style encoder to predict the style latent of a reference image. A face latent which controls the structure of the face is randomly sampled. The style latent and face latent are mixed in a fashion of style-mixing to generate the final output, where the degree of stylization is controlled by the number of layers blended with the style latent. The model is trained on FFHQ and a stylized face dataset the authors introduced simultaneously, where the face latent correspond to 
main_review: This work provides a clear explanation of their method, and the writing is easy to follow. Also, the proposed dataset (AAHQ) will be useful to the community. Overall, the proposed idea of allowing arbitrary style input and style blending is interesting. However, both ideas have been explored by prior work. Also, while most prior work uses real images as sources, the proposed method only shows results on synthesized source images. It will be more convincing to provide real image editing results, so this work can be applied in a more realistic scenario. In my opinion, these issues will need to be addressed to make this work strong. More details are provided in the limitation section.

Update: I read the feedback and all of my previous comments have been addressed. Also, after reading other reviewers' comments and authors' feedback, I agree that this work provides a novel way to combine stylization and generation, while also improving upon prior works. I will change my ratings to a 6.
limitations_and_societal_impact: 1. Prior work by Park et.al. [1] also provides a solution to the proposed problem but is not compared or discussed in this work. Park et.al. show capabilities of arbitrary stylization in multiple domains (e.g. LSUN church, FFHQ, AFHQ, etc.), while the proposed method conducted experiments on only the face domain. Also, Park et.al. allows structure and style separation by training on a single domain, while the proposed method requires collecting a real and stylized domain separately. It appears that Park et.al. provides a stronger yet more elegant solution to this problem.

2. No actual real image editing is shown in this work. Although real stylized images are used as a reference for the style, it is also important to show the capability of taking a real source image and stylize it.

3. In evaluation, all the source images are face images generated by the proposed method without stylization. These source images are then used as the inputs for the baselines. Since the baselines are image-to-image translation methods, the comparison is fairer if actual real images are used as the source images.

4. I'm confused why the completely stylized image ($i=0$) shares the same structure as the other faces ($i>0$). Does any of the face latent $w_f$ got blended into the final output? My guess is that the $\theta$ term in Eq. 5 allows this preservation, but will be great to verify this. 

[1] Park et.al. Swapping Autoencoder for Deep Image Manipulation.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposed a method for face generation across a wide range of artistic styles. The key contributions are:

1) a style encoder that maps the gram matrix of the style image to a latent space that can be consumed by the StyleGAN decoder. 

2) a blending module that accomplishes style transfer with linear interpolation between the style vector and the face vector, and a leant control vector that decides the “blending factor” for different layers. 

The paper also introduced a new dataset for artistic face images and demonstrated the method’s effectiveness in learning those artistic styles by a single network. 
main_review: The work is novel because it studies how to use the idea of blending the feature maps of two networks for artistic face generation with “arbitrary styles”. As a downstream application of StyleGAN for faces, the generated stylized images are in general of higher quality in comparison to the early style transfer methods that work on general subjects. In addition, unlike the recent StyleGAN based methods such as toonify networks, which have to be trained for each new style, the proposed method trained a single network that can be applied to a wide variety of styles. The limitation of out-of-distribution style transfer is studied and addressed by one-shot learning that fine-tunes the networks on a single out-of-distribution reference style image for 1000 iterations.

The training objectives seem intuitive, and there are sufficient technical details for reproduction. The proposed AAHQ dataset will be very interesting to the research community. I hope the authors can open it to the public. 

The results are encouraging, though I have a few comments in the following section. 

Overall the paper proposed a novel idea, and the method and results are solid. 
limitations_and_societal_impact: In general, the quality of the results is encouraging, especially considering the method is trained to work with many different styles and can be quickly adapted to new styles that haven't been trained with. However, it is also clear that the identity in the style reference image can leak into the results. This is especially obvious in Figure 7 (supplementary). It seems the gender of the source images can often be overwritten by the reference style images (rows 3 and 4).

I wonder why the i=18 results in Figure 4 (supplementary) are different between the Gram matrix approach and the style encoder approach? Did they use different face latent codes, or is there some differences in these two approaches (the values in the leaned alpha maybe?). Clarifying this would help the reader judge the likeness of the face identity in the results.

It would be helpful if the paper elaborates on the learned 18-dim alpha vector. What are the learned values in this vector? Is there any statistical bias that favors larger/smaller values in the certain layers for style transfer? Also, some comparisons between using the learned alpha and the results of using fixed alpha (such as 0.5 for midpoint interpolation) would be helpful.

Last but not least, it is not clear to me why the method restricts itself to reference style images that contain only portraits. The proposed style encoder first computes the Gram matrix, which has been shown to work well across arbitrary subjects and will drop most of the spatial content information of those subjects. So to me, it seems non-portrait reference style images probably will also work fine. On the other hand, if the objective is to make the reference style images and the face images better aligned in the latent space for blending, then a PSPNet fashion style encoder will make more sense. I would like to hear comments from the authors.

Minor: Line 142-143 says that the generator outputs stylized images when i=0 and natural images when i=18, but the rest of the paper (line 150-153 and eq. 6) seems to suggest the opposite.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose BlendGAN to perform arbitrary multi modal face stylization. They do so by pretraining a Style Encoder on an artistic dataset, then using weighted blending module (WBM) to blend face and style representations into an image using a StyleGAN-based decoder. They outperform previous methods qualitatively and quantitatively.  
main_review: One advantage of BlendGAN is its flexibility in using either randomly sampled style or a reference style to produce multimodal images. This allows arbitrary generation and unlike Toonify [1], does not require a new model for each new style.

I think the main novelty comes from learning two different distributions within a single StyleGAN. The style and face latent codes have its own separate distribution and when fed into the generator, with appropriate I, the generator can produce either real or artistic face. The use of WBM simply interpolates these 2 codes to generate a hybrid face.

In L143, it says i=0 corresponds to stylized face but in L151 natural faces are paired with i=0? Should this not be i=18?

Although Toonify is not multimodal, this is the closest baseline and I think it should be compared to. Otherwise the comparisons and quantitative results are sufficient and significant.

Qualitatively, BlendGAN captures textures well, but not global structures. For example, in Figure 1, all the face shapes are the same. This is a limitation due to the usage of layer swapping to main semantics. In Figure 5 if layer swapping is not used, it is closer to the style of the reference image. However, the semantics (such as expressions) are slightly off. Layer swapping is used to capture the semantics, at the cost of style similarity. The trade off seen in BlendGAN is the same as in Toonify. In this regard, there are no improvements over it. 

Overall, idea is not novel and is a combination of several previous work combining AdaIN [2], Toonify [1], and SimCLR [3]. However, there are some novel concepts in the implementation of the generator. The results are overall better than previous approaches, but BlendGAN still suffers similar problems faced by those approaches. 

[1] Pinkney, Justin NM, and Doron Adler. "Resolution Dependent GAN Interpolation for Controllable Image Synthesis Between Domains." arXiv preprint arXiv:2010.05334 (2020).

[2] Huang, Xun, and Serge Belongie. "Arbitrary style transfer in real-time with adaptive instance normalization." In Proceedings of the IEEE International Conference on Computer Vision, pp. 1501-1510. 2017.

[3] Chen, Ting, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. "A simple framework for contrastive learning of visual representations." In International conference on machine learning, pp. 1597-1607. PMLR, 2020.

limitations_and_societal_impact: yes
review_title: null

Meta-review: UPDATE: The revision of this paper has been reviewed and the paper has been accepted.  However, the following additional minor changes are required for the camera-ready:
- Acknowledge in the main text the issue that the new model has the same challenges as PULSE (i.e., input images with darker skin are now stylized as lighter skinned faces in the output).
- Update the "Broader Impacts" section to mention the fact that face data is biometric data and thus needs to be sourced and distributed more carefully, with attention to potential privacy, consent and copyright issues. 

----

It came to the attention of the program chairs and ethics review chairs very late in the review process that this paper deals with face generation, a sensitive application, but was not flagged for ethics review.  An emergency ethics review was obtained and the program chairs and ethics review chairs then discussed the paper in detail.

Based on this discussion, we have decided to conditionally accept the paper, given the ethical concerns related to face generation more broadly. In order for this paper to be fully accepted, authors need to address the following concerns:

- Provide examples beyond light-skinned faces and other potentially biased results or demonstrate some mitigation and reflection on any biased outcomes of the model. Include an analysis or disaggregated evaluation acknowledging any limitations in how the model handles faces of differing demographics. 
- Communicate any face data distribution restrictions or limitations to model distribution in light of privacy or malicious use concerns.
- Acknowledge any potential harmful applications or depictions that could arise from the use of this technology in an expanded “broader impacts and ethical considerations” section. 

We appreciate the cooperation of the authors in this process, and we hope that these adjustments and further reflection will improve the overall quality of the work. We hope authors can commit to these improvements as a requirement for inclusion at this conference. 

The original meta-review from the AC follows.

----

This paper proposes a model for synthesizing faces with diverse styles. The model can generate arbitrary stylized faces and their natural photo versions at the same time. In general, many reviewers find the idea interesting and the results encouraging. There are concerns regarding the technical novelty and comparisons against Swapping Autoencoder and Toonify. The rebuttal addressed most of the concerns and clarified the difference between the proposed work vs. Toonify and Swapping Autoencoder. The AC agreed with the reviewers’ consensus and recommended accepting the paper. 

Decision: Accept (Poster)