Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper presents a new research topic “interspecies face embedding”, which aims to extract common features from faces of several species, including humans, as a dense embedding. The process enables the discovery of other species' facial semantics without enough annotations by transferring knowledge from well-annotated human data. Algorithmically, a multi-teacher knowledge distillation paradigm is introduced to guide unsupervised embedding learning. Experimentally, based on the learned interspecies face embedding, the authors perform interspecies facial keypoint transfer on MAFL and AP-10K datasets.
strengths_and_weaknesses: Strengths: 
+ The problem addressed in the paper is interesting.
+ The proposed method of using multi-teacher knowledge distillation for interspecies face embedding learning is technically sound.
+ The paper is well-organized. 


Weaknesses:
Here are some concerns:
1. It’s not clear why the species-specific StyleGAN2 models share geometric consistencies in the learned W latent space.

i) In the pseudo-paired data synthesis stage, it is unclear to me why the pseudo-paired data x^d’ generated from the found latent code should have the same face geometry with the original image x^d (L178-179). In my understanding, there are not necessarily correlations in the learned species-specific W latent spaces. 

ii) Is the domain converter species-specific? Did you train separate StyleGAN2 models for different species? Or one universal model for all the animals in AFHQ.

iii) Have you considered training a single StyleGAN2 model by combining FFHQ and AFHQ? Would this make it easier to learn interspecies face embedding? Then the domain converter is not necessary.

2.	The experiment section is quite weak.

i) Qualitative and quantitative results on more keypoints are desirable. The authors only perform keypoint transfer on 3 landmarks. I suggest considering more semantic landmarks, i.e., corners of the mouse etc.

ii) The authors should include more baselines. The current baselines CSE and DVE are not designed for the face. The comparison is thus not convincing to me. I understand there is no literature addressing the same problem. But I think the authors could compare with visual semantic correspondence methods. Although these methods deal with intra-class semantic correspondence, the intra-class deformation of some categories (i.e., car) is more challenging than this interspecies face deformation. 

iii) I suggest the authors could add one experiment on interspecies face parsing, which can clearly demonstrate the effectiveness of the proposed method.

iv) More visual results are needed.

3. It’s unclear to me why the pre-trained CSE model is relevant to the face embedding learning and can improve the keypoint transfer performance (based on the results in Tab.2). In my understanding, the CSE model provides Interspecies body priors.

4. Discussions on limitations are needed. Such as, the proposed method may need to train species-specific StyleGAN2 models and domain converters.


questions: Please see the weaknesses.
limitations: Please see the weaknesses.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a face embedding method to obtain shared semantics between different animal species. The basic idea is to use a knowledge distillation paradigm to extract information from face synthesis (StyleGAN2) and interspecies surface embedding (CSE) models. Specifically, the main encoder is trained so that 1) the embedding is close to the one from CSE and 2) the domain-specific embeddings through converters are close to the features of StyleGAN2. The model also synthesizes pseudo-paired face images using the SyleGAN2 generator to enforce facial semantics correspondence further. 
strengths_and_weaknesses: ## Strengths
- The target task is interesting, and the proposed approach is reasonable to address this goal.
- The performance of the proposed method has been qualitatively and quantitatively verified from various angles, and the improvement from the baseline method is acknowledged.

## Weaknesses
- The proposed method takes the knowledge distillation approach and requires the pre-trained StyleGAN2 and CSE models for training. Compared to previous studies such as DVE [36], it can be seen that the data and annotations required for learning have increased. The performance of the final embedding is expected to be highly dependent on the performance of these two pre-trained models.
- Related to the above, the main contribution of the proposed method is the combination of these two models, and the pseudo-paired data synthesis is not that significant in terms of technical novelty.
questions: - As commented above, I would like a more detailed discussion of the training data and pre-training model that this method requires. For example, what images and annotations would be needed to target animals not included in AP-10K or AFHQ?
- Likewise, it would be good to see how much the performance of the pre-training model and the amount of data used for training affects the final performance.
- This is not necessarily a weakness, but it is interesting to note that the eyes are emphasized in the generated images seen in Figure 7 and elsewhere. It would be good to have a discussion on what reason this artifact occurs.
limitations: - The above issue of generalization to other animal categories is also mentioned in the text as a limitation.
- As the method deals with faces, sufficient consideration must be given to social impact, which is also mentioned in the text.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents an "Interspecies face understanding" task to predict unified spacial features of both human and animal faces.
To solve this problem, it presents a multi-teacher knowledge distillation framework that combines the advantages of different models, i.e. CSE for cross-domain features and StyleGAN2 for facial embedding.
It also explores the latent space of StyleGAN2 to synthesise paired data for semantic matching.
strengths_and_weaknesses: Strengths:
1. This paper explores the interspecies facial landmark detection model, which has many potential applications.
2. It is interesting to me to combine two orthogonal models to obtain the cross-domain facial embedding.

Weaknesses:
1. I think the "Interspecies Keypoint Transfer" experiment is not sufficient for the following points:\
  a) Dataset. I think 300W [1] or WFLW [2] for human facial landmarks, and [3] for animal facial landmarks, would be helpful to show the performance of this paper;\
  b) Point Number. Only 3 points are presented as the qualitative results. Can this method handle landmarks in mouth and face contour?\
  c) Comparison. If using the datasets mentioned in 1. a), there can be a comparison with previous supervised/unsupervised methods, which help readers better understand the performance of this paper. The current comparison looks isolated.\
  d) Ablation Study. The ablation study results are partly the same as the "Human+Dog" result in Table 1. So the CSE results in Table 1 are actually DIFE trained with L_{K1}? If so, I think it is improper to call it "CSE" here.
2. There is no constraint to enforce the spatial consistency of the input and output of the Domain Converter.
3. No real images are used when training the Encoder.

Reference

[1] Sagonas, Christos, et al. "300 faces in-the-wild challenge: The first facial landmark localization challenge." Proceedings of the IEEE international conference on computer vision workshops. 2013.

[2] Wu, Wayne, et al. "Look at boundary: A boundary-aware face alignment algorithm." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

[3] Khan, Muhammad Haris, et al. "Animalweb: A large-scale hierarchical dataset of annotated animal faces." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
questions: 1. Why not compare on the common face alignment datasets with commonly used metrics?
2. Can this method handle landmarks in mouth and face contour?
3. Are the CSE results in Table 1 actually DIFE trained with L_{K1}? If not, is there any illustration for the similarity between Table 2 and "Human+Dog" column in Table 1?
4. How to guarantee the spatial consistency between the input and output of the Domain Converter? How is this Domain Converter trained?
5. Only synthesised images are used to train the Encoder. If testing on a synthesised dataset, how is the performance? Do you have any experience with this?
limitations: The authors have discussed the broad impact and potential negative social impact of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper addresses the problem of dense interspecies face embedding. To cope with the lack of data, first multi-teacher knowledge distillation of Continuous Surface Embedding (CSE) and StyleGAN is used. Then, pseudo pair images are synthesized through the latent space exploration of StyleGAN2 to find implicit associations between different animal faces. Finally, a semantic matching loss is introduces to overcome the problem of extreme shape differences between species. Interspecies facial keypoint transfer is performed on MAFL and AP-10K. Results for applications in interspecies face image manipulation and dense keypoint transfer are also provided.

strengths_and_weaknesses: Strengths
The main contributions of the paper are:
- A cross-domain face understanding study is presented that considers human faces
as well as faces of animals. The approach domain adaptation to avoid tedious and expensive animal data collection.
- A multi-teacher knowledge distillation paradigm is proposed that extracts and combines the information from models with different architectures and data domains. The proposed framework learns continuous face embedding across interspecies data from CSE and StyleGAN2. With this solution dense interspecies face embedding (DIFE) are learnt.
- A method for synthesizing paired data is proposed for learning the semantic matching using the latent space exploration of StyleGAN2.

Weaknesses
The contribution of the paper is limited. It is more on the application side but it fails to convince about the effective transfer between human an animal faces and application domains.
In particular:
- The baselines used in the comparison of Table 1 are not designed for the same task proposed in this work and have been adapted to it so the comparison is somewhat inconsistent.
- the keypoint transfer is shown only for three keypoints (nose tip and eyes center, but the transfer on the animal face is not particularly accurate)
- the results in figure 6 also are not particularly accurate
- the applications are not particularly convincing

In general, the paper is well written, but there are some errors to check. See for example:
- page 3, line 124: "In StyleGAN2 distillation [? ]," --> missing reference
- page 5, line 177: "is an latent space" --> a latent

============= POST REBUTTAL ================
Authors have answered to most of my questions, so I changed my score to borderline accept. However, I still think the contribution is not very strong. 
questions: - Authors should convince more about the effective applicability of the method. In particular, I think it would be interesting to see a comparison with a solution that learns face keypoints from animal images rather than transferring them from face images. This would support more the method and is something missing in the paper. 

limitations: Authors reported about the impact of the work but in a too general way about limitations. I think they should have indicated more specifically the limitations possibly also providing failure cases.
review_title: null

Meta-review: metareview: This paper uses knowledge distillation to transfer learn facial embeddings across humans and animals. Helps when sufficient data for learning embeddings from animal faces is not available. An interesting application of standard concepts from domain adaptation, knowledge distillation, etc. While preparing the final paper the authors may highlight the novelty in this work. The paper is acceptable.
confidence: Less certain
Decision: Accept