Dense Interspecies Face Embedding
Authors: Sejong Yang
Subin Jeon
Seonghyeon Nam
Seon Joo Kim
Venue: NeurIPS 2022 Accept (2022)
Keywords: Face Understanding
Cross Domain
Knowledge Distillation

Abstract: Dense Interspecies Face Embedding (DIFE) is a new direction for understanding faces of various animals by extracting common features among animal faces including human face. There are three main obstacles for interspecies face understanding: (1) lack of animal data compared to human, (2) ambiguous connection between faces of various animals, and (3) extreme shape and style variance. To cope with the lack of data, we utilize multi-teacher knowledge distillation of CSE and StyleGAN2 requiring no additional data or label. Then we synthesize pseudo pair images through the latent space exploration of StyleGAN2 to find implicit associations between different animal faces. Finally, we introduce the semantic matching loss to overcome the problem of extreme shape differences between species. To quantitatively evaluate our method over possible previous methodologies like unsupervised keypoint detection, we perform interspecies facial keypoint transfer on MAFL and AP-10K. Furthermore, the results of other applications like interspecies face image manipulation and dense keypoint transfer are provided. The code is available at https://github.com/kingsj0405/dife.

Dense Interspecies Face Embedding
Sejong Yang Yonsei University sejong- wanoongyannyyoosei..e. ac kr
Subin Jeon
Yonsei University
subinjeon@yonsei. ac. kr
Seonghyeon Nam York University 1aan0331@gmail1 com
Scon Joo Kim
Yonsei University
seonjookim@yonneei. ac .r
Abstract
Dense Interspecies Face Embedding (DIFE) is a new direction for understand- ing faces of various animals by extracting common features among animal faces including human face. There are three main obstacles for interspecies face under- standing: (I) lack of animal data compared human, (2) ambiguous connection between faces of various animals, and (3) extreme shape and style variance. To cope with the lack of data, utilize multi-teacher knowledge distillation of CSE and StyleGAN2 requiring no additional data or label. Then we synthesize pseudo pair images through the latent space exploration of StyleGAN2 find implicit associations between different animal faces. Finally, we introduce the semantic matching loss to overcome the problem of extreme shape differences between species. To quantitatively evaluate our method over possible previous methodolo- gies like unsupervised keypoint detection, we perform interspecies facial keypoint transfer on MAFL. and AP-10K. Furthermore, the results of other applications like interspecies face image manipulation and dense keypoint transfer are provided. The code is available at http://github..oo/kiigggUU/KS
Introduction
Face understanding is a longstanding research topic in computer vision, and there have been extensive studies and applications such face recognition, facial expression estimation, face photo manipula- tion, and virtual humans (6] 47 4 [3. The vast majority of successful works on faces have focused on human faces due its practical implication for variety of applications and the accessibility to a large amount of annotated data. There have been few attempts for understanding faces of other animal species to facilitate more applications such as ecological research, pet identification, and farm automation 14 34 12]. However, the task of understanding animal faces is more challenging than that for the human face due the lack of sufficient data with annotations. In the literature on full-body understanding, joint learning of interspecies data has been studied as remedy by transferring the knowledge of well-annotated domain such human to other species [33] 26 27). However, the existing approaches are focused on full-body data, therefore their performance on facial data limited.
In this work, we are interested in the problem of learning the representation of interspecies facial semantics, which common representation shared across interspecies f'acial data. Learning such representation allows us to extend face understanding beyond the human face, as it enables discovering facial semantics other species without enough annotations through transferring knowledge from well-annotated human data. As many animal species share similar face geometry, the representation can also benefit from learning multiple species jointly. Since sharing photos of animals through social networks has become more popular, learning the interspecies face representation can be used for
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(a) Dense Interspecies Face Embedding (DIFE)
(b) Applications of DIFE
Figure I: (a) Dense Interspecies Face Embedding (DIFE) is a representation that contains common characteristies of faces of various species embedded to pixels in the feature space. (b) DIFE allows us to find visual correspondence across multiple face domain, so we can transfer facial keypoints between animals or generate different animal faces with the same posture.
various applications. For instance, one can search animal photos using a photo of human face as query to find animal faces with similar pose, An animal face animation can be synthesized by using that of a human template, which can be useful for media production.
To achieve our goal, we propose to learn Dense Interspecies Face Embedding (DIFE) shown in Fig. 1(a). Our approach is inspired by CSE [27). which was proposed to learn continuous surface embedding (CSE) of full-body shared across interspecies data. While the interspecies embedding in 277 is promising, it is difficult to use it for faces as the embedding in the facial region is too coarse to discriminate facial landmarks accurately.
overcome this problem, we propose a multi-teacher knowledge distillation method for learning DIFE. Our key idea is to enforce a student network to learn fine-grained interspecies facial embedding from two teacher networks: CSE and StyleGAN2 [19]. Since the CSE has useful representation of interspecies data, student network learn a common embedding space shared in multiple species. For a fine-grained face embedding, we simultaneously exploit StyleGAN2 as the secondary teacher. StyleGAN2 is originally trained to generate high-quality face images [9]. Recent studies have demonstrated that the features of StyleGAN2 have abundant information on facial attributes such as postures and expressions 11 36 37). Thus, the goal of our knowledge distillation is to transfer such information to the student network for learning tine-grined facial semantics for multiple species.
use it as teacher to have the
To further improve the performance DIFE, we propose data augmentation method to synthesize mheudo-paired images, and the semantic matching loss to enhance the semantic correspondence between images. For the data augmentation, we synthesize images, where the face poses of the synthesized images are similar to that of input images using StyleGAN2. This can be achieved in our framework as DIFE is tied to the latent features of multiple StyleGAN2 networks in different domains. Thus, DIFE of the input image can be used as face geometry condition for the latent space exploration of StyleGAN2 to generate images for both intra- and inter-domain pairs. In addition, inspired by [39). semantic matching loss is proposed to use the pseudo-paired data to enforce the DIFE to learn pixel-level fine-grained matching.
We demonstrate the effectiveness of our method on the cross-domain keypoint transfer task using interspecies data such as people, dogs, cats, and wild animals in MAFL (50) and AP-10K [45). We evaluate our method on both sparse and dense keypoint transfers. In addition, we apply our method for an image editing task of overlaying a virtual object on various animal faces.
Our contribution can be summarized as follows:
We present a cross-domain face understanding study that considers not only human faces but also the faces of animals. Our approach makes it possible to avoid tedious and expensive animal data collection through domain adaptation, setting the stage for a variety of cross- domain applications.
We present novel multi-teacher knowledge distillation paradigm that extracts and combines the information from models with different architectures and data domains. Our framework is designed to learn continuous face embedding across interspecies data from CSE and fine- grainad face embedding from StyleGAN2. Combining the knowledge from the two, our method learns dense interspecies face embedding (DIFE).

---

We further introduce a method for synthesizing paired data for learning the semantic matching using the latent space exploration of StyleGAN2. With the novel approach, we can refine the representation of our face embedding for semantic correspondence between various instances and species.
2 Related Works
2.1 Domain-specific Face Understanding
We first review works that aimed to identify the visual correspondence between various instances and scenes within one data category, human or animal. For faces, detecting facial landmarks such as the eyes and the nose has been studied extensively to find correspondences between subjects. Initial works [38 24 46 49 43 find facial landmarks using facial landmark annotations in a fully supervised way. Regarding animal faces, Yang et al. 44 detected sheep facial landmarks based 6 annotated keypoints, Although most works show impressive results under extreme conditions, the lack of annotated animal data limits the application compared to human faces.
17 on finding landmarks from collection of images in an unsupervised way, which is applicable to various domains, including human body, face, etc. This stream of works relies on the fact that landmarks are spatially equivariant to the transformation. Some works learn landmarks by comparing warped images with the original ones [40] (8), and others cast the problem into a conditional image generation using video frames depicting the same instances in different poses [ [6]. The most related work to method is DVE (39] in that they consider not only equivariance between the same instances but also intra- category variations. All of aforementioned methods can detect landmarks within the same object categories, but the landmarks are not convertible between different object categories (e.g. humans and cats).
Recently, there have been active researches 40
39
2.2 Interspecies Understanding
There are studies that aim to discover the relationship between animal species, whose structures are related but geometrically diverse. To the best of our knowledge, (32) is the only work that conducted semantic matching between animal faces. Animal facial landmark detector is trained by transferring the knowledge from the pre-trained human landmark detector. The network is finetuned using horses and sheep data with 5 keypoints annotations. In comparison, we propose an unsupervised approach for interspecies facial semantic matching with the higher granularity of the keypoints.
Closely relevant to our work is the stream works in [33 26] (27). as they find the relationship of body poses between animals. Sanakoyeu et al. (33 transfers the knowledge of the human body to the chimpanzee using DensePose [10). The CSE (26) utilizes the manually annotated 3D meshes relate several animal categories flexibly. In the following work (27). they utilize the cycle-consistency between meshes and images to further the performance. Although they show impressive results on the complete body, they have difficulty in achieving a fine-level sharpness on faces, making it impossible to discern different parts of faces. While our work also takes the advantage of CSE [27] detect interspecies correspondences, we improve the accuracy of facial correspondence with the help of the latent features of StyleGAN2 [9 without additional annotations.
of each category and continuous surface embedding
2.3 StyleGAN2 Utilization
Our work is also related to StyleGAN utilization works in that features of pretrained Style- GAN2 [19] to obtain fine-grained facial features. StyleGANs [8. [9]
show highly peoto-realistic results in the unconditional image generation problem with the power of well disentangled latent spaces. There have been abundant works to exploit the pre-trained StyleGAN for various tasks,
including image manipulation [1 B 37 35). 3D reconstruction
30) image segmentation 22
and semantic matching [3I]
Because StyleGAN2 has plenty of face geometry many works try to extract the information with
knowledge distillation 29 Labels4Free 2 and Segmentation in style [2] utilizes the feature
of StyleGAN2 to train segmentation model in unsupervised manner. In StyleGAN2 distillation [4].
3

---

Embedding
CSE
L
CSE.
DIFE
Enbbedlngg
Synthesized Image
CHs Feature of StyleGAN2
Facial Embeldiig
Domain-Specifie Embedding
Encoder Ø
Domain Converter
Lx2
StyleGAN2
Crosi
yrd
Facial Endbedling
x
E
D
F
(a) Knowledge Distillation
Original Domain
Puir Domatn
Synthesized Image
DIFE
Domain-Speciffi Embedding
Laions Space Exploniia
Pseudo-Pair
DIFE of Pscudo-Pair
S
LTr'
x
Ed
D
x
E
Semantle Matching Loss
(b) Pseudo-Pais Data Synthesis and Utilization
Figure 2: (a) Our encoder learns the cross domain association and the geometry of face with the responses and the intermediate feature from CSE and multiple StyleGAN2's The converters for each domain necessary to generate the domain-specific embeddings which is compatible with the intermediate features of StyleGAN2s. (b) The pseudo pair data of different domains can be synthesized with the domain-specific embedding and the latent exploration of StyleGAN2. The semantic matching loss is utilized overcome the shape difference between faces of different species.
they generate images and labels with StyleGAN2 generator and face attribute classifier, and image manipulation model is trained with synthesized data. However we can tell studies have used StyleGAN models learned in different data domains at the same time, while our work learns the relationship between separate StyleGAN2 models, which have different data domains.
3 Method
Let x" E HX''''s denote an image of face in domain E 1..., - 1}, where H and W the height and the width of the image and D denote the number of domains. The goal of our task obtain a dense interspecies face embedding E E R. x xn of the image x through an encoder &. which is described as E o(x*). ). Here, n is the embedding dimension for dense interspecies face embedding. To train the encoder + in an unsupervised manner, we propose two training approaches: the multi-teacher knowledge distillation, and the semantic matching through pscudo-paired data synthesis. Since all images and labels are synthesized by the teacher models whose training data are not suitable for task, the training can be considered as unsupervised learning. In the following, we describe our dense interspecies face embedding (DIFE) and its training detail.
3.1 Multi-Teacher Knowledge Distillation
Fig. 2(a) illustrates the overview of the multi-teacher knowledge distillation multiple domains consisting of humans, dogs, and cats. Desirable properties of the interspecies face embedding are two folds. First, the representation is shared by various face images of different domains. Faces in multiple domains should have similar embeddings if their geometry is similar. Second, the embedding contains fine- fine-grained details of facial semanties, that different parts of face distinguishable.
To learn such properties, we first train DIFE by distilling knowledge from CSE 27). the representation of CSE has high-quality co-embeddings of diverse animal species, exploit it to

---

learn a common space for cross-domain faces. Specifically, we optimize a knowledge distillation loss that minimizes the distance between the DIFE and the CSE embedding of the same input. Formally, the loss is described as
C Σ |
-x(x))12--
(1)
where Cd is the CSE embedding of x4. Note that x is an image synthesized by StyleGAN2 described as dotted arrow in Fig. 2
Although the above loss enforces our embedding to learn a shared space for multiple domains of faces, the representation of CSE is coarse in the region of the face since it is originally learned to distinguish different parts of a full body. To further learn fine-grained details of faces, we add another knowledge distillation loss using the features of StyleGAN2. Specifically, we exploit the semantic information of faces extracted from pre-trained StyleGAN2 to enhance the representation of DIFE. Using synthetic images generated by the StyleGAN2 as imput, we minimize the distance between the embedding of the input and its corresponding StyleGAN2 feature map. which is formulated as
Σ
Fd Dd1 2
(2)
where Fdd is the StyleGAN2 feature map, and Dd is the domain-speciin embedding of x for the domain d. Since the dimension of DIFE is different from that of StyleGAN2 and each feature of StyleGAN2s is in the different domain, we convert DIFE to a domain
embedding using a domain converter Vrl as follows: D = (E*). When learning from multiple domains, we
optimize L = Ea k and Lx2 = Ed kx for all of the domains.
3.2 Semantic Matching
Pseudo-paired data synthesis. Fig. 20b) shows how to synthesize greudo-paired data x" which has the identical face geometry with the original image x². The paired domain d' can be different from the original domain d. From a given human face image x" generated by StyleGAN2, we first extract DIFE using the encoder 4. Then, with the domain converter qpa"" we transform the interspecies embedding into domain-specifer embedding De In Fig. 2(b). we convert DIFE into the dog-specific embedding. Note that the domain-specifer embedding lies the similar vector space with StyleGAN2, since the domain converter is trained by the StyleGAN2 knowledge distillation.
Finally, we synthesize pseudo-paired image x"' by feeding the domain-specific embedding D to StyleGAN2. Rather than directly feeding the embedding, we introduce latent space exploration method, which generates more realistic images by adjusting the latent space of the domain-specifte embeddings into the natural image manifold.
In the latent space exploration, given a domain domain-specifie embedding Dd the goal is to find latent code wd' Wad which satisfies the following two conditions. Here, W is a latent space of StyleGAN2 trained on the data domain d'. First, the pseudo-paired data x" generated from the found latent code w should have the same face geometry with the original image x". This is achieved by finding w with small difference between the StyleGAN2 feature F (w that is generated from w"", and the domain-specific embedding Dd". Second, the seoudo-paired data needs to be realistic and natural. To this end, following []]. we take the mean of latent codes w, and enforce our latent code to be similar to w. w is calculated by sampling 4096 latent codes. With the regularization, the generated image looks natural by locating the latent code in the realistic image manifold. Finally, the latent code w" is found by optimizing the following equation:
arg min wEwW'
Fddd (w)
Dd 2
+
l|w
w/12
Here, the feature map Fd (w) is extracted from StyleGAN2 by feeding the latent code to optimize. After the optimization, the pseudo-paired data x"" is synthesized by StyleGAN2 with w".
Semantic matching loss. The purpose of the semantic matching loss is to boost the matching performance of DIFE at pixel level with the synthesized pseudo-paired data. The pixel location u E ? on 2D cartesian coordinate system Ω {0, .... H' 1} x {0, 10 W - 1} is used to explain
5

---

the loss. Theoretically, the pixel embedding E at location u in DIFE of the original image has the same semantic meaning with the corresponding pixel embedding E of the pseudo-paired image. However, the shape difference between different domains makes it infeasible to exactly match the pixel locations. For example, the eyes of the original image x and the eyes of the pseudo-paired data x" in Fig. 2(b) do not perfectly match. To alleviate this issue, we introduce soft distance measure similar to [39]. Intuitively, we compare the embedding from a pixel on the original image to the weighted embedding of a small region the pseudo-pane paired image, as described as the contour in Fig. bb. We find the region compare based on the similarity o (EE, Ed Uis Uj) between the pixel embeddings as:
E
(E E ui uz)
u.
(4)
E
Ed"
where u, and uy are the pixel coordinates of Ed and Ed respectively. Finally, the semantic matching loss ad is defined follows:
Cd.d' M
Σ E0
u4 Σ
(E Ed un, uz)
(5)
MyE
When learning from multiple domains, we optimize intra semantic matching loss Can Ed and inter semantic matching loss L2 = D Ed Car for all permutations of the domains.
3.3 Training Objective
We can formulate our final training objective with hyper-parameterd as:
L = A1 Lx1 + )2 LK2 + Xa LM1 Å LM2-
The matching losses Las and Lase cannot be applied in the early stage of training, since the domain- specific embedding is different from the features of StyleGAN2s. Therefore, we first impose Lx1 and Lx2 on the encoder and domain converter, and after initialization, we start to generate pseudo-paired data and utilize semantic matching losses L.1 and L
4 Experiments
4.1 Interspecies Keypoint Transfer
Given image in the source domain with keypoints, the goal of interspecies keypoint transfer is to find the corresponding locations of the keypoints on another image in the target domain. We mainly evaluate our method on this task to demonstrate that our embedding is not only high quality representation of cross-domain faces, but also beneficial for knowledge transfer to unlabeled domains. The task is basically achieved by finding the pixel-level correspondence between the embeddings of the source and the target images. The performance is quantitatively evaluated by calculating the error between the positions of the transferred keypoints and their corresponding ground truth. We further describe mathematical formulations of the evaluation in detail in the supplementary material.
Datasets. At the training time, DIFE is trained with synthetic images and their target labels generated by pre-trained CSE and StyleGAN2 networks. We use four datasets for evaluation: MAFL [50]. AP-I0K [45). WFLW (42) and AnimalWeb (20]. The MAFL is a dataset of human faces that is composed of 20k images. The test set of MAFL excluding the shared images with the training set of CelebA is used (400 images). The AP-10K dataset composed of 10k images of various animal species which is split into three disjoint subsets, i.e., train, validation, and test sets, with the ratio of 7:1:2 per animal species. As the images contain the full-body of animals, we crop the region of their faces by computing the bounding box using facial landmarks in the dataset. For the evaluation, we make pairs of two different species by randomly selecting images from each domain. As result, we collect 100 pairs of humans and dogs, 89 pairs of human and cats, 47 pairs of humans and wild animals, and pairs of dogs and cats. The same preprocessing applied to WFLW and AnimalWel to evaluate method more dense keypoint annotations including the mouth.

---

Table I: Quantitative results of interspecies keypoint transfer MAFL and AP-10K. The pixel error is obtained by calculating the euclidean distance between the ground truth and the transferred keypoints. indicates that the model is trained by the authors of this paper, the pre-trained weights are not provided. f indicates that the data domains for the learning and the evaluation do not match.
Pixel Error
Method
Human+Cat Hog
Dog+Cat
Human+ Dog+CCa
Human+ Wild f
CSE
19.00
18,15
8.00
15.20
17.69
D
17.78
17.40
6.75
14.34
15.58
CATs
14.78
17.63
8.47
13.67
14.05
Ours(DIFE)
11.73
11.00
6.51
13.16
10.37
Source
CSE
DVE
CATs
Ours
Figure 3: Qualitative results of interspecies keypoint transfer on MAFL and AP- 10K.
Implementation details. We pre-train CSE with LansePose-COCy which are the datasets for full-body keypoints of human and animal respectively. StyleGAN2 is pre-trained with FFHQ [19] for human and AFHQ [9] for animals. DIFE encoder uses the hourglass 339 and the domain converter consists of three consecutive 7 7 conv layers. We Adam optimizer (1 with the learning rate of 10-3 batch size of 12, and max training step of 1M with early stopping. The lambda values for each loss functions fixed to 10 10-2 X2 10", A3 10 and by 10-2 in all experiments. All experiments are carried out on one NVIDIA Titan XP.
23 and DensePose-LVis [1].
network 28 for a fair comparison with DVE
Baselines. We compare our method with three basclines: CSE (27 DVE 33 and CATs g Even though CSE is originally learned for full-body embeddings of interspecies data, compare the method to demonstrate the effectiveness of our method on face images. DVE is an unsupervised learning approach for learning pixel-level facial embeddings. which designed for human faces in the original work. For the comparison, we train the method with interspecies data. Both the training set of AP-10K and synthesized images of StyleGAN2 were used, and the latter is selected for better performance. We implemented the baselines using the original source code available on their websites with the default hyperparameters. CAT's the state-of-the-arte for finding visual semantic correspondence. We use the pre-trained model from the original paper trained on SPair-71k (5 that contains annotations including the landmarks of dogs, cats, and other animals.
Quantitative results. Table T presents the quantitative results CSE, DVE, and DIFE on humans, dogs, cats, and wild animals showing that our method achieves better performance in all data domains.
7

---

Table 2: The interspecies keypoint transfer on WFLW 42 and AnimalWeb 20
Human+ Dog
Human+Cat
Dog+Cal
Human+
CSE
14.43
14.95
16.85
16.08
DVE
14.16
13.28
13.40
15.74
CATs
20.84
18.84
19.35
22.20
Ours
12.01
12.84
11.70
14.03
As can be seen in the table, transferring between human and other animals is more challenging due to the larger shape differences between faces of human and animals. Compared to other baselines, DIFE makes significant improvement with the help of the semantic loss. The errors in all methods are lower for Dog+Cat, with DIFE performing better than other baselines. We have also tested a challenging case of the keypoint transfer between three domains by training the models at once with human, dogs, and cats, in which DIFE also outperforms other baselines. For the Human+ Wild case, utilized the model that was trained with human+cat, because it was difficult train with human and wild animals due to the extreme shape and style variance. Since the wild animal domain includes animals that resemble cats, such tigers, lions, and cheetahs, using model trained with human and cats performed reasonably. showing that our method is applicable for out-of-domain cases. Our method also outperforms the CAT' which indicates the applicability of our method to discovering landmarks in unlabeled animal domains.
Table shows the quantitative result of the interspecies keypoint transfer WFLW 42 and Animal Web [20] with landmarks including the corners of the eye and mouth. Our method shows the best performance compared to previous methods on every domain pair.
Qualitative results. Fig. shows some qualitative results of the keypoint transfer. As expected, CSE has difficulty in finely distinguishing different parts of faces as it has limited face prior for animals due the sparse annotation DensePose-L.VIS. While DVE tends to recognize the eyes and the noses some extent, it does not properly overcome the shape difference, showing some mismatches. DIFE, with its ability to understand cross-domain association and face embedding, can figure out exact visual correspondence on interspecies faces. More visualizations of the interspecies keypoint transfer including the results on WFLW and AnimalWeb can be found in the appendix.
Fig. 4 visualizes the embeddings of different methods. CSE shows similar values across domains, but less distinction between different parts of face. On the other hand, StyleGAN2 features exhibit distinctiveness between different parts of face, but corresponding parts between different domains show large difference in values, making difficult to align between different domains. DIFE makes the best of both worlds, showing distinctive values within face, while having similar values between corresponding face parts between different domains.
Embeedingg CSF Cid
Features StyleGAN2
Image
Layer5
Layer7
Layer9
d
Fd
E
Image
x
Features StyleGAN2 Fd
Coarse
Fine
Figure 4: Visualization of embeddings
Figure 5: Visualization of features of StyleGAN2

---

Table 3: Ablation study losses
Table 4: Quantitative results of choosing layers of StyleGAN2 for the knowledge distillation.
LKI
LK2
CMi
LM2
Pixel Error
19.33
13.59
/
/
11.73
Layer Number
n
Pixel Error
Layer5
512
13.80
17.45
Layer7
512
11.73
Layer9
256
14.78
13.03
12.22
Figure 6: The result of image manipulation
Figure 7: The face parsing results based on DIFE
4.2 Ablation Study
The effect of loss functions. Table 3 shows an ablation study to verify the role of each loss. Pixel errors are reduced by adding each loss, and any absence loss leads the performance drop. As we cannot synthesize pseudo-paired data without the knowledge distillation of StyleGAN2, the result for the absence of LK2 not provided. The absence of L leads large performance drop of DIFE, and we find that the role of knowledge distillation of CSE is essential in the early stages of learning. The semantic matching loss also helps to increase the performance by learning more fine visual correspondence across the data domains.
Which layer in StyleGAN2 is good for the knowledge distillation? Table 4 shows the result of selecting a different layer of StyleGAN2 for acquiring the features. Fig. shows that the features from the higher layer is more fine-grained but domain-speciffe which makes the domain converter to become confused. Also, because the dimension of the domain-specific embedding follows the dimension of StyleGAN2 features nd, selecting the ninth layer limits the representation power of the domain embedding. On the other hand, the feature from the lower layer is easy for the knowledge distillation with the abundant representation power with large nd, but the information about the face is limited because of its coarseness. With this ablation study, we selected the seventh layer of StyleGAN2 to learn face prior of each data domain.
4.3 Applications
Image manipulation. Fig. 6 presents image manipulation examples based on DIFE. Putting objects like glasses and masks requires the positions of face parts, and the result shows that DIFE finds exact locations to put the objects onto the images, even on animal faces.
Interspecies face parsing Fig. shows the results of interspecies face parsing based on DIFE. Following the segmentation in style [29]. we use k-means clustering by DIFE for unsupervised face parsing. The eye, nose, mouth, and hairy parts are discovered with simple method which means DIFE has proper semantic information for the interspecies face.
Dense keypoint transfer. Although DIFE extracts embeddings from all pixels, we cannot accu- rately evaluate all visual correspondences due to the lack of annotations. However, we can figure out more dense visual correspondence qualitatively by transferring generated grid keypoints... In Fig. g

---

Origin
Intra
Inter
Figure 8: The result of dense keypoint transfer
Figure 9: The examples of pseudo pair data
the result of transferring dense keypoints from human source another human, dog, cat, and wild animal targets can be found.
Pseudo-paired data synthesis. Fig. shows examples of sheudo-paired data with various data
domains and postures. Given original images, we perform pseudo-pant paiied data synthesis on the intra-domain and the inter-domain. We that the generated paired images in similar poses with the original ones, while having minor texture variations.
Conclusion
In this paper, we introduced study for extracting common features from faces of several species, including humans, as dense embedding. In this process, we presented a mechanism for extracting and learning desired information from separate teacher models, even though the teacher models have different architectures and data domains. Furthermore, we proposed a new data augmentation technique with latent space exploration of StyleGAN2 and the semantic matching loss to overcome the extreme shape difference between species. Various experiments and applications were offered for the evaluation, and our approach has been shown to be superior to other baselines. For the future research directions, we need to deal with multiple animals more than three domains efficiently, which not focused on this paper due to the absence pre-trained StyleGAN2. A research on the generalization method for the out-of-domain data is needed to address the situation where data cannot be obtained or the animal species is unknown. Our method dependent on the performance of pre-trained CSE and StyleGAN2. Lastly, there are typical face understanding failure cases like occlusion or dark illumination. As the first work to tackle the difficult problem of interspecies face embedding, more follow-up studies necessary to overcome current limitations. Nevertheless, believe that this work has set the stage for new research topic of cross-domain study for interspecies face understanding.
6 Broader Impact.
This study aims to understand the interspecies face, which can be potentially extended to various applications. For example, our method can assist to find lost pets, check the health status of farm animals through analyzing the facial expressions. Identifying wild animals for ecological experiments another expected application our method. Moreover, the proposed method can potentially be extended to the automatic motion capture, that can be used in the film production or content creation for AR/VR. We note that the studies using DIFE could potentially violate the personal privacy or animal rights. The studies extending DIFE should carefully consider the above potential misuses, and bring positive impacts to the society.
Acknowledgments
This work supported by Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) (No.2022-0-00124, Development of Artificial Intelligence Technology for Self-Improving Competency-Aware Learning Capabilities), and No. 2020-0-01361, Artificial Intelligence Graduate School Program (Yonsei University).
10

---

References
[1] Abdal, R., Qin, Y. Wonka, P.: Image2stylegan++: How to edit the embedded images? In: Proceedings of the IEEE/CVFConfert pp. 1196-88055 (2020)
Computer Vision and Pattern Recognition (CVPR).
[2] Abdal, R., Zhu, P.. Mitra, N.J., Wonka, P.: Labels4free: Unsupervised segmentation using stylegan. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 13970-13979 (2021)
[3] Abdal, R., Zhu, P., Mitra, N.J., Wonka, Styleflow: Attribute-conditionedd exploration of gyylegan-genertten images using conditional continuous normalizing flows. ACM Transactions on Graphics (TOG) 40(3), 1-21 (2021)
[4] Barsoum, E., Zhang, C., Canton Ferrer, C., Zhang, Z.: Training deep networks for facial expression recognition with crowd-sourced label distribution. In: ACM International Conference on Multimodal Interaction (ICMI) (2016)
[5] Bulat, A., Tzimiropoulos, G.: How far are from solving the 2d & 3d face alignment problem? (and dataset of 230,000 3d facial landmarks). In: Proceedings the IEEE/CVF International Conference Computer Vision (ICCV) (2017)
[6] Cao, Q.. Shen, I.,, Xie, W., Parkhi, O.M., Zisserman, Vggface2: A dataset for recognising faces across pose and age. In: IEEE International Conference on Automatic Face & Gesture Recognition (F&G), pp. 67-74, IEEE (2018)
[7] Cheng, Z., Su, J.C.,, Maji, S.: On equivariant and invariant learning of object landmark repre- sentations. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 9897-9906 (2021)
[8] Cho, S., Hong, S., Jeon, S., Lee, Y., Sohn, K., Kim, S.: Cats: Cost aggregation transformers for visual correspondence. Advances in Neural Information Processing Systems (NIPS) 34 (2021)
[9] Choi, Y., Uh, Y., Yoo, J., Ha, J.W.: Stargan v2: Diverse image synthesis for multiple domains. In: Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
[10] Güler, R.A., Neverova, N., Kokkinos, Densepose: Dense human pose estimation in the wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 7297-7306 (2018)
[11] Gupta, A., Dollar, P., Girshick, R.: Lvis: A dataset for large vocabulary instance segmentation. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR) (2019)
[12] Hansen, M.F., Smith, M.L., Smith, L.N., Salter, M.G., Baxter, E.M., Farish, M., Grieve, B.: Towards on- on-farm pig face recognition using convolutional ncural networks. Computers in Industry 98, 145-152 (2018)
[13] Härkönen, E., Hertzmann, A., Lehtinen, J., Paris, S.: Ganspace: Discovering interpretable gan controls. Advances in Neural Information Processing Systems (NIPS) 33, 9841-9850 (2020)
[14] Hummel, H.I., Pessanha, F,, Salah, A.A., van Loon, T.J., Veltkamp, R.C.: Automatic pain detection on horse and donkey faces. In: 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020). pp. 793-800. IEEE (2020)
[15] Jakab, T., Gupta, A., Bilen, H., Vedaldi, A.: Unsupervised learning of object landmarks through conditional image generation. Advances in Neural Information Processing Systems (NIPS) 31 (2018)
[16] Jakab, T., Gupta, A., Bilen, H., Vedaldi, A.: Self-supervised learning of interpretable keypoints from unlabelled videos. In: Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 8787-8797 (2020)

---

[17] Jiang, J., Ji, Y., Wang, X., Liu, Y., Wang, J. Long, M.: Regressive domain adaptation for unsupervised keypoint detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 0780-6789 (2021)
[18] Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4401-4410 (2019)
[19] Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of stylegan. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). pp. 8110-8119 (2020)
[20] Khan, M.H., McDonagh, J., Khan, S., Shahabuddin. M., Arora, A., Khan, F.S., Shao, L., Tzimiropoulos, G.: Animalweb: large-scale hierarchical dataset of annotated animal faces. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1939-6948 (2020)
[21] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. International Conference for Learning Representations (ICLR) (2015)
[22] Li, D., Yang, J., Kreis, K., Torralba, A., Fidler, S.: Semantic segmentation with generative models: Semi-supervised learning and strong out-of-domain generalization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
[23] Lin, T.Y., Maire, M., Belongie. S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Proceedings of Proceedings of European Conference on Computer Vision (ECCV) (2014)
[24] Lv, J., Shao, X., Xing, J., Cheng, C.,, Zhou, X.: A deep regression architecture with two-stage re- initialization for high performance facial landmark detection. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). pp. 3317-3326 (2017)
[25] Min, J., Lee, J., Ponce, J., Cho, M.: Spair-7lk: A large-scale benchmark for semantic corre- spondence. arXiv prepreint arXiv: 10543 (2019)
[26] Neverova, N., Novotny, D., Khalidov, V., Szafraniec, M., Labatut, P., Vedaldi, A.: Continuous surface embeddings (2020)
[27] Neverova, N., Sanakoyeu, A., Novotny, D., Labatut, P., Vedaldi, A.: Discovering relationships between object categories via universal canonical maps (2021)
[28] Newell, A., Yang, K., Deng, J.: Stacked hourglass networks for human pose estimation. In: Proceedings of European Conference on Computer Vision (ECCV). pp. (2016)
Springer
[29] Pakhomov, D.,, Hira, S., Wagle, N., Green, K.F.,, Navab, N.: Segmentation in style: Unsupervised semantic image segmentation with stylegan and clip. arXiv preprint arXiv:2107.1255 (2021)
[30] Pan, X., Dai, B., Liu, Z., Loy, C.C., Luo, P.: Do 2d gans know 3d shape? unsupervised 3d shape reconstruction from 2d image gans. In: International Conference for Learning Representations (ICLR) (2021)
[31] Peebles, W., Zhu, J.Y., Zhang, R., Torralba, A., Eiros, A., Shechtman, E.: Gan-supervised dense visual alignment. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)
[32] Rashid, M., Gu, X., Jac Lee, Y.: Interspecies knowledge transfer for facial keypoint detection. In: Proceedings of the IEEE/CV Conference Computer Vision and Pattern Recognition (CVPR). pp. 6894-6903 (2017)
[33] Sanakoyeu, A., Khalidov, V., McCarthy, M.S., Vedaldi, A., Neverova, N.: Transferring dense pose to proximal animal classes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5233-5242 (2020)
12

---

[34] Schneider, S., Taylor, G.W., Kremer, S.C.: Similarity learning networks for animal individual
re-identification- -identiiiiatioonbeyoddd beyond the capabilities of human observer. In: IEEE International Conference on Automatic Face & Gesture Recognition (F&G). pp. 44-52 (2020)
[35] Shen. Y.. Yang, C., Tang, X., Zhou, B.: Interfacegan: Interpreting the disentangled face representation learned by gans. IEFE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (2020)
[36] Shen. Y., Zhou, B.: Closed-form factorization of latent semantics in gans. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1532-1540 (2021)
[37] Shoshan, A., Bhonker, N., Kviatkovsky, I., Medioni, G.: Gan-control: Explicitly controllable gans. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 14093 (2021)
(38] Sun, Y., Wang. X., Tang. X.: Deep convolutional network cascade for facial point detection. In: Proceedings of the TEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3483 (2013)
[39] Thewlis, J., Albanie, S., Bilen, H., Vedaldi, A.: Unsupervised learning of landmarks by descriptor vector exchange. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 6361-6371 (2019)
[40] Thewlis, J., Bilen, H., Vedaldi, A.: Unsupervised learning of object landmarks by factorized spatial embeddings. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 5916-5925 (2017)
[41] Viazovetskyi, Y., Ivashkin, V., Kashin, E.: Stylegan2 distillation for feed-forward image manipulation. In: Proceedings of European Conference on Computer Vision (ECCV). pp. 170-186. Springer (2020)
[42] Wu, W., Qian, C., Yang, S., Wang. Q., Cai, Y., Zhou, Q.: Look at boundary: A boundary-awan face alignment algorithm. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2129-2138 (2018)
[43] Xiao, S., Feng, J., Xing, J., Lai, H., Yan, S., Kassim, A.: Robust facial landmark detection via recurrent ditentive-refinement networks. In: Proceedings of Proceedings of European Conference on Computer Vision (ECCV), pp. 57-72. Springer (2016)
[44] Yang, H., Zhang, R., Robinson, P:: Human and sheep facial landmarks localisation by triplet interpolated features. In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV). pp. 1-8. IEEE (2016)
[45] Yu, H., Xu, Y., Zhang, J., Zhao, W., Guan, Z., Tao, D.: 10-10k: A benchmark for animal pose estimation in the wild. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NIPS Datasets and Benchmarks) (2021)
[46] Zhang, J., Shan, S., Kan, M., Chen, X.:
Coarse-to-fine auto-encoder networks (cfan) for real- time face alignment. In: Proceedings of European Conference on Computer Vision (ECCV). pp. 1-16. Springer (2014)
[47] Zhang, K., Zhang, Z., Li, Z., Qiao, Y.: Joint face detection and alignment using multitask cascaded convolutional networks. IEBE Signal Processing Letters 23(10), 1499-1503 (2016)
[48] Zhang, Y., Guo, Y., Jin, Y., Luo, Y., He, Z., Lee, H.: Unsupervised discovery of object landmarks as structural representations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2694-2703 (2018)
[49] Zhang, Z., Luo, P., Loy, C.C., Tang. X.: Facial landmark detection by deep multi-task learning. In: Proceedings of European Conference on Computer Vision (ECCV). pp. 94-108. Springer (2014)
13

---

[50] Zhang, Z., Luo, P., Loy, C.C., Tang, X.: Learning deep representation for face alignment with auxiliary attributes. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (2015)
[51] Zhang. Z., Song, Y. Qi, H.: Age progression/regrssion by conditional adversarial autoencoder. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5810-5818 (2017)
14


Reviews:
[{"review_id": "nyMveVtmfjJ", "reviewer": "Reviewer_wX8P", "summary": "The paper presents a new research topic “interspecies face embedding”, which aims to extract common features from faces of several species, including humans, as a dense embedding. The process enables the discovery of other species' facial semantics without enough annotations by transferring knowledge from well-annotated human data. Algorithmically, a multi-teacher knowledge distillation paradigm is introduced to guide unsupervised embedding learning. Experimentally, based on the learned interspecies face embedding, the authors perform interspecies facial keypoint transfer on MAFL and AP-10K datasets.", "questions": "Please see the weaknesses.", "limitations": "Please see the weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n+ The problem addressed in the paper is interesting.\n+ The proposed method of using multi-teacher knowledge distillation for interspecies face embedding learning is technically sound.\n+ The paper is well-organized. \n\n\nWeaknesses:\nHere are some concerns:\n1. It’s not clear why the species-specific StyleGAN2 models share geometric consistencies in the learned W latent space.\n\ni) In the pseudo-paired data synthesis stage, it is unclear to me why the pseudo-paired data x^d’ generated from the found latent code should have the same face geometry with the original image x^d (L178-179). In my understanding, there are not necessarily correlations in the learned species-specific W latent spaces. \n\nii) Is the domain converter species-specific? Did you train separate StyleGAN2 models for different species? Or one universal model for all the animals in AFHQ.\n\niii) Have you considered training a single StyleGAN2 model by combining FFHQ and AFHQ? Would this make it easier to learn interspecies face embedding? Then the domain converter is not necessary.\n\n2.\tThe experiment section is quite weak.\n\ni) Qualitative and quantitative results on more keypoints are desirable. The authors only perform keypoint transfer on 3 landmarks. I suggest considering more semantic landmarks, i.e., corners of the mouse etc.\n\nii) The authors should include more baselines. The current baselines CSE and DVE are not designed for the face. The comparison is thus not convincing to me. I understand there is no literature addressing the same problem. But I think the authors could compare with visual semantic correspondence methods. Although these methods deal with intra-class semantic correspondence, the intra-class deformation of some categories (i.e., car) is more challenging than this interspecies face deformation. \n\niii) I suggest the authors could add one experiment on interspecies face parsing, which can clearly demonstrate the effectiveness of the proposed method.\n\niv) More visual results are needed.\n\n3. It’s unclear to me why the pre-trained CSE model is relevant to the face embedding learning and can improve the keypoint transfer performance (based on the results in Tab.2). In my understanding, the CSE model provides Interspecies body priors.\n\n4. Discussions on limitations are needed. Such as, the proposed method may need to train species-specific StyleGAN2 models and domain converters.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "n3mlSG42241", "reviewer": "Reviewer_pN5V", "summary": "This paper presents a face embedding method to obtain shared semantics between different animal species. The basic idea is to use a knowledge distillation paradigm to extract information from face synthesis (StyleGAN2) and interspecies surface embedding (CSE) models. Specifically, the main encoder is trained so that 1) the embedding is close to the one from CSE and 2) the domain-specific embeddings through converters are close to the features of StyleGAN2. The model also synthesizes pseudo-paired face images using the SyleGAN2 generator to enforce facial semantics correspondence further. ", "questions": "- As commented above, I would like a more detailed discussion of the training data and pre-training model that this method requires. For example, what images and annotations would be needed to target animals not included in AP-10K or AFHQ?\n- Likewise, it would be good to see how much the performance of the pre-training model and the amount of data used for training affects the final performance.\n- This is not necessarily a weakness, but it is interesting to note that the eyes are emphasized in the generated images seen in Figure 7 and elsewhere. It would be good to have a discussion on what reason this artifact occurs.", "limitations": "- The above issue of generalization to other animal categories is also mentioned in the text as a limitation.\n- As the method deals with faces, sufficient consideration must be given to social impact, which is also mentioned in the text.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The target task is interesting, and the proposed approach is reasonable to address this goal.\n- The performance of the proposed method has been qualitatively and quantitatively verified from various angles, and the improvement from the baseline method is acknowledged.\n\n## Weaknesses\n- The proposed method takes the knowledge distillation approach and requires the pre-trained StyleGAN2 and CSE models for training. Compared to previous studies such as DVE [36], it can be seen that the data and annotations required for learning have increased. The performance of the final embedding is expected to be highly dependent on the performance of these two pre-trained models.\n- Related to the above, the main contribution of the proposed method is the combination of these two models, and the pseudo-paired data synthesis is not that significant in terms of technical novelty.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Privacy and Security (e.g., consent)"]}, "extra_text": {}}, {"review_id": "VQstlncmXJv", "reviewer": "Reviewer_5viz", "summary": "This paper presents an \"Interspecies face understanding\" task to predict unified spacial features of both human and animal faces.\nTo solve this problem, it presents a multi-teacher knowledge distillation framework that combines the advantages of different models, i.e. CSE for cross-domain features and StyleGAN2 for facial embedding.\nIt also explores the latent space of StyleGAN2 to synthesise paired data for semantic matching.", "questions": "1. Why not compare on the common face alignment datasets with commonly used metrics?\n2. Can this method handle landmarks in mouth and face contour?\n3. Are the CSE results in Table 1 actually DIFE trained with L_{K1}? If not, is there any illustration for the similarity between Table 2 and \"Human+Dog\" column in Table 1?\n4. How to guarantee the spatial consistency between the input and output of the Domain Converter? How is this Domain Converter trained?\n5. Only synthesised images are used to train the Encoder. If testing on a synthesised dataset, how is the performance? Do you have any experience with this?", "limitations": "The authors have discussed the broad impact and potential negative social impact of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper explores the interspecies facial landmark detection model, which has many potential applications.\n2. It is interesting to me to combine two orthogonal models to obtain the cross-domain facial embedding.\n\nWeaknesses:\n1. I think the \"Interspecies Keypoint Transfer\" experiment is not sufficient for the following points:\\\n  a) Dataset. I think 300W [1] or WFLW [2] for human facial landmarks, and [3] for animal facial landmarks, would be helpful to show the performance of this paper;\\\n  b) Point Number. Only 3 points are presented as the qualitative results. Can this method handle landmarks in mouth and face contour?\\\n  c) Comparison. If using the datasets mentioned in 1. a), there can be a comparison with previous supervised/unsupervised methods, which help readers better understand the performance of this paper. The current comparison looks isolated.\\\n  d) Ablation Study. The ablation study results are partly the same as the \"Human+Dog\" result in Table 1. So the CSE results in Table 1 are actually DIFE trained with L_{K1}? If so, I think it is improper to call it \"CSE\" here.\n2. There is no constraint to enforce the spatial consistency of the input and output of the Domain Converter.\n3. No real images are used when training the Encoder.\n\nReference\n\n[1] Sagonas, Christos, et al. \"300 faces in-the-wild challenge: The first facial landmark localization challenge.\" Proceedings of the IEEE international conference on computer vision workshops. 2013.\n\n[2] Wu, Wayne, et al. \"Look at boundary: A boundary-aware face alignment algorithm.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.\n\n[3] Khan, Muhammad Haris, et al. \"Animalweb: A large-scale hierarchical dataset of annotated animal faces.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0K8kgcfEz9i", "reviewer": "Reviewer_tYdV", "summary": "This paper addresses the problem of dense interspecies face embedding. To cope with the lack of data, first multi-teacher knowledge distillation of Continuous Surface Embedding (CSE) and StyleGAN is used. Then, pseudo pair images are synthesized through the latent space exploration of StyleGAN2 to find implicit associations between different animal faces. Finally, a semantic matching loss is introduces to overcome the problem of extreme shape differences between species. Interspecies facial keypoint transfer is performed on MAFL and AP-10K. Results for applications in interspecies face image manipulation and dense keypoint transfer are also provided.\n", "questions": "- Authors should convince more about the effective applicability of the method. In particular, I think it would be interesting to see a comparison with a solution that learns face keypoints from animal images rather than transferring them from face images. This would support more the method and is something missing in the paper. \n", "limitations": "Authors reported about the impact of the work but in a too general way about limitations. I think they should have indicated more specifically the limitations possibly also providing failure cases.\n ", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\nThe main contributions of the paper are:\n- A cross-domain face understanding study is presented that considers human faces\nas well as faces of animals. The approach domain adaptation to avoid tedious and expensive animal data collection.\n- A multi-teacher knowledge distillation paradigm is proposed that extracts and combines the information from models with different architectures and data domains. The proposed framework learns continuous face embedding across interspecies data from CSE and StyleGAN2. With this solution dense interspecies face embedding (DIFE) are learnt.\n- A method for synthesizing paired data is proposed for learning the semantic matching using the latent space exploration of StyleGAN2.\n\nWeaknesses\nThe contribution of the paper is limited. It is more on the application side but it fails to convince about the effective transfer between human an animal faces and application domains.\nIn particular:\n- The baselines used in the comparison of Table 1 are not designed for the same task proposed in this work and have been adapted to it so the comparison is somewhat inconsistent.\n- the keypoint transfer is shown only for three keypoints (nose tip and eyes center, but the transfer on the animal face is not particularly accurate)\n- the results in figure 6 also are not particularly accurate\n- the applications are not particularly convincing\n\nIn general, the paper is well written, but there are some errors to check. See for example:\n- page 3, line 124: \"In StyleGAN2 distillation [? ],\" --> missing reference\n- page 5, line 177: \"is an latent space\" --> a latent\n\n============= POST REBUTTAL ================\nAuthors have answered to most of my questions, so I changed my score to borderline accept. However, I still think the contribution is not very strong. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept