Decompose to Generalize: Species-Generalized Animal Pose Estimation
Authors: Guangrui Li
Yifan Sun
Zongxin Yang
Yi Yang
Venue: ICLR 2023 poster (2023)
Keywords: Pose Estimation
Domain Generalization
Transfer Learning

Abstract: This paper challenges the cross-species generalization problem for animal pose estimation, aiming to learn a pose estimator that can be well generalized to novel species. We find the relation between different joints is important with two-fold impact: 1) on the one hand, some relation is consistent across all the species and may help two joints mutually confirm each other, e.g., the eyes help confirm the nose and vice versa because they are close in all species. 2) on the other hand, some relation is inconsistent for different species due to the species variation and may bring severe distraction rather than benefit. With these two insights, we propose a Decompose-to-Generalize (D-Gen) pose estimation method to break the inconsistent relations while preserving the consistent ones. Specifically, D-Gen first decomposes the body joints into several joint concepts so that each concept contains multiple closely-related joints. Given these joint concepts, D-Gen 1) promotes the interaction between intra-concept joints to enhance their reliable mutual confirmation, and 2) suppresses the interaction between inter-concept joints to prohibit their mutual distraction.  Importantly, we explore various decomposition approaches, i.e., heuristic, geometric and attention-based approaches. Experimental results show that all these decomposition manners yield reasonable joint concepts and substantially improve cross-species generalization (and the attention-based approach is the best). 

Published as a conference paper at ICLR 2023
DECOMPOSE TO GENERALIZE: OPECIES-GENERALIZER ANIMAL POSE ESTIMATION
Guangrui Li
Yifan Sun?, Zongxin Yang", Yi Yang"
ReLERR, AAII, University of Technology Sydney. Baidd Inc.
3 CCAI, College of Computer Science and Technology, Zhejiang University. guangrui 1i Li@outlook com, sunyf150 s singhua. org.en baangoongxin,
edu. on
ABSTRACT
This paper challenges the cross-species generalization problem for animal pose estimation, aiming to learn a pose estimator that can be well generalized to novel species. We find the relation between different joints important with two-fold impact: 1) on the one hand, some relation is consistent across all the species and may help two joints mutually confirm each other, egg. the eyes help confirm the nose and vice versa because they are close in all species. 2) the other hand, some relation inconsistent for different species due to the species variation and may bring severe distraction rather than benefit. With these two insights, we pro- pose a Decompose-to- -oe--t-aieeeeeeeeeeeeeee (D-Gen) pose estimation method to break the inconsistent relations while preserving the consistent ones. Specifically, D-Gen first decomposes the body joints into several joint concepts so that each concept contains multiple closely-related joints. Given these joint concepts, D D-Gen 1) pro- motes the interaction between intra-concept joints to enhance their reliable mutual confirmation, and 2) suppresses the interaction between inter-concept joints prohibit their mutual distraction. Importantly, we explore various decomposition approaches. i.e., heuristic, geometric and diention-based approaches. Experimen- tal results show that all these decomposition manners yield reasonable joint con- cepts and substantially improve cross pross-species generalization (and the attention- based approach is the best).
INTRODUCTION
Animal pose estimation (Cao al. 2019 Li & Lee 2021b Mu al. 2020 Mathis et al. 2021) aims to identify and localize the anatomical joints animal bodies, and has received increasing attention for its wide application, i.e., biology, zoology, and aquaculture. animal pose estimation is the cross-species problem, i.e., using the already-learned pose estimator for novel species. Specifically, it is infeasible collect and annotate all the animal species, because the animal kingdom is a vast group of millions of different species. Under this background, the cross-species generalization is of great value for realistic applications.
critical challenge in realistic
This paper tackles the cross-species animal pose estimation from the domain--enneraizzatoon (DG) viewpoint (i.e., species is respective domain) and reveals a unique factor for this cross-species generalization, i.e., the relation between different joints. The joint relation in our view can be visual (e.g., the color relation between neighboring joints), structural (e.g., .. the nose under the eyes) and many more. Our focus on the joint relation is different from the popular concern in general domain generalization (Ben-David et al.) 2010 Blanchard et al., 2021 201T: David et al. 2010), which mainly considers the distribution shift between the source and the unseen target domain(s). The generic DG methods usually learn domain-invarian representations (Muandet et al. 2013 Ghifary
al. 2015 Li et al. 2018b Bui et al. 2021 Yang et al. 2021 Gong al. 2021) ) enhance the generalizability through meta-lleaminn (Dou et al. [019 Balaji et al. [018]] Li et al. 2018a; 2019)
or data augmentation (Zhou al. 2021 Shankar et al. 2018 Carlucci al. 2019 Volpi et al.
WWrk done during internship Baidu.

---

Published as a conference paper at ICLR 2023
Figure 1: Two reasons that bring domain gap to the joint relation. 1) structural discrepancy: the part lengths (i.e. the distances between different joints) may vary for different species (left most); 2) the visual similarities between some different joints are inconsistent for different species, e.g., the visual similarities between faces and other body parts are different for tiger, fox, and the cow.
2018). While these methods are potential for cross-species generalization as well, the joint relation a unique viewpoint that has never been explored under other DG scenarios.
The importance of joint relation is two-fold: 1) on the one hand, some joint relation is consistent across all the species and beneficial. With consistent relation, two joints may mutually confirm each other, e.g., the eye helps confirm the nose and vice versa, because they consistently close in all species. 2) on the other hand, some joint relation is inconsistent for different species due to species variation and is thus harmful for generalization, e.g. the length of non-rigid body parts such as legs. Such inconsistent relation makes the already-learned mutual confirmation become a severe distraction rather than any benefit. We note that the latter (negative) impact has more less been recognized by some earlier literature (Cao et al. 2019), while the former (positive) impact was neglected. In contrast Cao et al. (2019), we argue that both two factors are important and should be considered in combination.
With these two insights, we propose Decompose-to-Gennneee Geneerlllee (D-Gen) pose estimation method to break the inconsistent relations while preserving the consistent ones. Specifically, D-Gen first decomposes the body joints into several joint concepts. The decomposition facilitates that each individual concept contains multiple closely-related joints and that the joints in different concepts are far away or prone inconsistent relations. Given these joint concepts, D-Gen promotes the interaction between intra-concept joints and meanwhile suppresses the interaction between inter- concept joints. The approach for interaction promotion suppression very simple: D-Gen splits the top layers of the backbone network into several pose-estimation branches, each of which is responsible for a corresponding joint concept. Intuitively, the joints in different branches have less interaction, compared to the joints in the same branch. Consequently, D-Gen suppresses the distraction from inconsistent joint relation and yet preserves the beneficial mutual confirmation of consistent joint relation, thus improving cross-species generalization.
We explore three strategies for joint decomposition, iee, heuristic, geometric and the attention- based manner. The geometric manner clusters the joints based their geometric distances. The attention- manner uses the attention mechanism to learn the affinity between joint features and uses the affinity matrix for decomposition. Experimental results show that all these three strategies substantially improve cross-species generalization, validating the effectiveness of our joint decom- position. Another interesting observation is that the iirention-baseen strategy surpasses the other two strategies, indicating that attention-based concepts are better than the concept derived from human intuition (i.e., heuristic) and the pure geometric relation. Since the attention-basee based approach com- bines deep feature and geometric priors, its superiority against the geometric manner suggests that there are multiple forms of joint relation beyond the structural relation.
RELATIVE WORKS
Our work is closely related to two research areas, i.e., pose estimation and domain generalization.
2D pose estimation for human and animals. 2D pose estimation refers to identifying all the anatomical joints of bodies for images. There are mainly two paradigms, top-down and bottom-up.
The top-down paradigm (Huang et al., 2017 Papandreou et al. 2017 Zhang et al.) 2020 Cai al.)
2020 Newell et al.) 2016: Moon et al. 2019 Khirodkar et al.] (021) fist detects the person and then
localize the joints for each detected person. The bottom-up paradigm (Sun et al. 2019a Wang al.
2

---

Published as a conference paper at ICLR 2023
2019 Geng et al. 2021 Brasó et al. 2021: Jin et al. 2020: Wang et al. 2021 Tang & Wu 2019 Cao et al. 2017 7 Kreiss et al. 2019; Newell et al. 2017) first identifies joints in an ddlentity-gnnsseee manner, then derive poses with different grouping strategies. The top-down solutions accurate while requiring higher computation due to the extra detection process, while the bottom-up solutions are more popular for their lower computation cost. Here adopt the bottom-up solution for its efficiency and simplicity.
more
In terms of single-species pose estimation, Tang & Wu (2019) performs joint decomposition to strengthen the part with close correlations. However, they only employ the geometric clue inside single species while neglecting the structure variations across species and other forms of joint rela- tions (e.g., visual). Empirical experiments prove it is sub-optimal for the cross-species scenario.
Domain adaptation and domain generalization. Domain adaptation aims transfer the knowl- edge learned from a labeled source domain to an unlabeled target domain, which has attained re- markable progress in the past decades. Researches in domain adaptation overcome the distribution shift via alignment in the feature space directly (Long al. 2015 Sun & Saenko 2016 2 et al.)
2014), or employing adversarial training in the input space (Hoftman et a2.110188 Gong et al. 2019
Li et al., 2020) or feature space (Luo et al. 2017: Chen et al. 2019 Ganin & Lempitsky 2015
ct al. 202 Hu et al. 2022 2023).
Domain generalization extends to more universal scenario, where multiple source domains and target domains are available and only source domains are accessible during the training. Researches in this area can be categorized into streams: stream seeks to derive domain-invariin
repre-
sentations (Zhao et al., 2020; Muandet et al., 2013 Ghifary et al.) 2015 Li et al., 2018b c Huang
ct al. 2020 Seo al. 2020] et al.. Kim al. [2021]. 1), and the other attempts to enhance
the generalizability of trained model parameters Wu & Gongl (2021 1); Iwasawa & Matsuo (2021) via
meta-learning (Dou al., 2019 Li al. 2018a| Zhang et al. 202 b). or data augmentations (Zhou
et al. 2021 Carlucci et al.
2019 Li et al. 2022 Chen et al. 2021 Zhou et al. 2020).
In terms of animal pose estimation, there are already some works considering the domain adaptation problem for it. Mu et al. 2020) and [i & Lee (2021a) focus on the synthetic-to-real domain adapta- tion inside the same species. Despite making progress, these methods typically mitigate the domain gap induced by the difference in appearance and background, while the structural discrepancies in- side the same species are negligible and are not considered as the main obstacle. Cao et al. (2019) made the first attempt to mitigate the structural discrepancies between different species and realize it with an adversarial training paradigm and pscudo labeling. However, we argue it is sub-optimal because their work mainly alleviates the impact resulting from the parts that exhibit large variation across species, while ignoring the parts holding stable inter-joint relationships.
METHOD
3.1 OVERVIEW
Preliminaries. From the perspective of domain generalization (DG), each animal species in pose es- timation is within an independent domain. Let us assume the training set D contains A species (do- mains), i.e., D = (D1. DM }. forming a joint sample space A' x y (A' is the image space and y is the corresponding label space). Specifically, the i-th species (domain) D; contains N, samples, i.e., D. = {(x My > The goal of cross-species pose estimation is to train a pose estimator on D = D2. D2...... Dar) and then directly apply it to novel unseen species Du(u & 21, .... Mf), which requires good cross-domain generalizability. The trained deep model consists of feature extractor F and an estimator head G, which are parameterized by df and ΦG. respectively.
The framework of the proposed D-Gen is illustrated in Fig. The key motivation that some joint relations are consistent across all the species and are thus beneficial for cross-species gener- alization, while some other joint relations are inconsistent and harmful. Therefore, D-Gen seeks to break the inconsistent relations while preserving the consistent ones. To this end, D-Gen consists of two stages, i.e., joints decomposition (Section 3.2) and network split (Section (33). In the joint decomposition stage, D-Gen divides the body joints into several joint concepts that each cept contains multiple closely-related joints. Section 3.2 introduces three different decomposition strategies, i.e., heuristic, geometric and attention-basee decomposition. Afterward, Section 3.3 pro-
3

---

Published as a conference paper at ICLR 2023
Network Split
Joint Dasompodiioo
botion Japer
Of
Drenge
Gesmetry-bssed
Mess yfo
Attentombbssed
Inference
Figure 2: Overview of "Decompose Generalize" (D-Gen) scheme. D-GGn consists of two stages, Le., joints decomposition (left) and the sub-sequential network split (right). 1) In the joints decompo- sition stage, D-Gen leverages different strategies (e.g. heuristic, geometry-based to divide the body joints into several joint concepts, so that each concept contains closely-related joints (Section 3.2 2) Given the decomposed joint concepts, D-Gen correspondingly splits the top layers of the baseline network into multiple concept-specific branches (Section (.3). This network split suppresses the interaction between inter-concept joints and yet preserves the interaction within each concept. 3) During inference, D-Gen concatenates the predictions of all the concept-specific branches (step i) and averages them with the baseline prediction (step ii).
attention- -aaso))))
poses very simple network split method that splits the top layers of the backbone network into several concept-specific branches, corresponding to the decomposed joint concepts. Importantly, Sec. 3.3 also investigates the mechanism through gradients analysis and reveals that the network split suppresses the gradient conflict among inconsistently-eeltte joints (i.e., different concepts).
3.2 JOINTS DECOMPOSITION
We employ three different concept decomposition strategies, iee,, the heuristic, geometry-based, and the prention-bascee strategy. The attention-basee based decomposition is learned based on the bascline network (i.e., we insert an attention module between the feature extractor and the estimation head in the baseline, in Fig. 3). The heuristic decomposition is drawn from the anatomy knowledge, and the geometry-base -aase decomposition learned through the geometric distance. Empirically, we find the latter two decomposition strategies are indeed inferior to the attention-based strategy. However, we think the corresponding exploration is valuable for showing that joint decomposition can bring general benefit cross-species generalization (regardless of the decomposition strategy).
3.2. ATTENTION-BASED DECOMPOSITION
We first learn a pixel-to-concept attention Fig. 3 (a) and then use the attention results to infer the decomposition 3 (b)).
Learning pixel-to-concept attention. As shown in Fig. (a), append attention module after the feature extractor to discover the correlation between different joint features. The attention module takes the feature maps Z Rooxd (h and channels) as its input. Meanwhile, we pre-define that there are & concepts (k is soper-parameter and correspondingly provide k concept embeddings, i.e., F. Rxd for learning pixel-to-concept attention. Given the feature maps Z and the concept embeddings F, the attention module uses three linear projection get the qu, Key and Va ve, respectively, which is formulated as:
the spatial size of the feature map with d
Query ZW Key EW2, Value EW.,
where Wg/Wa/ WW.WW, E dxd linear layers that project the inputs into the identical low dimension space. We note that the concept embeddings and the three linear projections are all learnable through the attention mechanism.
The pixel-to co-cccpt attention is defined as:
ZW ZWWEEWA (EW.)),
2 = Z + W Wmmsootmant
(2)
Vdh
4

---

Published as a conference paper at ICLR 2023
Joint Features
My
Query
Lake
Map
Festure Map
Arginox
Key
Enchskking
Esination Head
Cuscape Embeokiing
(a) The proposed attection module
(b) Concept generation with the learned attention
Figure 3: Attention-based decomposition. Left: Illustration of the proposed attention mechanism. The attention module takes the feature map as query, and learnable concept embedding as the key for learning the affinity matrix, which is leveraged to promote the inter-joints interactions. Right: Illustration of the concept generation. For the learned feature map, we first extract the joint features based on the predicted joint location. Then with simple nearest neighbor search with concept embedding, we could obtain the concept label for each joint. The final division of concepts is obtained from the voting from all training images. X denotes the matrix multiplication and + denotes the element- wise sum.
where Wm, is linear layer that maps the dimension of features back to d, and the residual connection added for training stability.
The above attention makes each pixel-Ievel features absorb information from the shared set of con- cept embeddings. The resulting feature maps Z are then fed into the estimation head. We note that while intention is use such pixel-to-concept attention the clues for joint decomposi- tion, the attention actually has another positive side-effect: already promotes interaction among closely-related joints in an implicit manner. It is because the closely-related joints are likely to absorb information from the same concept embedding(s), thus gaining implicit interactions.
Decomposition by comparing pixel-wise features to concept embedding. After the model fin- ishes learning the pixel-to-concept attention, we use the already-learned k concept embeddings illustrated in Fig. (b). Specifically, given image
E &xd to infer the decomposition, and the predicted position of each visible joint, we extract the corresponding joint features from the feature maps Z. Let ăςi, m) denotes the extracted feature of the m-th joint from the i-th image. According to the cosine similarity between âçú, m) and the concept embeddings = {e,}/=m the concept-of-ine ootie-ere--ttttttteeeee of the m-th joint in the i-th image is inferred by:
Ci.m arg max cos < 2i >, j 2....
where the <, > denotes the operation of getting the angle of two vectors.
Finally, the concept-of-intereed for the m-th joint is voted from all the training samples.
3.2.2 HEURISTIC AND GEOMETRY-BASEE DECOMPOSITION.
In addition to the attention-basee decomposition, we explore two alternatives, i.e., the heuristic and geometry-based decomposition.
Heuristic decomposition. From the perspective of human knowledge, the joints could be sim- ply divided according to anatomy. Therefore. the heuristic strategy draws the knowledge from the human prior, and decomposes the body joints into three parts, i.e., head, hind leg, and foreleg. Intu- itively, the joints within each single part have relatively consistent mutual relations. That being said, experimental results (Sec. (.2) show that the improvement achieved by heuristic decomposition is relatively small, indicating that relying on intuition is sub-optimal choice.
Geometry- based decomposition. considers the distances between every pair of joints and can be viewed as a pure-structural strategy. The joint distances are derived from the ground-trum joint position and are used to construct employ spectral clustering (Ng et al., (200T) Shi & Malik] (2000) to cluster all the joints into multiple concepts. Some examples of the geometry- based concepts shown in Fig. 6
affinity matrix of the joints. Given the affinity matrix,
5

---

Published as a conference paper at ICLR 2023
Gradient Contict (84
Altention-baae basee
80
Geomtry-bassd
Heerisstccbesed
70
70
70
60-
60
60
40
40
30
30
30
2
Model Block Index
Model Block Index
Model Block Index
Inter-Con
Figure 4: Gradient conflict rate for three decomposition strategies. Generally, the gradient flict among intra-concept joints is lower than the conflict among inter-concept joints. Moreover, compared with the other two strategies, the attention-based decomposition achieves lower conflict for intra-concept joints, indicating better decomposition. An effective approach to circumvent the gradient conflict is isolate the optimization of severely joints into different branches.
3.3 NETWORK SPLIT
Given the joint concepts from a decomposition strategy, D-Gen correspondingly splits the top layers of the baseline network into multiple concept-specific branches, illustrated in Fig. 2 (the right part). Specifically, the top layers refer to the last Kap residual blocks in the backbone (ResNet), while the bottom layers the rest. Here, analyze the rationale for this simple network split through gradient analysis.
To be concrete, during the optimization. Assuming Jiy is the angle between the gradient of joint and joint j. the conflict rate is defined the proportion of gradients holding different directions, i.e. cos0.j 0. As shown in Fig. 4 the joints from the same concept exhibit higher consistency in the gradient space conflict
gradient conflict (Yu al. 2020) to estimate the inter-joint relationships
(i.e., lower conflict rate) while joints from different concepts show higher possibility with each other, thus hampering the optimization. Therefore, intuitive approach to circumvent the gradient conflict is to isolate the optimization severely-confite joints (i.e., different concepts) into different network branches, resulting in the proposed network split.
The overall training objective is formulated as:
== + E
+ (msse(c((r(w))).)
(4)
where d6 denotes the feature extractor with the i th concept branch and y² denotes the ground truth for the i th concept. We use LMSE() to denote the mean squared error.
During inference, first concatenate the output of concept-specifie branches and then average them with the baseline branch to obtain the final prediction. Notably, before the concatenation, we first sort them according to their original joint index the joints are sequentially divided.
The improvement NOT mainly due to model ensemble. A side effect of network split is the increase model size. More specifically, D-Gen splits the top layers of the baseline network and thus can be viewed of the ensemble of multiple concept-specific branches. However, we note that model ensemble (or increasing the model size) is not the major reason for our improvement. An explicit evidence is that: if we replace the joints decomposition strategy with random strategy (i.e., randomly dividing the joints into several concepts, as detailed in Sec. (.22, the resulting D-Gen only gains very slight improvement, i.e., +1.5% on Animal Pose Dataset (Table 3). In contrast, using attention-ban based joint decomposition, D-Gen obtains +6.5 % improvement. This observation confirms that model ensemble is only trivial reason for the improvement of D-Gen.
Discussion:
4 EXPERIMENTS
4.1 SETUP
Dataset. We evaluate our method on two large-scale animal datasets. AP-10K (Yu et al. (021)) is a large-scale benchmark for mammal animal pose estimation, and contains 23 animal families and 54

---

Published as a conference paper at ICLR 2023
Table 1: Intra-family DG results (AP) on AP-10K (Yu et al. 202T with the leave-one-out manner. King Cheetah, S.L.=Snow Leopard, and A.S. = Argali Sheep. Num number of training samples for this species.
Prisizes SOTA
"The pripored T-rGen
Farils
Terget
Nam.
ERM
Oudle
MisSyle
GreiSar
SWAD
Fii
Raraken
Henritti
Atteniin
Betwan
151
$6.4
15.4
81.1
56.1
63.0
15.3
10.1
47.9
58.5
15.3
65.3
Car
307
90.1
-910
16.1
14.1
17.8
41.8
Jagur
187
01.3
60.9
98.7
518
16.3
K.C
223
51.1
10A
02.3
55.6
633
55.5
19.0
66B
62.
675
02.1
Tuldas
Loqued
843
97.8
02-3
51.2
64.3
96.
52-7
5711
19:2
Panteer
106
41.5
4007
52.2
43.9
4K.4
S.L.
73
65.4
54.9
95.4
6211
75.7
50.3
50.0
61.2
6211
635
64.0
Tiper
844
39.9
$7.6
57.1
9933
41/4
57N
26.2
64.2
63.1
624
57443:06
8:0.1
51. 1:00
93.7
559
$7.91 1.0
$3.7
3.00
554 BJ
Black Bear
3M
30.1
43.2
$5.2
32-2
31.5
HL.5
26.8
26.5
36.1
41.1
4117
Brown bezz
EX1
13.3
17.5
25.1
28,0
24.9
Polar Hear
136
19.0
$1.3
17.6
21.2
Avrrage
19.8 0.4
0.3
0.5
19.6
20.4
15.91 1 0.9
21.0
106 0/4
24.7
$1.3
$3.4
56.2
$3.4
A.S
258
69.7
78.1
95.6
71.3
14.1
57.5
15.1
72.65
23.6
74.3
17.3
Biben
208
42.3
$1.6
91.8
-0.8
43.6
46.2
4723
46.1
48.5
45.8
49.5
Bosilac Bulisko
228
61.0
64.8
62.3
61.1
Steep
355
41.7
$7.4
95.0
41.0
43.1
46.1
41.5
40.5
41.0
41.2
40.1
Ansrage
0.
99.1
341.4
346.0
34.6 0:04
species following the taxonomic rank. It annotates 10,015 images and provides annotations with 17 keypoints. Animal Pose Dataset (Cao et al., 2019) collects and annotates species, i.e., cat, dog, sheep, cow, and horse, from 3000+ images, where both bounding boxes and keypoints annotated (20 keypoints) are provided. Animal Kingdom (Ng et al. 2022) is another dataset with a higher diversity that includes mammals, fishes, birds, amphibians, and reptiles with 23 keypoint annotations.
Evaluation. Follow Sun et al. 2019b), the evaluation metric is based on Object Keypoint Similarity (OKS). We report the mean average precision (AP) at OKS=0.50, 0.55,..0.90,, 0.95. The mean value and standard error over three random runs are reported in all main results.
Implementation details. We start from ResNet50 (He et al. 2016) with the backbone pretrained on ImageNet (Deng et al. 2009).. The batch size is set to 64, and the learning rate for the first stage and the second is set to 5 x 10 and 5 x10- respectively. We optimize the model with Adam for 210 epochs, where the learned rate decrease (x 10 at 170 and 200, respectively. The size of the input image is 256 x 256 and the heatmap is with size 64 x 64. The number of the concept-specific blocks is set to 2, and l is set to 3 for all transfer tasks.
4.2
MAIN RESULTS.
We evaluate approach under two domain generalization (DG) scenarios, i.e., intra-family DG and fnter-family DG, where the latter involves larger domain gaps regarding both the visual and structural knowledge. Following the common practice, we perform domain generalization with the leave-one-out setting, í.e., selecting one species / family as the target domain and the rest as the source domains. Since these two datasets are relatively new and only few results have been reported, we re-implement representative DG methods. i.e., Fish (Shi et al.. 2022). SWAD (Chal ltt al.) 202T). MixStyle (Zhou al., 202T), and GradSur (Mansilla et al. 2021) for comparison and we adopt ERM (empirical risk minimization that trains on all source domains). Oracle (training on target domains), and Oracle-Full (train on both source and target domains) as baselines. Moreover, use random decomposition in addition to the three discussed decomposition strategies to show that model ensemble (through random-decomposed branches) does not promise improvement.
Intra-family DG. For intra-family domain generalization, we evaluate our approach on the three largest families in AP-10K, i.e., Bovidae, Canidac, and Ursidae. The results are summarized in Table I from which we draw three observations.
1) Comparing "Previous SOTA" against the "ERM" baseline, we find the improvement achieved by previous state-of-the-ar DG methods is very trivial (if there is any). We infer it is because these methods mainly focus on the style gap of the source and target domain, and are not capable to tackle the distribution shift in terms of structural and visual relations.
2) Comparing the proposed D-Gen against the "ERM" baseline, we observe that all the three decom- position strategies (heuristic, geometry- baase, and attention-based) achieve consistent improvement.
7

---

Published as a conference paper at ICLR 2023
Table 2: Inter-family DG results (AP) on AP-10K (Yu et al. Cerc.= Cercoppcrcpi...cciaa...
2021
with the leave-one-out manner.
Previou
proposed D-Gei
Tages
Nums
ERM
Onali
Onall-F
MisSyylc
GralSar
SWAD
Fish
Ranalen
Heuristic
Geunetry
Attention
Bosidxe
1467
45.2
64.5
90.3
49.5
47.0
45.8
69.3
45.6
46,0
19.9
53.1
Felidee
1457
39.7
64.4
87.3
43.1
34.2
-2.4
99.
43.0
42.8
41.9
47.7
Condie
1130
50.2
01.9
85.8
48.2
46.3
53.6
50.6
533
49.7
5339
5488
Unstre
10.3
08.3
X1.7
51.3
38.5
11.1
11.5
28.4
11.1
23.0
Equibee
04
661
85.7
1K4
35.
19
39.53.05
0M
42. 0066
Table 3: Results (AP) on Animal Pose Dataset (Cao et al. 2019) with the leave-one-out manner.
Previous SOTA
The proposed D.Gen
Target
Num.
BRM
Oracle
MinStyle
GradSur
SWAD
Fish
Random
Ileuristic
Geometry
Ameniion
29.6
Sheap
22.0
28.4
28.2
Averae
25.4::044
249
For example, within the Bovidae family, these three methods surpass the ERM baseline by +2.3%, +3.2% and +5.2%, respectively. This observation validates the effectiveness of D-Gen.
3) Comparing the three decomposition strategies against each other and the additional "Random" strategy, observe that the attention-based strategy is the best among all. It indicates that the deep network is capable to discover better joint concepts than human intuition and pure-geometric knowledge. Another important observation is that the "Random" strategy barely achieves any im- provement over the baseline. It indicates that D-Gen barely benefits from the model ensemble, because the random strategy already takes the advantage of model ensemble.
4) Under circumstances with fewer labeled samples, domain generalization can serve as powerful baseline for pose estimation. For example, our method surpasses the Oracle with 4.8 % on the species Bobcat from the family Felidae, where only 150 images are provided with annotations.
Inter-family DG. We further evaluate the pro- posed D-Gen under the inter-family DG sce- nario. Without loss of generality, we select several animal families with relatively large di- versity for each dataset (Table 2 for AP-10K and Table 3 for Animal Pose Dataset). We observe our method maintains its superiority against previous methods, which is consistent with the observation under the intra-family sce- nario. Moreover,
Table 4: Results (PCK @0.05) dom Dataset (Ng et al. 2022) with the leave-one-
Animal King-
manner. AM Amphimian
Family
Mammals
AM
Reptiles
Birds
Fishes
Avg.
ERM
11.5
15.0
11.4
13.2
12.4
12.7
Oracle
37.3
69.1
66.5
52.0
45.5
54.1
SWAD
13.4
18.3
15.5
17.4
12.6
15.4
17.9
17.7
17.4
Heur.
16.6
16.9
18.1
18.8
15.1
17.1
observe that under the fater-family scenario, the achieved results are lower than the intra-family scenario. It con- finms our intuition that inter-family generalization is more challenging due to larger cross-species distribution shifts.
Geom.
17.1
18.5
18.9
19.1
16.1
17.9
Att.
18.3
19.0
19.4
19.9
18.8
19.1
4.3 ABLATION STUDIES
In this section, we conduct ablation studies under two scenarios, ie., intra-family DG (within the Bovidae family, in particular) and the inter-family DG, to further investigate the proposed method.
Different number of concepts (k). In 5 (a)(b,, we report the results with varying A to evaluate its influence. With increasing, these decomposition solutions show different tendencies, i.e., the attention-based variant only fluctuates in small range, while the other two show degraded perfor- mance. Such a phenomenon verifies that the attention- based solution not sensitive to the selection of k. Further investigation reveals that, the & increased, the concept embedding still derives cept groups with stable number, i.e., from to 5. The reason is that, when is large, only part of the concept embeddings attend and modulate the inter-joint interactions while others not.
8

---

Published as a conference paper at ICLR 2023
(8)
[H 18 160
58
56
54
52
Figure 5: The impact of (the number of concepts) and hop (the number of concept-specifie blocks) with error bars. Both the intr-ffamlly and c) and the inter-family scenarios (b and are presented.
Heuristic
Geometry- based
Attention-
based
Figure Concept visualization on AP-10K, where different colors denote different mined concepts.
How does hop (number of concept-specific blocks) influence the generalization results? In Fig (c),d, we present the sensitivity analysis to the number concept-specific blocks (Lop)). As we could observe, with increasing (top. the performance improves constantly, justifying the necessity of the network split pipeline. Especially, even with the shared backbone, our method still achieves very competitive results, i.e,, 40.6 for inter-family DG, proving its superiority against previous methods. In contrast, the other two solutions only achieve very limited improvement with the increasing hop- The reason is that, illustrated in Fig. the joints of their mined concepts still exhibit high conflict with each other, which the network split cannot mitigate.
Visualization of the learned concepts. Fig. compares three joint decomposition strategies through visualization. For clear comparison, we adopt three concepts (painted in red, blue, and green color) for all the strategies. We observe that the attention-based decomposition tends to group joints from comparably distant positions into concept, benefiting from the learning of long-range dependencies with the attention module. In contrast, the heuristic and geometry- solutions mainly focus on the local structures and tend to associate joints with their neighbors.
5 CONCLUSION
In this paper. we propose a "decompose to generalize" (D-Gen) scheme for cross-species animal pose estimation. In contrast generic domain generalization methods, D-Gen focuses on the joint relation for improving the cross-species generalization. Specifically, D-Gen decomposes the body joints into several joint concepts, so that the joints in a single concept have relatively consistent rela- tions. Based on the joint concepts, D-Gen splits the feature extractor into multiple concept-specific branches. This simple network split suppresses the inconsistent interactions between inter-concept joints and yet maintains the consistent interactions between intra-concept joints. Experimental sults validate the effectiveness of D-Gen. We hope our work on cross-species pose estimation can provides new viewpoint for understanding the domain generalization problem.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
Our method does not induce bias but cannot alleviate the bias from the training data. Thus, with the proposed method, the model trained source domains may still preserve the bias from the source data, hence resulting inaccurate estimation results in the target domain.
Our method may fail under extreme scenarios with severe occlusion. The proposed method mainly assumes an ideal scenario with no or slight occlusion. However, wild environments and ani- mal habitats typically have various terrains and vegetation, which possibly causes severe occlusion. Our method may fail to deliver reliable pose estimations under such extreme circumstances.
Our method assumes well-align joint space. Under the diverse distribution of animal species, our method may fail to generalize to species with novel keypoints. Due to the limited diversity of current animal datasets, we cannot provide further analysis on the generalization to novel keypoints or species, e.g., from four-leg animals to birds, but hope to explore it in future works.
We hope our work can help protect the animals on our planet. Animal pose estimation can serve an important tool for analyzing animal behaviors and protecting them. However, training reliable pose estimator requires massive fine-annotated datasets, which is unrealistic for wild ani- mals, especially for rare species. Our method can help overcome this obstacle by learning species- generalizable representations, thus reducing the need for labor-extensive collections and annotations.
REPRODUCIBILITY STATEMENT
The proposed method is reproducible. We have provided the PyTorch-stee code for the proposed attention module, and the detailed training strategies in the main text and the appendix.
REFERENCES
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain gen- eralization using meta-regularization. Advances neural information processing systems, 31, 2018.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory of learning from different domains. Machine learning, 79(1)1151-175, 2010.
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to a new unlabeled sample. In Advances in Neural Information Processing Systems, 2011.
Gilles Blanchard, Aniket Anand Deshmukh, Urun Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning. Journal of Machine Learning Research, 2021.
Guillem Brasó, Nikita Kister, and Laura Leal-Tate Taixé. The center of attention: Center-keypoint group- ing via attention for multi-person pose estimation. ICCV, 2021.
Manh-Ha Bui, Toan Tran, Anh Tran, and Dinh Phung. Exploiting domain-specific features to en- hance domain generalization. In Advances in Neural Information Processing Systems, volume 34, pp. 119-2211 2021.
Yuanhao Cai, Zhicheng Wang, Zhengxiong Luo, Binyi Yin, Angang Du, Haoqian Wang, Xiangyu Zhang, Xinyu Zhou, Erjin Zhou, and Jian Sun. Learning delicate local representations for multi- person pose estimation. In European Conference on Computer Vision, pp. 455-472. Springer, 2020.
Jinkun Cao, Hongyang Tang, Hao-Sh Shu Fang, Xiaoyong Shen, Cewu Lu, and Yu-Wing Tai. Cross- domain adaptation for animal pose estimation. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2019.
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7291-7299, 2017.
10

---

Published as a conference paper at ICLR 2023
Fabio Maria Carlucci, Antonio D'Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Domain generalization by solving jigsaw puzzles. In CVPR, 2019.
Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yumsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Chaoqi Chen, Weiping Xie, Wenbing Huang, Yu Rong, Xinghao Ding, Yue Huang, Tingyang Xu, and Junzhou Huang. Progressive feature alignment for unsupervised domain adaptation. In CVPR, 2019.
Yang Chen, Yu Wang, Yingwei Pan, Ting Yao, Xinmei Tian, and Tao Mei. A style and semantic memory mechanism for domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9164-9173, October 2021.
Bowen Cheng, Bin Xiao, Jingdong Wang, Honghui Shi, Thomas S. Huang, and Lei Zhang. High- erhrnet: Scale-aware representation learning for bottom-up human pose estimation. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, 2020.
Shai Ben David, Tyler Lu, Teresa Luu, and Dávid Pál. Impossibility theorems for domain adaptation. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 129-136. JMLR Workshop and Conference Proceedings, 2010.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. CVPR, 2009.
Qi Dou, Daniel C. Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain generalization via model-agnostic learning of semantic features. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML, 2015.
Zigang Geng, Ke Sun, Bin Xiao, Zhaoxiang Zhang, and Jingdong Wang. Bottom-up human pose estimation via disentangled keypoint regression. In CVPR, 2021.
Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. Domain generaliza- tion for object recognition with multi-task autoencoders. In Proceedings the conference on computer vision, pp. 2551-2559, 2015.
the IEEE international
Rui Gong, Wen Li, Yuhua Chen, and Lue Van Gool. Dlow: Domain flow for adaptation and gener- alization. In CVPR, 2019.
Yunye Gong, Xiao Lin, Yi Yao, Thomas G Dietterich, Ajay Divakaran, and Melinda Gervasio. Confidence calibration for domain generalization under covariate shift. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Judy Hoffmann, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. CyCADA: Cycle-consistent adversarial domain adaptation. In ICML, 2018.
Zhengdong Hu, Yifan Sun, and Yi Yang. Switch to generalize: Domain-switch learning for cross- domain few-shot classification. International Conference on Learning Representations, 2022.
URL
net forum? i== AABMv I
Zhengdong Hu, Yifan Sun, and Yi Yang. Suppressing the heterogeneity: A strong feature extractor for few-shot segmentation. In International Conference on Learning Representations, 2023, URL https://openrevier net Forum 1ateummmdddCGuuu00000
Shaoli Huang, Mingming Gong, and Dacheng Tao. A coarse network for keypoint localization. In Proceedings of the IEEE International Conference Computer Vision, pp. 3028-3037, 2017.

---

Published as a conference paper at ICLR 2023
Zeyi Huang, Haohan Wang, Eric P. Xing, and Dong Huang. Self-challenging improves cross-domain generalization. In ECCV, 2020.
Yusuke Iwasawa and Yutaka Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. In Advances in Neural Information Processing Systems, volume 34, pp. 2427-2440, 2021.
Sheng Jin, Wentao Liu, Enze Xie, Wenhai Wang, Chen Qian, Wanli Ouyang, and Ping Luo. Differentiable hierarchical graph grouping for multi-person pose estimation. arXiv preprini arXiv:2007. 1864, 2020.
Rawal Khirodkar, Visesh Chari, Amit Agrawal, and Ambrish Tyagi. Multi-instance pose networks: Rethinking top-down pose estimation. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 3122-3131, October 2021.
Dachee Kim, Youngjun Yoo, Seunghyun Park, Jinkyu Kim, and Jaekoo Lee. Selfreg: Self- supervised contrastive regularization for domain generalization. In Proceedings the IEEF/CVF International Conference on Computer Vision (ICCV), 9619-9628, October 2021.
Sven Kreiss, Lorenzo Bertoni, and Alexandre Alahi. Pifpaf: Composite fields for human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 11977-11986, 2019.
Chen Li and Gim Hee Lee. From synthetic to real: Unsupervised domain adaptation for animal pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021a
Chen Li and Gim Hee Lee. Coarse-to-fine fine animal pose and shape estimation. In Advances Neural Information Processing Systems, volume 34, pp. 11757-11768, 2021b.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta-learning for domain generalization. In AAAI Conference on Artificial Intelligence, 2018a.
Da Li, Jianshu Zhang, Yongxin Yang, Cong Liu, Yi-Zhe Song, and Timothy M. Hospedales. Episodic training for domain generalization. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
Guangrui Li, Guoliang Kang, Wu Liu, Yunchao Wei, and Yi Yang. Content-consistent matching for domain adaptive semantic segmentation. European Conference on Computer Vision, pp- 440-456. Springer, 2020.
Guangrui Li, Guoliang Kang, Yi Zhu, Yunchao Wei, and Yang. Domain consensus clustering for universal domain adaptation. In IEEE/CVE Conference on Computer Vision and Pattern Recog- nition (CVPR), 2021a.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adver- sarial feature learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5400-5409, 2018b.
Pan Li, Da Li, Wei Li, Shaogang Gong, Yanwei Fu, and Timothy M. Hospedales. A simple feature augmentation for domain generalization. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision (ICCV), pp. 8886-8895, October 2021b.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 624-633 2018c.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation networks. In /CML,, 2015.
Zelun Luo, Yuliang Zou, Judy Hoffman, and Li F Fei-Fei. Label efficient learning of transferable representations acrosss domains and tasks. In NeurlPS, 2017.
12

---

Published as a conference paper at ICLR 2023
Lucas Mansilla, Rodrigo Echeveste, Diego H. Milone, and Enzo Ferrante. Domain generalization via gradient surgery. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6630-6638 October 2021.
Alexander Mathis, Thomas Biasi, Steffen Schneider, Mert Yuksekgonul, Byron Rogers, Matthias Bethge, and Mackenzie W Mathis. Pretraining boosts out-of-domain robustness for pose estima- tion. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1859-1868, 2021.
Gyeongsik Moon, Ju Yong Chang, and Kyoung Mu Lee. Posefix: Model-agnostic general human pose refmement network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7773-7781, 2019.
Jiteng Mu, Weichao Qiu, Gregory D. Hager, and Alan L. Yuille. Learning from synthetic animals. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International Conference Machine Learning, pp. 10-18. PMLR, 2013.
Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estima- tion. In European conference computer vision. pp. 483-499, Springer, 2016.
Alejandro Newell, Zhiao Huang, and Jia Deng. Associative embedding: End-to-end learning for joint detection and grouping. In I. Guyon, U. V. Luxburg, S. Ben- gio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, pp. 2277-2287. Curran Asso- ciates. Inc., 2017. URL https: httpptt/ppoceeediiiiii http://:/..ppccccig.s....m neurips.................. cc/papper/21///Eiz/ (B0o0722888222211777777222338383
Paper. pdf
Andrew Ng, Michael Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm.
Advances in neural information processing systems, 14, 2001.
Xun Long Ng, Kian Eng Ong. Qichen Zheng, Yun Ni, Si Yong Yeo, and Jun Liu. Animal kingdom: A large and diverse dataset for animal behavior understanding. In CVPR, 2022.
George Papandreou, Tyler Zhu, Nori Kanazawa, Alexander Toshev, Jonathan Tompson, Chris Bre- gler, and Kevin Murphy. Towards accurate multi-person pose estimation in the wild. In Pro- ceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4903-4911, 2017.
Seonguk Seo, Yumin Suh, Dongwan Kim, Geeho Kim, Jongwoo Han, and Bohyung Han. Learning to optimize domain specific normalization for domain generalization. In European Conference on Computer Vision, pp. 68-83. Springer, 2020.
Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross-gradient training. arXiv preprint arXiv:1804. 10745,, 2018.
Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Transactions on pattern analysis and machine intelligence, 22(8):888-90) 2000.
Yuge Shi, Jeffrey Seely, Philip H. S. Torr, Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. ICLR, 2022.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In ECCV, 2016.
Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high- representation learning for human pose estimation. In CVPR, 2019a.
Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Patlern Recognition, pp. 5693-5703, 2019b.
13

---

Published as a conference paper at ICLR 2023
Wei Tang and Ying Wu. Does learning specific features for related parts help human pose estima- tion? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. CoRR, 2014. URL
3474
Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. Advances in neural information processing systems, 31, 2018.
Dongkai Wang, Shiliang Zhang, and Gang Hua. Robust pose estimation in crowded scenes with direct pose-level inference. In Advances in Neural Information Processing Systems, volume 34, pp. 6278-6289, 2021.
Jingdong Wang. Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao. Deep high-resolution representation learning for visual recognition. TPAMI, 2019.
Guile Wu and Shaogang Gong. Collaborative optimization and aggregation for decentralized domain generalization and adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6484 6493, October 2021.
Fu-En Yang, Yuan-Chia Cheng, Yun Shiau, and Yu-Chiang Frank Wang. Adversarial teacher- student representation learning for domain generalization. In Advances in Neural Information Processing Systems, volume 34, pp. 19448-19460, 2021.
Hang Yu, Yufei Xu, Jing Zhang, Wei Zhao, Ziyu Guan, and Dacheng Tao. Ap-10k: A benchmark for animal pose estimation in the wild. In J. Vanschoren and S. Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
Tianhe Yu. Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances Neural Information Processing Systems, 33:5824-5836, 2020.
Feng Zhang, Xiatian Zhu, Hanbin Dai, Mao Ye, and Ce Zhu. Distribution-ane aware coordinate repre- sentation for human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7093-7102, 2020.
Guojun Zhang, Han Zhao, Yaoliang Yu, and Pascal Poupart. Quantifying and improving transferabil- ity in domain generalization. In Advances Neural Information Processing Systems, volume 34, pp. 10957-10970, 2021a.
Marvin Mengxin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: A meta-learning approach for tackling group shift. In NeurlPS, 2021b.
Shanshan Zhao, Mingming Gong, Tongliang Liu, Huan Fu, and Dacheng Tao. Domain generaliza- tion via entropy regularization. Advances in Neural Information Processing Systems, 33, 2020.
Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan- Michael Frahm (eds.), Computer Vision ECCV 2020, pp. 561-578, 2020.
Kaiyang Zhou, Yongxin Yang. Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In ICLR, 2021.
14

---

Published as a conference paper at ICLR 2023
Table 5: Ablation studies attention module for intra-family domain generalization (family Bovi- dae). Experiments here are conducted on AP-10K (Yu et al. 2021 with the leave-one-out manner. A.S. Argali Sheep.
Target Domain
Decomposition
With attention
Average
Antelope
A.S.
Bilson
Buffalo
Cow
Sheep
56.0
72.6
46.1
60.0
48.9
40.6
53.8
Random
52.6 (U)
67.7 (1)
51.1 (1)
60.0
43.7 (L)
42,8 (1)
53.0 (U)
Heuristic
53.4 49.6 (U)
73.6
48.8
61.1
49.7
41.0
54.6
69.8 (U)
48.4 (1)
58.9 (L)
47.8 (L)
39.7 (U)
52.4(L)
53.9
74.3
45.8
62.9
48.8
41.2
54.5
Geometry
55.4 (1)
70.3 (4)
47.0 (1)
63.8 (t)
48.8 (-)
44.8 (1)
55.1
Attention
60.9
77.3
49.8
68.1
51.8
40.1
57.9
Table 6: Ablation studies on attention module for inter-family domain generalization. periments here are conducted on AP-10K Yuu et al., 202T) with the leave-one-out manner. Cerc.:Ceroppithecida.
Ex-
Target Domain
Decomposition
With attention
Average
Bovidae
Felidae
Canidae
Ursidae
Cerc.
Equidae
Hominidae
48.6
43.0
53.3
31.5
24.7
39.7
38.4
39.6
Random
49.3 (1)
42.0
51.2 (U)
32.0
25.6 (1)
40.6
37.7
39.8 (L)
Heuristic
46.0
42.8
49.7
30.7
40,8
40.6
40.0
40.0
/
46.6 (f)
43.9 (t)
49.2 (U)
32.8 (t)
29.7 (4)
42.2 (f)
41.3 (1)
40.8 (1)
Geometry
41.9
53.9
23.0
42.8
v
50.8 (1)
42.6 (t)
54.8 (1)
36.4 (t)
25.2 (t)
39.9 (L)
42.6
41.8 (1)
Attenion
v
52.1
47.7
54.8
31.1
27.2
39.0
45.6
42.9
APPENDIX
A ADDITIONAL ANALYSIS AND RESULTS
Ablation the pixel-to-concept attention mechanism. We note that the pixel-to-conceed attention mechanism actually has two impacts on the attention bbase decomposition,, i.e., 1) it learns clues for joint decomposition, and 2) incurs interaction between joints and concept proxies, while the other two decomposition strategies do NOT have such effect. Therefore, although Table 1 and Table 2 in the manuscript show that the iitention-basedd decomposition achieves the largest improvement among all the four strategies, this question by adding the same attention module into the network under the other three decompo- sition strategies (without changing their joint decomposition results). The results summarized in Table 5 (intra-family) and Table (inter-family), from which we draw two observations as below:
not clear what is the exact reason for its superiority. We answer
First, we observe that adding the attention module for joint interaction does not promise improve- ment. For example, it slightly compromises the random and the heuristic decomposition (53.8+5330 and 54.6++5.44 under the intra-family DG scenario. Second, for geometry- -based decomposition, adding the attention module leads to slight improvement, i.e., + 0.6 for geometry- baased solution in Ta However, it is worth noting that, even with the attention, still cannot compete with the attention-based decomposition. Combining these two observations, we conclude that the superiority of the iitention-basee solution should not mainly ascribed to the pixel-to-concept interactions but to better joint concepts.
What if treat all joints as one concept or treat each joint as individual concepts? Table we complement the results when manually set & and & 17, which treats all joints one concept or treat each joint as one concept. Despite the network split, the manual concept division barely improves the performance over the ERM baseline, which again justifies the effectiveness of the proposed D-Gen paradigm. These results also validate that simply increasing the model capacity cannot promise improvement in the generalization.
15

---

Published as a conference paper at ICLR 2023
Table DG results (AP) on AP-10K (Yu et al. 2021) under varying k (number of concepts) with D-Gen paradigm or manual setting.
D-Gen
Manual
Decom
9
17
Atiention-basck
57.9
57.1
56.9
Intra-DG
Geometry-based
54.2
53.9
51.0
52.8
51.9
Random
53.8
52.9
53.1
Attention-basse
42.5
42.0
42.1
Inter-DG
Gcometry-basee
40.9
40.8
39.5
38.0
38.2
Random
39.6
40.3
38.8
Table Comparison on the training time of each iteration.
Method
ERM
MisStyle
GradSur
SWAD
Fish
Ours
Time iteration (ms)
250
251
310
290
2246
280
Comparison on more datasets. In Table 10 we present the generalization results to Horse-C dataset (Mathis al.) 202T). Following their evaluation protocol, we validate the effectiveness of the proposed method on this benchmark and report the results, and again attains superior results.
Analysis on joint feature extraction In Table II we testify the sensitivity to the feature extrac- tion process, i.e., localizing the joint features with ground truth the prediction. The negligible difference verifies that our method robust the choice on clues for feature extraction.
Analysis on model capacity In Table 9 we present the analysis of the model capacity, i.e, FLOPs and inference speed. And in Table g we compare the time for each training iteration Apparently, our method does not require longer time even with two stages when compared with some previous SOTAs. This is because some of them employ extra inner steps during the optimization. For exam- ple, Fish (Shi et al. 2022) requires to perform n inner steps to aggregate the gradients of domains independently.
Evaluation of uncertainty. In Table 12
repeat our experiments two more data splits with three different random seeds. Under such variation, we re-evaluate the uncertainty in two scenarios, ie., intra-family DG within Bovidae and fater-family DG AP-10K. The results show consistent uncertainty with the main paper on both scenarios.
16

---

Published as a conference paper at ICLR 2023
Table 9: Analysis on the model capacity.
Model configuration
FLOPs (G)
Inference speed(MS)
Original Network (ResNet-50)
21.0
9.2
with network split
24.8
12.4
Table 10: PCK@0.3(%) for out-of-distribution generalization to Horse-C Mathis et al. (FF=front foot; HF Hind foot; HH = Hind Hock).
2021
Method
Nose
Eye
Shoulder
Wither
Elbow
NeaFFF
OFFF
Hip
NeaHHH
NearHFF
OIHE
Average
et 2021
68.2
73.6
85.4
85.8
88.1
72.6
70.2
89.2
85.7
77.0 79.3
74.1
79.1
Daben conertion)
67.9
76.4
86.1
83.8
88.3
79.6
74.3
90.1
86.6
76.6
80.9
Table 11: Results with different clues for joint feature extraction. Experiments here are under the inter-family DG scenario on AP-10K (Yu et al. 202T). Cerc.=Cercopitheciiae
Target Domain
Extraction clue
Average
Bovidac
Felidae
Canidae
Unsidac
Cerc.
Equidac
Hominidac
Ground truth Prediction
53.1
47.2
52.3
35.9
28.7
47.5
46.6
43.2
52.1
47.7
54.8
31.1
27.2
39.0
45.6
42.9
Table 12: More random for uncertainty evaluation. Experiments here conducted AP- 10K (Yu et al. 2021) in the leave-one-out manner. The intra-family DG is performed on the family Bovidae. Transfer
Data Split
Data Split 2
Average
Average
Pam--Pamly DG
37.91
57.13
58.03
$7.69 0.49
$7.47
38.
$7.52
$7.81 =0.7
Inter Family DG
43.13
42.14
42.63
42.63 0.50
42.92
43.31
41.98
42.73 + 0.68
17

---

Published as a conference paper at ICLR 2023
More qualitative comparisons between different joint decomposition strategies. In Fig. 7 provide more visualizations of the mined concepts with different decomposition strategies, and we make the following observations:
1) Despite being intuitively reasonable, heuristic decomposition may induce ambiguity between some hard-to-distinn diitttn-dit joints. For example, the left and right forelegs are assigned the same concept, while distinguishing them within the same concept might not be easy.
2) For the geometry- boase solution, notice that it tends to focus and identify closely-related structures in the geometric space, i.e., the forelegs and the hind legs.
3) In Fig. 7 (c), find that the left eye and the right eye assigned to different concepts. This result may seem counter-intuitive at first glance but is actually reasonable from a closer look. is because the left and right eyes are very small and hard to distinguish. Therefore, associating the left and right eyes with some different easy-to-distinuussh joints helps to discriminate them, as well. On the contrary, both the heuristic and geometry -based strategies assign the left and right eyes to the same concept and are actually inferior.
Qualitative comparisons on pixel-to-concept atfention. To better understand the pixel-to-concept attention, we present the visualization of the learned attention maps in Fig. 8 As we could ob- server, each concept embedding apparently favors specific regions, and such preference consistent over different species. Such phenomenon verifies that the proposed attention mechanism indeed encourages the inter-joint relationships that can benefit the generalization.
Qualitative comparison with previous solutions on pose estimation results. In Fig. 2 and Fig. T0
we present the qualitative results on intra-family and inter-family DG, respectively, and compare with previous solutions. Compared with the previous solution, the proposed D-Gen paradigm can deliver more accurate estimations, proving the effectiveness of the proposed method. Concretely, the larger variation in appearance and shape renders previous solutions less effective, especially the joints from non-rigid parts, e-g. legs. In contrast, our method, especially the attention-based variant, shows stronger robustness them and delivers more accurate estimations both intra-DG and inter-DG scenarios.
Heuristic decomposition
(b) Geometry bused decomposition
(c) Attentionn- decomposition
Figure 7: Concept visualization on AP-10K (Yu et al. 2021), where different mined concepts are in different colors. noticeable observation that the attention bisention-based strategy assigns the left and right eyes to different concepts. This decomposition result seems counter-intuitive but is actually reasonable: it associates the handtto-dissinguish left and right eyes with different -i-istiuiit joints, so that the latter joints provide clues for distinguishing the left and right eyes.
18

---

Published as a conference paper at ICLR 2023
Figure 8: Attention weight visualization on AP-10K (Yu al., 2021). where the bottom two rows correspond to the heatmap of different concepts. With a variety of animal species, iee, cow, wolf, sheep, and dog, the concept embedding can effectively attend and associate specific keypoints, which further justifies the effectiveness of the proposed approach.
Fish
SWAD
Heuristic
Geometry
Atlention.
Ground
Figure 9: Pose estimation result under the intra-family DG on AP-10K (Yu al. 202T). Experi- ments here follow the leave-one-out protocol on Family Bovidae and with the Antelope as the target domain. Compared with solutions, our method demonstrates stronger capability on joint localization and identification, especially on joints from non-rigid part, egg., legs.
19

---

Published as a conference paper at ICLR 2023
Fish
SWAD
Heuristic
Geometry
Attention.
Ground Truth
Figure 10: Pose estimation result under the inter-family DG on AP-10K Yu et al. 2021 Experi- ments here follow the leave-one-out protocol and the target domain is Bovidae. Under larger gap between species, our method maintains its superiority against previous solutions.
20

---

Published as a conference paper at ICLR 2023
Algorithm A PyTorch-style pseudocode for pixel-to-concept attention
import torch
torch.nn
import torch. nn. functional
class Pixel2Concept Attention (nn. Module):
init (self, q dim, k dim, v v_dim, dim, out_ dim, dzopout=0.005 super (PAxel2concept
self) init ()
q_dim dim =-dimm dim out_dim 256
self.hid self. N_query nn Linear (k_ dim, dim) self.w_key (in chanzadiinccannnesngg....
out channelsshid dim, kernel size=1, stride=l, padding=0) (adidigl,,
self.w Faif.w_valiee value self
nn Lin.ee dim, hid dim)
nn. (in, chiiiia__aaneesshiddd dim, channelssout dim, Bernel_size=l stride=1, padding=0)
self dropout nn. Dropout (dropout) self = an. (out, dim, out dim, kernel sanel_size=,, padding=0) the number enn.Parameter (torc....aateeeeeeeeeeee (self, conff..on 256))
Concept embedding, where
self welf.concept.i..
def fsel,, feats):
B, C, H, shape
= self. quezy wwwelffconcett.eee emb)
self. 4_kay (feats)
v welf.w_valll
ceeelf.concet_emme
k = k-permute (0, 2,
q.permute (1, (1.Quuuusueeeee((
mascch.mmmml1111 (k,
simi simi (self. hillf.hid.ce dime+ (-)5))
simi F. (simi, dim
dropout (simi)
out = torch. beech.matmul (simi, v)
out out out.permute perputereeee 3, 1, 2)
selt.
return out
B MORE IMPLEMENTATION DETAILS
Experimental setup. All the presented solutions are re-implemented with their released code.
MixStyle (Zhou al. 2021h: We find the default config works best: set the probability 0.5, set to 0.3, and employ the random style augmentation.
GradSur (Mansilla al. 2021): We adopt the default module, i.e., "PCGrad" function from the original code, which mitigates the gradient conflicts according to their inner product. No extra hyper-parameters is required
SWAD (Cha et al. 2021 )
We report the best results from its presented variants, and no extra hyper-
parameters are required to tune.
Fish (Shi et al. y022): The meta-learning rate is set to 0.001. Other parameters for optimization the same our default setting.
Training. The training process consists of twos stages, i.e., the first stage derives the concept divi- sion with the learned concept embeddings, and the second stage optimizes with the concept-speciff branches. To be specific, the first stage takes 60 epochs and the second takes 150 epochs. As for our one-stage solutions, we optimize them for 210 epochs. For the attention- decomposition, the attention module is activated for both stages.
Evaluation. Following (Sun al. 2019b; Cheng et al. 2020 we evaluate the pose similarities with OKS Σ. exp( (apppd///="""aeeee Σ. (0,,00
Here d, denotes the Fuclidean distance between detected keypoint and its corresponding ground truth, v; the visibility flag of the ground truth, & is object scale, and K; is per- ker-iitt constant that controls falloff. The reported average precision the average of AP scores at OKS (0.50, 0.55, 0.90, 0.95.)
Data augmentation. In training, we employ the following data augmentation: random rotation ([-30%, 30")), random scale ([0.75, 1.5]). random translation ([-40, 40]) to crop an input image patch of size 256 x 256 and random flip. In inference, only resize and normalization is performed.
Attention module. Algo. A we provide the implementation of the pixel-to-concept attention.
21


Reviews:
[{"review_id": "vO03JdqMYZc", "reviewer": "Reviewer_2nfE", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes a new approach for domain generalization in cross-species pose estimation using different approaches to partition joints to be estimated into different ‘concepts’, and separately training parts of the network to estimate the position of each joint concept group. Three approaches for separating joint concepts are used: based on the proximity of joints to one another (heuristic), based on the joint affinity matrix (geometric), and based on a learned attention mechanism (attention). In each images are processed using common feature extraction layers, and then joint concepts are separated and processed in separate branches. The manuscript introduces the details of the attention based approach, which is novel and shown to perform superior to the others in several cases. They show that the attention approach minimizes gradient conflicts, i.e. does a better job of separating weight gradient vectors across joint groups. \nThey then perform benchmark experiments on two datasets that allow for testing domain generalization approaches within and across animal family. They compare the performance of each approach, along with a random split to account for ensemble gains, to competing domain generalization approaches and oracle approaches, in particular training and testing within a species or family. In many cases the attention approach performs superior to other domain generalization approaches. They close by investigating the effect of salient hyperparameters: the number of layers that are concept-specific and the number of concepts. \n", "summary_of_the_review": "Overall I thought this was an interesting approach and seemed to show some promise compared to other approaches. The manuscript was clear and contained both fairly thorough benchmark comparisons, some hyperparameter optimization and some explanations for performance gains. I have reservations about the overall effect size/improvement and variance that I would like to see addressed, but otherwise I think it is a good contribution. ", "strength_and_weaknesses": "Strengths\n1) Attention Approach is novel, and compared against several other reasonable baselines and datasets. The experiments are well documented and well performed. I think the advantages are somewhat appreciable in the intra-family case, and less so in the inter-family case, but this will be a nice result to build on. \n2) Manuscript is clear and easy to follow, trained on standard benchmarks in the field and fairly comprehensive. I appreciated the didactic figure 5\n3) I thought the gradient conflict analysis was interesting and a nice empirical motivation for the work.\n\n\nWeaknesses\n1) Treatment of uncertainty in tables. I am not sure how large the uncertainty is for the experiments in Tables 1,2,3 are. It would help to have reporting of this to evaluate the significance of some of the results seen – I am not sure how seriously to take 0.5 differences in AP. The error bars reported below each family also seem too low given the spread of values (e.g. Table 1, attention/average has a s.e.m of 0.5 ?). These could be computed from different model seeds or training data subsets. \n\n2) The oracle shows worse performance than the domain generalization approaches at times in Table 1. Is this because of a lack of data for rarer classes (e.g. king cheetah)? In these cases the oracle would have much less training data. It might make sense to also try training the oracle on all datasets in the family, and report the number of instances of each animal. \n3) Its worth noting that none of the approaches seem very good on the inter-family experiments (Tables 2 and 3) \n4) It would help potential users to have representation qualitative visualization of some of these performance differences. What does a mAP gain of 63 vs 59 look like for Bobcat in Table 1? What are typical inter-family results? Without these it is hard for prospective users to gauge the quality of the performance gains. \n5) Why simply average the loss with the baseline network and why not include a learnable weight for combining the predictions? \n\nComments:\n I think the approach could have more broad generality, e.g. for facial keypoint tracking where separate groups of keypoints co-vary.  \nI would be curious if any of the domain adaptation approaches can be combined, e.g. those that seek to match distributions should be fairly orthogonal to the present work. \n\n\nNits\nFigure 1:  What are the methods for the bar graphs on the left, why are there no error bars? \nTable 1: A.S. undefined. \nViewpoint dependence. Could conceive of as a style difference. \nFigure 6 and 5 switched\nIts worth noting that these 2D style transfer approaches will be sensitive to camera viewing angle, and the relation between joints and joint concepts may depend on viewing angle. \nWhy is the domain generalization performance so poor for panda bears and polar bears? \nCan you provide more detail on the empirical risk minimization approach. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "mIGF0JPDm5A", "reviewer": "Reviewer_SQkn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an animal-pose estimation method which can be applied to multiple animal species. Firstly the proposed method divides a set of animal keypoints into several groups. The keypoints in one group are supposed to have close relation, i.e. intra-group keypoints may provide a localization cue to each other. On the other hand, keypoints in one group may hinder the localization of the ones in another group due to the independence between groups. Then, the authors build a separate branch for each keypoint group for the keypoint detection network in order to encourage the interaction between intra-group keypoints and suppress the interaction between inter-group keypoints. The authors propose three grouping methods, among which two are based on rules and one is attention-based clustering method. The experimental results show that the proposed method using attention-based clustering outperforms some SOTAs , and thus the effectiveness the proposed method is verified.", "summary_of_the_review": "The attention-based keypoint clustering method is interesting, but novelty is still not impressive. Besides, specific characteristic of a species is not actually considered by the proposed work as claimed by the authors.", "strength_and_weaknesses": "Strength: \n1. The attention-based keypoint clustering method is interesting, and the effectiveness is verified well via comparison with SOTAs and ablation study. \n2. Gradient conflict is used to analyze the effectiveness of the proposed idea.\n\nWeaknesses:\n1. The authors claim that some joint relations are consistent across all the species while some other joint relations are inconsistent and harmful. But the proposed method divides the keypoints into constant groups without considering species, which means the relations between keypoints are constant cross species. I think this work aims to build keypoint groups and estimate each group of keypoints relatively independently. This does not provide capability of handling  the diversity of species. \n2. The main novelty lies in keypoint grouping, while the network-split's novelty is relatively limited. I expect the authors to perform multiple methods to encourage interaction between intra-group keypoints and suppress interaction between inter-group, in order to verify the effectiveness of the idea of keypoint grouping\n3. The keypoint grouping result shown in Fig. 5 indicates that the grouping does not comply with prior knowledge very well. I doubt that this is relevant with training data's distribution.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Jm6D9SIQrAe", "reviewer": "Reviewer_e7pK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a domain generation approach, D-Gen, for cross-species animal pose estimation. D-Gen first learns k concept embeddings using an attention-based module. Then, previously-learned concept embeddings are used to decompose the body joints into k joint concepts with nearest neighbor search. Finally, D-Gen split the top layer of the network into multiple concept-specific branches based on the joint concepts. Experiments are conducted in both intra-family and inter-family settings to show the effectiveness of D-Gen.", "summary_of_the_review": "The paper addresses an interesting problem but is hampered by the limited novelty of the approach and the lack of experiments for cross-species animal pose estimation with a larger domain gap(e.g. fish, birds).", "strength_and_weaknesses": "Pros:\n\n-The motivation in this paper is reasonable in that some of the relationships between specific joints may be shared by multiple animal species; hence, considering such relationships aids in cross-species animal pose estimation.\n\nCons:\n\n-K (Number of concept embedding) is a fixed number and is set to 3 for all tasks. Concept embedding in Fig 3 (b) is similar to the codebook[1]. When estimating the poses of various animal species, it is better to select a larger value for k and let the network dynamically select the concept embeddings to utilize.\n\n-The gap between the source and target domain is relatively small in both intra-family and inter-family (basically a four-legged mammal). A larger domain gap would be more convincing (e.g., fish, birds).\n\n-Lack experiments on existing large-scale animal pose datasets, e.g. [2]\n\n【1】 Van Den Oord A, Vinyals O. Neural discrete representation learning[J]. Advances in neural information processing systems, 2017, 30.\n\n【2】 Ng, Xun Long, et al. \"Animal Kingdom: A Large and Diverse Dataset for Animal Behavior Understanding.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JDJRB4GzOU2", "reviewer": "Reviewer_uMhs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for animal pose estimation able to generalize to different species. The method approaches the problem from the domain generalization point of view. It is based on the observation that some relations among joints remain consistent among different species, while others change drastically. According to this, the paper proposes to separate joints to concepts/groups and process them using a network with split top layers, as to break inconsistent relations while maintaining consistent ones.", "summary_of_the_review": "Based on the discussion above, I think that the proposed method proposes an interesting and significant contribution to the less explored problem of animal pose estimation (in comparison to human pose estimation). I propose acceptance, although there are some aspects which could be improved (see weaknesses above)\n\n## Comments after the rebuttal\nI find the proposed approach novel in the specific domain. The contributions are well motivated and interesting, especially the distinction between consistent and not-consistent joints is intuitive and supported by the results. Nevertheless, the type of consistency needs to be better defined to avoid confusion (e.g., geometric/topological vs semantic consistency).\n\nI am quite satisfied with the author responses and I think that the paper has become stronger after the revisions. However, the increase in performance between different animal families is somewhat limited. I keep my score to 6 as I think that the paper is acceptable for publication.", "strength_and_weaknesses": "## Strengths\nThe proposed approach is principled and well motivated. Moreover, there are improvements in the animal pose estimation performance, regardless of the decomposition method, i.e. attention-based, heuristic or geometric. It is also important that the attention-based performs significantly better with respect to the other two methods. The network split is also well motivated based on gradient analysis.\nThe method is compared against state-of-the-art domain generalization methods.\n\n## Weaknesses\nThe fact that the proposed approach is based on two distinct stages, namely joint decomposition and network split, can be considered a weakness. This is mainly because the two steps appear to be strictly coupled, yet they are treated independently. It would not be a weakness if being disjoint the two stages could offer better compositionality (e.g. some step could be replaced with some alternative) but this is not discussed in the paper. \n\nMoreover, the use of the split network seems to require some relatively involved bookkeeping for composing the per-concept features. A short discussion on this subject could help reproducibility.\n\nAdditionally, inference is only briefly discussed in the paper, mainly in Figure 2. It would help if more details were provided. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster