Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method for animal pose estimation able to generalize to different species. The method approaches the problem from the domain generalization point of view. It is based on the observation that some relations among joints remain consistent among different species, while others change drastically. According to this, the paper proposes to separate joints to concepts/groups and process them using a network with split top layers, as to break inconsistent relations while maintaining consistent ones.
strength_and_weaknesses: ## Strengths
The proposed approach is principled and well motivated. Moreover, there are improvements in the animal pose estimation performance, regardless of the decomposition method, i.e. attention-based, heuristic or geometric. It is also important that the attention-based performs significantly better with respect to the other two methods. The network split is also well motivated based on gradient analysis.
The method is compared against state-of-the-art domain generalization methods.

## Weaknesses
The fact that the proposed approach is based on two distinct stages, namely joint decomposition and network split, can be considered a weakness. This is mainly because the two steps appear to be strictly coupled, yet they are treated independently. It would not be a weakness if being disjoint the two stages could offer better compositionality (e.g. some step could be replaced with some alternative) but this is not discussed in the paper. 

Moreover, the use of the split network seems to require some relatively involved bookkeeping for composing the per-concept features. A short discussion on this subject could help reproducibility.

Additionally, inference is only briefly discussed in the paper, mainly in Figure 2. It would help if more details were provided. 
clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to read. The description of the method is sufficiently clear. Nevertheless, some aspects like inference (see above) could be described in more detail. 

Although the approach is based on established solutions (domain generalization, network splitting, etc.) their application to pose estimation, and animal pose estimation in particular, is novel.

Regarding reproducibility, a code snippet is provided that can help in implementing the method and reproducing the results.
summary_of_the_review: Based on the discussion above, I think that the proposed method proposes an interesting and significant contribution to the less explored problem of animal pose estimation (in comparison to human pose estimation). I propose acceptance, although there are some aspects which could be improved (see weaknesses above)

## Comments after the rebuttal
I find the proposed approach novel in the specific domain. The contributions are well motivated and interesting, especially the distinction between consistent and not-consistent joints is intuitive and supported by the results. Nevertheless, the type of consistency needs to be better defined to avoid confusion (e.g., geometric/topological vs semantic consistency).

I am quite satisfied with the author responses and I think that the paper has become stronger after the revisions. However, the increase in performance between different animal families is somewhat limited. I keep my score to 6 as I think that the paper is acceptable for publication.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper describes a new approach for domain generalization in cross-species pose estimation using different approaches to partition joints to be estimated into different ‘concepts’, and separately training parts of the network to estimate the position of each joint concept group. Three approaches for separating joint concepts are used: based on the proximity of joints to one another (heuristic), based on the joint affinity matrix (geometric), and based on a learned attention mechanism (attention). In each images are processed using common feature extraction layers, and then joint concepts are separated and processed in separate branches. The manuscript introduces the details of the attention based approach, which is novel and shown to perform superior to the others in several cases. They show that the attention approach minimizes gradient conflicts, i.e. does a better job of separating weight gradient vectors across joint groups. 
They then perform benchmark experiments on two datasets that allow for testing domain generalization approaches within and across animal family. They compare the performance of each approach, along with a random split to account for ensemble gains, to competing domain generalization approaches and oracle approaches, in particular training and testing within a species or family. In many cases the attention approach performs superior to other domain generalization approaches. They close by investigating the effect of salient hyperparameters: the number of layers that are concept-specific and the number of concepts. 

strength_and_weaknesses: Strengths
1) Attention Approach is novel, and compared against several other reasonable baselines and datasets. The experiments are well documented and well performed. I think the advantages are somewhat appreciable in the intra-family case, and less so in the inter-family case, but this will be a nice result to build on. 
2) Manuscript is clear and easy to follow, trained on standard benchmarks in the field and fairly comprehensive. I appreciated the didactic figure 5
3) I thought the gradient conflict analysis was interesting and a nice empirical motivation for the work.


Weaknesses
1) Treatment of uncertainty in tables. I am not sure how large the uncertainty is for the experiments in Tables 1,2,3 are. It would help to have reporting of this to evaluate the significance of some of the results seen – I am not sure how seriously to take 0.5 differences in AP. The error bars reported below each family also seem too low given the spread of values (e.g. Table 1, attention/average has a s.e.m of 0.5 ?). These could be computed from different model seeds or training data subsets. 

2) The oracle shows worse performance than the domain generalization approaches at times in Table 1. Is this because of a lack of data for rarer classes (e.g. king cheetah)? In these cases the oracle would have much less training data. It might make sense to also try training the oracle on all datasets in the family, and report the number of instances of each animal. 
3) Its worth noting that none of the approaches seem very good on the inter-family experiments (Tables 2 and 3) 
4) It would help potential users to have representation qualitative visualization of some of these performance differences. What does a mAP gain of 63 vs 59 look like for Bobcat in Table 1? What are typical inter-family results? Without these it is hard for prospective users to gauge the quality of the performance gains. 
5) Why simply average the loss with the baseline network and why not include a learnable weight for combining the predictions? 

Comments:
 I think the approach could have more broad generality, e.g. for facial keypoint tracking where separate groups of keypoints co-vary.  
I would be curious if any of the domain adaptation approaches can be combined, e.g. those that seek to match distributions should be fairly orthogonal to the present work. 


Nits
Figure 1:  What are the methods for the bar graphs on the left, why are there no error bars? 
Table 1: A.S. undefined. 
Viewpoint dependence. Could conceive of as a style difference. 
Figure 6 and 5 switched
Its worth noting that these 2D style transfer approaches will be sensitive to camera viewing angle, and the relation between joints and joint concepts may depend on viewing angle. 
Why is the domain generalization performance so poor for panda bears and polar bears? 
Can you provide more detail on the empirical risk minimization approach. 

clarity,_quality,_novelty_and_reproducibility: The manuscript is clear and the experiments are well described. The network design appears novel. 
summary_of_the_review: Overall I thought this was an interesting approach and seemed to show some promise compared to other approaches. The manuscript was clear and contained both fairly thorough benchmark comparisons, some hyperparameter optimization and some explanations for performance gains. I have reservations about the overall effect size/improvement and variance that I would like to see addressed, but otherwise I think it is a good contribution.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a domain generation approach, D-Gen, for cross-species animal pose estimation. D-Gen first learns k concept embeddings using an attention-based module. Then, previously-learned concept embeddings are used to decompose the body joints into k joint concepts with nearest neighbor search. Finally, D-Gen split the top layer of the network into multiple concept-specific branches based on the joint concepts. Experiments are conducted in both intra-family and inter-family settings to show the effectiveness of D-Gen.
strength_and_weaknesses: Pros:

-The motivation in this paper is reasonable in that some of the relationships between specific joints may be shared by multiple animal species; hence, considering such relationships aids in cross-species animal pose estimation.

Cons:

-K (Number of concept embedding) is a fixed number and is set to 3 for all tasks. Concept embedding in Fig 3 (b) is similar to the codebook[1]. When estimating the poses of various animal species, it is better to select a larger value for k and let the network dynamically select the concept embeddings to utilize.

-The gap between the source and target domain is relatively small in both intra-family and inter-family (basically a four-legged mammal). A larger domain gap would be more convincing (e.g., fish, birds).

-Lack experiments on existing large-scale animal pose datasets, e.g. [2]

【1】 Van Den Oord A, Vinyals O. Neural discrete representation learning[J]. Advances in neural information processing systems, 2017, 30.

【2】 Ng, Xun Long, et al. "Animal Kingdom: A Large and Diverse Dataset for Animal Behavior Understanding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
clarity,_quality,_novelty_and_reproducibility: The motivation in this article is well-explained and generally reasonable, but the technical novelty in the individual parts are limited.
summary_of_the_review: The paper addresses an interesting problem but is hampered by the limited novelty of the approach and the lack of experiments for cross-species animal pose estimation with a larger domain gap(e.g. fish, birds).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an animal-pose estimation method which can be applied to multiple animal species. Firstly the proposed method divides a set of animal keypoints into several groups. The keypoints in one group are supposed to have close relation, i.e. intra-group keypoints may provide a localization cue to each other. On the other hand, keypoints in one group may hinder the localization of the ones in another group due to the independence between groups. Then, the authors build a separate branch for each keypoint group for the keypoint detection network in order to encourage the interaction between intra-group keypoints and suppress the interaction between inter-group keypoints. The authors propose three grouping methods, among which two are based on rules and one is attention-based clustering method. The experimental results show that the proposed method using attention-based clustering outperforms some SOTAs , and thus the effectiveness the proposed method is verified.
strength_and_weaknesses: Strength: 
1. The attention-based keypoint clustering method is interesting, and the effectiveness is verified well via comparison with SOTAs and ablation study. 
2. Gradient conflict is used to analyze the effectiveness of the proposed idea.

Weaknesses:
1. The authors claim that some joint relations are consistent across all the species while some other joint relations are inconsistent and harmful. But the proposed method divides the keypoints into constant groups without considering species, which means the relations between keypoints are constant cross species. I think this work aims to build keypoint groups and estimate each group of keypoints relatively independently. This does not provide capability of handling  the diversity of species. 
2. The main novelty lies in keypoint grouping, while the network-split's novelty is relatively limited. I expect the authors to perform multiple methods to encourage interaction between intra-group keypoints and suppress interaction between inter-group, in order to verify the effectiveness of the idea of keypoint grouping
3. The keypoint grouping result shown in Fig. 5 indicates that the grouping does not comply with prior knowledge very well. I doubt that this is relevant with training data's distribution.
clarity,_quality,_novelty_and_reproducibility: The proposed method is well verified by the experiments. 

Although the attention-based keypoint clustering method is interesting, the whole novelty is limited since the divide-and-conquer is a common approach In computer vision. Besides, specific characteristic of a species is not actually considered by the proposed work as claimed by the authors.
summary_of_the_review: The attention-based keypoint clustering method is interesting, but novelty is still not impressive. Besides, specific characteristic of a species is not actually considered by the proposed work as claimed by the authors.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper addresses the problem of cross-animal pose estimation (generalization to novel animals). The authors propose a Decompose-to-Generalize method to break the inconsistent relations (relations that are inconsistent for different species due to species variation and bring distraction) while preserving the consistent ones (relations that are consistent across all species). Reviewers think the method is novel and interesting. Reviewer uMhs: Approach is principled and well motivated. Network split is also well motivated based on gradient analysis. Reviewer 2nfE: Gradient conflict analysis was interesting. Attention Approach is novel. Though Reviewer e7pK commented that paper has limited novelty, but during the AC-Reviewer discussion,  Reviewer e7pK pointed out that keypoint grouping is novel in this field, and the effectiveness of keypoint grouping has been supported in experiments. Reviewer SQKn gives negative score, but also mentioned that the attention-based keypoint clustering method is interesting. Overall, the paper is clearly presented, and the efficacy is supported by the experiments. The weaknesses include the relatively marginal performance improvement and the lack of some ablation studies as mentioned by reviewer SQKn. Considering the novelty of the work, the AC recommends acceptance. Authors need to improve the final version by incorporating the promised revisions to the paper, and add the ablation studies (e.g., performing multiple methods to encourage interaction between intra-group key-points and suppress interaction between inter-group). 
justification_for_why_not_higher_score: Some ablation studies are not included, e.g., performing multiple methods to encourage interactions between intra-group key-points and suppress interactions between inter-group.
justification_for_why_not_lower_score: The proposed method is novel, and the experiments have supported the efficacy.
Decision: Accept: poster