Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduced AP10k, a dataset for 2D animal pose detection. The dataset consists of 10,000 images of 60 animals (all mammals, I believe) with up to 17 pose keypoints labeled, and an additional 50,000 images for self-supervised learning applications. The authors present statistics on the distribution of images per animal group. They then evaluate the performance of different human pose estimation algorithms for animal pose estimation in multiple relevant domains: testing the importance of pretraining on imagenet and COCO, testing the dependence on the number of epochs, and pretraining on different groups of species. The number of images and species is well beyond what is present in the current literature, and there is a growing interest in animal pose estimation, and so the dataset could be a valuable addition to the literature. 
strengths: The manuscript fills a need in the field that is of growing interest to diverse communities (ecology, biomechanics, neuroscience). The dataset is far larger and more diverse than what is present in the literature, and I think will be useful for algorithm development. The pre-training and cross-domain examples are useful for gauging how well the approach will generalize, although some of the results with few numbers of epochs are less interesting. In general I think that this manuscript is a good fit for the track and would vote for acceptance and increase the score if some of the issues of clarity and correctness discussed below are addressed. 
weaknesses: I think the manuscript could be improved before acceptance. 

L47: “ manually labeled and carefully double-checked”. I have questions about the documentation of the annotation quality. Since multiple annotators were used, it would be nice to have a sense of the variation in performance across annotators. This is important because the position of many of these joints is actually fairly ambiguous (eg the neck keypoint of the bison and many other animals). The actually mAP and other metrics should be compared to the inter-annotator variability. 

There are also at least some errors present - eg in the polar bear image can see a left/right color swap. In the skunk example on the github page one of the left/right limbs is mislabeled. 

How were the keypoints instructed to be labeled? How were occlusions marked if part of a limb was absent in the frame (eg the polar bear in Figure 1).

Choice of images and backgrounds. Animal behavior tracking is a broad field, encompassing ecology research in the wild, zoo settings, and laboratory settings. It is clear what types of animals were selected, but not clear what type of backgrounds were selected. This is important in thinking about what type of setting the result will generalize to. 

Multiple Animals. How are multiple animals handled (e.g. Figure 1) ? Are there a sufficient number of examples to afford a benchmark on multiple animal detection? 

Figure 4 and Table 9: Are the images for testing in the Animal Pose 10k dataset? This is a reasonable comparison, but it would be nice to know the performance on animals within class. More generally, I think I would want to see performance outside the training animals (ie well outside the dataset) to test generalization performance. 

Performance: can you provide representative examples of success and failures of the pose estimation? Are there still challenging species? How is the performance on rare species? It seems from the inter-family DG track that the performance could be improved. 

Quantification metrics: the mAP threshold should be fine, and a reporting of the exact means of calculating the AP50, 75 etc. should be introduced. I looked them up in the HRNet paper, but would nice to be have here. 


correctness: The knee in ungulates is actually much higher than is marked in Figure 2 – what is actually labeled is the ankle. This needs to be made clear.   More broadly, some species, such as bats, may have a very different body plan, and it is not clear how choices for labeling were made in cases when the body plan of animals deviates substantially from ‘typical’ domesticated mammals. 

Long tail distribution. Spinning this as a feature, not a problem, is a bit of an odd claim. Obviously if an investigator wanted to to gauge algorithm performance on rare classes it would be better to have the ability to manipulate the full statistics of the training dataset, ie having balanced classes and being able to undersample from them. 
clarity: The manuscript should be thoroughly checked for grammar, and especially tense. e.g. “limiting the generalization ability”, should be limiting their generalization ability;  “to facilitate the research” should be to facilitate research. The tense throughout is problematic as well (ie the Data Annotation section should be written in past tense)

L3 – previous works doing what? The antecedent is unclear. 

It is not clear exactly how the human pose transfer is being performed, ie how the weights of the network are being transferred and which layers are being reused. 

Table 6/7 – the metric used could be directly stated in the caption (assumed to be mAP).

relation to prior work: This manuscript actually only concerns mammals, which is a small fraction of the animal kingdom. This also informs an unstated point of the dataset, which is that all of these animals have a common body plan. It is less clear how this would work for pose estimation in say, arthropods or fish. 
additional feedback: L298 ‘general pose estimation’ – general pose estimation is not well defined. This dataset is restricted to mammals.
review_title: Large animal dataset in need of some additional documentation

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposes a large-scale benchmark for animal pose estimation, consisting of over 10,000 images and 60 species. The data are organized following the taxonomic rank. Standard methods are benchmarked in this dataset.
strengths: * The paper is well written and is easy to read.
* This dataset is organized following the taxonomic rank. I think this is beneficial for future works.
* The authors benchmark three different tracks in this dataset, including SL, CD-TL, and DG track. Experiments seem technically sound.
weaknesses: I think the authors should explain how you choose the settings in the experimets. For example, in 4.4.1, why using Canidae and Felidae? If you choose other families, will the conclusion change?
correctness: All the claims made in the submission are correct.
clarity: The paper is well written.
relation to prior work: It is clearly discussed how this work differs from previous contributions.
additional feedback: Overall, I think this is a good paper. But the documentation should be further improved.
review_title: Official Review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper proposes AP-10k, a dataset for animal pose estimation, where the raw images originate from a collection of existing, publicly available datasets. It consists of 10k images from various animal families and species, with each instance being annotated with 17 keypoints. To date, this is the largest dataset for this task considering the number of samples and number of species. In addition, the dataset includes over 40k unlabeled images for semi- or self-supervised learning. The authors’ experiments aim to answer questions such as how models for human pose estimation transfer to this new dataset, as well as inter- and intra-family generalization.  
strengths: S1. The dataset will accelerate research in this area. Unlike previous datasets that consist of a specific category alone (or a limited number of categories), it includes a number of different animal species, which is essential in developing more generalizable methods. 

S2. While a significant number of collected images remain unlabeled, this can enable more semi-supervised learning approaches. 

S3. The paper is well-organized and easy to follow.

weaknesses: W1. I find the experiments conducted rudimentary, especially in Sec. 4.2 (w/ and w/o ImageNet pretraining) and 4.3 (pretraining on COCO for human pose estimation). The results here are unsurprising: ImageNet pretraining helps, and so does fine-tuning longer in the transfer experiment from human pose estimation on COCO. In addition, I would have liked to see experiments that answer perhaps less obvious questions. For example,
(a) cross-domain transfer across species, i.e. how well does the model trained on a subset of AP-10k transfer to new categories? Along these lines, the authors have already studied the generalizability (w/o finetuning), but have not investigated transfer performance. 
(b) also related is few-shot learning, i.e. learning pose estimation on new categories (with the same or even different keypoints) from only a few samples. The dataset is ideal for such experiments due to its long tail, i.e. since some animal families are already under-represented in the dataset.         

W2. Some more details regarding the annotation process would be welcome. The authors mention that domain knowledge may be required to provide labels for this task and that “13 well-trained annotators” were recruited. What is the background of the annotators and how were they trained prior to the annotation process?

W3. I have some concerns related to the taxonomy, please see “Correctness”. 

-- post rebuttal --
I have read the other reviews and the authors' responses. My mild concerns have been clarified and I have improved my score from 6 to 7 and recommend acceptance. The additional experiments are interesting and should be included in the paper, as promised. 
correctness: The dataset construction seems sound. However, there is one point regarding the data organization and taxonomy that remains unclear to me. Why are “Chihuahua”, “Collie”, “Dalmatian” and “German Shepherd” considered as different species than Dog (and similarly for cats)? There is a footnote explaining that this is due to different appearances and sizes. However, I cannot fully understand the reasoning behind choosing to single out these specific species. I would appreciate it if the authors could share some more details or observations that possibly led to this.  
clarity: The paper is clear and easy to follow. One point that could be improved is in Table 3: I suppose ResNet-50 and ResNet-101 are the backbones for the SimpleBaseline [42] mentioned in the text. This could be made more clear.   
relation to prior work: Related work is sufficiently discussed. In Section 4.4.3 there is also an interesting cross-dataset experiment with an already existing dataset [6]. 
additional feedback: typos:
L62: on the human pose estimation dataset →  on a … 

L91, L123: facilitate the research → facilitate research

L122: makes them difficult → makes it difficult

L123: regarding → considering 

L134: biological a prior → a biological prior

L211: start → starting
review_title: Technical Review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces a new data set for animal pose. The data set consists of over 10k images with 60 species of animals. The authors conducted experiments on the data sets in a supervised and cross-domain fashion.
strengths: The paper is well written and easy to follow.

The experiments are enough and the description of the dataset is detailed.
weaknesses: The evaluation method for the experiments is not mentioned (It is use mAP, but how it was used it is not specified). 
The baseline models are not detailed at all. A very short description should be introduced.

correctness: The data set is constructed in a sound way.
clarity: The paper is very well written and easy to follow. 
relation to prior work: Yes, the authors mentioned the previous work and compare to them in terms of  number of images and number of species.
additional feedback: It would be great if the final version would have the following items:

- The evaluation method for the experiments is not mentioned (It is use mAP, but how it was used it is not specified). 
- The baseline models are not detailed at all. A very short description should be introduced.
review_title: Review

Meta-review: All reviewers recommend acceptance. The AC sees no reason to overturn the reviewers' decision. Congratulations to authors!
Decision: Accept