Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces a new large scale dataset for multi-animal 3D pose estimation. It consists of approximately 22 million frames capturing interactions between 18 pairs of laboratory rats. The videos are captured from 30 viewpoints, using 6 cameras placed at 5 different locations and annotations are collected using body markers that provide information about the heads, spines, trunks, hips, and shoulders of the animals. The videos are also annotated for the action recognition task, with 11 behavioural categories and 3 inter-animal interaction categories. Furthermore, a strong baseline for 3D pose estimation is provided.
strengths: In stark contrast with previous multi-animal pose estimation datasets, the main strength of this dataset is that it provides three-dimensional annotations. Additionally, the number of annotated videos and their dimension are bigger than similar datasets. Finally, the authors performed a strong benchmark by adapting DANNCE[1], a model for individual animal pose estimation to the multi-animal setting.

[1] Timothy W Dunn, Jesse D Marshall, Kyle S Severson, Diego E Aldarondo, David GC Hildebrand, Selmaan N Chettih, William LWang, Amanda J Gellis, David E Carlson, Dmitriy Aronov, et al. Geometric deep learning enables 3d kinematic profiling across species and environments.
weaknesses: As the authors already stated, one of the main concerns consists in the number of markers used to keep track of the pose. Since they do not cover all the important points on the body of the animal, there might be problems in identifying the corresponding behaviour. Another important issue, that is a general challenge for this type of dataset, is the occlusion that inevitably happens while the animals interact with each other. Furthermore, sharing more insights about the way the labels of the behaviour were selected would better position this work.  Were the categories defined by the authors or are they a standard in the field? Also, a benchmark on the behaviour classification would have been nice.
correctness: Regarding the pose estimation part, the dataset is constructed in an objective way, using body landmarks to collect the corresponding information. With respect to the specific and interaction behavioural categories, I consider more details should have been provided.
clarity: The paper is well organized and the writing is clear. However, I would suggest changing the name of Section 3, as in my opinion "Results" sounds too general and not really correlated to creating a dataset. 
relation to prior work: The differences with previous works are clearly discussed in the related work section. It presents 2D and 3D pose estimation datasets, both in terms of single and multi-animal settings, together with their limitations. The contributions and the importance of the current dataset are also mentioned. 
additional feedback: -
review_title: The PAIR-R24M Dataset for Multi-animal 3D Pose Estimation

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: To address the lack of standardized benchmarks for multi-animal 3D pose estimation, the author collects the first large-scale multi-animal 3D pose benchmark with multiple viewpoints and distinct rat pairs. Ground truth 3D pose annotation is provided via motion capture. Also, behavioral labels are given in a semi-automated way. A strong baseline for multi-animal 3D pose estimation without motion capture is given via an adapted version of DANNCE.
strengths: a. It is valuable as the first multi-animal 3D pose benchmark to advance the progress in this area.

b. The large scale and diversity of the collected data could advance progress in this area.

c. The semi-automated labeling of behavior is interesting and meaningful for analysis on animals, and further fine-grained labeling of behavior is promised which would be very helpful.
weaknesses: a. The lack of hand, foot and limb annotation limits full understanding of animal pose as the authors stated in Section 4. I'm looking forward to the completion of these annotations.

b. It seems to me that the markers totally occlude the rats' head. Would it harm the generalization when the rats wear no markers?

Updated after rebuttal:
The authors response addressed my main concerns.
correctness: I believe the claims are correct. The dataset is constructed soundly. The experiment is well-designed and appropriate.
clarity: The paper is overall well-written and clear to follow.
relation to prior work: Sufficient discussion on prior works is given.
additional feedback: It would be better if some of the failure cases (frames discarded for severe occlusion) could be illustrated.

Updated after rebuttal:
The authors promised to provide these samples.
review_title: Review on Submission 172

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes a new benchmark for multi-animal 3D pose and behaviour estimation. It is the first benchmark to study multi-animal in 3D cases, while providing individual and interaction behavioral labels.
strengths: + The paper is well written and is easy to read.
+ The paper provides large scales 3D pose annotations for rats.
+ The paper also provides behaviour annotations.
+ The way to collect annotations is clearly elaborated.
+ The statistics details of the dataset is provided.
weaknesses: - The authors mention that they recorded extra data when paired with animals not bearing markers, which can prevent swaps of motion capture markers across animals. (Line 156-158) Does it mean that currently, there exist lots of swap problems in the dataset? I think it will damage the reliability of this benchmark.
- Is it necessary to provide behavioural labels in each frame? I think not all motion in a time period relates to the labelled behaviour.
- The occlusions that lasted for over 3s will be dropped. The data preserved in the final datasets will be the cases that are less occluded. Therefore, the benchmark will be not challenging enough.
correctness: All the claims made in the submission are correct.
clarity: The paper is well written.
relation to prior work: It is clearly discussed how this work differs from previous contributions.
additional feedback: Line 261, the authors mention human annotations of distal limbs can be added in future datasets. But I think manually annotating the 3D keypoints is quite difficult, especially under sub-mm accuracy. Is there any technique to ease this problem?
review_title: Official Review of Paper172

Meta-review: Reviewers have an overall positive opinion regarding the paper contribution. Overall the provided data set is large, unique in 3D animal poses, and provides useful complementary annotations. This can be a valuable resource for the community.
Decision: Accept