SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In essence, ChimpACT emerges as an invaluable resource for researchers spanning the domains of primatology, comparative psychology, computer vision, and machine learning. It furnishes a comprehensive and varied array of annotations, paving the way for in-depth analysis of multifaceted chimpanzee behaviors and catalyzing the development of advanced machine learning algorithms. The inherent long-tail distribution not only presents a formidable challenge for chimpanzee identification and behavior recognition but also beckons explorations into few-shot learning in future endeavors.
To rigorously assess ChimpACT, we benchmark a suite of representative methods across the aforementioned three tracks: (i) detection, tracking, and ReID, (ii) pose estimation, and (iii) spatiotemporal action detection. Our computational framework leverages four NVIDIA GeForce RTX 3090 GPUs (24GB) for both training and evaluation across all tracks. In the subsequent sections, we delve into the implementation details, baseline methods, and evaluation metrics for each track.
4.1 Detection, tracking, and ReID
Setting We evaluate several prominent Multiple Object Tracking (MOT) algorithms on ChimpACT, including both classical methods such as SORT (Bewley et al., 2016), DeepSORT (Wojke et al., 2017), and Tracktor (Bergmann et al., 2019), as well as the state-of-the-art methods such as ByteTrack (Zhang et al., 2022), and OC-SORT (Cao et al., 2023). All implementations are based on the MMTracking (Contributors, 2020a) codebase. For those methods supporting flexible detection backbones, we trial two typical detectors, including the two-stage detector Faster R-CNN (Ren et al., 2015) and the one-stage detector YOLOX (Ge et al., 2021). Each method undergoes training for 10 epochs, adhering to the official configurations, which encompass optimizer settings, batch size, data augmentation techniques, and pre-trained models. Given that the three classical methods lack inherent ReID
Table 3: Results of the detection, tracking, and ReID track on the ChimpACT test set. The row highlighted in light blue is the performance reference on the human tracking dataset MOT-17 (Milan et al., 2016). - denotes not applicable.  $\oslash$  denotes unreported.
Method	Detector	ReID	HOTA ↑	MOTA ↑	MOTP ↑	IDF1 ↑	mAP ↑	nFP ↓	nFN ↓	nIDs ↓
SORT(Bewley et al., 2016)	Faster R-CNNYOLOX	ResNet-50	42.6±1.039.8±0.8	47.4±0.643.2±1.3	22.9±1.320.3±0.5	42.7±1.237.7±1.7	70.7±1.671.4±1.6	19.1±0.316.1±3.1	31.4±0.537.8±1.7	2.1±0.02.8±0.5
DeepSORT(Wojke et al., 2017)	Faster R-CNNYOLOX	ResNet-50	47.6±0.440.2±1.0	46.7±0.543.2±1.2	23.0±1.220.3±0.5	52.8±1.538.4±1.9	70.7±1.671.4±1.6	19.0±0.316.1±3.1	31.4±0.537.8±1.7	2.9±0.12.9±0.6
Tracktor(Bergmann et al., 2019)	Faster R-CNN	ResNet-50	49.5±0.7	50.5±1.1	22.6±1.1	55.6±1.2	70.7±1.6	13.8±0.5	35.2±0.7	0.5±0.0
QDTrack(Pang et al., 2021)	Faster R-CNN	-	50.3±3.2	54.2±4.6	22.2±1.4	55.8±3.6	77.8±2.0	19.7±3.6	24.6±0.8	1.4±0.2
ByteTrack(Zhang et al., 2022)	Faster R-CNNYOLOX	-	43.7±0.349.2±0.8	36.9±2.243.9±1.3	24.6±0.320.3±1.0	48.8±1.355.2±1.1	68.2±1.170.3±1.0	27.7±1.118.0±7.4	34.2±1.037.4±6.1	1.2±0.20.7±0.0
OC-SORT(Cao et al., 2023)	Faster R-CNNYOLOX	-	43.4±1.047.9±0.4	38.2±1.942.1±2.6	24.3±0.220.5±0.8	48.7±2.253.3±0.8	68.7±0.870.5±0.8	25.0±1.620.3±1.3	35.6±1.536.6±2.1	1.2±0.11.1±0.3
OC-SORT(Cao et al., 2023)	YOLOX	-	63.2	78.0	∅	77.5	∅	2.7	19.0	0.3
modules, we supplement with a dedicated ReID network built on ResNet-50 (He et al., 2016). The training curves of select methods (refer to Fig. A2a) affirm convergence within the training epochs.
We split the video clips in ChimpACT into  $80\%$  train,  $10\%$  validation, and  $10\%$  test. Both the train set and test set cover all the individuals. Models are trained on the training set, with performance metrics reported on the test set. We employ widely-accepted evaluation metrics, drawing from convention in human/object detection, tracking, and ReID (Bewley et al., 2016; Pang et al., 2021; Zhang et al., 2022). Specifically, we utilize (i) mean Average Precision (mAP) Lin et al. (2014) to gauge the detection accuracy, and (ii) the CLEAR metrics (Bernardin and Stiefelhagen, 2008) (MOTA, MOTP, FP, FN, IDs), IDF1 (Ristani et al., 2016), and HOTA (Luiten et al., 2021) to evaluate various facets of the tracking performance. It is worth noting that for FP, FN, and IDs, we report normalized values and denote these metrics as nFP, nFN, and nIDs, respectively.
Results Tab. 3 summarizes these tracking algorithms' performances on the ChimpACT test set. We conducted three runs for each method and reported the average and variance of these metrics. Notably, the variance across multiple runs is minimal, underscoring the robust reproducibility of our benchmarking. A holistic view of the results reveals that QDTrack (Pang et al., 2021) emerges as the top performer. However, it does suffer from a higher count of identity switches compared to other methods. In terms of detection performance, the YOLOX algorithm (Ge et al., 2021) stands toe-to-toe with Faster R-CNN (Ren et al., 2015). A discernible trend is evident among contemporary tracking methods, which seem to excel in identity association capabilities over their classical counterparts. This is corroborated by marked improvements in tracking metrics like IDF1 and IDs. Such a trend intimates that the latest tracking methods might be adept at maintaining consistent object identities, a pivotal aspect when tracking and analyzing individual trajectories within chimpanzee cohorts.
While the results garnered by the array of tracking algorithms are commendable, they still lag behind the benchmarks set on human-centric datasets (Zhang et al., 2022; Pang et al., 2021; Cao et al., 2023). This disparity can be attributed to challenges like the low contrast and low color variation of the body fur of chimpanzees, compounded by intricate self-occlusions. Nonetheless, this very observation accentuates the significance of ChimpACT. It not only offers a challenging arena for tracking algorithms but also stands as an ideal platform for pioneering and refining tracking methods tailored for chimpanzees and other non-human primates.
4.2 Pose estimation
Setting We benchmark several state-of-the-art human pose estimation methods on ChimpACT, including CPM (Wei et al., 2016), SimpleBaseline (Xiao et al., 2018), HRNet (Sun et al., 2019), DarkPose (Zhang et al., 2020). Broadly, human pose estimation methods can be bifurcated into two primary paradigms: heatmap-based and regression-based. We harness the MMPose (Contributors, 2020b) framework for implementing these methods. Please refer to Appx. C for more implementation details. All the models undergo training for 210 epochs, maintaining the official configurations for optimizers, batch sizes, and learning rates. To gauge any potential model overfitting, we present the validation curve on the AP metric in Fig. A2b, reassuringly suggesting an absence of overfitting.
Table 4: Results of the pose estimation track on ChimpACT test set. The row highlighted in light blue is the performance reference on the human pose estimation dataset COCO (Lin et al., 2014).  $\varnothing$  denotes unreported.
Method	Backbone	PCK@0.05	PCK@0.1	AP	\( AP^{50} \)	\( AP^{75} \)	\( AP^M \)	\( AP^L \)	AR
SimpleBaseline(Xiao et al., 2018)	ResNet-50	25.3±0.5	46.2±0.5	8.6 ±0.4	27.4±1.3	3.9 ±0.4	0.3±0.1	12.5±0.5	17.3±0.7
ResNet-101	26.2±1.0	46.4±1.1	8.7 ±0.4	27.5±0.6	4.2 ±0.5	0.3±0.0	12.9±0.2	17.7±0.4
ResNet-152	26.3±0.4	47.3±0.8	9.3 ±0.1	29.2±1.1	4.7 ±0.3	0.5±0.0	13.4±0.2	18.6±0.0
RLE(Li et al., 2021)	MobileNetV2	27.5±1.4	48.1±1.7	16.7±0.8	43.1±2.7	11.1±0.8	2.0±0.7	17.7±0.8	19.5±0.9
ResNet-50	28.2±1.7	47.1±3.1	16.3±2.5	41.2±6.9	11.4±1.4	1.3±0.8	17.4±2.8	20.0±1.6
ResNet-101	28.2±3.5	46.5±4.3	16.2±2.6	41.1±5.7	10.8±2.4	2.1±0.1	17.3±2.8	20.1±2.1
ResNet-152	30.0±1.3	48.4±2.2	18.1±2.8	43.0±7.9	13.5±0.6	1.4±0.3	19.2±3.2	22.3±1.1
CPM(Wei et al., 2016)	CPM	40.7±0.2	60.4±0.0	21.6±0.1	51.0±0.4	17.1±0.1	9.5±0.6	22.4±0.1	25.4±0.1
Hourglass(Newell et al., 2016)	Hourglass-4	44.6±0.5	60.8±0.1	20.6±0.3	48.9±0.1	16.0±0.4	4.6±0.1	23.7±0.6	28.2±0.2
MobileNetV2(Sandler et al., 2018)	MobileNetV2	39.8±0.4	59.4±0.4	19.4±0.1	48.5±0.6	14.3±0.8	2.3±0.1	20.6±0.1	23.2±0.1
SimpleBaseline(Xiao et al., 2018)	ResNet-50	43.3±0.2	61.7±1.2	22.1±0.2	51.5±0.4	17.7±0.2	3.7±0.4	23.4±0.2	26.3±0.1
ResNet-101	42.8±0.3	60.7±0.2	21.7±0.1	52.5±0.4	16.7±0.0	4.3±0.2	23.0±0.1	26.2±0.2
ResNet-152	43.9±0.4	61.6±0.1	22.7±0.4	53.4±0.6	18.3±0.4	5.3±0.5	23.9±0.4	27.1±0.1
HRNet(Sun et al., 2019)	HRNet-W32	48.6±0.9	65.6±0.6	25.9±0.4	58.2±0.8	22.1±0.4	6.1±0.4	27.0±0.6	30.3±0.5
HRNet-W48	47.3±0.2	64.5±0.2	25.1±0.1	57.2±0.6	21.0±0.1	6.9±0.9	26.2±0.3	29.6±0.1
DarkPose(Zhang et al., 2020)	ResNet-50	43.7±0.0	62.1±0.6	22.8±0.1	53.8±0.8	18.8±0.6	3.4±0.2	24.1±0.0	27.1±0.1
ResNet-101	43.1±0.9	61.2±1.4	22.1±0.3	52.6±0.6	17.6±0.7	4.0±0.4	23.4±0.3	26.5±0.3
ResNet-152	43.5±0.3	61.2±0.2	22.4±0.1	53.2±0.1	17.4±0.3	4.6±0.0	23.7±0.1	26.7±0.1
HRNet-W32	48.7±0.5	65.6±0.9	25.7±0.4	58.4±0.8	21.3±0.8	5.6±0.4	26.9±0.2	30.1±0.2
HRNet-W48	47.6±0.7	64.5±1.0	25.8±0.4	58.0±1.7	21.5±0.3	6.6±0.5	27.0±0.4	30.2±0.5
HRFormer(Yuan et al., 2021)	HRFormer-S	45.1±0.4	61.4±0.4	23.0±0.0	53.1±0.4	19.7±0.2	5.5±1.6	24.1±0.2	27.1±0.1
HRFormer-B	46.4±0.3	63.0±0.1	24.1±0.6	55.3±1.0	20.1±0.1	5.2±0.4	25.4±0.5	28.2±0.4
HRNet(Sun et al., 2019)	HRNet-W32	∅	∅	74.4	90.5	81.9	70.8	81.0	79.8
The train/test partitioning mirrors that of the first track. We use mAP with various thresholds, adhering to the conventions of human pose estimation (Lin et al., 2014). Additionally, we report the Percentage of Correctly estimated Keypoints (PCK) metric (Andriluka et al., 2014; Ng et al., 2022). PCK@α quantifies the fraction of accurately predicted keypoints within a distance threshold defined as  $\alpha \times max(\text{height}, \text{width})$ , derived from the bounding box of the chimpanzee. This metric is widely recognized for its accuracy in body joint localization in both human and animal pose estimation.
Results Tab. 4 consolidates these pose estimators' performances on the ChimpACT test set. Notably, the heatmap-based DarkPose (Zhang et al., 2020) with an HRNet (Sun et al., 2019) backbone emerges as the top-performing model. This trend aligns with observations in human pose estimation, where heatmap-centric methods (Wei et al., 2016; Xiao et al., 2018; Newell et al., 2016; Sun et al., 2019) predominantly lead the pack, attributed to their robustness against pose and appearance variations. However, the heatmap representation may be less accurate in scenarios where multiple joints are occluded or closely spaced, and it demands heftier computational and memory resources. Conversely, the newer regression-based methods (Li et al., 2021) are computationally leaner but tend to be more susceptible to overfitting and generally lag in performance.
These results underscore that the task of chimpanzee pose estimation is distinct and nuanced, and cannot be seamlessly addressed by merely repurposing human-centric pose estimation methods. We believe there are two primary reasons for this: (i) chimpanzees exhibit unique joint flexibility and a broader range of motion, and (ii) the visual texture and appearance of chimpanzee fur diverge significantly from human skin. These insights emphasize the need for chimpanzee specific pose estimation strategies.
4.3 Spatiotemporal action detection
Setting We benchmark four representative human action detection baselines on ChimpACT using the MMAction2 (Contributors, 2020c) codebase, including ARCN (Sun et al., 2018), LFB (Wu et al., 2019), and SlowFast with its variant SlowOnly (Feichtenhofer et al., 2019). All models undergo training for 20 epochs with a batch size of 32. Convergence is evident from the training curves
Table 5: Results of spatiotemporal action detection track on ChimpACT test set. The row highlighted in light blue is the performance reference on the human action dataset AVA (Gu et al., 2018). - denotes not applicable. "w. NL/Max/Avg LFB" denotes using non-local, max, or average LFB module. "w. Ctx" indicates using both the RoI feature and the global pooled feature for classification. "mAP," "mAPL," "mAPO," "mAPS," and "mAPo" represent the overall mAP and mAP for Locomotion, Object interaction, Social interaction, and others.
Method	Frame sampling	Module	mAP	\( \mathrm{mAP}_L \)	\( \mathrm{mAP}_O \)	\( \mathrm{mAP}_S \)	\( \mathrm{mAP}_o \)
ACRN (Sun et al., 2018)	\( 8\times 8\times 1 \)	\( 24.4_{\pm 0.5} \)	\( 58.7_{\pm 0.7} \)	\( 33.8_{\pm 1.7} \)	\( 14.7_{\pm 0.4} \)	\( 0.0_{\pm 0.0} \)
\( 4\times 16\times 1 \)	\( 23.9_{\pm 1.3} \)	\( 57.8_{\pm 0.4} \)	\( 35.0_{\pm 4.0} \)	\( 13.8_{\pm 1.6} \)	\( 0.0_{\pm 0.0} \)
LFB (Wu et al., 2019)	\( 4\times 16\times 1 \)	w: NL LFB	\( 22.0_{\pm 0.9} \)	\( 50.1_{\pm 0.8} \)	\( 32.3_{\pm 0.9} \)	\( 13.5_{\pm 1.6} \)	\( 0.6_{\pm 0.1} \)
\( 4\times 16\times 1 \)	w: Max LFB	\( 23.2_{\pm 0.7} \)	\( 45.0_{\pm 1.5} \)	\( 31.2_{\pm 0.8} \)	\( 17.7_{\pm 1.4} \)	\( 0.5_{\pm 0.0} \)
\( 4\times 16\times 1 \)	w: Avg LFB	\( 21.3_{\pm 1.6} \)	\( 45.0_{\pm 3.6} \)	\( 29.8_{\pm 1.1} \)	\( 14.7_{\pm 2.6} \)	\( 0.5_{\pm 0.0} \)
SlowOnly (Feichtenhofer et al., 2019)	\( 8\times 8\times 1 \)	\( 20.9_{\pm 1.9} \)	\( 48.1_{\pm 7.0} \)	\( 36.2_{\pm 2.8} \)	\( 11.5_{\pm 1.0} \)	\( 0.0_{\pm 0.1} \)
\( 4\times 16\times 1 \)	\( 19.2_{\pm 1.1} \)	\( 47.0_{\pm 2.5} \)	\( 28.3_{\pm 2.5} \)	\( 11.0_{\pm 1.2} \)	\( 0.0_{\pm 0.1} \)
\( 8\times 8\times 1 \)	w: Ctx	\( 22.3_{\pm 1.9} \)	\( 52.3_{\pm 3.2} \)	\( 31.2_{\pm 1.3} \)	\( 13.8_{\pm 2.4} \)	\( 0.1_{\pm 0.1} \)
\( 4\times 16\times 1 \)	w: Ctx	\( 21.4_{\pm 0.9} \)	\( 47.6_{\pm 2.0} \)	\( 33.0_{\pm 1.2} \)	\( 13.2_{\pm 2.2} \)	\( 0.2_{\pm 0.1} \)
SlowFast (Feichtenhofer et al., 2019)	\( 8\times 8\times 1 \)	\( 21.9_{\pm 1.0} \)	\( 53.0_{\pm 0.7} \)	\( 30.6_{\pm 2.2} \)	\( 12.9_{\pm 1.2} \)	\( 0.0_{\pm 0.1} \)
\( 4\times 16\times 1 \)	\( 22.0_{\pm 0.8} \)	\( 52.9_{\pm 2.3} \)	\( 33.1_{\pm 2.3} \)	\( 12.6_{\pm 0.9} \)	\( 0.0_{\pm 0.0} \)
\( 8\times 8\times 1 \)	w: Ctx	\( 24.3_{\pm 0.6} \)	\( 56.8_{\pm 1.6} \)	\( 31.5_{\pm 2.0} \)	\( 15.6_{\pm 0.8} \)	\( 0.1_{\pm 0.1} \)
\( 4\times 16\times 1 \)	w: Ctx	\( 24.1_{\pm 0.9} \)	\( 56.6_{\pm 2.0} \)	\( 34.7_{\pm 2.7} \)	\( 14.6_{\pm 0.4} \)	\( 0.1_{\pm 0.1} \)
SlowFast (Feichtenhofer et al., 2019)	\( 8\times 8\times 1 \)	25.8	-	-	-	-
in Fig. A2c. We maintain consistent optimizers and learning rates as in official implementations. Ground-truth bounding boxes for each chimpanzee are provided during both training and testing, as per Tang et al. (2020). Please refer to Appx. C for further details on ablative modules.
We adopt the same train-test split as previous tracks. Performance is gauged using mAP across 23 action classes, as per standard (Feichtenhofer et al., 2019; Tang et al., 2020). Additionally, we evaluate the mAP within the four behavioral types separately.
Results Tab. 5 summarizes the action detection algorithms' performances on the ChimpACT test set. The overall mAP aligns with results on human action datasets, underscoring the feasibility of automated action detection for video coding and further analyses. Locomotion behaviors achieve a notably higher mAP, likely due to their solitary nature and distinct patterns. Conversely, Conversely, the "others" category registers the lowest mAP, attributed to its limited data—comprising just  $0.14\%$  of action instances across two fine-grained classes. This imbalance suggests the potential benefit of few-shot learning methods in the future. The results highlight both the promise and areas for improvement in the dataset, positioning it as a valuable platform for advancing spatiotemporal action detection algorithms. We anticipate that ChimpACT will further studies into the social dynamics of non-human primates in semi-naturalistic environments.
In this work, we introduced ChimpACT, a novel longitudinal video dataset capturing the intricate behaviors of group-living chimpanzees, focusing on the juvenile chimpanzee, Azibo. Our meticulous annotations and diverse social interactions within the dataset offer a unique view into the world of our closest evolutionary relatives. Through comprehensive experiments, we underscored the challenges and nuances of applying human-centric computer vision algorithms to the distinct behaviors and interactions of chimpanzees. The dataset's depth, combined with its long-tail distribution, not only emphasizes its significance but also paves the way for interdisciplinary research bridging primatology, comparative psychology, computer vision, and machine learning. By making this resource available, our aspiration is to catalyze advancements in video understanding, inspire the research community to craft specialized techniques for non-human primates and deepen our collective insights into their intricate social fabric and dynamics.
Limitation and future work ChimpACT is based on captive chimpanzees living in a semi-natural environment, limiting the observable range of behaviors. Natural foraging, responses to predators, and intergroup encounters are absent. Focusing on Azibo overrepresents certain individuals and underrepresents others, limiting the assessment of the full social network. Nevertheless, we plan to contribute more data and labels to create a larger and more comprehensive chimpanzee dataset.
The authors would like to thank the Wolfgang Kohler Primate Research Center, BasicFinder CO., Ltd., and Keyue Zhang for annotations and quality check, Zihao Yin for discussions and preliminary experiments on the chimpanzee detection models, Guangyuan Jiang and Yuyang Li for their technical support on the GPU cluster, and NVIDIA for their generous support of GPUs and hardware. X. Ma, J. Su, W. Zhu, Y. Zhu, and Y. Wang are supported in part by the National Key R&D Program of China (2022ZD0114900), and Y. Zhu is supported in part by the Beijing Nova Program and the National Comprehensive Experimental Base for Governance of Intelligent Society, Wuhan East Lake High-Tech Development Zone.
Altmann, J. (1974). Observational study of behavior: sampling methods. Behaviour, 49(3-4):227-266. 3,5
Andriluka, M., Iqbal, U., Insafutdinov, E., Pishchulin, L., Milan, A., Gall, J., and Schiele, B. (2018). Posetrack: A benchmark for human pose estimation and tracking. In Conference on Computer Vision and Pattern Recognition (CVPR). 6
Andriluka, M., Pishchulin, L., Gehler, P., and Schiele, B. (2014). 2d human pose estimation: New benchmark and state of the art analysis. In Conference on Computer Vision and Pattern Recognition (CVPR). 3, 9
Bain, M., Nagrani, A., Schofield, D., Berdugo, S., Bessa, J., Owen, J., Hockings, K. J., Matsuzawa, T., Hayashi, M., Biro, D., et al. (2021). Automated audiovisual behavior recognition in wild primates. Science Advances, 7(46):eabi4883. 4
Bain, M., Nagrani, A., Schofield, D., and Zisserman, A. (2019). Count, crop and recognise: Fine-grained recognition in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. 2, 3, 4
Baker, T. A. (2022). Wolfgang köhler primate research center. Encyclopedia of Animal Cognition and Behavior, page 7310. 5
Bala, P. C., Eisenreich, B. R., Yoo, S. B. M., Hayden, B. Y., Park, H. S., and Zimmermann, J. (2020). Automated markerless pose estimation in freely moving macaques with openmonkeystudio. Nature Communications, 11(1):4560. 2, 3
Bard, K. A., Dunbar, S., Maguire-Herring, V., Veira, Y., Hayes, K. G., and McDonald, K. (2014). Gestures and social-emotional communicative development in chimpanzee infants. American Journal of Primatology, 76(1):14-29. 5
Bergmann, P., Meinhardt, T., and Leal-Taixe, L. (2019). Tracking without bells and whistles. In International Conference on Computer Vision (ICCV). 7, 8, A5
Bernardin, K. and Stiefelhagen, R. (2008). Evaluating multiple object tracking performance: the clear mot metrics. EURASIP Journal on Image and Video Processing, 2008:1-10. 8
Bewley, A., Ge, Z., Ott, L., Ramos, F., and Upcroft, B. (2016). Simple online and realtime tracking. In IEEE International Conference on Image Processing (ICIP). 4, 7, 8, A5
Boesch, C. (1996). The emergence of cultures among wild chimpanzees. In Proceedings-British Academy. 6
Bohnslav, J. P., Wimalasena, N. K., Clausing, K. J., Dai, Y. Y., Yarmolinsky, D. A., Cruz, T., Kashlan, A. D., Chiappe, M. E., Orefice, L. L., Woolf, C. J., et al. (2021). Deepethogram, a machine learning pipeline for supervised behavior classification from raw pixels. *Elife*, 10:e63377. 4
Cao, J., Pang, J., Weng, X., Khirodkar, R., and Kitani, K. (2023). Observation-centric sort: Rethinking sort for robust multi-object tracking. In Conference on Computer Vision and Pattern Recognition (CVPR). 7, 8, A5
Contributors, M. (2020a). MMTracking: OpenMMLab video perception toolbox and benchmark. https://github.com/open-mmlab/mmtracking.7
Contributors, M. (2020b). Openmmlab pose estimation toolbox and benchmark. https://github.com/open-mmlab/mmpose.8
Contributors, M. (2020c). Openmmlab's next generation video understanding toolbox and benchmark. https://github.com/open-mmlab/mmaction2.9
A Additional details on ChimpACT
A.1 Ethogram
We detail the ethogram definition in Tab. A1, which systematically describes the daily behaviors of chimpanzees.
2015 indoor
2016 indoor
2016 outdoor
2017 indoor
2017 outdoor
2018 indoor
2018 outdoor
Figure A1: Example frames from the ChimpACT dataset. ChimpACT possesses rich social interactions of the complex everyday life of group-living chimpanzees and contains several environmental enrichment.
A.2 Dataset details
Collection and organization 405 hours of video footage of the Leipzig A-group chimpanzees were collected between 2015 and 2018. To create a representative sample of the footage, 163 video clips were selected, with 15, 35, 86, and 27 clips taken from each year. These video clips cover the four seasons. Each clip is 1000 frames long, with only 3 clips being shorter than 1000 frames. Visual examples from six clips, featuring both indoor and outdoor enclosures, are shown in Fig. A1. The dataset covers a diverse range of physical scenarios, camera views, and social behaviors, as demonstrated in these examples. For instance, in the third row of the figure, an adult chimpanzee is shown grooming an infant chimpanzee in her arms, while later on, the same infant is nursed.
Annotation process and quality The annotation process was conducted using BasicFinder CO., Ltd.'s private labeling platform, which involved a team of 15 annotators and 2 managers. Prior to commencing the annotation work, our team developed comprehensive guidelines that explicitly outlined the requirements for labeling. These guidelines covered several aspects, including:
(i) Assigning a bounding box for each chimpanzee in the image. (ii) Specifying the visibility of the bounding boxes. (iii) Assigning tracking IDs to each bounding box for tracking purposes. (iv) Localizing 2D keypoints within each bounding box. (v) Indicating the visibility of each 2D keypoint. (vi) Assigning behavior labels for each bounding box.
To ensure that the annotators followed these guidelines accurately, the project managers provided training based on the guidelines. Following the training, the annotators performed a trial annotation on a small dataset. We actively sought feedback from the annotators during this phase, which allowed us to address any issues and make necessary improvements. We conducted a thorough review of the trial annotations to verify that the quality met our standards.
During the trial labeling phase, we reached out to three labeling companies and ultimately selected BasicFinder CO., Ltd. based on their exceptional labeling quality. It is worth noting that BasicFinder CO., Ltd. has previously led the annotation efforts for the BDD100K (Yu et al., 2020) dataset, which is a substantial dataset used for autonomous driving purposes. This experience demonstrates their ability to maintain high annotation standards for complex and extensive datasets. Consequently, their involvement improves the reliability of our ChimpACT dataset annotations as well.