ChimpACT: A Longitudinal Dataset for Understanding Chimpanzee Behaviors
Authors: Xiaoxuan Ma
Stephan Paul Kaufhold
Jiajun Su
Wentao Zhu
Jack Terwilliger
Andres Meza
Yixin Zhu
Federico Rossano
Yizhou Wang
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Computer Vision
Non-human primates behavior
TL;DR: We propose ChimpACT, a comprehensive dataset for investigating chimpanzee social behavior in a semi-naturalistic and social environment with abundant annotations of instance bounding boxes, body poses, and spatial-temporal action labels.

Abstract: Understanding the behavior of non-human primates is crucial for improving animal welfare, modeling social behavior, and gaining insights into distinctively human and phylogenetically shared behaviors. However, the lack of datasets on non-human primate behavior hinders in-depth exploration of primate social interactions, posing challenges to research on our closest living relatives. To address these limitations, we present ChimpACT, a comprehensive dataset for quantifying the longitudinal behavior and social relations of chimpanzees within a social group. Spanning from 2015 to 2018, ChimpACT features videos of a group of over 20 chimpanzees residing at the Leipzig Zoo, Germany, with a particular focus on documenting the developmental trajectory of one young male, Azibo. ChimpACT is both comprehensive and challenging, consisting of 163 videos with a cumulative 160,500 frames, each richly annotated with detection, identification, pose estimation, and fine-grained spatiotemporal behavior labels. We benchmark representative methods of three tracks on ChimpACT: (i) tracking and identification, (ii) pose estimation, and (iii) spatiotemporal action detection of the chimpanzees. Our experiments reveal that ChimpACT offers ample opportunities for both devising new methods and adapting existing ones to solve fundamental computer vision tasks applied to chimpanzee groups, such as detection, pose estimation, and behavior analysis, ultimately deepening our comprehension of communication and sociality in non-human primates.

ChimpACT: A Longitudinal Dataset for
Understanding Chimpanzee Behaviors
Xiaoxuan Ma
Stephan P. Kaufhold
Jiajun Sur sujiajunêpku edu. en
maxiacxuan@pku edu. en
spkaufho@ucsd.cm
Wentao Zhu wtzhu@pku. edu. en
Jack Terwilliger jterwillggrrucsd.eer
Andres Meza anmeza@ucsd. edu
Yixin Zhu 35,04
Federico Rossano 2,04
Yizhou Wang' 3,3,4
mokin.zhu@pku eetepkuuedu.cco on
frossano@ucsd. edu
y zhou wwng@pwu..ua.... edu. cn
X. Ma, S. Kaufhold, and J. Su contributed equally.
X Corresponding authors
CFCS, School of Computer Science, Peking University, China
Department of Cognitive Science, University of California, San Diego, USA
3 Institute for Artificial Intelligence, Peking University, China
Nat'l Eng. Research Center of Visual Technology, China PKU-WUHAN Institute for Artificial Intelligence, China
https: http:///shrirraameeee github i ChimpAcT/
Swedk resting
Azbo being carted
being cariedd
Comec TISting
Figure I: Sample frames and annotations from ChimpACT clip. While we also annotate visibility for both the bounding box and the kcypoint, these are omitted here for clarity.
Abstract
Understanding the behavior of non-human primates is crucial for improving animal welfare, modeling social behavior, and gaining insights into distinctively human and phylogenetically shared behaviors. However, the lack of datasets on non-human primate behavior hinders in-depth exploration of primate social interactions, posing challenges to research on our closest living relatives. To address these limitations, we present Chi ChimpACT, comprehensive dataset for quantifying the longitudinal behavior and social relations of chimpanzees within social group. Spanning from 2015 to 2018, ChimpACT features videos of a group of over 20 chimpanzees residing at the Leipzig Zoo, Germany, with particular focus documenting the developmental trajectory of one young male, Azibo. Ch ChimpACT is both com- prehensive and challenging, consisting of 163 videos with cumulative 160,500 frames, each richly annotated with detection, identification, pose estimation, and fine-grained spatiotemporal behavior labels. We benchmark representative methods of three tracks on Ch impACT: (i) tracking and identification, (ii) pose estimation, and (iii) spatiotemporal action detection of the chimpanzees. Our experiments reveal that ChimpACT offers ample opportunities for both devising new methods and adapting existing ones solve fundamental computer vision tasks applied to chimpanzee groups, such as detection, pose estimation, and behavior analy-
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

sis, ultimately deepening our comprehension of communication and sociality in non-human primates.
Introduction
Studying the behavior of non-human primates essential for gaining evolutionary insights (Langer- graber et al., 2012), conducting biomedical research (Schapiro et al., 2005), and improving animal welfare (Dawkins, 2003; Gonyou, 1994). Furthermore, given the close phylogenetic proximity be- tween humans and non-human primates, it provides ethically sound and effective avenue to probe the roots of human sociality (The Chimpanzee Sequencing and Analysis Consortium, 2005). Tradi- tional field research typically requires researchers to enter wildlife conservation areas for extended durations, sometimes spanning multiple years. This involves habituating primate groups to human presence, capturing video footage, and laboriously manually coding these videos for subsequent statistical analysis (Hobaiter et al., 2017; Fröhlich et al., 2020; Surbeck et al., 2017; Luncz et al., 2018; Sirianni et al., 2015). While video coding is heralded as the gold standard for distilling rich, nuanced behavioral patterns (Wiltshire et al., 2023), its practical utility hinges on the efficiency of the coding process. This not only demands researchers with specialized expertise but is also prone to attentional biases.
Recent strides in computer vision offer promise for the automated analyses of non-human primate behaviors, especially those of chimpanzees. Nevertheless, the scarcity of high-quality longitudinal datasets remains bottleneck. Assembling chimpanzee behavioral data is formidable endeavor, necessitating substantial resources and expertise. This process entails continuous video recording and meticulous manual annotation, with keen emphasis annotation accuracy and consistency. While some datasets (Marks et al., 2022; Bala et al., 2020) confine subjects to indoor enclosures, resulting in atypical and constrained environments, others resort to sourcing and labeling chimpanzee images online (Labuguen et al., 2021; Desai et al., 2022; Ng et al., 2022; Yao et al., 2023). Unfortunately, these often overlook the intricate social dynamics inherent chimpanzee groups, hindering comprehensive study of their social behaviors and social relationships.
Addressing the existing limitations, introduce ChimpACT, comprehensive longitudinal dataset tailored for the in-depth study of chimpanzee social behavior in a semi- natirnaauralistic setting, replete with annotations of instance bounding boxes, body poses, and spatial-temporal action labels. A comparison with other datasets is provided in Tab. 1. ChimpACT encompasses footage of a specific chimpanzee group residing Leipzig Zoo, Germany, with particular focus juvenile male named Azibo (refer to Fig. 1). The data, gathered between 2015 and 2018, employs focal sampling
Table 1:
Comparison of ChimpACT with existing primate behavioral datasets. Aquare-bracketed numbers denote label counts for the chimpanzee category. Ø denotes undocumented. the "Species" row, G represents general, P for primates, M for macaque, and C for chimpanzee. In the "Source" row, stands for Internet, for 200, C for cage, W for wild, and CP for captive.
Track
Track 2 pose estimation
Track 3
Dataset
Species
detection, tracking, RelD
action recognition
Source
frame
box
track
frame
pose #
track dim.
class #
label
AP-10K
13,028
G
X
X
X
10,015
X
2D
X
X
I
al., 2021)
[<500]
AingaoK
99,297
30,100
(Ng et al., 2022)
G
X
33,099
[576]
X
140
[2]
I
OpenApePose
71,868
(Desai et al., 2022)
X
X
71,868
[18,010]
X
X
OpenMonkeyChalleng
111.529
P
X
X 111,529
x 2D
X
X
I& Z
(aa al., 2023)
[<10,000]
OpenMonkeyStudio
33,
C
(Bala 2020)
M
X
X
194,518
3D
X
X
(6.7m²)
MacaquePose
16,393
(Labuguen al., 2021)
M
X
13,083
[0]
X
X
& Z
SIPEC
2,200
C
M
4
191
/
X
X
Ø
(Marks al., 2022)
(15m²)
CCR (Bain et al., 2019)
936.914 1,937,585
/
X
X
W
ChimpACT
C
23
160,500
56,324
/
16,028
56,324
/
2D
23
64,289
CP
(Ours)
(4400m²

---

(Altmann, 1974). Born in April 2015, Azibo' has been living in the group since birth, providing unique perspective on the development of an individual within a chimpanzee group characterized by well-defined kin relationships. (also depicted in Fig. 2a). The footage covers the daily lives of over 20 chimpanzees in a group, aggregating to 163 video recordings, approximately 160,500 frames, and spanning around 2 hours.
Our annotations on ChimpACT are extensive, marking each individual's detection, tracking, identifi- cation, pose estimation, and spatiotemporal action detection. Sample frames with their corresponding annotations are illustrated in Fig. Each chimpanzee's 's identity is confirmed by a seasoned behav- ioral researcher familiar with the Leipzig chimpanzees, ensuring data precision and trustworthiness. Crucially, we employ an ethogram (detailed in Fig. 2b) devised by the same expert for fine- fine-grained action labels. To our knowledge, Ch ChimpACT is the first to furnish ethogram annotations for the machine learning and computer vision community. This bespoke ethogram delineates behaviors into four categories: locomotion, object interaction, social interaction, and others, with each encompassing several detailed actions we diligently annotate.
While ad advancements in computer vision have notably addressed human-centric tasks, such as human pose estimation (Sun et al., 2019; Xiao et al., 2018), the dearth of chimpanzee datasets has curtailed progress on chimpanzee-specific challenges. Despite their genetic closeness to humans (The Chim- panzee Sequencing and Analysis Consortium, 2005), deciphering chimpanzee behaviors is intricate due to their unique morphology, appearance, and keypoint articulation. Highlighting the importance of crafting sophisticated chimpanzee perception models, we evaluate prominent human perception methods on three tracks: (i) detection, tracking, and identification (RelD), (ii) pose estimation, and (iii) spatiotemporal action detection. Our findings underscore Chi ChimpACT's potential as a platform for the community to pioneer advanced techniques for better perception of the chimpanzees and ultimately contribute to a deeper understanding of non-human primates.
2 Related work
Computer vision for animals A myriad of datasets and benchmarks have emerged, harnessing computer vision techniques to advance animal research. For instance, 3D-ZeF20 (Pedersen et al., 2020) introduces 3D tracking of zebrafish to the MOT benchmarks. AnimalTrack (Zhang et al., 2023) emphasizes multi-animal tracking across a spectrum of species. AP-10K (Yu et al., 2021) and APT- 36K (Yang et al., 2022) venture into animal pose estimation for diverse species. AnimalKingdom (Ng et al., 2022) extends its focus to fine-grained multi-label action recognition. Moreover, several studies have delved into multi-agent behavior understanding from a social interaction perspective (Sun et al., 2021, 2023). Distinctively, ChimpACT stands out as a holistic benchmark, encompassing three varied downstream tasks and boasting rich annotations of social interactions.
Human video datasets In contrast to animal-centric video datasets, a more substantial collection is tailored to human subjects, addressing diverse numan-centrin video understanding tasks. For instance, the MOT Challenge (Milan et al., 2016) is curated for multi-person tracking. Other benchmarks like COCO (Lin et al., 2014) and MPII (Andriluka al., 2014) cater to human pose estimation. Meanwhile, datasets such as Kinetics (Kay et al., 2017), ActivityNet (Fabian Caba Heilbron and Niebles, 2015), and AVA (Gu et al., 2018) are dedicated human action recognition. With Ch ChiCT, we encompass analogous tasks but introduce challenges specific to chimpanzee behavior.
Datasets on primate behavioral understanding Most existing primate datasets are tailored towards individual primate detection and pose estimation. These either stem from confined indoor settings (Bala et al., 2020; Marks et al., 2022) are amassed and labeled from online sources (Labuguen et al., 2021; Desai et al., 2022; Ng et al., 2022; Yao et al., 2023). The former can induce atypical behavioral patterns, while the latter often omits longitudinal interactions, rendering them suboptimal for analyzing chimpanzee social dynamics. A notable exception is the CCR dataset (Bain al., 2019), chronicling 13 chimpanzees in the Bossou forest over two years. Yet, it primarily focuses on individual detection and recognition, lacking behavioral annotations, which limits its efficacy for probing the social nuances of wild primates. Tab. 1 offers comprehensive comparison. The narrow focus of most primate datasets on singular tasks restricts their breadth and adaptability to diverse research inquiries. Contrarily, Chi ChimpACT presents a multifaceted approach, encompassing identities, kinship, detection labels, pose annotations, ethograms, and line-graine action labels. This richness
Details about Azibo can be found at https: //t iinuurlyuir..m com/ az azibo-chiipp/
3

---

1970
Corn
Preuige
Natsscha
Maja
Derlen
mediio
Swela
Siplegigg
Ethagram
2000
Bambari
Lone
Ted
Bangolo
Bagging
Yours
AsDo
(a)
(b)
Figure 2: (a) Kinship of the observed chimpanzee group. Rectangles and ellipses represent males and females, respectively, with arrows flowing from the parents to the child. Their vertical position relative the time axis indicates the year birth. (b) Ethogram with annotated behaviors.
positions it as an indispensable tool for devising advanced chimpanzee behavior analysis methods and enriching the overarching comprehension of primate behavior.
Methods for primate behavioral analysis Deciphering primate behavior is instrumental in un- derstanding their social dynamics and cognitive abilities. Behavioral analysis often encompasses subtasks like individual detection, tracking, and identification (Bain et al., 2019; Marks et al., 2022), pose estimation (I (Labuguen et al., 2021; Desai et al., 2022; Mathis er al., 2018; Wiltshire et al., 2023), and behavior recognition (Ng al., 2022; Bain et al., 2021). While each task has specialized techniques, many are rooted in human behavioral research. Numerous algorithms exist for human tracking (Bewley et al., 2016; Pang et al., 2021), pose estimation (Sun et al., 2019; Xiao et al., 2018), and behavior recognition (Feichtenhofer et al., 2019). However, due to the dearth of primate datasets, primate behavioral analysis often repurposes algorithms designed for humans, including:
Detection, tracking, and ReID identify individual primates in videos, often leveraging established object human detection algorithms like Mask-RCNN (He et al., 2017). For instance, SIPEC (Marks et al., 2022) employs Mask-RCNN with ResNet backbone (He et al., 2016) to track and segment macaque. Bain et al. (2019) utilize CNNs to crop and identify individual chimpanzees.
Pose estimation discerns primate poses, frequently adapting human pose estimation methods like SimpleBaseline (Xiao et al., 2018). DeepLabCut (Mathis et al., 2018; Lauer et al., 2022), for instance, employs ResNet-50 with ImageNet pre-trained weights for 2D animal pose estimation. SIPEC (Marks et al., 2022) modifies SimpleBaseline for 2D macaque poses.
Behavior recognition identifies primate actions and interactions. Contemporary methods (Bain al., 2021; Bohnslav al., 2021) often derive from human action recognition algorithms like SlowFast (Schindler and Steinhage, 2021). Notably, Bain et al. (2021) integrates audio cues for classifying two simple non-interactive behaviors: nut cracking and buttress drumming. In contrast, ChimpACT encompasses over 20 daily behaviors under an ethogram hierarchy, capturing both solitary actions and intricate social interactions.
In essence, primate behavioral analysis is a multifaceted endeavor, intertwining computer vision, machine learning, and primatology. The advent of Ch impACT marks significant stride towards unraveling the intricate social tapestry of our primate kin.
3 ChimpACT
3.1 Dataset description
ChimpACT comprises about 2-hour video footage of chimpanzees recorded at the Leipzig Zoo in Germany between 2015 and 2018. The videos focus on one male chimpanzee, Azibo, who was born in April 2015 to Swela and has lived with the A-chimpanzee group² at the Leipzig Zoo ever since.
'The A chimpaazzee group is among the most extensively studied zoo-residing chimpanzee cohorts. Its members have been subjects of both behavioral and cognitive studies, spanning observational and experimental

---

The longitudinal observation of Azibo offers a rare lens into his behavioral evolution, social dynamics, and gtra-group relationships. With over 20 individuals in the group, ChimpACT serves as a treasure trove of insights into chimpanzee behavior and social intricacies. Key attributes of ChimpACT are delineated below.
Longitudinal data Spanning four years, Ch ChimpACT chronicles the life of a stable zoo-residing chimpanzee group, offering rare glimpse into the nuances of chimpanzee social behavior develop- ment. Tracking the growth and interactions of a young chimpanzee within this group sheds light on chimpanzee socialization, the evolution of social skills (Matsuzawa, 2013), the formation of social bonds and integration into the dominance hierarchy (Matsuzawa et al., 2006), and the acquisition of group-specific cultural behaviors (Van Leeuwen, 2021; Musgrave et al., 2021).
Semi-naturalistic and social environment The videos in Ch ChimpACT capture chimpanzees in their semi-naturalistic habitats at Leipzig Zoo, split between indoor (96 videos) and outdoor (67 videos) enclosures. The indoor space, spanning roughly 400 m² features a plethora of environmental enrichments, ranging from wooden climbing structures and hammocks to vegetation and foraging boxes. When weather permits, the chimpanzees have access to a 4000 m outdoor area, replete with vegetation, surrounded by an artificial river, and complemented by enrichments similar the indoor space. This blend of environments ensures the dataset's ' relevance for both naturalistic and artificial environments. The multifaceted physical and social surroundings of the chimpanzees further imbue the dataset with intricate behaviors and social dynamics.
Ethogram with solitary and social behaviors Ch i mpACT captures the daily life of group-living chimpanzees, offering invaluable insights into the evolution and sustenance of their social behav- iors and relationships (Nishida al., 2010). By focusing on juvenile chimpanzee, Ch i imAACT illuminates facets of social learning, communication, bonding, and more, all pivotal in the social and ecological life of chimpanzees (Bard et al., 2014). To systematically represent these behaviors, we composed an ethogram- detailed catalog of behavioral categories, depicted in Fig. 2b (further details in Appx. A). This ethogram organizes behaviors into four primary categories, like locomotion and social interaction, each further subdivided into several fine- tne-grained actions, meticulously anno- tated and validated with expert oversight. By delving into these behaviors, ChimpACT elucidates not only the social dynamics shaping social relationships but also the cognitive and ecological influences on juvenile chimpanzee behaviors.
3.2 Dataset collection
The focal video data were collected with the Chimpanzee-A group housed at Leipzig Zoo, Germany, using focal sampling (Altmann, 1974). Videographers were instructed focus on Azibo and his mother, Swela, but also on capturing the environmental context and his interactions with other chimpanzees. Videos from ChimpACT were sampled from a larger set of around 405 hours of longitudinal focal video recordings of the dyad between 2015 and 2018. These videos were recorded by several research assistants during the daytime (7am-4pp) 4-m)) using tripod-mounted RGB cameras. Two JVC Everio camera models were utilized across the years, filming with framerate of 25 (Codec H.264) and with resolutions of 720 x 578 and 1280 x 720, respectively. The mother-infant dyad was filmed for about five hours each week during the observation period. The footage contains both optical zoom and camera movements.
3.3 Dataset tasks and annotations
ChimpACT supports three tracks: (i) chimpanzce detection, tracking, and ReID, (ii) chimpanzee pose estimation, and (iii) spatiotemporal action detection. We provide fine-grain fine-grained annotations for each track. From the extensive footage, we curated 163 video clips, each approximately 1000 frames in length. Fifteen adept annotators were then tasked with annotating bounding boxes, body keypoints, and ine-grained behavioral classes for each chimpanzee at intervals of every 10 frames. To ensure accuracy and consistency, a behavioral researcher familiar with the chimpanzee group meticulously reviewed and refined the identity and behavioral class annotations. For a deeper dive into the annotation process and its quality, please refer to Appx. and our dedicated website.
designs, conducted by researchers aftiliated with the MPI for Evolutionary Anthropology (Baker, 2022; McEwen
al., 2022).
5

---

Detection, tracking, and RelD This task encompasses the detection and tracking of individual chimpanzees across video sequences, subsequently coupled with their re-identification. ChimpACT features over 23 distinct chimpanzee individuals, each identified by a primate expert familiar with the Leipzig A-group chimpanzees. Initially, annotators were instructed to delineate the bounding box of each chimpanzee, ensuring consistent box IDs for the same individual throughout video clip. Subsequently, the expert matched these box IDs with the corresponding true names of the chimpanzees, resulting in the identification of 23 unique individuals. Additionally, every annotated bounding box is attached with a visibility attribute, indicating if the chimpanzee is fully visible, truncated, occluded in given frame. Such visibility annotations chimpanzee behavior, potentially bolstering tracking robustness. Fig. 3a illustrates the occurrence frequency (on log scale) of each individual, revealing a Nong-tail distribution. This pattern aligns with the focal sampling strategy, where Azibo is the primary subject. Notably, Swela, Azibo's mother, also exhibits high occurrence frequency, resonating with prior studies (Boesch, 1996).
support the reasoning of the
Pose estimation Pose esti- mation aims to predict the loca-
21.
tions of the chimpanzee joints that have semantic meaning, such as the knee and shoulder, from an input image. There are four keypoints on the chim- panzee's 's face (i.e., two for the eyes, and one each for the up- per and lower lips), for to- tal of 16 chimpanzee keypoints (refer to Sec. 3.3 and Fig. 4). Annotators are tasked with marking the 2D joint coordi- nates and the visibility status of each joint. We adopt the visi- bility protocol from the COCO 2D human keypoint annota- tions (Lin et al., 2014), where value of 0 indicates joint outside the image frame, sig- nifies an obscured joint within the image, and 2 designates clearly visible joint. Such an annotation protocol affords son about chimpanzee's orientation and action based on facial joint visibility. For instance, the chimpanzee might be cating something if the two lips apart. Sample frames showcasing pose annotations are depicted in Fig. Notably, Ch i immACT holds the potential for future expansion to encompass pose tracking tasks, analogous the Pose'Track (Andriluka et al., 2018) for humans.
Frequency (8g)
5.5
3.3
3.1
3.0
2.4
2.8
1.8
1.0
#oito
Dawla
Kista
ahiit
Ree
Frode
Carrie
Bengolo
Loto
Derrnv
Aateriy
Hatasotte
Fresjje
Fal
Lane
Moja
Slavtra
Baniarri
Youma
Makertl
Kemo
Chimpanzee names
30.9
18'
6.8
6.1
5.5
Frequency (%)
10a
1.5
1.0
0.4
0.1
6.02
0.62
104
0.02
resting
tasching
sleaping
anstrectng
plaiin
grooning
moving
being
prooneed
climiing
mure
batrr
purred
otd
lniing
carrecd
poreeeiing Tokied oaj
sunn obl
eneu
Behavlors
(b)
Figure 3: (a) Distribution (in log scale) of annotations for each indi-
vidual. (b) Distribution (in log scale) of annotations for each behavior. Vector graphics; zoom for details.
Table 2: Keypoint definitions for chimpanzee.
No.
Definition
No.
Definition
0
Root of hip
8
Right eye
Right knee
9
Left eye
2
Right ankle
10
Right shoulder
3
Left knee
11
Right elbow
4
Left ankle
12
Right wrist
0
5
Neck
13
Left shoulder
6
Upper lip
14
Left elbow
7
Lower lip
15
Left wrist
Figure 4: Keypoint definitions for chimpanzee.
Spatiotemporal action detection Spatiotemporal action detection seeks to attribute one or multiple behavioral labels to each bounding box containing chimpanzee, leveraging the spatiotemporal context within video clip. Our ethogram, detailed Fig. 2b, delineates 23 nuanced subcategories of behaviors and guides the fine- fine-grained annotations of chimpanzee behavior, such as "climbing"

---

within the "locomotion" category. Notably, within the realm of social interactions, we meticulously differentiate between the action performer and receiver. For instance, the grooming behavior is bifurcated into "grooming" and "being groomed." Every chimpanzee in a frame has its subcategory behavior annotated. It not uncommon for an individual simultaneously exhibit multiple behaviors, exemplified by Swela's "carrying" and "moving" actions in Fig. 1. The distribution of these behavioral annotations, visualized in Fig. 3b on a log scale, reveals a long-tail distribution, mirroring the authentic behavioral tendencies of chimpanzees in their natural habitats.
manipulating object
1.00%
enection
diepleying
O. 1.1%
toaching Baginn
being marsen
losing object 0.62%
racsing
Jaking dbject O. 0.02%
reeting
0.09%
20
being porieed
begging
Behavioo
baing begging
lucamotion 45.01%
lse Suryen
Ethogram
baing cariie 0. 1.58%
baing groomned
carriing 0.69%
Sccital Interestion
1,20%
48%
34 69%
nursing 1.56%
plyeng
ensoeeinn 8.0%%
Artion
Loama
clibbing 1.88%
Chimpanzee
(b)
Figure 5: (a) Distribution of the annotated behavior categories. (b) Distribution showcasing individuals alongside their respective social behaviors. Vector graphics; zoom for details.
Fig. 5a showcases the distribution of the annotated behaviors, with social interactions constituting approximately 35% of the total annotations. Furthermore, Fig. 5b delineates the distribution of labeled social behaviors across distinet individuals, highlighting grooming, playing, and touching as predominant activities within the social dynamics the group-living chimpanzees.
invaluable resource for researchers spanning the domains of primatology, comparative psychology. computer vision, and machine learning. furnishes comprehensive and varied array of annotations, paving the way for in-depth analysis of multifaceted chimpanzee behaviors and catalyzing the development of advanced machine learning algorithms. The inherent long-tail tail distribution not only presents formidable challenge for chimpanzee identification and behavior recognition but also beckons explorations into few-shot learning in future endeavors.
In essence, ChimpACT emerges
Experiments
To rigorously assess Ch. ChimpACT, we benchmark a suite of representative methods across the afore- mentioned three tracks: (i) detection, tracking, and ReID, (ii) pose estimation, and (iii) spatiotemporal action detection. Our computational framework leverages four NVIDIA GeForce RTX 3090 GPUs (24GB) for both training and evaluation across all tracks. In the subsequent sections, we delve into the implementation details, baseline methods, and evaluation metrics for each track.
4.1 Detection, tracking, and RelD
Setting We evaluate several prominent Multiple Object Tracking (MOT) algorithms on ChimpACT, including both classical methods such as SORT (Bewley et al., 2016), DeepSORT (Wojke al., 2017), and Tracktor (Bergmann et al., 2019), as well as the state-of state-of-the-ar art methods such as Byte'Track (Zhang al., 2022), and OC-SORT (Cao et al., 2023). All implementations are based on the MMTracking (Contributors, 2020a) codebase. For those methods supporting flexible detection backbones, we trial two typical detectors, including the two-stage detector Faster R-CNN (Ren et al., 2015) and the one-stage detector YOLOX (Ge et al., 2021). Each method undergoes training for 10 epochs, adhering to the official configurations, which encompass optimizer settings, batch size, data augmentation techniques, and pre-trained models. Given that the three classical methods lack inherent ReID
7

---

Table Results of the detection, tracking, and ReID track the ChimpACT test set. The row highlighted light blue is the performance reference on the human tracking dataset MOT-17 (Milan et al., 2016). denotes not applicable. 2 denotes unreported.
Method
Detector
RelD
HOTA
MOTA 1
MOTP1
IDF1 1
MAP
nHP
nFN 1 niDs
SORT
Faster R-CNN YOLOX
42.6.10
47.4..06
22.9-1.
42.741a
70.7.1.
19.
31...0
2.1400
ResNet-50
(Bewley et al, 2016)
39.840s
43.2-1a
20.3-0.
37.7417
71.411
16.143.1
37,8+17
2.84..
DeepSORT
Faster R-CNN YOLOX
ResNet-50
47.6404
46.7-0.
23.0-1a
52.8+1
70.741.
19,040a
314405
2.4..11
(Wojke al. 2017)
40.2g1.
43.2g12
20.3-0.
384421
71.4g1
16.1g3.1
37.841.7
2.9
Tracktor
Paster R-CNN
ResNet-50
49.5g0.7
50.5g1.1
22.6-1.1
55.6g12
70.7g1.
13.8g0.
35.2g0.7
0.5g0.0
(Bergmann et al., 2019)
QDTmack
(Pang et al., 2021)
R-CNN
$0.3
44.2...6
22.2.
45.8
77.8,a0
19.7.a.
24.6.
1440a
Taakkrack
R-CNN
43.7.0a
36.9.9a
24.6....
48.8
68.2
27.7.1.1
34.2
1.2
(Zhang et al., 2022)
YOLOX
49,2408
43.9-1.
20.3-1.
55.2+11
70.3e1.
18.0274
37.4+0.1
0.740.
OC-SORT
Faster R-CNN
43,441.
38.2-1.9
24.3-0.
48,74a
68.7.0.
25.0+1.
35.6115
1.2+0a
(Cao 2023)
YOLOX
47.9404
42.1g2.
20.5.0.8
53.3go.
70.5
20.3g1.
36.6g2.1
1.1g0.
OC-SORT
YOLOX
78.0
77.5
2.7
19.0
0.3
(Cao et al., 2023)
modules, we supplement with a dedicated ReID network built on ResNet-50 (He al., 2016). The training curves of select methods (refer to Fig. A2a) affirm convergence within the training epochs.
We split the video clips in ChimpACT into 80% train, 10% validation, and 10% test. Both the train set and test set cover all the individuals. Models are trained on the training set, with performance metrics reported on the test set. We employ videly-accepted evaluation metries, drawing from convention in human/object detection, tracking, and ReID (Bewley et al., 2016; Pang et al., 2021; Zhang et al., 2022). Specifically, we utilize (i) mean Average Precision (mAP) Lin et al. (2014) to gauge the detection accuracy, and (ii) the CLEAR metrics (Bernardin and Stiefelhagen, 2008) (MOTA, MOTP, FP, FN, IDs), IDFI (Ristani et al., 2016), and HOTA (Luiten al., 202.1 evaluate various facets of the tracking performance. It is worth noting that for FP. FN, and IDs, we report normalized values and denote these metrics as nFP, nFN, and nIDs, respectively.
Results Tab. 3 summarizes these tracking algorithmss' performances on the Chi ChimpACT test set. We conducted three runs for each method and reported the average and variance of these metrics. Notably, the variance across multiple runs is minimal, underscoring the robust reproducibility of our benchmarking. A holistic view of the results reveals that QDTrack (Pang et al., 2021) emerges as the top performer. However, it does suffer from a higher count of identity switches compared to other methods. In terms of detection performance, the YOLOX algorithm (Ge et al., 2021) stands toe-to-toe with Faster R-CNN (Ren et al., 2015). A discernible trend is evident among contemporary tracking methods, which seem to excel in identity association capabilities over their classical counterparts. This is corroborated by marked improvements in tracking metrics like IDFI and IDs. Such a trend intimates that the latest tracking methods might be adept at maintaining consistent object identities, a pivotal aspect when tracking and analyzing individual trajectories within chimpanzee cohorts.
While the results garnered by the array of tracking algorithms commendable, they still lag behind the benchmarks set on human-centric datasets (Zhang et al., 2022; Pang et al., 2021; Cao et al., 2023). This disparity can be attributed to challenges like the low contrast and low color variation of the body fur of chimpanzees, compounded by intricate self-occlusions. Nonetheless, this very observation accentuates the significance of ChimpACT. It not only offers a challenging arena for tracking algorithms but also stands as an ideal platform for pioneering and refining tracking methods tailored for chimpanzees and other non-human primates.
4.2 Pose estimation
Setting We benehmark several state-of-the-art human pose estimation methods on ChimpACT, including CPM (Wei et al., 2016), SimpleBaseline (Xiao et al., 2018), HRNet (Sun et al., 2019), DarkPose (Zhang et al., 2020). Broadly, human pose estimation methods can be bifurcated into two primary paradigms: heatmap-based and regression-based. We harness the MMPose (Contributors, 2020b) framework for implementing these methods. Please refer to Appx. C for more implementation details. All the models undergo training for 210 epochs, maintaining the official configurations for optimizers, batch sizes, and learning rates. To gauge any potential model overfitting, we present the validation curve the AP metric in Fig. A2b, reassuringly suggesting absence of overfitting.
8

---

Table 4: Results of the pose estimation track ChimpACT test set. The row highlighted in light blue is the performance reference on the human pose estimation dataset COCO (Lin et al., 2014). Ø denotes unreported.
Method
Backbone
PCK@0.05
PCK@0.1
AP
AP
APT
APM
AP
AR
ResNet-50
25.3
46.220.5
20.4 20.4
27.4g1.3
3.9 20.4 0,3
12.5 17.
SimplcBaseline
Regression
(Xiao et al., 2018)
ResNet-101
ResNet-152
26.2.1.0
26.3+0.4
47.3-0.
46.4-11
8.7
+0.1
20.4
27.510.6
29.2+1.1
4,2 20.5
4.7 =0.3
0.3
0.5
12.9
13.4
17.7-u.4
18.6.0.0
MobileNetV2
27.521.4
48.1217
16.720.8
43.1227
11.1
2.0
17.7
19.5
RLB
ResNet-50
28.2
47.1=3.1
16.3
41.2g69
11.44.4
1.3
17.422
20.0
(Li al., 2021)
ResNet-101
28.2+3.6
46.524.3
16.242.6
41.1
10.842.4
2.1
17.3+2.8
20.1
ResNet-112
30.0
48.4_a.2
18.1
43.0g7.9
13.5
1.4003
19.213.2
22.3
CPM
CPM
40.7.0:2
60.4
21.6go.
51.020.4
17.1
9.5
22.4go.a
25.4go.1
(Wei et al., 2016)
Hourglass
Hourglass-4
44.6
60.8
20.6.0.3
48.9
16.020.4
4.6
23.720.6
28.220.2
(Newell et al., 2016)
MobileNetV2
MobileNetV2
39.810.4
59.42a.
19.4
48.5ga.
14.3
2.3
20.61o.1
23.2.g0.1
(Sandler et al., 2018)
ResNet-50
43.3
61.7212
22.1
51.5go.4
17.7.0.2
3.7
23.4
26.3
SimplcBascline
ResNet-101
42.8
60.7
21.7
52.5go.4
16.740.0
4.3
23.0go.1
26.2.go.2
(Xiao et al., 2018)
ResNet-152
43.9+0.4
61.6-0.1
22.7+0.4
53.4+0.6
18.3+0.4 5.3
23.9+0.4
27.1
Heatmop based
HRNet
HRNet-W32
48.610.9
65.6ga.6
25.920.4
58.22g.s
22.1
6.1
27.02o.6 30.3g0.5
(Sun et al., 2019)
IRNNet-W48
47.3
64.5...2
25.1
57.2
21.0go.1
6.9.0.v
26.2.0.3
29.6....1
ResNet-50
43.7+0.0
62.1=a.4
22.8+0.1
53.8+0.8
18.8+0.6
3.4
24.1+0.0
27.1=0.1
R0sNet-101
43.1
61.2-14
22.1
$2.6.a.6
17.6
4.0
23.4 26.5
DarkPose
ResNct-152
43.5
61.2-0.
22.4
53.2+0.1
17.4
4.6
23.740.1 26.7
(Zhang et al., 2020)
HRNet-W32
48.7
65.6_0.9
25.7
58.420.8
21.3+0.6
5.640.4
26.94a.2 30.1
HRNet-W48
47.610.7
64.5g1.
25.810.4
58.0g17
21.5
6.6
27.020.4
30.2g0.5
HRFormer
FRForeesSS
45.1
61.4-04
23.0+0.0
53.1
19.740.a
5.5+1.6
24.1
27.1+0.1
(Yuan et al., 2021)
IRRForme-BB
46.420.8
63.0_a.1
24.1
55.3+1.
20.1
5.2
25.4
28.2
HRNet
HRNet-W32
74.4
90.5
81.9
70.8
81.0
79.8
(Sun et al., 2019)
The train/test partitioning mirrors that of the first track. We use mAP with various thresholds, adhering to the conventions of human pose estimation (Lin et al., 2014). Additionally, we report the Percentage of Correctly estimated Keypoints (PCK) metric (Andriluka et al., 2014; Ng et al., 2022). PCK @ a quantifies the fraction of accurately predicted keypoints within distance threshold defined as a x mar mar(height, width), derived from the bounding box of the chimpanzee. This metric is widely recognized for its accuracy in body joint localization in both human and animal pose estimation.
Results Tab. 4 consolidates these pose estimators' performances on the Ch impACT test set. Notably, the heatmap-based DarkPose (Zhang et al., 2020) with an HRNet (Sun et al., 2019) backbone emerges as the top-performing model. This trend aligns with observations in human pose estimation, where heatmap-centric methods (Wei et al., 2016; Xiao et al., 2018; 80 Newell et al., 2016; Sun et al., 2019) predominantly lead the pack, attributed to their robustness against pose and appearance variations. However, the heatmap representation may be less accurate in scenarios where multiple joints are occluded or closely spaced, and it demands heftier computational and memory resources. Conversely, the newer regression-based based methods (Li et al., 2021) computationally leaner but tend to be more susceptible to overfitting and generally lag in performance.
These results underscore that the task of chimpanzee pose estimation is distinct and nuanced, and cannot be seamlessly addressed by merely repurposing human-centric pose estimation methods. We believe there are two primary reasons for this: (i) chimpanzees exhibit unique joint flexibility and a broader range of motion, and (ii) the visual texture and appearance of chimpanzee fur diverge significantly from human skin. These insights emphasize the need for chimpanzee specific pose estimation strategies.
4.3 Spatiotemporal action detection
Setting We benchmark four representative human action detection baselines on ChimpACT using the MMAction2 (Contributors, 2020c) codebase, including ARCN (Sun et al., 2018), LFB (Wu et al., 2019), and SlowFast with its variant SlowOnly (Feichtenhofer al., 2019). All models undergo training for 20 epochs with a batch size of 32. Convergence is evident from the training curves

---

Table S: Results of spatiotemporal action detection track ChimpACT test set. The row highlighted in light blue is the performance reference on the human action dataset AVA (Gu et al., 2018). denotes not applicable. "w. NL/Max/Avg LFB" denotes using non-local, max, or average LFB module. "w. Ctx" indicates using both the Rol feature and the global pooled feature for classification. "mAP." "mAPz." "mAPo." "mAPs." and "mAP," represent the overall mAP and mAP for Locomotion, Object interaction, Social interaction, and others.
Method
Frame sampling Module
mAP
mAP,
mAPo
mAPs
mAP.
ACRN
8 x 8 x 1 x 16
24.440.5
58.7.0.7
33.821.7
14.7=0.4
0,0ga.0
(Sun al., 2018)
2..
57.8
35.0
13.8
0.0
4 x 16 1
W. NL LFB
22.0+0.9
50.1
32.3+0.9
13.5-1.6
0.6
1.FB
4 x 16 x 1
W. Max LFB
23.2+0.7
45.0+1.5
31.2+0.8
17.7+1.4
0,5+0.0
(Wu et al., 2019)
x 16 x
W: Avg LFB
11.3+
45.0+3.6
29.8
14.7g2.6
0.5
x x 4 x 16
20.9
48.1
36.2+2.
11.5
0.0go.1
SlowOnly (Feichtenhofer et al., 2019)
19.2
47.0;2.
28.3
11.0
g.0
x x 1 4 x 16 x 1
W: Ctx
22.3
52.313.2
31.211.3
13.812.4
0.1
W: Cix
21.4+0.9
47.6-20
33.0+1.2
13,2g2.2
0,2g0.1
x x x 16
21.9
53.020.7
30.612.2
12.9
0.0
SlowFast (Feichtenhofer et al., 2019)
22.0
52.922.
33.122.3
12.6g0.9
0.0ga.0
x x
W: Cix
24.3
56.8+1.6
31.5+2.0
15.6
0.1
4 x 16
W: Cix
10.9
56.6:20
34.712.
14.6:0.4
0.1
SlowFast
(Feichtenhofer al., 2019) x x
25.8
in Fig. A2c. We maintain consistent optimizers and learning rates as in official implementations. Ground-truth bounding boxes for each chimpanzee are provided during both training and testing, as per Tang et al. (2020). Please refer to Appx. C for further details ablative modules.
We adopt the same train-tee test split as previous tracks. Performance is gauged using mAP across 23 action classes, per standard (Feichtenhofer et al., 2019; Tang et al., 2020). Additionally, evaluate the mAP within the four behavioral types separately.
Results Tab. 5 summarizes the action detection algorithms' performances on the Ch ImpACT test set. The overall mAP aligns with results on human action datasets, underscoring the feasibility of automated action detection for video coding and further analyses. Locomotion behaviors achieve a notably higher mAP, likely due to their solitary nature and distinct patterns. Conversely, Conversely, the "others" category registers the lowest mAP, attributed to its limited data data-comprising just 0. 0.14% of action instances across two fine-grained classes. This imbalance suggests the potential benefit of sew-shot learning methods in the future. The results highlight both the promise and areas for improvement in the dataset, positioning it as a valuable platform for advancing spatiotemporal action detection algorithms. We anticipate that Ch ChimpACT will further studies into the social dynamics of non-human primates in eemi-naturalistic environments.
5 Conclusion
In this work, we introduced ChimpACT, novel longitudinal video dataset capturing the intricate behaviors of group-living chimpanzees, focusing on the juvenile chimpanzee, Azibo. Our meticulous annotations and diverse social interactions within the dataset offer a unique view into the world of our closest evolutionary relatives. Through comprehensive experiments, we underscored the challenges and nuances of applying numan-centre computer vision algorithms to the distinct behaviors and interactions of chimpanzees. The dataset's 's depth, combined with its long-tail distribution, not only emphasizes its sigmificance but also paves the way for interdisciplinary research bridging primatology. comparative psychology, computer vision, and machine learning. By making this resource available, our aspiration is catalyze advancements video understanding, inspire the research community to craft specialized techniques for non-human primates and deepen our collective insights into their intricate social fabric and dynamics.
Limitation and future work ChimpACT is based on captive chimpanzees living in a semi-natural environment, limiting the observable range of behaviors. Natural foraging, responses to predators. and intergroup encounters are absent. Focusing on Azibo overrepresents certain individuals and underrepresents others, limiting the assessment of the full social network. Nevertheless, we plan to contribute more data and labels to create a larger and more comprehensive chimpanzee dataset.
10

---

Acknowledgement
The authors would like to thank the Wolfgang Köhler Primate Research Center, BasicFinder CO., Ltd., and Keyue Zhang for annotations and quality check, Zihao Yin for discussions and preliminary experiments on the chimpanzee detection models, Guangyuan Jiang and Yuyang Li for their technical support on the GPU cluster, and NVIDIA for their generous support of GPUs and hardware. X. Ma, Su, Zhu, Y. Zhu, and Y. Wang are supported in part by the National Key R&D Program of China (2022DD01 14900), and Y. Zhu is supported in part by the Beijing Nova Program and the National Comprehensive Experimental Base for Governance of Intelligent Society, Wuhan East Lake High-Tech Development Zone.
References
Altmann, J. (1974). Observational study of behavior: sampling methods. Behaviour, 49(3-4):227-266 3.5
Andriluka. M.. Igbal, U., Insafutdinov, E., Pishchulin, L.. Milan, A., Gall. J. and Schiele, B. (2018). Posetrack: A benchmark for human pose estimation and tracking. In Conference Computer Vision and Pattern Recognition (CVPR). 6
Andriluka, M., Pishchulin, L.,, Gehler, P., and Schicle, (2014). 2d human pose estimation: New benchmark and state of the analysis. In Conference Computer Vision and Pattern Recognition (CVPR). 3,
Bain, M., Nagrani, A., Schofield, D., Berdugo, S., Bessa, J., Owen, J., Hockings, K. J., Matsuzawa, T., Hayashi, M., Biro, D., al. (2021). Automated audiovisual behavior recognition in wild primates. Science Advances, 7(46)reai44883
Bain, M., Nagrani, A., Schofield, D., and Zisserman, A. (2019). Count, crop and recognise: Fine-grained recognition in the wild. Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. 2, 3,
Baker, A. (2022). Wolfgang köhler primate research center. Encyclopedia of Animal Cognition and Behavior, page 7310. 5
Bala, P. C., Eisenreich, B. R., Yoo, B. M., Hayden, B. Y., Park, S., and Zimmermann, J. (2020). Automated markerless pose estimation in freely moving macaques with openmonkeystudio. Nature Communications, 11(1):4560. 3.3
Bard, K. A., Dunbar, S., Maguire- V., Veira. Y., Hayes, K. G., and McDonald, K. (2014). Gestures and social-emotional communicative development in chimpanzee infants. American Journal of Primatology. 76(1):14-29.
Bergmann, P., Meinhardt, T., and Leal- Taixe, L. (2019). Tracking without bells and whistles. In International Conference on Computer Vision (ICCV). 8,8, A5
Bernardin, K. and Stiefelhagen, R. (2008). Evaluating multiple object tracking performance: the clear mot metries, EURASIP Journal Image and Video Processing, 2008:1-10. 8
Bewley, A., Ge. Z., Ott, L., Ramos, F., and Uperoft, B. (2016). Simple online and realtime tracking. In IEEE International Conference on Image Processing (ICIP). 4, 7, AS
Boesch, C. (1996). The emergence of cultures among wild chimpanzees. In Proceedings-Briicee British Academy. 6
Bohnslav, J. P., Wimalasena, N. K., Clausing, K. J., Dai. Y. Y., Yarmolinsky, D. A., Cruz, T., Kashlan, A. D., Chiappe. M. E., Orefice, L. L., Woolf, C. J., et al. (2021). Deepethogram, machine learning pipeline for supervised behavior classification from raw pixels. Elife. 10:e63377. 4
Cao, Pang, J., Weng. X., Khirodkar, R., and Kitani. K. (2023). Observation sort: Rethinking sort for robust multi-object tracking. In Conference on Computer Vision and Patiern Recognition (CVPR). 7, 8, A5
Contributors, M. (2020a). MMTracking: OpenMMLab video perception toolbox and benchmark. https:
//github. com/ wwwn-ppeniiie ataaaab///tttrre htg.e/mm.n.maa//mmtatoo....... 7
Contributors. M. (2020b). Opemmmlab pose estimation toolbox and benchmark. gpen-mmaab//mpoon /mmpse...e 8
Contributors, M. (2020c). Openmmlab's next generation video understanding toolbox and benchmark. https: //github. com/ open- mm www.w//www...aiiaooooomm /ww.atmoac.tn....mmmmmmmmmo

---

Dawkins, M. S. (2003). Behaviour tool in the assessment of animal welfare. Zoology, 106(4):383-877. 2
Desai, N., Bala, Richardson, R., Raper, J.. Zimmermann, J., and Hayden, B. (2022). Openapepose: database of annotated ape photographs for pose estimation. arXiv preprint arXiv:22/2.00771. 2, 2,3,
Fabian Caha Heilbron, Vietor Escorcia, B. G. and Nichles, J. C. (2015). Activitynet: A large-scale video benchmark for human activity understanding. In Conference on Computer Vision and Pattern Recognition (CVPR). 3
Feichtenhofer, C., Fan, H., Malik, J., and He, K. (2019). Slowfast networks for video recognition. International Conference on Compuder Visian (ICCV). 4, 9, 10, A9,
Fröhlich, M., Müller, G., Zeiträg, C., Witig, R. M.. and Pika, S. (2020). Begging and social tolerance: Food solicitation tacties in young chimpanzees (pan troglodytes) in the wild. Evolution and Human Behavior, 41(2):126 135.
Ge, Z., Liu, S., Wang. F. Li, Z., and Sun, J. (2021). Yolox: Exceeding yolo series in 2021. arXiv preprint WiXii:2007.08430. 8
Gebru, T., Morgenstern, J.. Vecchione, B., Vaughan, J. W., Wallach, H., Iii H. D., and Crawford, K. (2021). Datasheets for datasets. Communications of the ACM, 64(12):66-2. A14
Gonyou, H. W. (1994). Why the study of animal behavior is associated with the animal welfare issue. Journal of Animal Science, 2218)22171-211 11777 2
Gu. C., Sun, C., Ross, D. A., Vondrick. C., Pantofaru, C., Li. Y., Vjjyanarasimman, S., Toderici, G., Ricco, S.. Sukthankar, R., al. (2018). Ava: video dataset of spatio-temporally localized atomic visual actions. In Conference on Computer Vision and Pottern Recognition (CVPR). 3,
Ho, K., Gkioxari, G., Dollár, P., and Girshick, R. (2017). Mask r-cnn. In International Conference on Computer Vision (ICCV). 4
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning image recognition. Conference on Computer Vision and Pattern Recognition (CVPR). 4, 8
Hobaiter, C., Samuni, I.., Mullins, C., Akankwasa, W. J., and Zuberbühler, K. (2017). Variation in hunting behaviour in neighbouring chimpanzee communities in the hudongo forest, uganda. PloS One, 12(6):001880 78065.
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vjjaannraaiinhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al. (2017). The kinetics human action video dataset. arXiv preprint arXiv:1705.06955
Labuguen, R., Matsumoto, J., Negrete, S. B., Nishimaru, H., Nishijo, H., Takada, M., Go, Y., Inoue, K.-i., and Shibata, T. (2021). Macaquepose: novel "in the wild" macaque monkey pose dataset for markerless motion capture. Frontiers Behavioral Neumoscience, 14:58115 1.541 2,
Langergraher, K. E., Prüfer, K., Rowney, C., Boesch, C., Crockford, C., Fawcett, K., Inoue, E., Inouc-Nuruyama, M.. Mitani, J. C., Muller, M. N., et al. (2012). Generation times in wild chimpanzees and gorillas suggest earlier divergence times in great ape and human evolution. Proceedings the National Academy of Sciences (PNAS), 197636)11771
Lauer, J., Zhou, M., Ye, S., Menegas, W., Schneider, S., Nath, T., Rahman, M. M., Di Santo, V., Soberanes. D., Feng. G., et al. (2022). Multi-animal pose estimation, identification and tracking with decplabcut. Nature Methods, 19(4):49-50kk
Li, J., Bian, S., Zeng. A., Wang. C., Pang, B., Liu, W., Lu, C. (2021). Human pose regression with residual logliikeldhood estimation. In International Conference on Computer Vision (ICCV). 9, A8, A9
Lin. T.-Y.. Maire, M., Belongie. S.. Hays. J., Perona, P., Ramanan, D., Dollár, P., and Zitickk, C. L. (2014). Microsoft coco: Common objects in context. In European Conference Computer Vision (ECCV). 3, 6, 8,
Luiten, Osep, A., Dendorfer, P., Tor,, P., Geiger, A., Leal- Laixe, L., and Leibe, (2021). Hota: A higher order metric for evaluating objei- tracking. International Journal of Computer Vision (ICC),, 129:548- 578.
Luncz, L. V., Sirianni, G., Mundry, R., and Boesch, C. (2018). Costly culture: differences in nut-cracking efficiency between wild chimpanzee groups Animal Behaviour, 137:63-73..
12

---

Marks, M., Jin, Q.. Sturman, O., von Ziegler, L.. Kollmorgen, S., von der Behrens, W. Mante. V., Bohacek. J.. and Yanik, M. F. (2022). Deep-learning-basee based identification, tracking, pose estimation and behaviour classification interacting primates and mice complex environments. Nature Machine Intelligence. 4(4):331-340. 2,
Mathis, A., Mamidanna, P. Cury, K. M., Abe, T., Murthy, V. N., Mathis, W. Bethge. (2018). Deeplabeut: markerless pose estimation of user-defined body parts with deep learning. Nature Neuroscience, 21(9):1281-128, 4, A16
Matsuzawa, T. (2013), Evolution of the brain and social behavior in chimpanzees. Current Opinion in Neurobiology, 23(3):443 449. 5
Matsuzawa, T., Tomonaga, M., and Tanaka, M. (2006). Development in chimpanzees. Springer. 5
McEwen, E. S., Warren, E., Tenpas, S., Jones, B., Durdevic, K., Rapport Munro, E.,, and Call, J. (2022). Primate cognition in zoos: Reviewing the impact of zoo-based research over 15 years. American Journal of Primatology, 84(10);e3369.
Milan, A., Leal-Taixé, L.,, Reid, L., Roth, S., and Schindler, K. (2016). Mot16: A benchmark for multi-object tracking. arXiv preprint a603: 600.0003.. 3, 8, A5
Musgrave, S., Lonsdorf, E., Morgan. D., and Sanz, C. (2021). The ontogeny of termite gathering among chimpanzees in the goualougo triangle. republic of congo. American journal of physical anthropology. 174():187-200.3
Newell, A., Yang. K., and Deng, J. (2016). Stacked hourglass networks for human pose estimation. In European Conference on Computer Vision (ECCV). 9, A8, A9
Ng. X. L., Ong, K. E., Zheng, Q. Ni, Y., Yeo, S. Y., and Liu, J. (2022). Animal kingdom: A large and diverse dataset for animal behavior understanding. In Conference on Computer Vision and Pattern Recognition (CVPR). 2, 3, 4,9
Nishida, T., Zamma, K., Matsusaka. T., Inaba, A., and McGrew, W. C. (2010). Chimpanzee behavior in the wild: an audio-visual encyclopedia. Springer Science & Business Media. 5
Pang. J., Qiu, L., Li, X., Chen, H., Li, Q., Darrell, T., and Yu, F. (2021). Quasi-dense similarity learning for multiple object tracking. In Conference on Computer Vision and Pattern Recognition (CVPR). 4, 8, A5
Pedersen, M., Haurum, B., Bengtson, H., and Moeslund, T. B. (2020). 3d-zef: 3d zebrafish tracking benchmark dataset. In Conference on Computer Vision and Pattern Recognition (CVPR).
Ren, S., He, K., Girshick, R., and Sun, J. (2015). Faster Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems (NeurIPS). 7,
Ristani, E., Solera, F., Zou, R., Cucchiara, and Tomasi, C, (2016). Performance measures and a data set for multi-target, multi-camera tracking. ECCV Workshops.
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, I.-C. (2018). Mobilenetv2: Inverted residuals and linear bottlenecks. In Conference on Computer Vision and Pottern Recognition (CVPR). 9, A8, A9
Schapiro, J.. Perlman, E., Thiele, E., Lambeth, (2005). Training nonhuman primates to perform behaviors useful biomedical research. Lab Animal, 34(5):37-42,
Schindler, F. and Steinhage, V. (2021). Identification of animals and recognition of their actions in wildlife videos using deep learning techniques. Ecological Informatics, 61:101215.
Sirianni, G. Mundry, R., and Boesch, C. (2015). When to choose which tool: multidimensional and conditional selection of nut cut-kcacking hammers in wild chimpanzees. Animal Behaviour, 150-1.66665.
Sun, C., Shrivastava, A., Vondrick, C., Murphy, K., Sukthankar, R., and Schmid, C. (2018). Actor-centric relation network. In European Conference on Computer Vision (ECCV). 9, 10, A12
Sun, J. J.. Karigo. T.. Chakraborty, D., Mohanty, P., Wild, B., Sun, Q.. Chen, C., Anderson, D. Perona, P.. Yue, Y., et al. (2021). The multi-agent behavior dataset: Mouse dyadic social interactions. arXiv preprint 1041:2200..0270..
Sun, J. J.. Marks. M., Ulmer, A. W., Chakraborty, D., Geuther, B., Hayes, E., Jia, H., Kumar, V., Oleszko, S., Partridge, Z. et al. (2023). Mabe22: A multi-species multi-task benchmark for learned representations of behavior. In International Conference on Machine Learning (ICML).
13

---

Sun, K., Xiao, B., Liu, D., and Wang. J. (2019). Deep high-resolution representation learning for human pose estimation. In Conference on Computer Vision and Pattern Recognition (CVPR). 3, 4, 8, 9, A8, A9
Surbeck, M., Boesch, C., Girard-Buttin C., Crockford, C., Hohmann, G., and Wittig, R. M. (2017). Comparison of male conflict behavior in chimpanzees (pan trolodytes) and bonobos (pan paniscus), with specific regard to coalition and post-conflict behavior. American Journal of Primatology. 79(6)=22241. 2
Tang, J., Xia, J., Mu, X., Pang. B., and Lu, C. (2020). Asynchronous interaction aggregation for action detection. In European Conference on Computer Vision (ECCV). 10
The Chimpanzee Sequencing and Analysis Consortium (2005). Initial sequence of the chimpanzee genome and comparison with human genome. Nature. 437(7055):69-33 2.3
Van Leeuwen, (2021). Temporal stability of chimpanzee social culture. Biology Letters, 17(S)22021001. 5
Wang, X., Girshick, R., Gupta, A., and He, K. (2018). Non-local neural networks. In Conference Computer Vision and Pattern Recognition (CVPR). A9
Wei, S.-F.., Ramakrishna, V., Kanade, T., and Sheikh, Y. (2016). Convolutional pose machines. In Conference on Computer Vision and Pattern Recognition (CVPR). 9, A8, A9
Wiltshire, C., Lewis-Cheethan Cheetham, J., Komedová, V., Matsuzawa, T., Graham, K. E., and Hobaiter, C. (2023). Deepwild: Application of the pose estimation tool deeplabeut for behaviour tracking in wild chimpanzees and bonobos. Journal of Animal Ecology. 2, 4
Wojke, N., Bewley, A., and Paulus, D. (2017). Simple online and realtime tracking with deep association metrie. In IEEÉ International Conference on Image Processing (ICIP). 7, 8, AS
Wu, C.-Y., Feichtenhofer, C., Fan, H., He, K., Krahenbuhl, P., and Girshick, R. (2019). Long-term feature hanks for detailed video understanding. Conference on Computer Vision and Pattern Recognition (CVPR). 9, 10, A9, A12
Xiao, B., Wu, H., and Wei, Y. (2018). Simple baselines human pose estimation and tracking. In European Conference on Computer Vision (ECCV). 3, 4, 8, 9, A8, A9
Yang, Y., Yang, J., Xu, Y., Zhang, J., Lan, I.., and Tao, D. (2022). Apt-36k: A large-scale benchmark for animal pose estimation and tracking. In Advances in Neural Information Processing Systems (NeurIPS). 3
Yao, Y., Bala, P., Mohan, A., Bliss-Moreau, E., Coleman, K., Freeman, S. M., Machado, C. J., Raper, J.,
Zimmermann, J., Hayden, B. Y. et al. (2023). Openmonkeychallengee Dataset and benchmark challenges for pose estimation of non-human primates. International Journal of Computer Vision (ICV), 131(1):243-258 2,
Yu, F., Chen, H., Wang, X., Xian, W., Chen, Y., Liu, F., Madhavan, V., and Darrell, T. (2020). Bdd100k: A diverse driving dataset for heterogeneous multitask learning. Conference Computer Vision and Pattern Recognition (CVPR). A3
Yu, H. Xu, Y., Zhang, J., Zhao, W., Guan, Z., and Tao, D. (2021). Ap-10k: A benehmark for animal pose estimation in the wild. Proceedings of the Neural Information Procexsing Systems (NeurIPS) Track on Datasets and Benchmarks. 2.3
Yuan, Y., Fu, R., Huang, L.,, Lin, W., Zhang, C., Chen, X., and Wang. J. (2021). Hrformer: High-resolution vision transformer for dense predict. In Advances in Neural Information Processing Systems (NeurIPS). 9, A8, A9
Zhang, F., Zhu, X., Dai, H., Ye, M., and Zhu, C. (2020). Distribution-aware coordinate representation for human pose estimation. In Conference on Computer Vision and Pattern Recognition (CVPR). 8, 9, A8, A9
Zhang, L., Gao, J., Xiao, Z., and Fan, H. (2023). Animaltrack: A benchmark for multi-animal tracking in the wild. International Journal of Computer Vision (UCV), 131(2)496-5133,
Zhang. Y., Sun, P., Jiang, Y., Yu, D., Weng. F., Yuan, Z., Luo, P., Liu, W., and Wang, X. (2022). Bytetrack: object tracking by associating every detection box. In European Conference on Computer Vision (ECCV). 7, AS
14

---

A Additional details on ChimpACT
A.1
Ethogram
We detail the ethogram definition in Tab. A1, which systematically describes the daily behaviors of chimpanzees.
Table A1: The ethogram used for the ChimpACT dataset.
category
definition
subcategory
subeategory definition
moving
moving horizontally, eg. walking, running
pallerns sell-inititted movement of an individunl
1. climbing
moving vertically, egg, climbing up down stracture
locomotion
resting
remaining stationary, e.R., standing, sitting, lying resting and keeping eyes clused
3. sleeping
4. solitary object playing non social and non goal directed object interaction and explo-
ration
direct physical interactions with object interaction inanimate stationary or movable objects by hands, feet or mouth
5. cating
consuming and processing food
manipulating object
manipulation of any kind inanimatc object excluding cating
7. grommnng
chimpanzee, the gromer, is cleaning the fur, head, hand,
feet, or genitals of another chimpanzoe, usually using their hands and/or mouth
being groomed
one chimpanzee, the groomee, getting their skin or eleaned by another chimpanzee
9. aggressing
a chimpanzee showing agonistic behavior towards another chimgaanzee. This range from charging and chasing other chimpanzee to direct physical contact such as shipping hitting, and biting
10. embracing
a chimpanzoe is embracing another chimpanzee with their arms, be confused with carrynng
11. begging
a chimpanzee is requesting food or another object from an- other chimpanzee, oftentimes extending their arm, reach- ing, or using an palm begging gesture
at least two chimpanzees are
being hegged
chimpanzee requested or another object by another chimpanzee
interacting in differentiated noles:
with one individual initiating the social behavior finitiator) one taking object individual receiving the social behavior (recipicat)
social interaction
taking an object from the possession of another chimpanzee, the transter might resisted
14. losing abject
the possession is laken by another chimpanzee
15. carrying
chimpanzee (usually an adult) carries another chimpanzee (usually an infant or juvenile) on the back, front, side, arm, or leg for than steps
16. being carried
a chimpanzee (usually an infant or juvenile) is carried by another chimpanzee (usually an adult) the back, front, side, arm, or leg for mere than 2 stops.
nursing
a female chimpanzee is nursing (hreastfed, ie., making physi- cal contact with the nipple) an infantijuvenile
18. being nursed
an infans//uvvnille being nursed (breastfod, iee. making phys-
ical contact with the nipple) by a female chimpanzee
playing
a chimpanzee is physically interacting with another individual friendly, teasing, mock lighting way (e.g.. play lighting and other behaviors)
20. touching
chimpanzee makes hody contact with another chimpanzee (e.e.. holding hands) and it does not fit with any of the other social interaction categories deseribed above
21. erection
a male chimpanzee has an erect penis
others
other behaviors
displaying
chimpanzee, usually with puffed up (piluerection) and an erection. performs a dominance display, which includes walking with swagger, swinging their arns to the sides, and making calls with increasing amplitude, commonly ending by stomping against or slapping objects. Displays can be directed another chimpanzse or be undirected
A1

---

2015 indoor
2015 indoor
2016 indoor
2016 outdoor
2017 indoor
2017 outdoor
2018 indoor
2018 outdoor
Figure A1: Example frames from the ChimpACT dataset. ChimpACT possesses rich social interactions of the complex everyday life of group-living chimpanzees and contains several environmental enrichment.
A2

---

A.2 Dataset details
Collection and organization 405 hours of video footage of the Leipzig A-group chimpanzees were collected between 2015 and 2018. To create representative sample of the footage, 163 video clips were selected, with 15, 35, 86, and 27 clips taken from each year. These video clips cover the four seasons. Each clip is 1000 frames long, with only clips being shorter than 1000 frames. Visual examples from six clips, featuring both indoor and outdoor enclosures, are shown in Fig. A1. The dataset covers a diverse range of physical scenarios, camera views, and social behaviors, as demonstrated in these examples. For instance, in the third row of the figure, an adult chimpanzee is shown grooming an infant chimpanzee in her arms, while later on, the same infant is nursed.
Annotation process and quality The annotation process was conducted using BasicFinder CO., Ltd.'s private labeling platform, which involved team of 15 annotators and managers. Prior to commencing the annotation work, our team developed comprehensive guidelines that explicitly outlined the requirements for labeling. These guidelines covered several aspects, including:
(i) Assigning bounding box for each chimpanzee in the image. (ii) Specifying the visibility of the bounding boxes. (iii) Assigning tracking IDs to each bounding box for tracking purposes. (iv) Localizing 2D keypoints within each bounding box. (v) Indicating the visibility of each 2D keypoint. (vi) Assigning behavior labels for each bounding box.
To ensure that the annotators followed these guidelines accurately, the project managers provided training based on the guidelines. Following the training, the annotators performed a trial annotation on a small dataset. We actively sought feedback from the annotators during this phase, which allowed us to address any issues and make necessary improvements. We conducted a thorough review of the trial annotations to verify that the quality met our standards.
During the trial labeling phase, we reached out to three labeling companies and ultimately selected BasicFinder CO., Ltd. based on their exceptional labeling quality. It worth noting that BasicFinder CO., Ltd. has previously led the annotation efforts for the BDD100K (Yu et al., 2020) dataset, which is substantial dataset used for autonomous driving purposes. This experience demonstrates their ability to maintain high annotation standards for complex and extensive datasets. Consequently, their involvement improves the reliability of our ChimpACT dataset annotations as well.
Once we were confident in the quality of the trial annotations, we proceeded with the large-scale annotation process. To manage the annotations efficiently, each video clip was designated as an annotation task, and our managers assigned these tasks to individual annotators using BasicFinder CO., Ltd.'s 's platform, ensuring that there has implemented rigorous quality management practices throughout the annotation process. These practices include a customized workflow, complete job traceability, precise performance tracking, multiple levels of auditing, and scientific personnel management. By adhering to these practices, we were able maintain high standards of quality and accuracy while ensuring efficient processing speed. The annotation process followed a sequential workflow of execution, review, and quality control. Experienced annotators were responsible for executing the annotations, while the manager, as well as our team, conducted thorough reviews and quality control checks, Any annotations that did not meet the required standards were sent back to the annotators for corrections. The quality control phase involved a comprehensive review and verification of all data by both the managers and our own team, ensuring the integrity and accuracy of the annotations. Once all the data had been confirmed to meet our standards of quality, we concluded the annotation process.
overlap in assignments. BasicFinder CO., Ltd.
More specifically, label chimpanzee identities, annotators only needed assign tracking ID) to each chimpanzee, which was then reviewed by the primatologist in our team, who assigned the apes' names based on his knowledge of the observed Leipzig A A-group chimpanzees. The process of localizing 2D keypoints within each bounding box and assigning behavior labels for each chimpanzee presented bigger challenges than other tasks. To overcome these challenges, we implemented several measures to ensure accuracy and consistency. For the labeling of 2D keypoints, we provided detailed instructions accompanied by visual illustrations, aiming to provide clear guidelines for annotators to precisely identify and mark the keypoints. For labeling of behaviors, we supplied example videos showcasing different chimpanzee behaviors, created by our team's experienced primatologists. These videos served as valuable references, enabling annotators to accurately assign behavior labels based on observed actions. Throughout the annotation process, the primatologists actively participated, offering their expertise and providing valuable feedback to ensure the annotations aligned with
A3

---

scientific standards. Finally. the behavioral primatologists in our team manually reviewed all labeled frames ensure data reliability. These measures and the involvement of the primatologists were instrumental in enhancing the overall quality and reliability of the annotations.
For more information the dataset, including pre-processing scripts, and visualized annotations, please refer to our project website.
B Discussion on ChimpACT
Intended uses The ChimpACT dataset is a versatile resource that can be used for studying al- gorithms for chimpanzee detection, tracking, identification, pose estimation, and spatiotemporal action detection. Therefore, the dataset is both relevant for questions in computer vision and primate behavior. In the context of computer vision, it lends itself other research topics, including but not limited to pose tracking, few-shot learning, reakly-supervised learning, and transfer learning. Considering primate behavior, the dataset shares numerous features with other video data commonly collected with captive and wild chimpanzee populations. This makes it an ideal resource for fine- grained investigations of social (e.g., grooming, nursing, aggression) and nonsocial (e.g.. locomotion, object interactions) chimpanzee behaviors. We strongly encourage researchers to utilize our dataset solely for research purposes that promote animal welfare and conservation. We firmly discourage any use of the dataset for harmful activities such as poaching, hunting or any other exploitation of primates. It is crucial for researchers to approach the data with a focus on positive societal impacts and to refrain from any potential negative consequences.
The Ch ChimpACT dataset raises no ethical concerns regarding the privacy information of human subjects, as it solely focuses on chimpanzees. Studying the social behavior of chimpanzees provides an ethical and efficient means to explore aspects of human sociality due to our phylogenetic proximity. By analyzing their behaviors, we can gain insights into the evolution of human social behavior and potentially contribute to both the scientific and ethical understanding of the human condition. The ethics committee of the Wolfgang Köhler Primate Research Center approved the observational data collection for this project.
Ethics
Maintenance, distribution, and license The Ch ChimpACT dataset will be maintained by the authors and made publicly available with total of 160.500 frames (around 2 hours) on our project website. The Ch ChimpACT dataset will be distributed under the CC BY-NC 4.0 license.
Wage paid to annotators We collaborated with BasicFinder CO., Ltd. for the annotation process. The labeling was carried out by 15 annotators, and they were offered a fair wage as per the prearranged contract. The total expenditure for the labeling process was approximately 70,000 RMB.
C Experiments
We trained all the models with officially-used training configurations for each of the three tracks. Please refer to the code implementation on our Github for details. Although we trained the models for different epochs experiments conducted on different tracks, these choices were made based on conventional practices, Based on the training loss curves provided in Figs. A2a and A2c, it can be observed that all tracking and spatiotemporal action detection methods have reached convergence within the chosen training epochs. To assess the potential overfitting of the pose estimation models, we have included the validation curve on the AP metric in Fig. A2b. The validation curve demonstrates the performance of the pose models on the validation set, which indicates that the pose estimation models are not exhibiting signs overfitting. Therefore, based on the training loss curves and the validation curve, it can be concluded that the chosen training epochs are appropriate for both tracking and pose estimation methods.
C.1 Detection, tracking, and ReID
We partitioned the dataset of 163 videos into three sets: 127 videos for training. 17 for validation, and 19 for testing. Of note, all individual chimpanzees are present in both the training and testing sets. In the test set, there are 12 and 7 videos for indoor and outdoor scenes, respectively.
A4

---

Tracking
Estimaation
Spatiotempunil Actio Detection
CPhtc Pioter NCM00
www wwww
14
Strviist (e2000
20
10
10
Noc
50
230
Iberattoo
Egoch
Iberatio
(a)
(b)
(c)
Figure A2: Training or validation curves on three tracks of example methods. (a) Training loss curve of example tracking methods. The training iterations correspond to 10 epochs. (b) Validation curve on the AP metric of example pose estimation methods. (c) Training loss curve of example spatiotemporal action detection methods. The training iterations correspond to 20 epochs.
For the evaluation metries, MOTA (Multiple Object Tracking Accuracy) takes into account FP (False Positives), FN (False Negatives), and IDs (IDentity switches). Usually, FP and FN are larger than IDs; therefore, MOTA mainly assesses the detection performance. IDFI evaluates the ability to preserve subject identities to assess identification association performance. HOTA (Higher Order Tracking Accuracy) is recently proposed metric that considers accurate detection, association, and localization equally important, and balances their effects explicitly.
Results We additionally evaluated the performance on the indoor and outdoor test set in Tabs. A2 and A3, respectively. Notably, the results indicate that these approaches achieve consistently better performance on the indoor test set compared to the outdoor test set. This may be attributed to the greater complexity of outdoor scenarios and the presence of varying camera views, which can significantly increase the difficulty of detecting and tracking chimpanzees. Furthermore, the presence of occlusions, similar appearances, and other environmental factors can further exacerbate the challenges of chimpanzee tracking in outdoor settings.
Table A2: Results of the detection, tracking, and RelD track on ChimpACT indoor test set.
Method
Detector
ReID
HOTA
MOTA
MOTP
IDFI
mAP
FP
FN
IDs
SORT (Bewley et al., 2016)
Faster R CNN YOLOX
ResNet-50
49.1
52.7
21.0
49.2
76.2
8275
9396
731
41.3
46.7
18.9
38.4
77.2
6440
13163
1105
DeepSORT (Wojke al., 2017)
Faster R-CNN YOLOX
ResNet-50
53.2
51.6
21.0
58.3
76.2
8277
9398
1144
43.3
46.8
18.9
40.8
77.2
6440
13163
1092
Tracktor (Bergmunn al., 2019)
Faster R-CNN
ResNet-50
53.6
54.5
20.6
76.2
146
QDTrack (Pang al., 2021)
Faster R-CNN
53.6
53.6
20.9
58.5
76.7
8121
9591
332
Paster R-CNN YOLOX
48.8
38.9
22.1
52.3
72.7
11599
11799
372
ByteTrack (Zhang al., 2022)
51.0
48.0
17.7
55.6
76.2
5080
14893
245
OC-SORT (Cao et al., 20233
Faster R-CNN YOLOX
48.6
40.5
21.6
71.8
10022
431
49.8
47.9
19.3
53.6
75.9
7550
12292
422
Table A3: Results of the detection, tracking, and ReID track on ChimpACT outdoor test set.
Method
Detector
RoID
HOTA
MOTA
MOTP
IDFI 1 MAP1
FN 1
IDs
SORT (Bewley et 2016)
Faster R-CNN YOLOX
ResNet-50
31.3
43.1
25.2
35.0
63.3
3288
8142
422
34.8
31.9
22.9
35.0
3649
751
DeepSORT (Wojke al., 2017)
Faster CNN YOLOX
ResNet-50
39.4
41.7
25.2
47.8
63.3
3280
8134
726
36.6
31.8
22.9
37.0
61.5
3649
9786
788
Tracktor (Bergmann al., 2019)
Faster R-CNN
ResNet-50
38.8
42.5
24.5
45.0
63.3
2734
9146
94
QDTrack (Pang et al., 2021)
Faster R-CNN
40.0
50.5
27.1
49.6
73.3
3705
6067
534
ByteTrack (Zhung et al., 2022)
R-CNN
32.5
32.7
30.1
42.9
62.1
5346
312
YOLOX
44.2
37.5
23.1
51.5
60.4
1842
11042
139
Faster R-CNN
28.3
27.4
29.8
39.6
60.5
5342
9341
448
OC-SORT (Cao al., 2023)
YOLOX
42.7
31.6
22.6
47.8
60.5
4298
252
We visualize the tracking results in Figs. A3 and A4, with the ground-truth bounding boxes and chimpanzee identities shown the last row. We visualized the confidence scores of the estimated bounding boxes and their associated IDs in each frame obtained by the evaluated methods. It is worth noting that we do not require individual identification of each chimpanzee, but rather assign the same ID to the same animal across frames, following the common practice in humtn tracking (Milan
AS

---

SORT (Faster R-CNN)
DeepSORT (Faster R-CNN)
Tracktor
QDTrack
TracTrack (YOLOX)
OC-SORT (YOLOX)
GT
Figure A3: Qualitative results of representative methods the ChimpACT test set the tracking task. For each method, visualize the estimated confidence score ("conf") and the associated IDs ("boxID") cach bounding box in cach frame. The ground-truth bounding boxes and chimpanzee names are shown in the last row, and we add a number left to the name to make easier to track. Please zoom in for details.
et al., 2016). The estimated box ID is therefore used solely for evaluating the tracking performance. We observed that the evaluated methods performed well in scenarios with minimal occlusion, but struggled to detect and associate the same individual chimpanzee when heavy occlusion occurred. For instance, Fig. A3, the infant chimpanzee's 's bounding box is lost in some frames, and its identity is erroneously switched later due to heavy occlusion. This is challenging task in chimpanzee detection and tracking, as occlusions frequently occur group-living habitats. Please refer to supplementary video for more experimental results. In conclusion, the experimental results reveal the limitations of existing methods for chimpanzee detection and tracking, underscoring the need for more robust algorithms to be developed. We believe that our dataset can make valuable contribution
A6

---

SORT (Faster R-CNN)
DeepSORT (Faster R-CNN)
Tracktor
QDTrack
BrteTrack (YOLOX)
OC-SORT (YOLOX)
GT
Figure A4: More qualitative results of representative methods the ChimpACT test set on the tracking task. For each method, visualize the estimated confidence score ("conf") and associated IDs ("boxID") of each bounding box in each frame. The ground- truth bounding boxes and chimpanzee names are shown in the last row, and we add number left to the name to make it easier to track. Please zoom in for details.
to the advancement of this field, by providing challenging benchmark for evaluating and comparing different methods.
C.2 Pose estimation
We followed the partition of the dataset as the first track to train and evaluate the methods.
Results We report the PCK @0.1 for the 16 keypoints in Tab. A4. The results reveal that the keypoints on the face, such as the eyes and lips, exhibited better estimation compared to the arms and legs. This could be attributed to the fact that eyes and lips have more distinctive visual patterns than
A7

---

Table A4; Results of the pose estimation track for each keypoint on ChimpACT test set. We report @C metric. We abbreviate the keypoint names. Please refer to Sec. 3.3 for the keypoint definition.
Mathal
Backhore
Dlep
Lrkeer
Lmnki
Slneez
<lankie
Stack
6ulpp
2.8pp
Krrge
9less
10mbal
L1mbbow
12rune
14.klhww 15.bnn
50
85.7
31.5
Simpllbbattio at,
ResNec
47.4
35.8
MoNieNeVV
53
46.9
53.8
19:1
08.7
01.8
77.1
76.7
86.3
95.1
59.3
41.6
13.5
59.0
45.3
31.
RLE (Li et 2021)
Rechae-101
ResNes-00
42.7
49.4
42.6
12.8
46.6
42.7
56.0
46.2
61.0
52.2
75.5
73.
77.4
87.1
81.
79.3
60.3
59.0
46.0
30.3
63.0
55.9
46.5
13.6
50.5
1900
Reshie-153
CPM
CPM
61.1
11.7
59.7
06.7
85.5
91.3
601
67.8
66.3
53.4
Houzhas Clicwell et 2010)
Barnisss-4
62.4
65.3
70.6
65.2
67.5
06.4
84.0
$5.9
86.9
87.3
68.5
61.7
60.4
65.3
66.0
56.0
MohiteNarV cf it al, 2018
MoNteNarV2
58.8
64.8
71.3
61.
66.38
07.0
81.88
65.4
91.0
1991
69.3
50.
56.9
67.4
64.8
57.1
ResSa-50
54.6
62.5
16.8
85.1
92.5
67.6
StepleBoscliie OXiao 2018)
Reshee-101
91.9
55.3
NRN (Sur or at., 2019:
HRNer W32
IRNNs-W-6
06.8
61.5
69.1
69.5
36.6
74.5
N5.1
66.1
70.4
09.2
70.5
70.7
58.2
87.5
66.9
90.4
93.5
92.5
92.1
92.2
75.3
76.3
56.7
67.7
64.4
61.1
72.1
72.4
69.5
70.8
62.8
58.9
Rechis-90
64.4
68.6
K1.
06.6
00.9
06-1
91:3
61.3
69.6
65.0
5800
DukPPee (Zhang ct. 2020)
Reshiet 152
02.5
92.6
69.5
53.4
IRRN--WW2
65.5
67.3
36.0
67.2
71.6
89.1
93.4
92.1
75.6
65.5
73.1
69.1
82.6
IRRNs-W-66
65.9
69.7
73.5
87.1
72.8
72.0
80.6
91.3
94.5
91.8
73.3
62.6
61.2
71.6
70.6
82.8
HIlFormen (Vaan 20213
IRlvamen- 6511
HRForuer
06.1
70.7
71.9
64.2
66.3
08.5
70.9
67.5
84.5
936
91.0
21.3
61.0
591
08.1
70.8
64.9
58.0
460
limbs, which are often surrounded by heavy fur. Tab. A5 further reports the PCK@0.1 I for each action category on the test set. We observe that different action types exhibit variations in pose accuracy, for example, with climbing generally achieving slightly higher accuracy compared to resting in most methods. This observation can be attributed to the higher potential for self-occlusion during resting, as chimpanzees tend to exhibit sigmificant self-occlusion due to their flexible joints. This is evident in the visualized examples in Fig. A5, where (a) and (c) depict resting poses with pronounced self-occlusion. In contrast, during climbing, the body is mostly in an extended state, as shown in (b) and (d). Consequently, the PCK tends to be slightly higher for climbing compared to resting as shown in Tab. A6. To validate this assumption, we further evaluate the performance of all the methods for non-occluded poses in Tab, A7. It is interesting to note that all the methods achieve high PCK accuracy when all the keypoints are visible. This demonstrates their effectiveness in accurately estimating poses when occlusions are minimal or absent.
These observations highlight the unique and intricate nature of chimpanzee pose estimation, which is complicated by their flcxible joint articulations and extended range of motion, as well as the dissimilar physical appearances of their fur in comparison to that of humans. Consequently, developing accurate pose estimation algorithms for chimpanzees requires careful consideration and specialized techniques that account for their unique characteristics.
Figs. A6 and A7 present the qualitative results of several models the Ch ChimpACT test split, with the ground-truth poses displayed in the last row. It is promising to observe that directly transferring human pose estimation algorithms to chimpanzees yielded decent performance. However, due to self-ocelusion lusions and different physical appearance and joint articulations, these models are susceptible to errors in estimating the positions of limbs, as seen in the misaligned right arm and leg of the young chimpanzee in the first column of Fig, A6 and the third column of Fig. A7.
Table A6: Results of the pose estimation by HRNet-W32 model. We report PCK @0.05 and PCK @0.1 metries.
(>)
(b)
No.
Action
PCK @0.05
PCK
(a) resting
43.8
62.5
(b)
climbing
81.2
93.8
(c)
resting
68.8
93.8
(d)
climbing
75.0
100.0
(c)
(d)
Figure A5: Visualization of predicted pose
by HRNet-W33 (Sun et al., 2019).
A8

---

Table AS: Results of the pose estimation track for each action category on ChimpACT test set. We report PCK PCK@0.1 metric. The action category number is consistent with Tab. Al. Method
Bockhone
0
4
5
6
10
11
12
15
16
18
19
20
21
22
ResNa-90
39.8
47.
48.0
56.
44.4
64.2
56.7
35.7
26.1
61.3
67.0
51
450
26
14.5
57
5.
29.9
26.
87.5
866
SumpleBasctiee (Niao et st, 2008)
RosNet-101
39.3
49.0
48.1
59.5
46.3
60.9
57.5
36.
21.7
75.0
09.5
57.5
52
35.5
29
6.6
28.8
26.6
62.5
525
Regrrssio
R55Net-152
40.5
45.
49.9
56.2
46.5
63.9
8
37.7
23.6
68
67
62
5
30
34.3
66
5.3
29
26
75,0
56.6
MokiieNerV2 40.8
68.5
53
28
298
62.5
RLB et 2021)
Re-Ner90
50
35
29
ResNet 1esNet-152
43.3
46.6
68.0
55.8
34.1
48.5
83.
29.7
31.6
25.0
62.
41.4
57.2
64.2
56.3
34.1
48.8
35.6
31
30.2
S0L1
75.0
59.8
CPM (Wes et al., 2016)
CPM
49.4
59.4
99.0
60.9
53.9
73.1
65.2
46.6
28.4
H1
66.6
52.5
601
41.6
34
10.6
3.8
36.1
41.8
68.8
66.2
Hourghass (Newell et al., 2016)
Hourglass 4
48.1
66.5
55.3
63.2
58.5
71.9
67.4
50.3
27.8
61.3
72.8
47.5
65.0
44.0
38.2
14.1
18
40.6
35.3
81.3
63.6
MobilcNerV2 (Sandler et at. 2018)
MobilcNe122
49.8
58.4
56.1
59.5
54.8
71.2
05.1
52.8
25.8
75.0
72.8
00.0
48.8
39.8
35.8
11.7
1.5
35.0
30.2
81.3
01.4
Ro
5233 60.0
57.2
60.9
56.3
73.9
66.0
53.3
25.2
81
72.0
62.5
65.0
45.0
35.4
5.4
2.4
39.1
37.6
75,0
65.7
SimpleBaseline (Kio
ResNet-10
52.0
71.9
36.9
08.8
1eNNa-112
Heonaags
HeN (Sun et al,, 2019)
HKNG--48
HRNee-Wi2
56.7
56.9
66.1
65.0
60.9
60.2
60.3
16..3
75.7
69.3
70.3
54.8
20.2
61.3
66.3
63.8
67.5
50.6
52.9
2.4
2.9
41.3
40.2
41.4
30.7
87.5
65.9
ResNa-50
52.1 60.9
57.5
62.1
57.4
72.6
66.1
56,0
26.8
75.0
72.9
56.3
6
42.
31
96
04
35.3
30.0
75.0
66.4
ResNet-10
52.6
62.6 57.6
61.4
56
71.8
67.7
$1.7
26.3
31
73.4
65.0
62.5
44.0
3
6.3
2.6
36.7
35.7
87,5
61.1
DarkPPss (Chang et al., 2020)
RosNet- 152
52.
63.3
57.8
59.4
57.9
73.2
67.9
53,
25.7
H1
76.3
57.5
0
45.
35.0
8.7
1.7
35.9
37.1
87.5
083
HRN
56.9
68.9
62.6
62.5
61.5
74.0
69.7
56.5
36.0
81
81
58
72.5
52.2
423
98
21
41.6
44.5
81.3 67.
IUNNet W48
57.6
67.7
59.2
73.7
69.5
56.3
28.5
53.8
52
36.7
39.7
40.1
75,0
639
HRFoemer (Yiun sl., 202
HRFemme-5
68.2
52.5
45.1
54.5
[[Foormer D 54.2
58.8
72.4 68.2
52.4
77.5
55.0
67.5
46.8
0.7
40.8
37.2
Table A7: Results of the pose estimation track for non-occluded poses ChimpACT test set. We report the PCK metrics. The non-oceluded poses denote those with all keypoints visible.
Method
Backbene
PCK@0.05
PCK@0.1
ResNet-50
47.6
80.6
SimpleRoseline (Ninn al., 2018)
ResNet-101
47.2
77.9
Regression
ResNet-152
54.5
MobilaNaV2
47.7
82.4
RLE (Li al., 2021)
ResNet-50
RssNa-101
28,4
52,9
$2.4
ResNe-152
60.0
85.5
CPM (Wei et ol., 2016)
CPM
74,0
894
Hourglass (Newell 2016)
Hourglave.4
77.6
MobilcNe (Sandler et al., 2018)
MobileNaa22
67.4
89.0
ResNet-50
75.2
$9.5
SimpleBaseline (Xian al., 2018)
ResNet-101
68.7
84.0
ResNet1122
71.4
Heamopp hoood
HRNec (Sun et 2019)
HRNer-W32
77.6
92.1
HRNet-Wa W48
79.4
90.2
ResNet-50
74.6
ResNet-101 101
74.6
$3.7
DarkPose iZhang et al., 2020)
ResNet-155 152
73.0
$9.1
HRNet W32
80.7
93.0
HRNet
78.4
HRFormer (Yuan et al., 2021)
HRForrer-B
Hororrer-S
70.9
75.6
88.5
88.4
C33 Spatiotemporal action detection
We adopted the same dataset partition as the first track. The frame sampling strategy was defined as T x x N. We ablated two strategies that continuously sample one frame every I frames and finally get input clip with T frames by setting T * 1. N denotes the number of clips which is used only when T 1. For the four representative methods, we ablated different modules. For LFB (Wu et al., 2019), we ablated different ways of the feature bank operator instantiations, by using non-local (NL) blocks (Wang et al., 2018) or average (Avg) or max (Max) pooling. For SlowFast (Feichtenhofer et al., 2019) and the variant SlowOnly, we ablated the context module (Ctx), which indicates that using both the Rol feature and the global pooled feature for the action classification.
We report the mAP for each model's best configuration several subcategory behaviors in Tab. A8. The models exhibit better performance in detecting locomotion and solitary object interactions, possibly because these actions are relatively simple and involve less interaction between individuals, making it easier for the model to distinguish between action patterns. However, there is still considerable room for improvement in existing models for action categories with higher levels of interaction, such as social interactions.
Results
We provide qualitative results in Figs. A8 and A9. All methods recognized the playing action of the two chimpanzees in Fig. A8, but incorrectly classified the touching actions as grooming in Fig, A9, These two action patterns exhibit subtle differennces that significantly challenge the models
A9

---

RLE
(Re5Net-152)
CPM
Hourglass
MobileNetV2
SimpleBaseline (ResNet-152)
HRNet-W48
DarkPose (ResNet-152)
DarkPose (HRNet-W48)
HRFormer-B
GT
Figure A6: Qualitative results of representative methods on the ChimpACT test set on the pose estimation task. The ground -ruth poses are shown in the last row.
to distinguish them accurately. We recommend referring to the supplementary video for the video results to observe the difference. The challenges of such distinctions highlight the need for stronger algorithms to address these issues effectively.
A10

---

RLE (ResNet-152)
CPM
Hourglass
MobileNetV2
SimpleBaseline (R55Net-152)
HRNet-W48
DarkPose (ResNet-152) 152)
DarkPose (HRNet-W48)
HRFormer-B
GT
Figure A7: More qualitative results of representative methods on the ChimpACT test set on the pose estimation task. The ground-truth poses are shown in the last row.
Overall, we hope that our work will inspire further research and development in the area of chimpanzee behavior recognition, with the ultimate goal of improving our understanding of chimpanzee and primate behaviors and ecology.
A11

---

plo-ing
ACRN
rooing
recii
LFB (w. NL LFB)
recti
ployig
plaving
SlowOnly (w. Ctx)
pleving
reofie
playlng
Fro Ping
plodig
plasing
plrting
SlowFast (w. Ctx)
wroning
resioe
restfre
rerths
playlng
sering
GT
roorg
refi
Figure A8: Qualitative results of representative methods on the ChimpACT test set on the spatiotemporal action detection task. The ground- tound actions shown in the last
Table A8: Results of spatiotemporal action detection track on ChimpACT test set.
Method
mAP moving climbing sol. obj, playing cating grooming playing being begged from apgressing being nursed
ACRN (Sun et al., 2018)
24.4
60.2
23.2
35.2
51.3
7.7
42.9
0.0
0.0
4.4
LFB (Wa 2019)
22.4
45.3
10.0
34.4
56.1
51.0
0.0
32.1
SlowOnly (Feichtenhofer 2019)
SlowFast (Feichtenhofer al., 2019)
24.5
60.9
37.2
47.3
35.3
10.4
49.2
0.0
0.0
7.5
A12

---

rertin
La OPATT
ACRN
ponding
rusting
racking
Harking
ructing
LFB (w. NL LFB)
DAtng groomss
runting
reeting
prooning
resting
groombng
SlowOnly (w. Ctx)
rwiting
Pestes
proentig
gaaming
grooming
bere grovel
SlowFast (w. Ctx)
Partin
rerting
buuting
rufry leadoy
louchng
renking
eentraaing madi
natin Necching
wirk: HOUOTE
retro tucting
GT
Figure A9: More qualitative results of representative methods the ChimpACT test the spatiotem- poral action detection task. The ground-truth actions are shown last row.
A13

---

D Data documentation
We follow the datashect proposed Gebru et al. (2021) for documenting our ChimpACT and associated benchmarks:
1. Motivation
(a) For what purpose was the dataset created?
This dataset was created to facilitate the study of chimpanzee behaviors, and ultimately advance our understanding of communication and sociality in non-human primates.
(b) Who created the dataset and behalf of which entity?
This dataset was created by Xiaoxuan Ma. Stephan P. Kaufhold, Jiajun Su, Wentao Zhu, Jack Terwilliger, Andres Meza, Yixin Zhu, Federico Rossano, and Yizhou Wang. Xiaoxuan Ma, Jiajun Su, Wentao Zhu, Yixin Zhu, and Yizhou Wang are with Peking University. Stephan P. Kaufhold, Jack Terwilliger, Andres Meza, and Federico Rossano are with the University of California. San Diego.
(c) Who funded the creation of the dataset?
The creation of this dataset was funded by Peking University and the University of California, San Diego.
(d) Any other Comments?
None.
2. Composition
(a) What do the instances that comprise the dataset represent? For video data, each instance video clip regularized from the raw video. Each instance contains video footage focusing on group of chimpanzees collected in Leipzig Zoo, Germany, For benchmarking, each instance has rich annotations of chimpanzee identities, poses, and actions. See Sec. 3 and Appx. A.
(b) How many instances are there in total?
We have 163 video instances in total.
(c) Does the dataset contain all possible instances or sample (not necessarily random) of instances from larger set?
No, this a brand-new dataset.
(d) What data does each instance consist of? See Sec. 3 and Appx. A.
(e) there label or target associated with each instance?
Yes. See Sec. 3 and Appx. A.
(f any information missing from individual instances?
No.
(g) Are relationships between individual instances made explicit? Yes.
(h) Are there recommended data splits?
Yes, we have separated the whole dataset into training, validation, and test set. See Sec. 4.1, Appx. C and the project website for details.
Are there any errors, sources of noise, or redundancies in the dataset? There are almost certainly some errors in video annotations. We did our best to minimize these, but some certainly remain.
(i) Is the dataset self-contained, or does link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)?
The dataset is self-contained.
(k) Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by pactor-patient confidentiality, data that includes the content of individuals' non-public communications)?
No.
()) Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? No.
A14

---

(m) Does the dataset relate to people?
No.
nn) Does the dataset identify any subpopulations (e.g., by age, gender)?
No.
(0) Is it possible to identify individuals (iee,, one or more natural persons), either directly or indirectly (Le., in combination with other data) from the dataset? Not applicable. Our dataset only contains chimpanzees.
(p) Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals racial ethnic origins, sexual orientations, religious beliefs, political opinions or union memherships, or locations; financial or health data; biometric genetic data; forms of government identification, such as social security numbers; criminal history)? No.
(q) Any other comments?
None.
3. Collection Process
(a) How was the data associated with each instance acquired? See Sec. 3.2 and Appx. A for details.
(b) What mechanisms or procedures were used to collect the data (e-g., hardware apparatus sensor, manual human curation, software program, software API)? We used JVC Everio cameras to collect video footage (Codec H.264), See Sec. 3.2 for details.
(c) If the dataset a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)?
See Sec. 3.3 and Appx. A for details.
(d) Who was involved in the data collection process (e.g students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)?
The video data was collected by the authors. The annotations were performed by the workers in BasicFinder CO., Ltd., and the workers were offered fair wage per the prearranged contract. See Sec. 3 and Appx. B for details.
(c)
Over what timeframe was the data collected?
The data were collected from 2015 2018, and labeled in 2022.
(f) Were any ethical review processes conducted (e.g., by an institutional review board)?
Not applicable. The Ch ChimpACT dataset raises no ethical concerns regarding the privacy information of human subjects, as it solely focuses on chimpanzees.
(g) Does the dataset relate people?
No.
(h) Did you collect the data from the individuals in question directly, obtain it via third parties or other sources (e.g., websites)?
Not applicable.
Were the individuals in question notified about the data collection? Not applicable.
Did the individuals in question consent to the collection and use of their data? Not applicable.
(k) If consent was obtained, were the consenting individuals provided with mecha- nism to revoke their consent in future or for certain uses?
Not applicable.
()) Has analysis of the potential impact of the dataset and its data subjects (e.g., data protection impact analysis) been conducted? Yes, Appx. B.
(m) Any other comments?
None.
4. Preprocessing, Cleaning and Labeling
A15

---

(a) Was any preprcesingddeainggabbiiii of the data done (e.g,, discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)?
Yes, see Sec. 3.
(b) Was the "raw" data saved in addition to the preprocessed/ccennc//hbbeed data (e.g,, to support unanticipated future uses)?
Yes, we provide the raw data on our project website.
(c) Is the software used to preproces//cean/aabe the instances available?
No. The annotation software is the private labeling platform provided by BasicFinder CO., Ltd. However, existing open-source annotation software such as DeepLabCut (Mathis al., 2018) could also be used to preprocess/clean/labe the instances.
(d) Any other comments?
None.
5. Uses
(a) Has the dataset been used for any tasks already?
No, the dataset is newly proposed by
(b) there a repository that links to any or all papers or systems that use the dataset?
Yes, we provide the link to all related information on our project website.
(c) What (other) tasks could the dataset be used for?
This dataset could be used for other research topics, including but not limited to pose tracking, few-shot learning, and transfer learning.
(d) Is there anything about the composition of the dataset or the way it was collected and hwwpr/eeeeeedeeeeee////aaaaa
that might impact future uses?
We propose to annotate the keyframe every 10 frames for the pose track and action detection track. For tracking track, we label all the frames.
(e) Are there tasks for which the dataset should not be used? The usage of this dataset should be limited to the scope understanding chimpanzec/non hhwmum primate behaviors.
(I)
Any other comments?
None.
6. Distribution
(a) Will the dataset be distributed third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created?
Yes, the dataset will be made publicly available.
(b) How will the dataset be distributed (e.g., tarball on website, API, GitHub)? The dataset could be accessed our project website.
(c) When will the dataset be distributed?
The dataset will be released by the end of 2023 on our project website.
(d) Will the dataset be distributed under copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)?
We release our benchmark under CC BY-NC 4.0 license.
(e) Have any third parties imposed IP-hased other restrictions the data associ- ated with the instances? No.
(f) Do any export controls or other regulatory restrictions apply to the dataset or to individual instances?
No.
(g) Any other comments? None.
7. Maintenance
(a) Who is supprringhhsttig/miinaainng the dataset? Xiaoxuan Ma is maintaining.
A16

---

(b) How can the ovner/cuurator/maaager of the dataset be contacted (e.g., email address)?
maxiaoxuan @ pku.edu.cn
(c) Is there erratum?
Currently, no. As errors are encountered, future versions of the dataset may be released and updated
website.
(d) Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances")?
Yes, applicable.
(c) If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were individuals in question told that their data would be retained for fixed period time and then deleted)?
Not applicable. The dataset does not relate to people.
(f) Will older versions of the dataset continue to be suppotee//nstcdmmainninee??? Yes, older versions of the benchmark will be maintained on our website.
(g) If others want to extend/augment/buid on/contribute to the dataset, is there mechanism for them to do so?
Yes, please get in touch with us by email.
(h) Any other comments?
None.
A17


Reviews:
[{"review_id": "2DcIGx2KZP", "reviewer": "Reviewer_H6yA", "summary": "", "questions": "", "limitations": "1. The dataset is 2D. I understand that the animal dataset is very hard to obtain, however, 2D limits its usefulness.\n2. The dataset size is not large.\n3. The writing and the figures can be improved.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The dataset on animals is very hard to obtain.\n2. The paper is well organized.\n3. The proposed dataset is interesting.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "br1gUvWoem", "reviewer": "Reviewer_nT7y", "summary": "", "questions": "", "limitations": "From an ethical perspective it could be noted that pose estimation, tracking etc. can be used for surveillance. ", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "•\tThe writing is clear and well organized. The manuscript is well-contextualized within the primate ethology field. \n•\tThe comparisons to human SOA algorithms are comprehensive. \n•\tThe dataset contains a rich range of behavioral annotations and a diversity of different annotations\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "mtlwvBsyhF", "reviewer": "Reviewer_8sjD", "summary": "", "questions": "", "limitations": "Some of the issues mentioned above in opportunities for improvement could be incorporated into the discussion of limitations.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "Datasets such as ChimpACT will be critical for the field to develop sophisticated new methods to study animal behavior in the wild. Such studies will have a range of scientific, and societal, and ecological impacts. These datasets are not trivial to collect and annotate.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "I4XOcfHvNb", "reviewer": "Reviewer_zjvm", "summary": "", "questions": "", "limitations": "The paper discusses limitations but does not discuss any potential negative societal impacts (authors might want to consider application of behavior models on humans, since some references were made to how this dataset might relate to other human models & datasets (ex: L202))", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- This chimp dataset is recorded over a long period of time, and manually labeled for a broad set of tasks (whereas existing primate datasets only focus on a subset of the tasks as reviewed by the authors). The recording conditions also appear to be more natural than indoor recordings.\n\n- The authors perform a large set of experimental evaluations with representative models from the machine learning and computer vision community. The evaluations seem comprehensive.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "SlaS60DymQ", "reviewer": "Reviewer_TrcR", "summary": "", "questions": "", "limitations": "Limitations are discussed (captivity, focus on one individual).", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The contribution fills a gap as datasets providing behavioral annotations for animals are scarce. \n- A large number of baselines are provided for three tasks.\n- Good explanation of the dataset statistics.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: All reviewers rate the submission above the acceptance threshold and they acknowledge the importance of the dataset for the study of chimpanzee social behavior in a semi-naturalistic environment. The challenges posed by the collection and annotation of such a benchmark are well addressed in this work. The authors are advised to make good use of the thoughtful recommendations provided by reviewers, especially with regard to discussion of: 1) occlusion handling, 2) suitability of the data for longitudinal analysis, 3) similarity to existing animal behavior datasets, 4) model selection and hyperparameter tuning.
Decision: Accept (Poster)