SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiaoxuan Ma  $^{1,\star}$
maxiaoxuan@pku.edu.cn
Stephan P. Kaufhold<sup>2,∗</sup>
spkaufho@ucsd.edu
Jiajun Su<sup>1, *</sup>
sujiajun@pku.edu.cn
Wentao Zhu
wtzhu@pku.edu.cn
Jack Terwilliger
jterwilliger@ucsd.edu
Andres Meza
anmeza@ucsd.edu
Yixin Zhu3,5,
Federico Rossano2,
frossano@ucsd.edu
Yizhou Wang  $^{1,3,4}$
yizhou.wang@pku.edu.cn
* X. Ma, S. Kaufhold, and J. Su contributed equally. Corresponding authors
<sup>1</sup> CFCS, School of Computer Science, Peking University, China
$^{2}$  Department of Cognitive Science, University of California, San Diego, USA
<sup>3</sup> Institute for Artificial Intelligence, Peking University, China
$^{4}$  Nat'l Eng. Research Center of Visual Technology, China
$^{5}$  PKU-WUHAN Institute for Artificial Intelligence, China
https://shirleymaxx.github.io/ChimpACT/
Figure 1: Sample frames and annotations from a ChimpACT clip. While we also annotate visibility for both the bounding box and the keypoint, these are omitted here for clarity.
Understanding the behavior of non-human primates is crucial for improving animal welfare, modeling social behavior, and gaining insights into distinctively human and phylogenetically shared behaviors. However, the lack of datasets on non-human primate behavior hinders in-depth exploration of primate social interactions, posing challenges to research on our closest living relatives. To address these limitations, we present ChimpACT, a comprehensive dataset for quantifying the longitudinal behavior and social relations of chimpanzees within a social group. Spanning from 2015 to 2018, ChimpACT features videos of a group of over 20 chimpanzees residing at the Leipzig Zoo, Germany, with a particular focus on documenting the developmental trajectory of one young male, Azibo. ChimpACT is both comprehensive and challenging, consisting of 163 videos with a cumulative 160,500 frames, each richly annotated with detection, identification, pose estimation, and fine-grained spatiotemporal behavior labels. We benchmark representative methods of three tracks on ChimpACT: (i) tracking and identification, (ii) pose estimation, and (iii) spatiotemporal action detection of the chimpanzees. Our experiments reveal that ChimpACT offers ample opportunities for both devising new methods and adapting existing ones to solve fundamental computer vision tasks applied to chimpanzee groups, such as detection, pose estimation, and behavior analy-
sis, ultimately deepening our comprehension of communication and sociality in non-human primates.
Studying the behavior of non-human primates is essential for gaining evolutionary insights (Langergraber et al., 2012), conducting biomedical research (Schapiro et al., 2005), and improving animal welfare (Dawkins, 2003; Gonyou, 1994). Furthermore, given the close phylogenetic proximity between humans and non-human primates, it provides an ethically sound and effective avenue to probe the roots of human sociality (The Chimpanzee Sequencing and Analysis Consortium, 2005). Traditional field research typically requires researchers to enter wildlife conservation areas for extended durations, sometimes spanning multiple years. This involves habituating primate groups to human presence, capturing video footage, and laboriously manually coding these videos for subsequent statistical analysis (Hobaiter et al., 2017; Fröhlich et al., 2020; Surbeck et al., 2017; Luncz et al., 2018; Sirianni et al., 2015). While video coding is heralded as the gold standard for distilling rich, nuanced behavioral patterns (Wiltshire et al., 2023), its practical utility hinges on the efficiency of the coding process. This not only demands researchers with specialized expertise but is also prone to attentional biases.
Recent strides in computer vision offer promise for the automated analyses of non-human primate behaviors, especially those of chimpanzees. Nevertheless, the scarcity of high-quality longitudinal datasets remains a bottleneck. Assembling chimpanzee behavioral data is a formidable endeavor, necessitating substantial resources and expertise. This process entails continuous video recording and meticulous manual annotation, with a keen emphasis on annotation accuracy and consistency. While some datasets (Marks et al., 2022; Bala et al., 2020) confine subjects to indoor enclosures, resulting in atypical and constrained environments, others resort to sourcing and labeling chimpanzee images online (Labuguen et al., 2021; Desai et al., 2022; Ng et al., 2022; Yao et al., 2023). Unfortunately, these often overlook the intricate social dynamics inherent to chimpanzee groups, hindering a comprehensive study of their social behaviors and social relationships.
Addressing the existing limitations, we introduce ChimpACT, a comprehensive longitudinal dataset tailored for the in-depth study of chimpanzee social behavior in a semi-naturalistic setting, replete with annotations of instance bounding boxes, body poses, and spatial-temporal action labels. A comparison with other datasets is provided in Tab. 1. ChimpACT encompasses footage of a specific chimpanzee group residing at Leipzig Zoo, Germany, with a particular focus on a juvenile male named Azibo (refer to Fig. 1). The data, gathered between 2015 and 2018, employs focal sampling
Table 1: Comparison of ChimpACT with existing primate behavioral datasets. Square-bracketed numbers denote label counts for the chimpanzee category.  $\varnothing$  denotes undocumented. For the "Species" row, G represents general, P for primates, M for macaque, and C for chimpanzee. In the "Source" row, I stands for Internet, Z for zoo, C for cage, W for wild, and CP for captive.
Dataset	Species	Track 1 detection, tracking, ReID	Track 2 pose estimation	Track 3 action recognition	Source
ID #	frame #	box #	track	frame #	pose #	track	dim.	class #	label #
AP-10K (Yu et al., 2021)	G	X	X	X	X	10,015	13,028 [<500]	X	2D	X	X	I
AnimalKingdom (Ng et al., 2022)	G	X	X	X	X	33,099	99,297 [576]	X	2D	140	30,100 [∅]	I
OpenApePose (Desai et al., 2022)	P	X	X	X	X	71,868	71,868 [18,010]	X	2D	X	X	I
OpenMonkeyChallenge (Yao et al., 2023)	P	X	X	X	X	111,529	111,529 [<10,000]	X	2D	X	X	I & Z
OpenMonkeyStudio (Bala et al., 2020)	M	X	X	X	X	194,518	33,192 [0]	✓	3D	X	X	C (6.7m²)
MacaquePose (Labuguen et al., 2021)	M	X	X	X	X	13,083	16,393 [0]	X	2D	X	X	I & Z
SIPEC (Marks et al., 2022)	M	4	191	2,200 [0]	✓	X	X	X	X	4	∅	C (15m²)
CCR (Bain et al., 2019)	C	13	936,914	1,937,585	✓	X	X	X	X	X	X	W
ChimpACT (Ours)	C	23	160,500	56,324	✓	16,028	56,324	✓	2D	23	64,289	CP (4400m²)
(Altmann, 1974). Born in April 2015, Azibo<sup>1</sup> has been living in the group since birth, providing a unique perspective on the development of an individual within a chimpanzee group characterized by well-defined kin relationships. (also depicted in Fig. 2a). The footage covers the daily lives of over 20 chimpanzees in a group, aggregating to 163 video recordings, approximately 160,500 frames, and spanning around 2 hours.
Our annotations on ChimpACT are extensive, marking each individual's detection, tracking, identification, pose estimation, and spatiotemporal action detection. Sample frames with their corresponding annotations are illustrated in Fig. 1. Each chimpanzee's identity is confirmed by a seasoned behavioral researcher familiar with the Leipzig chimpanzees, ensuring data precision and trustworthiness. Crucially, we employ an ethogram (detailed in Fig. 2b) devised by the same expert for fine-grained action labels. To our knowledge, ChimpACT is the first to furnish ethogram annotations for the machine learning and computer vision community. This bespoke ethogram delineates behaviors into four categories: locomotion, object interaction, social interaction, and others, with each encompassing several detailed actions we diligently annotate.
While advancements in computer vision have notably addressed human-centric tasks, such as human pose estimation (Sun et al., 2019; Xiao et al., 2018), the dearth of chimpanzee datasets has curtailed progress on chimpanzee-specific challenges. Despite their genetic closeness to humans (The Chimpanzee Sequencing and Analysis Consortium, 2005), deciphering chimpanzee behaviors is intricate due to their unique morphology, appearance, and keypoint articulation. Highlighting the importance of crafting sophisticated chimpanzee perception models, we evaluate prominent human perception methods on three tracks: (i) detection, tracking, and identification (ReID), (ii) pose estimation, and (iii) spatiotemporal action detection. Our findings underscore ChimpACT's potential as a platform for the community to pioneer advanced techniques for better perception of the chimpanzees and ultimately contribute to a deeper understanding of non-human primates.
Computer vision for animals A myriad of datasets and benchmarks have emerged, harnessing computer vision techniques to advance animal research. For instance, 3D-ZeF20 (Pedersen et al., 2020) introduces 3D tracking of zebrafish to the MOT benchmarks. AnimalTrack (Zhang et al., 2023) emphasizes multi-animal tracking across a spectrum of species. AP-10K (Yu et al., 2021) and APT-36K (Yang et al., 2022) venture into animal pose estimation for diverse species. AnimalKingdom (Ng et al., 2022) extends its focus to fine-grained multi-label action recognition. Moreover, several studies have delved into multi-agent behavior understanding from a social interaction perspective (Sun et al., 2021, 2023). Distinctly, ChimpACT stands out as a holistic benchmark, encompassing three varied downstream tasks and boasting rich annotations of social interactions.
Human video datasets In contrast to animal-centric video datasets, a more substantial collection is tailored to human subjects, addressing diverse human-centric video understanding tasks. For instance, the MOT Challenge (Milan et al., 2016) is curated for multi-person tracking. Other benchmarks like COCO (Lin et al., 2014) and MPII (Andriluka et al., 2014) cater to human pose estimation. Meanwhile, datasets such as Kinetics (Kay et al., 2017), ActivityNet (Fabian Caba Heilbron and Niebles, 2015), and AVA (Gu et al., 2018) are dedicated to human action recognition. With ChimpACT, we encompass analogous tasks but introduce challenges specific to chimpanzee behavior.
Datasets on primate behavioral understanding Most existing primate datasets are tailored towards individual primate detection and pose estimation. These either stem from confined indoor settings (Bala et al., 2020; Marks et al., 2022) or are amassed and labeled from online sources (Labuguen et al., 2021; Desai et al., 2022; Ng et al., 2022; Yao et al., 2023). The former can induce atypical behavioral patterns, while the latter often omits longitudinal interactions, rendering them suboptimal for analyzing chimpanzee social dynamics. A notable exception is the CCR dataset (Bain et al., 2019), chronicling 13 chimpanzees in the Bossou forest over two years. Yet, it primarily focuses on individual detection and recognition, lacking behavioral annotations, which limits its efficacy for probing the social nuances of wild primates. Tab. 1 offers a comprehensive comparison. The narrow focus of most primate datasets on singular tasks restricts their breadth and adaptability to diverse research inquiries. Contrarily, ChimpACT presents a multifaceted approach, encompassing identities, kinship, detection labels, pose annotations, ethograms, and fine-grained action labels. This richness
<sup>1</sup>Details about Azibo can be found at https://tinyurl.com/azibo-chimp/.
(b)
Figure 2: (a) Kinship of the observed chimpanzee group. Rectangles and ellipses represent males and females, respectively, with arrows flowing from the parents to the child. Their vertical position relative to the time axis indicates the year of birth. (b) Ethogram with annotated behaviors.
positions it as an indispensable tool for devising advanced chimpanzee behavior analysis methods and enriching the overarching comprehension of primate behavior.
Methods for primate behavioral analysis Deciphering primate behavior is instrumental in understanding their social dynamics and cognitive abilities. Behavioral analysis often encompasses subtasks like individual detection, tracking, and identification (Bain et al., 2019; Marks et al., 2022), pose estimation (Labuguen et al., 2021; Desai et al., 2022; Mathis et al., 2018; Wiltshire et al., 2023), and behavior recognition (Ng et al., 2022; Bain et al., 2021). While each task has specialized techniques, many are rooted in human behavioral research. Numerous algorithms exist for human tracking (Bewley et al., 2016; Pang et al., 2021), pose estimation (Sun et al., 2019; Xiao et al., 2018), and behavior recognition (Feichtenhofer et al., 2019). However, due to the dearth of primate datasets, primate behavioral analysis often repurposes algorithms designed for humans, including:
- Detection, tracking, and ReID identify individual primates in videos, often leveraging established object or human detection algorithms like Mask-RCNN (He et al., 2017). For instance, SIPEC (Marks et al., 2022) employs Mask-RCNN with a ResNet backbone (He et al., 2016) to track and segment macaque. Bain et al. (2019) utilize CNNs to crop and identify individual chimpanzees.
- Pose estimation discerns primate poses, frequently adapting human pose estimation methods like SimpleBaseline (Xiao et al., 2018). DeepLabCut (Mathis et al., 2018; Lauer et al., 2022), for instance, employs ResNet-50 with ImageNet pre-trained weights for 2D animal pose estimation. SIPEC (Marks et al., 2022) modifies SimpleBaseline for 2D macaque poses.
- Behavior recognition identifies primate actions and interactions. Contemporary methods (Bain et al., 2021; Bohnslav et al., 2021) often derive from human action recognition algorithms like SlowFast (Schindler and Steinhage, 2021). Notably, Bain et al. (2021) integrates audio cues for classifying two simple non-interactive behaviors: nut cracking and buttress drumming. In contrast, ChimpACT encompasses over 20 daily behaviors under an ethogram hierarchy, capturing both solitary actions and intricate social interactions.
In essence, primate behavioral analysis is a multifaceted endeavor, intertwining computer vision, machine learning, and primatology. The advent of ChimpACT marks a significant stride towards unraveling the intricate social tapestry of our primate kin.
3 ChimpACT
3.1 Dataset description
ChimpACT comprises about 2-hour video footage of chimpanzees recorded at the Leipzig Zoo in Germany between 2015 and 2018. The videos focus on one male chimpanzee, Azibo, who was born in April 2015 to Swela and has lived with the A-chimpanzee group<sup>2</sup> at the Leipzig Zoo ever since.
2The A-chimpanzee group is among the most extensively studied zoo-residing chimpanzee cohorts. Its members have been subjects of both behavioral and cognitive studies, spanning observational and experimental
The longitudinal observation of Azibo offers a rare lens into his behavioral evolution, social dynamics, and intra-group relationships. With over 20 individuals in the group, ChimpACT serves as a treasure trove of insights into chimpanzee behavior and social intricacies. Key attributes of ChimpACT are delineated below.
Longitudinal data Spanning four years, ChimpACT chronicles the life of a stable zoo-residing chimpanzee group, offering a rare glimpse into the nuances of chimpanzee social behavior development. Tracking the growth and interactions of a young chimpanzee within this group sheds light on chimpanzee socialization, the evolution of social skills (Matsuzawa, 2013), the formation of social bonds and integration into the dominance hierarchy (Matsuzawa et al., 2006), and the acquisition of group-specific cultural behaviors (Van Leeuwen, 2021; Musgrave et al., 2021).
Semi-naturalistic and social environment The videos in ChimpACT capture chimpanzees in their semi-naturalistic habitats at Leipzig Zoo, split between indoor (96 videos) and outdoor (67 videos) enclosures. The indoor space, spanning roughly  $400~m^2$ , features a plethora of environmental enrichments, ranging from wooden climbing structures and hammocks to vegetation and foraging boxes. When weather permits, the chimpanzees have access to a  $4000~m^2$  outdoor area, replete with vegetation, surrounded by an artificial river, and complemented by enrichments similar to the indoor space. This blend of environments ensures the dataset's relevance for both naturalistic and artificial environments. The multifaceted physical and social surroundings of the chimpanzees further imbue the dataset with intricate behaviors and social dynamics.
Ethogram with solitary and social behaviors ChimpACT captures the daily life of group-living chimpanzees, offering invaluable insights into the evolution and sustenance of their social behaviors and relationships (Nishida et al., 2010). By focusing on a juvenile chimpanzee, ChimpACT illuminates facets of social learning, communication, bonding, and more, all pivotal in the social and ecological life of chimpanzees (Bard et al., 2014). To systematically represent these behaviors, we composed an ethogram—a detailed catalog of behavioral categories, depicted in Fig. 2b (further details in Appx. A). This ethogram organizes behaviors into four primary categories, like locomotion and social interaction, each further subdivided into several fine-grained actions, meticulously annotated and validated with expert oversight. By delving into these behaviors, ChimpACT elucidates not only the social dynamics shaping social relationships but also the cognitive and ecological influences on juvenile chimpanzee behaviors.
3.2 Dataset collection
The focal video data were collected with the Chimpanzee-A group housed at Leipzig Zoo, Germany, using focal sampling (Altmann, 1974). Videographers were instructed to focus on Azibo and his mother, Swela, but also on capturing the environmental context and his interactions with other chimpanzees. Videos from ChimpACT were sampled from a larger set of around 405 hours of longitudinal focal video recordings of the dyad between 2015 and 2018. These videos were recorded by several research assistants during the daytime (7am-4pm) using tripod-mounted RGB cameras. Two JVC Everio camera models were utilized across the years, filming with a framerate of 25 (Codec H.264) and with resolutions of  $720 \times 578$  and  $1280 \times 720$ , respectively. The mother-infant dyad was filmed for about five hours each week during the observation period. The footage contains both optical zoom and camera movements.
3.3 Dataset tasks and annotations
ChimpACT supports three tracks: (i) chimpanzee detection, tracking, and ReID, (ii) chimpanzee pose estimation, and (iii) spatiotemporal action detection. We provide fine-grained annotations for each track. From the extensive footage, we curated 163 video clips, each approximately 1000 frames in length. Fifteen adept annotators were then tasked with annotating bounding boxes, body keypoints, and fine-grained behavioral classes for each chimpanzee at intervals of every 10 frames. To ensure accuracy and consistency, a behavioral researcher familiar with the chimpanzee group meticulously reviewed and refined the identity and behavioral class annotations. For a deeper dive into the annotation process and its quality, please refer to Appx. A and our dedicated website.
designs, conducted by researchers affiliated with the MPI for Evolutionary Anthropology (Baker, 2022; McEwen et al., 2022).
Detection, tracking, and ReID This task encompasses the detection and tracking of individual chimpanzees across video sequences, subsequently coupled with their re-identification. ChimpACT features over 23 distinct chimpanzee individuals, each identified by a primate expert familiar with the Leipzig A-group chimpanzees. Initially, annotators were instructed to delineate the bounding box of each chimpanzee, ensuring consistent box IDs for the same individual throughout a video clip. Subsequently, the expert matched these box IDs with the corresponding true names of the chimpanzees, resulting in the identification of 23 unique individuals. Additionally, every annotated bounding box is attached with a visibility attribute, indicating if the chimpanzee is fully visible, truncated, or occluded in a given frame. Such visibility annotations can support the reasoning of the chimpanzee behavior, potentially bolstering tracking robustness. Fig. 3a illustrates the occurrence frequency (on a log scale) of each individual, revealing a long-tail distribution. This pattern aligns with the focal sampling strategy, where Azibo is the primary subject. Notably, Swela, Azibo's mother, also exhibits a high occurrence frequency, resonating with prior studies (Boesch, 1996).
Pose estimation Pose estimation aims to predict the locations of the chimpanzee joints that have semantic meaning, such as the knee and shoulder, from an input image. There are four keypoints on the chimpanzee's face (i.e., two for the eyes, and one each for the upper and lower lips), for a total of 16 chimpanzee keypoints (refer to Sec. 3.3 and Fig. 4). Annotators are tasked with marking the 2D joint coordinates and the visibility status of each joint. We adopt the visibility protocol from the COCO 2D human keypoint annotations (Lin et al., 2014), where a value of 0 indicates a joint outside the image frame, 1 signifies an obscured joint within the image, and 2 designates a clearly visible joint. Such an annotation protocol affords rea
(b)
Figure 3: (a) Distribution (in log scale) of annotations for each individual. (b) Distribution (in log scale) of annotations for each behavior. Vector graphics; zoom for details.
son about chimpanzee's orientation and action based on facial joint visibility. For instance, the chimpanzee might be eating something if the two lips are apart. Sample frames showcasing pose annotations are depicted in Fig. 1. Notably, ChimpACT holds the potential for future expansion to encompass pose tracking tasks, analogous to the PoseTrack (Andriluka et al., 2018) for humans.
Table 2: Keypoint definitions for chimpanzee.
No.	Definition	No.	Definition
0	Root of hip	8	Right eye
1	Right knee	9	Left eye
2	Right ankle	10	Right shoulder
3	Left knee	11	Right elbow
4	Left ankle	12	Right wrist
5	Neck	13	Left shoulder
6	Upper lip	14	Left elbow
7	Lower lip	15	Left wrist
Figure 4: Keypoint definitions for chimpanzee.
Spatiotemporal action detection Spatiotemporal action detection seeks to attribute one or multiple behavioral labels to each bounding box containing a chimpanzee, leveraging the spatiotemporal context within a video clip. Our ethogram, detailed in Fig. 2b, delineates 23 nuanced subcategories of behaviors and guides the fine-grained annotations of chimpanzee behavior, such as "climbing"
within the "locomotion" category. Notably, within the realm of social interactions, we meticulously differentiate between the action performer and receiver. For instance, the grooming behavior is bifurcated into "grooming" and "being groomed." Every chimpanzee in a frame has its subcategory behavior annotated. It is not uncommon for an individual to simultaneously exhibit multiple behaviors, exemplified by Swela's "carrying" and "moving" actions in Fig. 1. The distribution of these behavioral annotations, visualized in Fig. 3b on a log scale, reveals a long-tail distribution, mirroring the authentic behavioral tendencies of chimpanzees in their natural habitats.
(b)
Figure 5: (a) Distribution of the annotated behavior categories. (b) Distribution showcasing individuals alongside their respective social behaviors. Vector graphics; zoom for details.
Fig. 5a showcases the distribution of the annotated behaviors, with social interactions constituting approximately  $35\%$  of the total annotations. Furthermore, Fig. 5b delineates the distribution of labeled social behaviors across distinct individuals, highlighting grooming, playing, and touching as predominant activities within the social dynamics of the group-living chimpanzees.