BigMaQ: A Big Macaque Motion and Animation Dataset Bridging Image and 3D Pose Representations
Authors: Lucas Martini
Alexander Lappe
Anna Bognár
Rufin Vogels
Martin A. Giese
Venue: ICLR 2026 Poster (2026)
Keywords: surface tracking
action recognition
benchmark
animals
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: BigMaQ, a large-scale motion capture dataset of macaques, demonstrates the benefits of shape-based pose descriptions for behavioral recognition.

Abstract: The recognition of dynamic and social behavior in animals is fundamental for advancing several areas of the life sciences, including ethology, ecology, medicine and neuroscience. Recent progress in deep learning has enabled an automated recognition of such behavior from video data. However, an accurate reconstruction of the three-dimensional (3D) pose and shape has not been integrated into this process. Especially for non-human primates, the animals phylogenetically closest to humans, mesh-based tracking efforts lag behind those for other species, leaving pose descriptions restricted to sparse keypoints that are unable to fully capture the richness of action dynamics. To address this gap, we introduce the $\textbf{Big Ma}$ca$\textbf{Q}$ue 3D Motion and Animation Dataset ($\texttt{BigMaQ}$), a large-scale dataset comprising more than 750 scenes of interacting rhesus macaques with detailed 3D pose descriptions of skeletal joint rotations. Recordings were obtained from 16 calibrated cameras and paired with action labels derived from a curated ethogram. Extending previous surface-based animal tracking methods, we construct subject-specific textured avatars by adapting a high-quality macaque template mesh to individual monkeys. This allows us to provide pose descriptions that are more accurate than previous state-of-the-art surface-based animal tracking methods. From the original dataset, we derive BigMaQ500, an action recognition benchmark that links surface-based pose vectors to single frames across multiple individual monkeys. By pairing features extracted from established image and video encoders with and without our pose descriptors, we demonstrate substantial improvements in mean average precision (mAP) when pose information is included. With these contributions, $\texttt{BigMaQ}$ establishes the first dataset that both integrates dynamic 3D pose-shape representations into the learning task of animal action recognition and provides a rich resource to advance the study of visual appearance, posture, and social interaction in non-human primates. The code and data are publicly available at [https://martinivis.github.io/BigMaQ/](https://martinivis.github.io/BigMaQ/).

Published as a conference paper at ICLR 2026
BIGMAQ: A BIG MACAQUE MOTION AND ANIMATION DATASET BRIDGING IMAGE AND 3D POSE REPRESEN- TATIONS
Lucas Martini Alexander Lappe Anna Bognár Hertie Institute, University of Tübingen IIMPR-IS lucas. mact ini -uniinneen de
Rufin Vogels
Martin Giese
KKU Leuven
ABSTRACT
The recognition of dynamic and social behavior in animals is fundamental for ad- vancing several areas of the life sciences, including ethology, ecology, medicine and neuroscience. Recent progress in deep learning has enabled an automated recognition of such behavior from video data. However, an accurate reconstruc- tion of the three-dimensional (3D) pose and shape has not been integrated into this process. Especially for non-human primates, the animals phylogenetically closest to humans, mesh-based tracking efforts lag behind those for other species, leaving pose descriptions restricted to sparse keypoints that are unable to fully capture the richness of action dynamics. To address this gap, we introduce the Big MacaQue 3D Motion and Animation Dataset (BigMaQ), a large-scale dataset comprising more than 750 scenes of interacting rhesus macaques with detailed 3D pose de- scriptions of skeletal joint rotations. Recordings were obtained from 16 calibrated cameras and paired with action labels derived from curated ethogram. Extend- ing previous surface-based animal tracking methods, we construct subject-specific textured avatars by adapting a high-quality macaque template mesh individual monkeys. This allows to provide pose descriptions that are more accurate than previous state-of- the-art surface-bart based animal tracking methods. From the orig- inal dataset, we derive B00MaQ500, an action recognition benchmark that links surface-based pose vectors to single frames across multiple individual monkeys. By pairing features extracted from established image and video encoders with and without our pose descriptors, we demonstrate substantial improvements in mean average precision (mAP) when pose information is included. With these con- tributions, Bi gMa establishes the first dataset that both integrates dynamic 3D pose-shape representations into the learning task of animal action recognition and provides a rich resource to advance the study of visual appearance, posture, and social interaction in non-human primates. The code and data publicly available at Intttts::/// /tte::/mmartiiiiii github. 1o/ooigga0 BigMaC/
Figure 1: Bi ga A motion capture dataset with surface-based modeling for non-human primate behavior recognition. The dataset provides colored avatars that can be rendered from arbitrary views, including dynamic action reconstructions (left), rich individual and social interactions (right).

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Poses and shapes of human bodies have been effectively parameterized by 3D surface models using large amounts of data (Mahmood et al., 2019; Xu et al. 2020; Yu et al. (020), enabling even the modeling of person-specifhe anatomy (Guo et al. 2022). The realization of a comparable represen- tation for - its wide utility in the biological sciences is hampered by the scarcity of accurate 3D data and motion recordings. As a result, surface-based models either come a deformable generic shape space (Zuffi et al. 2017) or they are tailored specifically to individual species (Badger et al., 2020; An et al., 2023). Although surface descriptions exist for a variety of quadrupeds, behavioral research in species such rodents, flies, or fish, typically relies on less de- tailed representations, most often in the form of 2D keypoints (Hsu & Yitri 202T] MMnseees et al.) 2022 Lauer et al. 2022 An et al. 2023 Wu et al. 2024). The same holds for non-human primates (NHPs), (HHPss),artte particularly macaques, which provide critical insights that translate to humans (Gibbs et al.)
2007 Cooper al. [022 Miller et al.)
2012), especially in visual and behavioral Hesse & Tsao 2020)
1996; Bethell
neuroscience (Rushworth et al.) 2013
In recent years, considerable progress has been made in capturing macaque body movements from images and videos (Bala et al. [2200] Labuguen et al. 202 Yao et al. 2022). However, compared humans and other animals, existing approaches lack the integration of detailed 3D shape repre- sentations of macaques. Since access to these animals is limited, most annotated videos are sampled either from wildlife scenarios (Ma et al., 2023 Brookes et al., 2024). where pose, if described at all, is represented by sparse 2D keypoints, or from captive environments, where movements are rep- resented by 3D positions or keypoints (Bala et al., 2020: Marks et al.] 2022, Martini et al. 2024] Matsumoto et al.) 2025). Other approaches have incorporated synthetic data to couple 3D informa- tion with 2D image keypoints (Debnath et al., (025). However, none of these efforts have included an accurate model of macaque body shape, nor have generic shape models adequately addressed this gap. Therefore, recent neurocomputational studies relied on human-based models to reconstruct 31) shape in monkey images (Yilmaz et al., 2025).
In this work, we address these limitations by augmenting the pose tracking of macaques with re- constructions of the complete 3D body surface, yielding more expressive models. In particular, we present Big g, a large-scale macaque action dataset with realistic dynamic surface reconstruc- tions covering more than 750 scenes of interacting individuals. A representative example is shown in Figure| I By disentangling pose and shape. our dataset enables a detailed description of social interactions beyond previous keypoint-basee
approaches. In addition to the surface models, we pro- vide extensive annotations including individual identities, segmentation masks, 2D keypoints, 16 calibrated viewpoints, per-frame poses, and action labels.
Our dynamic pose and shape representations describe actions across different body shapes, while also modeling finer movements such as hand rotations. By extending previous work in dynamic surface modeling of animals, we further incorporate a symmetric time loss, cropped differential rendering, and integrate texture into a sped-up optimization process, which makes the processing of the large amount of video data possible. The resulting dataset composed of dynamic and realistic avatars constitutes a unique resource, enabling detailed studies in behavior and perception for non- human primates. We surface-based pose representations with action recognition in videos. In comparison with ofate-of- the-art mesh-based tracking approaches, our approach results in higher quality reconstructions both qualitatively and quantitatively. Finally, we create BigMaQ500, a benchmark of over 8k labeled videos with associated poses for action recognition in macaques. Using this dataset, we show that incorporating our pose descriptors along with video features from various foundation models not only substantially improves action recognition performance, but also outperforms commonly used descriptors in 2D and 3D.
aware of similar dataset in other animal species combining realistic
2 RELATED WORK
While there exists plethora of datasets for animal pose estimation, only subset of these also provides annotations for behavior recognition. Fewer still predict poses with mesh-basee models (see Table I)
2

---

Published as a conference paper at ICLR 2026
Shape and Pose Reconstruction of Animals. In animals, the simultancous estimation of pose and shape from images can be split into mooee--rree (Li al. 2022 Yang et al. 2022a Li et al. 2024) and
model-basee approaches (Zuffi et al. 2017: Wang et al. [02 While model-free approaches promising for 3D shape retrieval from images or videos, model-based methods yield higher quality results for complex poses and real (non-synthetic) data (Rüegg et al., 2023). Therefore, we focus our discussion on these approaches. For many common animal species, model-based approaches rely on a generic quadruped shape space that was designed by scanning variety of animal toy figurines, called SMAL (Zuthi et al. 2017) For dogs, this shape space has been fine-tuned to capture a greater diversity of species (Biggs et al. Rüegg et al. (022). The shape model has also been extended to track other species deviating from its original shape space, like bears or camels (Biggs) ett al. 2019 Zuffi et al., 2018). However, SMAL's utility is limited for animals whose shape or naturalistic poses differ substantially from those of quadrupeds. To address this limitation, recent work has focused on customizing template meshes for specific animal groups. For example, Wang et al.] (202Ib) incorporated deformations from template mesh of bird, creating unique shape space for different bird species. The model-based approach has also been extended to estimate shape and pose in videos using the SMAL shape model (Biggs et all,, [019 Sabathier et al.] (025). or by using template meshes of pigs, mice and dogs with multi-view recordings instead (An et al., 2023]. Without access accurate 3D data, even SMAL-based approaches rely on forming more informative priors by synthetic data or numan-feedbaed (Zuffi et al., 2019; Xu et al.] 2023). A recent line of work therefore extended the SMAL shape space to a wider range of mammal species using transformer models and synthetic data (Lyu et al. 2025b a). Given that the pose space of NHPs differs substantially from other quadrupeds (Bala et al., 2020; Yao et al., [2022] Labuguen
et al. 202T]] Ma et al., (2023), these species ideally need rich motion and shape recordings similar to
those in humans for realistic modeling of body and movement.
Pose and Action Recognition in NHPs. The aforementioned surface-based pose descriptions have only recently been introduced to describe animal poses in datasets (Xu et al. 2023
Lyu al.,
20256). Traditionally, the pose is defined by 2D keypoints for both arbitrary species Yang et al,
20226 Ng et al.] (022) and NHPs in particular (Yao et al. 2022 Desai al. 2023
Labuguen
et al. 2021). l). Although datasets exist that enable the training of models for automated behavior
recognition in conjunction with pose information (Marshall et al. 2021; Ng et al. 2022 Liu al.
2023 Ma et al. 2023), the tasks of behavior recognition and pose estimation are typically treated
as distinct training tasks (Luvizon et al. 2018 Duporge et al. 2024). Attempts to integrate pose information into action recognition (Luvizon et al. 2018 Choutas et al. 2018 Shah et al. 2022) have received less attention in recent research, largely due to rapid progress in model architectures for video classification, ranging from 3D convolutional (Carreira & Zisserman 2017), two-stream (Feichtenhofer et al. 2019), to transformer-
networks (Arnab et al. [202 Rajasegaran et al,
3023]Zhas et al., 2024). Further recent developments in this domain include large vision-languees
models (Zhong et al. 2025), or foundational encoders for video understanding (Zhao et al. (024). which surpass the performance of species-specifie models in behavior recognition of NHPs (Mal
jet al. 2023). Therefore, datasets that primarily focus on behavior recognition provide only minimal
tracking annotations, such as bounding boxes (Brookes et al., 2024| Duporge et al. 2024 Fuchs
ct al. 2025 Vogg et al. 2025 Huang et al. 2025). or none at all (Chen et al. 2023).
Despite these advances in deep-learning based feature extraction from video, pose descriptions re- main widely used to characterize, interpret and understand animal behavior (Mathis & Mathis 2020)
Bala et al. [020; Hsu & Yttri [02T]] Lauer et al.) 2022, Monsees et al.) 2022] An et al. 2023 Mat-
sumoto et al., 2025| Raman et al. 2025). In laboratory studies of key model species (c.g., rodents),
higher-cost 3D keypoint representations are increasingly paired with behavioral annotations (Mar- shall et | 2021 Patel et al. (023). Beyond keypoints, 3D surface models have only recently been incorporated into the behavioral analysis of pigs (An et al.. 2023), but as absolute, surface-level descriptors. A natural next step is, however, to include model-based pose representations into be- havioral recognition, which has been shown to yield state-of of-te-of-the-rrt the--at art performance in human action recognition (Rajasegaran et al. 2023). Apart from humans, we are not aware of any systematic dataset for learning such combined representations of poses and visual features in action recognition tasks.
3

---

Published as a conference paper at ICLR 2026
3 BIGMAQ
The core feature distinguishing BigMaQ from other animal datasets, particularly those of NHPs, is that we provide accurate skeletal pose features of a 3D surface model in combination with action labels for large amounts of data. Unlike other datasets that provide shape and pose information for static images through manual annotation synthetic data, derive this information from video recordings of real behavior using multi-view, markerless motion capture (see Table
Table I: Comparison of BigMaQ with existing pose estimation and action recognition datasets for NHPs, and animals in general. Species-specific datasets other than those containing non-human primates are not included. In the "Species" column, G denotes general, primates, C chimpanzees, and M macaques. The column "Type" differentiates between 2D keypoint, 3D keypoint, and 3D- shape (3D-S) representations. The origin of this 3D data is further specified by for synthetic and R for real recordings.
Pose
Dataset
Species
Action Rec.
Video
Type
3D-Data
Frame#
APT-36K
Yang et al.
2022b
G
x
/
2D
X
36,000
AnimalKing Kingdom
G
/
2D
X
33,099
(Ng et al. 2022)
LoTE Eniimal
(Liu al.
2023)
G
V
/
2D
X
35,000
Animal
G
X
X
3D-S
3,400 S
et al.
2023
AniMer er
G
X
X
3D-S
41,300 S
et al.
2025b
OpenMonkey Challlene
P
X
X
2D
X
111,529
(Yao et al.
(022)
OpenApe
P
X
X
2D
X
71,868
(Desai et al. 2023
ChimpACT
/
2D
X
16,028
(Ma al.) 2023
OpenMonkeyStudio
M
X
3D
R
195,228
(Bala al. 2020)
Macaqueloose Pose
X
2D
X
13,083
Labuguen al. 2021)
AacaauueMttionMoniie
/
X
X
(Huang al. 2025)
BigMaQ (Ours)
M
3D-S
R
173,543
Actions and Labels. The video data for this study was captured using 16 high-precision color cameras at a frame rate of 40 frames per second (FPS), recording eight different male rhesus macaques (Macaca mulatta) in neuroscientific laboratory (see Appendix [.I]. The cameras were geometrically calibrated and synchronized, with light panels surrounding the enclosure, resulting high-quality video footage of 2464 2056 pixels. More details are provided in Appendix [A.2] From these recordings, 763 distinct actions were selected, featuring the monkeys engaging in either solitary behaviors or interactions with another conspecific. Except for feeding the monkeys with fruits providing water, the recorded actions not induced by humans but were based on the monkeys' own spontaneous behavior, and interactions with individuals in the same and neighboring enclosures.
We mapped individual behaviors to multiple action labels from a macaque ethogram (see Appendix A.3), since single video contain several simultaneous behavioral displays and is therefore in- herently multi-label. To align these labels with existing action- trackingg literature for NHPs (Mal al. 2023), grouped these actions into the following categories: (1) Locomotion; (2) Object
4

---

Published as a conference paper at ICLR 2026
Interaction including also eating and drinking behavior; (3) Social Interactions between multiple individuals in scene; and (4) Other kinds of behavior with more subtle social meaning. Individual action categories
aggregated from existing ethological studies in macaques, respectively (Alt-
mann 1962. Sade 1973 Gunter et al., 2022). A more detailed description the action categories and their distribution across the dataset shown in Fig
given in Appendix A.3
Shaking (0.31%
Exploration (30.2)%
Moving (17.91%
Erection
Grooming (3.3%%
grooming
Climbing (1.5%%
grther
Walking (0.81%
Jumping (2.11%
Fleeing 10.47%
Standing upldown (1.1%%
Aveiding (1.5%%
Locomation
Ethogram
Social Interaction
16.3%
Anciety 15.37%
Intor Alli
Ignoring
12.09 4%
Nestral Approach (1.6%%
Holding Tail Erect (1.31%
Hounting (9.71%
Dominance Display (2.23%,
Besting (28.01%
Aggressiion on
Drinking (1.6)%
Eating 110.8)%
Solitary 0tj. Playing
Figure 2: Distribution of the ethogram-aligned action category labels.
To align the surface model with the video data and individual specific action categories, we trained detection model (YOLOv8, Jocher et al. (2023)) to predict monkey identities, and widely used 2D keypoint estimator (An et al.) 2023 Ma et al. 2023) Matsumoto et al.) (025). HRRNet-W4y (Wang et al., 2021a), to predict 20 keypoints on cropped images. We included the end positions of hand and feet, thus extending usual marker set ups for NHPs (Labuguen et al. 2021 Yao et al., 2022 to hand and feet postures. An overview of the predicted keypoints is given in Appendix Figure Annotations were generated for 306 monkey bodies or 3,712 images with internal software that projects already annotated 3D keypoints in remaining views to minimize manual labor. After training both models with this data (for model statisties Appendix (A.4). we generated keypoints (kps) and bounding boxes (bbox), each accompanied by confidence scores, for all 12,208 videos (764 16 views), and prompted zero-shot foundational model (SAM Ravi al. (2024) retrieve segmentation masks. Additional details on annotation, quality checks and uncertainty propagation of the labels are described Appendix A.5 In the following, describe how the surface models are aligned with these label data and videos. An overview of the full pipeline and resulting datasets shown in Figure
3D Mesh- TrackTracking. For our macaques, we use high-poly, artist-created template model (see Appendix Figure 9 composed of 10,632 vertices VR R- x3, and low-poly version fit the entire dataset, where Ny = 3625. These vertices are deformed and posed by an underlying rig via linear blend skinning (LBS); for more details, we refer the reader Appendix B and to et al. (2015). This rig consists of N, 115 joints in their template locations Jn E x where the influence of each joint on a single vertex determined by the skinning weights W E IR Ny xN,, To articulate the model, the pose parameters 0 E 3 define the relative rotations for each joint in axis-angle representation, specifying how each joint is oriented with respect to its parent in the kinematic hierarchy. After LBS, the vertices further rotated, scaled and shifted in global space by R E R y E R, and R3, respectively. The entire process that computes posed vertices V can be written
5

---

Published as a conference paper at ICLR 2026
Mulioion sozninns
Fitteing Behaviar Categ.
BigMaQ
750 (inter-lactions / 12,000 mull--tew videos
3D Labeling Tool
20 Kaypoints
Masks
(inci, HandFeett
Ethogram
Actions Labels
SAM 2
YOLOv6
FFRNet-W48
monko
Licoo id, cont
kip,
und
Video Inferencel Dynamic Pose Optimizaiion
Individual Optimication
BigMaQ500 benchmark
500 actions
8. 0.000 mulli--eww videos
InagerVideo Encodings
Trained with annolatione from 3D labeling lool (300 Postures/ / 3,700 images)
Mosn Correspondenees
Figure 3: Pipeline overview for generating Bia and the BigMaQ500 action-pose recognition benchmark. The 3D Labeling Tool provides annotations used to train the detection and keypoint models as well as optimize subject-specific avatars. These optimized avatars then combined with video-inferred labels to obtain dynamic pose reconstructions. BigMaQ500 includes all annota- tions available in MMa and additionally contains video encodings for more than 500 actions for which complete video-to-3D pose correspondences could be established.
V -p=yo LBS(O. W) +t.
(1)
The posed vertices and joints of the rig then rendered mesh via differentiable rendering Ravi et al.) (2020). resulting in predicted keypoints and silhouettes in the same perspective as our calibrated cameras. For the alignment of the model with the original videos, define frame-wise composite objective function
L(O) Ap + doLa + Aom Lem + Σ
Dkp + deaL
cam
over the entire set of learnable parameters Q 10, R. cr Instead of fixed joint positions, we use learnable bone length parameters a as well as vertex offsets & to adapt the template mesh to individual specific differences. To this end, Lp penalizes extreme joint rotations, L, ensures that adaptive bone lengths lie in reasonable range, I.gm handles smooth vertex deformations, and L and Lap align the articulated mesh with keypoints and silhouettes. The X; positive weighting coefficients. Since these adaptations and individual error terms have been discussed in prior work Badger et al. (2020): Rüegg al. (2022), further details are given in Appendices [B.] and B.2 We Tocus in the following on the features that make our pipeline feasible for the entire dataset, and discuss contributions beyond most recent efforts (An al. 2023
Large Data and Temporal Adjustments. To reduce memory and computational load, we do not operate on full image frames fit the surface model, but instead use cropped views for each frame and camera view, as given by our detection model (YOLOv8). We ensure a fixed maximum size for these cropped views and pass through the cameras sequentially in a batched format, which would be infeasible in high resolution. This allows us to define a temporal loss between all time points within a batch to enforce temporal consistency of the pose parameters. For rotations, we minimize angular velocities exploiting axis-angle rotations instead of Euclidean distances. Specifically, we define the angular velocity loss by the mean squared angular velocities across all joints and timesteps given as
W (m)
Long
(T - 1)J

---

Published as a conference paper at ICLR 2026
where w(r) denotes the instantaneous angular velocity (see Appendix B.3 for more details). For global translation, however, we compute mean squared temporal differences in Euclidean space. Therefore, let t())) IR3 denote the global translation at the n-th timestep. Its smoothness loss is defined as the mean squared finite differences, yielding the combined temporal regularization objective
T-1 LT Lang(O.r)) + (X.T) + T- Σ R=1
where F,T denotes the axis-angle representation of the rotation R for timesteps. This loss is added to the per-frame loss defined in equation 1 for all timesteps within a batch. In practice, we use six cameras and batches of 80 timesteps with an overlap of 10 frames, using cropped frames with maximum side length of 100 pixels. Generally, the labeled data used for training the detection and pose estimation models, yielding &. a, and the colors per individual. Then, optimize the poses of the individuals for each action following the optimization scheme given in Appendix [B.4] That section also provides an overview of the resulting subject-specific meshes and the ranges of the learnable parameters estimated for each individual.
first optimize the individual meshes
Mesh Coloring. To color the mesh, each vertex is associated with its own color vector, yielding the matrix C RR (RGB). Given posed vertices Vp and faces connecting these vertices, differentiable renderer produces a color image I under perspective camera II, and lighting &
f(e) = RTT,, Vr, F, C, e).
We estimate C Per individual by minimizing masked photometric objective on the training views:
Ephot
Σ ( (p) (p) (()) (p)
PEo
where S(() is the foreground silhouette mask, and are image pixels. We ensure that C remains in the range [0, 255] by applying scaled sigmoid function to the learned parameters.
RESULTS
Comparison to State of the Art. We compare B gMaQ with the most recent approaches in both multi-view and single-view surface tracking. For multi-view evaluation, we consider MAMMAL (An al. 2023) using our low-poly template model, which was also used generate the pose data for all actions in BigMaQ. In addition, we evaluate AniMer+, recent extension to SMAL that was designed for generic mammals (Lyu et al., 2025a b). Qualitative results for four different actions, each performed by different individuals, are shown in Figure AniMer+ fails to align the sur- face model with the image, estimating shapes that resemble entirely different animal species. For example, this is particularly apparent in the side-view shot where the mesh resembles more lion or tiger. The template model in MAMMAL yields more plausible alignments, but it does not cap- ture individual-specinn Comparing the intersection-over-nee approach outperforms (An et al." 2023) in the multi-view setting (Table 2}. To further evaluate the 3D alignment of our meshes relative to MAMMAL, we additionally report two standard metrics: the mean per-joint position error (MPJPE; Li al. (2015); Zhou et al. (0016). in mm) and the mean per-joint temporal deviation (MPJTD or mean joint velocity error: Li et al. (202);; Pavllo et al.) (2019) in mm/frame). which increases for jerkier or less temporally consistent keypoint trajecto- nes. Similar to the IoU measured across viewpoint projections, our method achieves more accurate and smoother skeletal alignment for all actions shown. To demonstrate that these improvements are not limited to the examples in Figure 4 we also evaluated fits on single frames extracted from all single-subject actions Bi 2 (Table 3). Across these frames, our method achieves higher IoU scores than previous approaches, and 3D skeletal alignment further confirms this advantage (26.907 mm for ours vs. 31.661 mm for MAMMAL). Evaluating the performance of BigMaQ over the en- tire time sequence of these single actions shows similarly strong performance (26.013 mm MPJPE and 8.463 mm/frame MPJTD). Further examples of dynamic surface captures are available in the Supplementary Video.
differences and generally produces lower-quality fits compared to Bi gMaQ. (IoU) of mesh fits over time quantitatively confirms that our
7

---

Published as a conference paper at ICLR 2026
Image
BigMaQ-C
B-gMaQ-M
MAMMAL
AniMer+
Figure 4: Qualitative comparison of different surface-tracking approaches for BigMaQ. illustrated for four actions of different individuals (rows). Only perspectives that were not used for multi-view optimization are shown. The second and third columns show the surface fits of B BigMaQ with color texture (BigMaQ-C) and without (Bi.MaQ-M).
Mean sequence-level surface reconstruction metrics for the actions shown in Figure MPJPE is reported in and MPJTD in mm/frame. IoU scores are averaged over time and across cameras involved in the surface reconstruction.
Table 2:
IoUf
MPJPEL [mm]
MPJTD| [mm/frame]
Action
MAMMAL
Ours
MAMMAL
Ours
MAMMAL
Ours
Walk
0.771
0.883
23.493
20.402
9.961
6.875
Food Picking
0.756
0.855
25.521
16.489
13.362
9.062
Branch Shake
0.757
0.831
22.163
13.633
17.676
15.515
Scratch
0.722
0.843
28.843
20.481
8.498
4.422
While renderings of our low-poly meshes may occasionally reveal triangular artifacts the back due to their lower vertex density, we additionally provide high-poly subject-specific meshes that can be rendered in the same poses without such artifacts. The tracked silhouettes from SAM 2, detections and keypoints are generally of high quality, but errors in these labels can impair the accuracy of our pose reconstructions, illustrated in Appendix B.7 Such errors typically occur in more complex multi-individual scenes or when monkeys leave the volume covered by the cameras. Keypoint prediction errors, in particular, lead stronger misalignments, since keypoints provide more informative constraints than silhouettes in the optimization process, as also demonstrated by loss ablation for dynamic mesh fitting (Appendix B.6).
Table 3: Mean IoU across single-subject actions in BigMaQ. aggregating all viewpoints used in optimization for the first frame of the respective action.
BigMaQ
MAMMAL
AniMer+
loU
0.844
0.714
0.591
8

---

Published as a conference paper at ICLR 2026
Table 4: Comparison of action recognition based on pose representations 0 and visual features including confidence intervals (mean + std) for multiple training runs: Including pose features im- proves performance across visual backbones. The overall mean average precision (mAP) further broken down by the categories defined in our ethogram: the subscripts L, OI, SI, and O denote Lo- comotion, Object Interaction, Social Interaction, and Others, respectively.
Vision Model
Feat.
mAP
mAPt
mAPor
mAPsi
mAPo
Pose
43.5 + 1.4
57.3 + 2.1
54.4 + 1.3
28.7 + 1.3
47.4 = 4.0
ResNe150
Vis
34.3 + 0.5
50.9 1.7
38.6 1.3
22.2 + 0.9
35.9 0.9
Vis+Pose
44.0 - 0.8
58.1 + 4.5
5.07
28.8 1.1
48.5 2.3
ViT-base-cls
Vis
32.9
51.8 1..811 1.0
1.5
18.0
35.9 = 1.1
Vis+
44.0 0.1
60.6 4.5
5.0
29.9
47.1 = 3.9
DINOv2-base-cls
Vis
37.8
0.8
4.8
25.1 1.2
37.8 = 1.4
Vis+Pose
41.8 2.6
2.0
1.4
26.2 - 2.9
45.5 = 4.9
ViT-base
Vis
34.5
52.3 0.9
41.8
20.0 + 1.3
36.9 = 0.9
Vis+Pose
1.7
1.8
0.8
28.4 0.9
40.9 = 5.5
DINOv2-base
Vis
40.4 41.4
55.0
0.6
27.2 2.7
42.1 = 1.2
Vis+Pose
59.5 1.6
51.4
27.5
42.0 0.8
TimeSformer
Vis
1.2
2.1
31.8 2.7
16.7 + 0.4
35.7 = 1.8
Vis+Pose
42.6
2.6
49.6
29.6 1.7
41.9 3.9
VideoPrism-
Vis
38.3
0.8
49.1 0.2
20.9 0.7
40.7 0.9
Vis+Pose
43.
60.9
2.8
5.21
29.7 4.8
46.3 = 5.6
Action Recognition. assess whether the pose vectors improve action recognition, we curated subset of BigMa that retains only scenes in which for more than 95% of all timesteps and individuals successful pose reconstructions can be provided (see also Appendix AA.5). We refer to this subset BigMaQ500, which contains 511 actions 8176 multi-view videos with long associated pose sequences. On this dataset, we trained three groups of transformer- each combining video features with pose vectors. Videos and surface reconstructions
models, both sampled at 10 Hz, following previous work in humans (Rajasegaran et al., (023). The first group of models used per-frame global embeddings: for ResNet50 (He al. 2016). we extracted features just before the classification layer, whereas for vision transformer models ViT-base (Dosovitskiy al. 2021) pre-trained on ImageNet-21k (Ridnik et (202)) and the more recent DINOv2 (Oquab et al. 2023) class tokens were used as global embeddings. The second group operated on patch tokens of the same vision transformer models, thereby retaining spatial information. The final group leveraged video encodings obtained from TimeSformer (Bertasius et al. (021)) and the foundation- scale VideoPrism model (Zhao et al., 2024). In addition, we trained a pose-only stream, absent of any visual input, to directly evaluate the contribution of pose features. Data splits to train these models included all individuals and action categories. More information on the model architecture and training is given in Appendix C We evaluated all models using mean average precision (mAP), a standard metric in action recognition (Rajasegaran et al., 2023: Ma et al.) 2023 ZZhao et al. 2024). categories and actions, further report sategory-speciiee mAP, following (Ma et al.) 2023). Our pose-only stream establishes a strong baseline, demonstrating that high-quality poses already provide sufficient cues distinguish many actions (Table second row). The combination of visual and pose features further improves performance, achieving a maximum overall mAP of 44.0 in two models. At the category level, different models yield the best results, but all models improve in action discrimination when augmented by our pose vectors. Similar to previous work NHPs (a et al.) (202)), social interactions remain the most challenging action category, underscoring the importance of pose features for modeling multi-individual behavior.
To account for the imbalance
The Effects of Pose Representation. not defined in terms of 3D generative parameters as in our vectors 0, but rather utilizing 2D key- points or 3D positional data [Mthiss & Mathis 2020; Bala et al. 2020 Lauer et al., 2022 Patel al. 2023 An et al. 2023: Wu et al., 2024 Matsumoto et al. 2025). Therefore, we evaluate how the action recognition performance is influenced by different pose descriptions, including 2D and 3D keypoints, surface points, and pose vectors represented as rotation matrices (see Table 5 Among these, the latter achieves the best results not only in pose-only stream models, but also when combined with visual embeddings based on class, spatial, and spatiotemporal tokens, indicating
the behavioral analysis of animals, the pose is usually

---

Published as a conference paper at ICLR 2026
that it is not the 3D information alone that drives recognition improvements. Instead, these results highlight the substantial benefits in behavior recognition of pose representations that construct 3D structure, rather than describing it. This perspective closely aligns with recent neuroscience studies that advocate incorporating generative approaches into recognition models (Yildirim et al. 2020 Yilmaz et al. 2025).
Table 5: Overall mAP scores for different pose representations in the BigMaQ action recognition lask. Results are shown first for the pose-only stream, followed by ViT-base-cls, DINOv2-base, and VideoPrism visual features in combination with pose features. KP denotes keypoint positions in 2D and 3D, M the posed mesh's vertex positions, and Rot the matrix form of A that consistently outperforms other pose descriptors.
Features
ViT-base-cls
DINOv2-base
VideoPrism
2D-KP
35.6 = 0.3
35.5 + 0.4
40.2 + 1.2
40.0 + 0.6
3D-KP
40.8 = 0.7
1.1
34.5 + 1.4
34.0 + 2.0
3D-M
35.2 = 4.4
2.4
36.6 + 0.8
36.0 = 3.3
3D-Rot
43.5 1.4
44.0 + 0.1
41.4 + 1.7
43.8 + 2.9
5 CONCLUSION
In this paper, introduced BigMaQ, video dataset combining markerless motion capture with 3D body surface and joint angle reconstruction, capturing a large variety of body movements and social interactions of macaques. Building upon previous methods of mesh-based animal tracking, pro- vide over 750 actions with rich annotations as multi-camera videos, individual detections, masks, keypoints, and most notably pose vectors per individual per-frame basis. Each individual mon- key is associated with a subject-specific surface model, enabling a more accurate characterization of pose and body motion than basic image features or keypoints. These resources are unique in their composition and inspired by available motion capture data in humans. In making this dataset available, we hope to not only demonstrate the utility of more accurate descriptors for non-hu non-human primates, but also to promote a shift in action recognition to incorporate 3D knowledge about bodies into the action recognition models themselves. While we have demonstrated the capabilities of these descriptors for action recognition and tracking, the same representation can also be used for realistic, controlled animation. This paves the way for in-depth studies of perception and neural encoding of generative pose, shape, and social interactions in ecology and neuroscience.
Limitations and Future Work. Behavioral categories were labeled by only two researchers on this specific dataset, and extending category prediction in-the-wild macaques or other NHPs would require broader consensus across behavioral experts. More challenging multi-individdal scenes would benefit from riew-consistent detection agreement, and SAM 2 silhouettes could be further improved by training a dedicated mask-quality discriminator. While our results demonstrate the advantages of pose-enabled action recognition primarily in multi-view scenarios, they are already informative for other species across the life sciences where 3D data or thape-spaces, are available
Karashchuk et al.) 2021 Marshall et al. 2021 An et al. 2023: Lyu et al., (2025)). Especially for quadrupeds, other approaches for in-the-wild image reconstructions do not rely on 3D data. While flexible, these models do not always yield faithful reconstructions (Li et al., 2024). In contrast, BigMaQ provides large-scale, high-quality 3D motion data for macaques. A natural next step is to derive a pose prior from this unique resource to regularize single-view reconstruction methods, as demonstrated for dogs in (Rüegg al. 2022: Kearney et al. 20200) Such prior could substantially improve generalization to complex poses and challenging in-the-wild imagery of NHP species.
Reproducibility Statement. We provide code and an anonymous URL to tracking labels in the Supplementary Materials. Additional dynamic surface estimates are included in the Supplemen- tary Video. Furthermore, the Appendix provides a detailed account of the individual steps of the BicMaQ processing pipeline, including the recording setup, ethogram, keypoint annotation, quality control, pose/detector model statistics, loss terms, individual avatar statistics, ablations, optimization procedure, action-recognition training protocol, and computational requirements.
10

---

Published as a conference paper at ICLR 2026
Ethics Statement. This work involves multi-view video recordings of captive macaques and the construction 3D motion dataset to support research on animal tracking and behavior understand- ing. All data collection approved by the relevant animal ethics committee. The macaques were part of an existing neuro- science facility, and no additional interventions, manipulations, or behavioral tasks were introduced specifically for this study.
carried out in accordance with regional animal care guidelines and was
The dataset contains only non-human primates and does involve human subjects beyond the personnel recording the behavior. Care was taken to ensure that the video material does not reveal sensitive information about facility staff.
We acknowledge that automatic behavior recognition technologies may raise concerns related to surveillance or misuse applied outside scientific and welfare contexts. We explicitly discourage any use of our methods or dataset for intrusive monitoring of humans or for applications that could negatively impact individual rights, privacy, or well-being. Similarly, we discourage deployments that could adversely affect animal welfare.
Acknowledgements. Funding was provided by the European Research Council (2019-SyG- RELEVANCEE-556995) The authors thank the International Max Planck Research School for In- telligent Systems (IMPRS-IS) for supporting Lucas M. Martini. We are further grateful to S. Po- likovsky for assistance with the motion capture hardware design and the Max Planck Institute for Intelligent Systems and M. J. Black for providing the template macaque mesh model. We thank M. V. Carrera and M. U. Syed for support video data annotation and processing, N. Taubert for mesh modeling, and I. Puttemans, A. Hermans, C. Ulens, S. Verstraeten, J. Helin, W. Depuydt, and M. De Paep for technical and administrative support.
REFERENCES
Stuart A. Altmann. A field study of the sociobiology of rhesus monkeys, macaca mulatta. Annals of the New York Academy of Sciences, 102(2):338-435, 1962. doi: https://doi.org/10..o 1749-6632.1 196...11...600
Liang An, Jilong Ren, Tao Yu, Tang Hai, Yichang Jia, and Yebin Liu. Three-dimensional surface motion capture of multiple freely moving pigs using MAMMAL. Nat Commun, 14, November 2023. ISSN 2041-1723. doi: 10.
Anurag Arna,, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: video vision transformer. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6816-6826, 2021. doi: 10. N0.11009CCC44992222222..11
Mare Badger, Yufu Wang, Adarsh Modh, Ammon Perkes, Nikos Kolotouros, Bernd G. Pfrommer, Mare F. Schmidt, and Kostas Daniilidis. 3D Bird Reconstruction: A Dataset, Model, and Shape Recovery from Single View. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020, volume 12363, pp. 1-17. Springer International Publishing. 2020. doi: 10.1007/978-
Praneet C. Bala, Benjamin R. Eisenreich, Seng Bum Michael Yoo, Benjamin Y. Hayden, Hyun Soo Park, and Jan Zimmermann. Automated markerless pose estimation freely moving macaques with StunnMookyySuudio. Nat Commun, 11:4560, December 2020. ISSN 2041-1723. doi: 10. 1038/s41467-020 18441-5.
Olivier A. Bauchau and Lorenzo Trainelli. The Vectorial Parameterization of Rotation. Nonlinear Dynamics, 32(1):71-92, April 2003. ISSN 1573-269X. doi:
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In Proceedings of the International Conference on Machine Learning (ICML), July 2021.
EJ Bethell, A Holmes, A MacLarnon, and S Semple. Cognitive bias in non-human primate: Hus- bandry procedures influence cognitive indicators of psychological well-being in captive rhesus macaques. Animal Welfare, 21:185-195, 2012. doi: 10.7 10.71200996222862211 21.2..1...8....

---

Published as a conference paper at ICLR 2026
Benjamin Biggs, Thomas Roddick, Andrew Fitzgibbon, and Roberto Cipolla, Creatures Great and SMAL: Recovering the Shape and Motion of Animals from Video. In C.V. Jawahar, Hongdong Li, Greg Mori, and Konrad Schindler (eds.), Computer Vision ACCV 2018, pp. 3-19. Springer International Publishing, 2019. doi: 10. 00110079988-3-00--0087---- 1.
Benjamin Biggs, Oliver Boyne, James Charles, Andrew Fitzgibbon, and Roberto Cipolla. Who left the dogs out?: 3D animal reconstruction with expectation maximization in the loop. In ECCV, 2020.
Otto Brookes, Majid Mirmehdi, Colleen Stephens, Samuel Angedakin, Katherine Corogenes, Dervla Dowd, Paula Dieguez. Thurston C. Hicks, Sorrel Jones, Kevin Lee, Vera Leinert. Juan Lapuente, Maureen S. McCarthy, Amelia Meier, Mizuki Murai, Emmanuelle Normand, Vir- ginie Vergnes, Erin G. Wessling, Roman M. Wittig, Kevin Langergraber, Nuria Maldonado, Xinyu Yang. Klaus Zuberbühler, Christophe Boesch, Mimi Arandjelovic, Hjalmar Kühl, and Tilo Burghardt. PanAf20K: A Large Video Dataset for Wild Ape Detection and Behaviour Recogni- tion. International Journal of Computer Vision 132(8))3086-311 August 2024. ISSN 1573- 1405. doi: 10.1007/s 223--02-02203--22
João Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 4724 4733, 2017. doi: 10.1 00.100//CPR.2017..3000
Jun Chen, Ming Hu, Darren J. Coker, Michael L.. Berumen, Blair Costelloe, Sara Beery, Anna Rohrbach, and Mohamed Elhoseiny. Mammalnet: A large-scale video benchmark for mammal recognition and behavior understanding. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13052-13061, 2023. doi: 10. 111109999952799.2223.011
Vasileios Choutas, Philippe Weinzaepfel, Jérôme Revaud, and Cordelia Schmid. Potion: Pose mo- tion representation for action recognition. In 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2018. doi: 10.1 10.100/CVPR.2008.0000
MMPose Contributors. Openmmlab pose estimation toolbox and benchmark. https: //gbt Com/open-min mm. ab/mmpose] 2020.
Eve B Cooper, Lauren JN Brent, Noah Snyder-Mackler, Mewa Singh, Asmita Sengupta, Sunil Khatiwada, Suchinda Malaivijtmond, Zhou Qi Hai, and James P Higham. The natural history of model organisms: The rhesus macaque as a success story of the anthropocene. eLife, 11: e78169, July 2022. ISSN 2050-084X. doi: 10.7554/eLife.78166
Soumyaratna Debnath, Harish Katti, Shashikant Verma, and Shanmuganathan Raman. L3d-pose: Lifting pose for 3d avatars from single camera in the wild. In ICASSP 2025 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, 2025. doi: 10. 10.1009/CASSP49666.000 2000S249966..22...1... 10890158.
Nisarg Desai, Praneet Bala, Rebecca Richardson, Jessica Raper, Jan Zimmermann, and Benjamin Hayden. OpenApePose, database of annotated ape photographs for pose estimation. eLife, 12:RP86873, December 2023. ISSN 2050-084X. doi: 10.7544/LLife.8687.. Publisher: eLife Sciences Publications, Ltd.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Debghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021.
Isla Duporge, Maksim Kholiavchenko, Roi Harel, Dan Rubenstein, Meg Crofoot, Tanya Berger- Wolf, Stephen Lee, Scott Wolf, Julie Barreau, Jenna Kline, Michelle Ramirez, and Chuck Stewart. Baboonland dataset: Tracking primates in the wild and automating behaviour recognition from drone videos, 2024.
Christoph Feichtenhofer. Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In 2019 JEEE/CVF International Conference on Computer Vision (ICCV), pp. 6201- 6210, 2019. doi: 10. 00.1109/CCC2220190006300
12

---

Published as a conference paper at ICLR 2026
Melissa Franch, Sudha Yellapantula, Arun Parajuli, Natasha Kharas, Anthony Wright, Behnaam Aazhang, and Valentin Dragoi. Visuo-frontal interactions during social learning in freely mov- ing macaques. Nature, 627(8802):174-11 March 2024. ISSN 1476-4687. doi: 10.1038/ s4 1415860024-07044--
Publisher: Nature Publishing Group.
Michael Fuchs, Emilie Genty, Adrian Bangerter, Klaus Zuberbühler, Jean-Mare Odobez, and Paul Cotofrei. From Forest Zoo: Great Ape Behavior Recognition with ChimpBehave. International Journal of Computer Vision, June 2025. ISSN 1573-1405. doi: 10. 1..1007//1123-025--222 1263-025-0248--66
R. A. Gibbs et al. Evolutionary and Biomedical Insights from the Rhesus Macaque Genome. Sci- ence, 316:222-224, April 2007. doi: 10. 126/sciene. 1132447.
Chris Gunter, R. Alan Harris, Zsofia Dovacs-Balint Muthuswamy Raveendran, Vasiliki Mi- chopoulos, Jocelyne Bachevalier, Jessica Raper, Mar M. Sanchez, and Jeffrey Rogers. Her- itability of social behavioral phenotypes and preliminary associations with autism spec- trum disorder risk genes in rhesus macaques: A whole cxome sequencing study. Autism Research, 15(3):447-433, 2022. ISSN 1939-3806. doi: 10.1002/aur2265. eprint:
Hengtao Guo, Benjamin Planche, Meng Zheng, Srikrishna Karanam, Terrence Chen, and Ziyan Wu. Smpl-a: Modeling person-specific deformable anatomy. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 20791, 2022. doi: 10.1109/
20022226
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Janis K. Hesse and Doris Y. Tsao. The macaque face patch system: a turtle's underbelly for the brain. Nature Reviews Neuroscience, 21(12):695-716 December 2020. ISSN 1471-0048. doi: 10. 10.10388/415833000000033
Alexander I. Hsu and Eric A. Yutri. B-SOiD, an open-source unsupervised algorithm for identifica- tion and fast prediction of behaviors. Nature Communications, 12(1):5188, August 2021. ISSN 2041-1723. doi: 1.11039/41447--222-5552220
Jiawei Huang, Zhiyuan Chen, Wenxuan Fan, Xiaomei Zhang, and Xibo Ma. Macaque-motion- monitor dataset: A new benehmark for macaque action recognition. In ICASSP 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, 2025. doi: 10. www.119//CASSS49960..O. 2025. 10887625.
2025
Glenn Jocher, Ayush Chaurasia, and Jing Qiu. Ultralytics yolov8. https://githhb: com/ 2023. Version 8.0.0.
ult ailtaalytics/ultral//e
Pierre Karashchuk, Katie L. Rupp, Evyn S. Dickinson, Sarah Walling- Bell, Elischa Sanders, Eiman Azim, Bingni W. Brunton, and John C. Tuthill. Anipose: A toolkit for robust markerless 3d pose estimation. Cell Reports, 36(13):109730, 2021. celrep.2021.199730.
ISSN 2211 2211-1247. doi: https://doi.orr http///oooorg/1... 10.110..
Sinéad Kearney, Wenbin Li, Martin Parsons, Kwang In Kim, and Darren Cosker. Rghd-dog: Pre- dicting canine pose from rgbd sensors. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8333-8342, 2020. doi: 10. 101100/CVPR4220000200000..3
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization, January 2017. arXiv:1412.6980 [es].
Rollyn Labuguen, Jumpei Matsumoto, Salvador Blanco Negrete, Hiroshi Nishimaru, Hisao Nishijo, Masahiko Takada, Yasuhiro Go, Ken-ichi Inoue, and Tomohiro Shibata. MacaquePose: A Novel "In the Wild" Macaque Monkey Pose Dataset for Markerless Motion Capture. Front Behav Neu- rosci, 14:581 1554,, January 2021. ISSN 1662-5153. doi: h..3389/fnhhh.0222 1554.
13

---

Published as a conference paper at ICLR 2026
Jessy Lauer, Mu Zhou, Shaokai Ye, William Menegas, Steffen Schneider, Tanmay Nath, Mo- hammed Mostafizur Rahman, Valentina Di Santo, Daniel Soberanes, Guoping Feng, Venkatesh N. Murthy, George Lauder, Catherine Dulac, Mackenzie Weygandt Mathis, and Alexander Mathis. Multi-animal pose estimation, identification and tracking with DaCu.. Nature Methods, 19 (4):496-504, April 2022. ISSN 1548-7105. doi: 10. (0088/411922002--044433
Ruilong Li. Julian Tanke, Minh Vo, Michael Zollhofer, Jurgen Gall, Angjoo Kanazawa, and Christoph Lassner. Tava: Template-free free animatable volumetric actors. In European Conference on Computer Vision (ECCV), 2022.
Sijin Li, Weichen Zhang, and Antoni B. Chan. Maximum-margin structured learning with deep networks for 3d human pose estimation. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 2848 2856, 2015. doi: 10. 10.1109/...2.11..3226
Tianqing Li, Kyle S. Severson, Fan Wang, and Timothy W. Dunn. Improved 3D Markerless Mouse Pose Estimation Using Temporal Semi-supervision. International Journal of Computer Vision, 131: 1389-11894 June 2023. ISSN 1573-1405.
Zizhang Li, Dor Litvak, Ruining Li, Yunzhi Zhang, Tomas Jakab, Christian Rupprecht, Shangzhe Wu, Andrea Vedaldi, and Jiajun Wu. Learning the 3d fauna of the web. In CVPR, 2024.
Dan Liu, Jin Hou, Shaoli Huang, Jing Liu, Yuxin He, Bochuan Zheng, Jifeng Ning, and Jingdong Zhang. Lote-animal: A long time-span dataset for endangered animal behavior understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 20064- 20075, October 2023.
Matthew Loper. Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graph., 34:1-16, November 2015. ISSN 0730-0301, 1557-7368.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https:
//arxiv.org/abatbbb /ttbb////11...0 05101
Diogo C.
Luvizon, David Picard, and Hedi Tabia. 2d/3d pose estimation and action recognition using multitask deep learning. In 2018 JEEE/CVF Conference on Computer Vision and Pattern doi: 10.1109/CVPR.2218.000
Recognition, pp. 5137-5146, 2018.
Jin Lyu, Liang An, Li Lin, Pujin Cheng, Yebin Liu, and Xiaoying Tang. Animer+: Unified pose and shape estimation across mammalia and aves via family-aware transformer, 2025a. URL https://arxxie org/ abs 2508 00298
Jin Lyu, Tianyi Zhu, Yi Gu, Li Lin, Pujin Cheng, Yebin Liu, Xiaoying Tang, and Liang An. Ani- mer: Animal pose and shape estimation using family aware transformer. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 17486-17496, 2025b.
Xiaoxuan Ma, Stephan Kaufhold, Jiajun Su, Wentao Zhu, Jack Terwilliger, Andres Meza, Yixin Zhu, Federico Rossano, and Yizhou Wang. Chimpact: A longitudinal dataset for understanding chimpanzee behaviors. Advances in Neural Information Processing Systems, 36:27501-27531, 2023.
Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black. AMASS: Archive of motion capture surface shapes. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
Markus Marks, Qiuhan Jin, Oliver Sturman, Lukas von Ziegler, Sepp Kollmorgen, Wolfger von der Behrens, Valerio Mante, Johannes Bohacek, and Mehmet Fatih Yanik. Deep-learning based iden- tification, tracking, pose estimation and behaviour classification of interacting primates and mice in complex environments. Nature Machine Intelligence, ((4):331-340) April 2022. ISSN 2522- 5839. doi: 10. 10.10301442556-0222-007777
Jesse Marshall, Ugne Klibaite, amanda gellis, Diego Aldarondo, Bence Olveczky, and Timothy W Dunn. The PAIR-R24M Dataset for Multi-animal 3D Pose Estimation. In J. Vanschoren and S. Yeung (eds.). Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021.
14

---

Published as a conference paper at ICLR 2026
Lucas M. Martini, Anna Bognár, Rufin Vogels, and Martin A. Giese. MacAction: Realistic 3D macaque body animation based on multi-camera markerless motion capture, April 2024.
Mackenzie Weygandt Mathis and Alexander Mathis. Deep learning tools for the measurement of animal behavior in neuroscience. Current Opinion in Neurobiology, 60:1-11, 2020. ISSN 0959- 4388. doi: https://doii.rgg wwt..oogg/....///...... wt/6//.conb.201999 10.00...0..0.... Neurobiology of Behavior.
Jumpei Matsumoto, Takaaki Kaneko, Kei Kimura, Salvador Blanco Negrete, Jia Guo, Naoko Suda- Hashimoto, Akihisa Kaneko, Mayumi Morimoto, Hiroshi Nishimaru, Tsuyoshi Setogawa, Ya- suhiro Go, Tomohiro Shibata, Hisao Nishijo, Masahiko Takada, and Ken-ichi Inoue. Three- dimensional markerless motion capture of multiple freely behaving monkeys toward automated characterization of social behavior. Science Advances, 11:eadn1355 June 2025.
Earl K. Miller, Cynthia A. Erickson, and Robert Desimone. Neural mechanisms of visual working memory in prefrontal cortex of the macaque. Journal of Neuroscience, 16(16):114--167, 1996. ISSN 0270-6474. doi: 10.1523/NNERROSCI. 16--0..1116----115119999999
Arne Monsees, Kay-Michael Voit, Damian J. Wallace, Juergen Sawinski, Edyta Charyasz, Klaus Scheffler, Jakob H. Macke, and Jason N. D. Kerr. Estimation of skeletal kinematics in freely moving rodents. Nature Methods, 19(11):100-1100, November 2022. ISSN 1548-7105, doi:
Xun Long Ng, Kian Eng Ong. Qichen Zheng, Yun Ni, Si Yong Yeo, and Jun Liu. Animal kingdom: A large and diverse dataset for animal behavior understanding. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19001-19012, 2022. doi: 10.1109/
CVPR52688.2022. 201.6688.002011866 844.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez. Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, Mahmoud Assran, Nico- las Ballas, Wojciech Galuba, Russell Howes, Po- Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Ar- mand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
Mahir Patel, Yiwen Gu, Lucas C. Carstensen, Michael E. Hasselmo, and Margrit Betke. An- imal Pose Tracking: 3D Multimodal Dataset and Token-based Pose Optimization. Interna- tional Journal of Computer Vision, 131(2)5514-53) February 2023. ISSN 1573-1405. doi: 10. 10.11077/11223-022-01777
Dario Pavllo, Christoph Feichtenhofer, David Grangier, and Michael Auli. 3D human pose estima- tion in video with temporal convolutions and semi-supervised training. IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR), 2019.
Jathushan Rajasegaran, Georgios Pavlakos, Angjoo Kanazawa, Christoph Feichtenhofer, and Ji- tendra Malik. On the benefits of 3d pose and tracking for human action recognition. In 2023 JEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 640-649, 2023. doi: N0.1109/CVPR5272..2222.. 200.:7229..020000063
Rajani Raman, Anna Bognár, Ghazaleh Ghamkhari Nejad, Albert Mukovskiy, Lucas Martini, Martin Giese, and Rufin Vogels. Keypoint-based modeling reveals fine-grained body pose tuning in superior temporal sulcus neurons. Nature Communications, 16(1):5796, July 2025. ISSN 2041- 1723. doi: 10.1038/441467-025. 549444555...
Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan- Yen Lo, Justin Johnson, and Georgia Gkioxari. Accelerating 3D deep learning with PyTorch3D. arXiv:2007.0850) 2020.
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Va- sudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Fe- ichtenhofer. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.007/ 2024. URL URL.https. //arxi/rxvv org abs /000 00714
15

---

http://0...11ff..oo.200000001110 Published as a conference paper at ICLR 2026
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-22 21k prrreennni for the masses. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URL
Vc26o1
Nadine Rüegg, Silvia Zuffi, Konrad Schindler, and Michael J. Black. BARC: Learning To Regress 3D Dog Shape From Images by Exploiting Breed Information. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 3876-3884, 2022.
Nadine Rüegg, Shashank Tripathi, Konrad Schindler, Michael J. Black, and Silvia Zuffi BITE:
Beyond Priors for Improved Three-D Dog Pose Estimation. In 2023 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8867-8876. IEEE, June 2023. doi: 10.
1109/CVPR522292223..0888
Matthew FS Rushworth, Rogier B Mars, and Jerome Sallet. Are there specialized circuits for social cognition and are they unique to humans? Current Opinion in Neurobiology, 2363)4436442, 2013. ISSN 0959-4388. doi: neuroscience.
Social and emotional
Remy Sabathier, Niloy J. Mitra, and David Novotny. Animal Avatars: Reconstructing Animatable 3D Animals from Casual Videos. Ales Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (eds.), Computer Vision - ECCV 2024, pp. 787, Cham, 2025.
Springer Nature Switzerland. ISBN 978- 778-3-031-7729867
D. S. Sade. An ethogram for Rhesus monkeys. I. Antithetical contrasts in posture and movement. American Journal of Physical Anthropology, 38(2)5337-542, March 1973. ISSN 0002-9483. doi: 10. 10.1002/ajpa.133030 1330380263.
Anshul Shah, Shlok Mishra, Ankan Bansal, Jun- Lun-Cheng Chen, Rama Chellappa, and Abhinav Shri- vastava. Pose and joint-aware action recognition. In 2022 IEEE/CVF Winter Conference on Appli- cations of Computer Vision (WACV), pp. 141-151, 2022. doi: 10. (09//AACV51458..2222.00000
Richard Vogg, Timo Lüddecke, Jonathan Henrich, Sharmita Dey, Matthias Nuske, Valentin Hassler, Derek Murphy, Julia Fischer, Julia Ostner, Oliver Schülke, Peter M. Kappeler, Claudia Fich- tel, Alexander Gail, Stefan Treue, Hansjörg Scherberger, Florentin Wörgötter, and Alexander S. Ecker. Computer vision for primate behavior analysis in the wild. Nature Methods, 22:1 154- 1166, June 2025. ISSN 1548-7105.
Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao. Deep high-resolution representation learning for visual recognition. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 43(10):3349-3366 2021a. doi: 10.11009/PAMM..2220000990 29836.....666.......
Yufu Wang, Nikos Kolotouros, Kostas Daniilidis, and Marc Badger. Birds of Feather: Capturing Avian Shape Models from Images. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14734-14744. IEEE, June 2021b. doi: 10.1109/CVR444437. 2021.01450.
Baihui Wu, Haiyang Yu, Jia Yi, Pengyu Lei, Jiaxuan He, Jing Ruan, Peiye Xu, Runchao Tao, Libo Jin, Wei Wu. Qinsi Yang, Da Sun, and Xiaoqun Zhang. Behavioral studies of zebrafish reveal new perspective on the reproductive toxicity of micro- and nanoplastics. Toxics, 12(3), 2024.
ISSN 2305-6304. doi: 10.3390/txxic 12300 78.0178.
Hongyi Xu, Eduard Gabriel Bazavan, Andrei Zanfir, William T. Freeman, Rahul Sukthankar, and Cristian Sminchisescu. GHUM & GHUML: Generative 3D Human Shape and Articulated Pose Models. In 2020 JEEE/CVF Conference on Computer Vision and Patern Recognition (CVPR), pp. 6183-6192, June 2020. doi: 10. 10.1109/VVPR22200.2200 2020.0062..
Jiacong Xu, Yi Zhang. Jiawei Peng, Wufei Ma, Artur Jesslen, Pengliang Ji, Qixin Hu, Jiehua Zhang, Qihao Liu, Jiahao Wang. Wei Ji, Chen Wang, Xiaoding Yuan, Prakhar Kaushik, Guofeng Zhang, Jie Liu, Yushan Xie, Yawen Cui, Alan Yuille, and Adam Kortylewski. Animal3d: A compre- hensive dataset of 3d animal pose and shape. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9099-9109, October 2023.
16

---

Published as a conference paper at ICLR 2026
Gengshan Yang, Minh Vo, Natalia Neverova, Deva Ramanan, Andrea Vedaldi, and Hanbyul Joo. BANMo: Building animatable 3D neural models from many casual videos. In CVPR, 2022a.
Yuxiang Yang. Junjie Yang, Yufei Xu, Jing Zhang, Long Lan, and Dacheng Tao. Apt-36k: A large-scale benchmark for animal pose estimation and tracking. Advances in Neural Information Processing Systems, 35:17301-17313, 2022b.
Yuan Yao, Praneet Bala, Abhiraj Mohan, Eliza Bliss-Moreau, Kristine Coleman, Sienna M. Free- man, Christopher Machado, Jessica Raper, Jan Zimmermann, Benjamin Y. Hayden, and Hyun Soo Park. OpenMonkeyChalleenge Dataset and Benchmark Challenges for Pose Esti- mation of Non-human Primates. Int Comput Vis, October 2022. ISSN 1573-1405. doi: 10.1007/s
liker Yildirim, Mario Belledonne, Winrich Freiwald, and Josh Tenenbaum. Efficient inverse graphics in biological face processing. Science Advances, 6(10)=eaxx597, 2020. doi: 10.1 126/scacc. aax5979.
Hakan Yilmaz, Aalap D. Shah, Ariadne Letrou, Satwant Kumar, Rufin Vogels, and IIker Yildirim. Multiarea processing in body patches of the primate inferotemporal cortex implements inverse graphics. Proceedings of the National Academy of Sciences, 122129):2420028712,, 2025. doi: 10. 00.1073/pnas24220877 122.
Hang Yu, Yufei Xu, Jing Zhang, Wei Zhao, Ziyu Guan, and Dacheng Tao. Ap-10k: A benchmark for animal pose estimation the wild. In Thirty-fiith Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Zhixuan Yu, Jae Shin Yoon, In Kyu Lee, Prashanth Venkatesh, Jaesik Park, Jihun Yu, and Hyun Soo Park. HUMBI: A large multiview dataset of human body expressions. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Long Zhao, Nitesh B. Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer J. Sun, Luke Friedman, Rui Qian, Tobias Weyand, Yue Zhao, Rachel Hormung, Florian Schroff, Ming- Hsuan Yang, David A. Ross, Huisheng Wang. Hartwig Adam, Mikhail Sirotenko, Ting Liu, and Boqing Gong. VideoPrism: A foundational visual encoder for video understanding. In International Conference on Machine Learning (ICML), 2024.
Enmin Zhong, Carlos R. del Blanco, Daniel Berjón, Fernando Jaureguizar, and Narciso García. An- imalMotionCLIPP Embedding motion in CLIP for Animal Behavior Analysis, April 2025, URL
2505 http://arxiv. org. abs 25005 00569 ar arXiv:2505.0055 [cs].
Xiaowei Zhou, Menglong Zhu, Kosta Derpanis, and Kostas Danilidis. Sparseness meets deepness: 3d human pose estimation from monocular video. In CVPR, 2016.
Silvia Zuffi, Angjoo Kanazawa, David W. Jacobs, and Michael J. Black. 3D Menagerie: Modeling the 3D Shape and Pose of Animals. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5524-5532. IEEE, July 2017. doi: 10. 10.1 10.1109/CPP..211..00
Silvia Zuffi, Angjoo Kanazawa, and Michael J. Black. Lions and tigers and bears: Capturing non- rigid, 3D, articulated shape from images. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3963. IEEE Computer Society, 2018
Silvia Zuffi, Angjoo Kanazawa, Tanya Berger- and Michael J. Black. Three-d safari: Learning to estimate zebra pose, shape, and texture from images "in the wild". In International Conference on Computer Vision, October 2019.
17

---

Published as a conference paper at ICLR 2026
A DATASET SPECIFICATIONS
A.1 ANIMALS AND HUSBANDRY
The animals were housed in enclosures at the KU Leuven Medical School and experienced a natural day-night cycle. Each monkey shared its enclosure with at least one other cage companion and has access to toys and other enrichment. The monkeys received water and dry food ad libitum each day. supplemented by fresh fruits on friday and the weekend. Since the recorded monkeys are subjects in neuroscience studies, they were implanted with a plastic headpost and recording chamber. Both the animal care and experimental procedures adhere to regional (Flanders) and European guidelines and have been approved by the Animal Ethical Committee of KU Leuven.
A.2 RECORDING SETUP
The 16 cameras used for the recording of the BigMaQ dataset (Victorem 51B 163CCX) were pre- cisely synchronized and recorded with video recorders (CORE2CXPLUS) from IO Industires, ex- ploiting low-latency TTL signals. Together with the cameras also eight LED light panels (Strobo- Mini2, Norka Automation) were controlled by these TTL signals. This allowed to light the scene in synchrony with the individual camera shutters, and limiting the light intensity to avoid disturbing the animals. The cameras were mounted on 12 tripods with additional clamps as close as possible to the enclosures' glass in order minimize reflections. An overview of the camera setup with correspond- ing camera IDs is shown in Figure 5 The enclosure measured 2.3 x 3.1 x 2.1 m. All cameras were oriented toward the center of the enclosure, focusing on the table where the monkeys predominantly resided (0.9 m wide, 1.8 m long). Since the setup was not completely protected against mechani- cal perturbations, the cameras were re-calibrated at the start and end of each recording session by a ChArUco board manufactured from calib.io (600x400 mm) with associated calibration software. To potentially reduce discrepancies between video recordings and colored meshes in post-production, also captured color checkers (Calibrite ColorChecker Classic XL) and a white- baatnebbllncing panel (Calibrite ColorChecker White Balance) at the beginning each session. Videos were recorded as longitudinal data across eight daytime recording sessions. Actions in collaboration with the neuroscience experimenters, due to the vast amount of data generated in a single day.
cut immediately on-site,
15
13
2.3m
14
2,1
3,1
Figure 5: The recording setup of BigMaQ includes 16 cameras positioned around the monkey en- closure. The cameras were oriented toward the center of the space, specifically focusing the table area. Camera IDs are shown in black adjacent to each camera, with odd- numbereed IDs indicating cameras providing top-down view.
18

---

Published as a conference paper at ICLR 2026
A.3 ETHOGRAM
We constructed an ethogram based on previous ethological studies in macaques (Altmann) 1962 Sade 1973 Gunter al. 2022), and grouped individual actions similar to recent action tracking in NHPs Ma et al. 2023). The resulting actions and categories are as follows:
Locomotion.
Patterns self-initiated movement of an individual.
1. Moving Horizontal movement, e.g., walking, running.
2. Climbing Vertical movement, e.g., climbing up or down structure.
3. Standing up/down Raising or lowering the body from or into bipedal stance.
4. Jumping Moving the body by propelling it into the air and landing elsewhere.
5. Resting Remaining stationary or only little movement, e.g., sitting or lying down, stand- ing still on four legs.
Object Interaction. Direct physical interactions with inanimate stationary or movable objects us- ing hands, feet, or mouth.
1. Solitary Object Playing - Non-social and non-goal-directed object manipulation (c.g., handling objects like poop or a cage).
2. Eating Preparing and consuming food.
3. Drinking Consuming liquids.
Social Interaction. Behavior that affects the behavior of at least another rhesus macaque. Loosely arranged from "Most aggressive", over "Neutral" to "Most submissive".
1. Aggression - Hostile interaction between two animals, with or without physical contact (includes push, chase, slap). In single monkeys include branch shake slapping the ground.
2. Dominance Display Bouncing the body, bobbing the head, open-jaweedlopen-moutt ges- tures conspicuous staring (often accompanied with beetled brows).
3. Mounting - Same-sex mounting interactions to assert dominance, reinforce status, or re- solve conflicts.
4. Holding Tail Erect "The tail is held up from the base, curled over backwards in the distal portion." (Altmann)
1962) Rank is communicated through the position of the tail.
5. Neutral Approach Smacking lips, whipping tail, or walking toward another monkey in a neutral, non-hostin manner.
6. Ignoring - Continuation of a monkey's ' behavior even though it clearly perceived another monkey's behavioral pattern directed at it.
7. Self-directed behavior Anxiety Behaviors such as scratching, yawning, or self- grooming; can be associated with anxiety and can include exploratory actions.
8. Avoiding Walking away, looking apprehensively. or grimacing toward another individ- ual (e.g., opening the mouth widely, pulling back lips). Often accompanied by lateral flexion, stretched posture, grimacing face signal submission. Also includes avoiding to stare at a dominant individual close to it.
9. Fleeing - Running away from, or escaping another monkey.
Others. Other behaviors, if social, more subtle.
1. Touch -
Approaching and touching another animal, distinct from grooming.
2. Follows Persistent trailing another animal, or continuous approach as the other moves away. Includes turning to gaze follow another monkey's actions.
3. Walking past - Passing by an individual; might be considered as first walking toward and then walking away from the other.
19

---

Published as a conference paper at ICLR 2026
4. Presentation for grooming - Posture to solicit grooming from another animal, can include more than one animal or ambiguous intent.
5. Grooming -
Picking through fur; one animal combing through another's hair, usually with hands, but sometimes with mouth.
6. Erection - Subtle social relevance, if any.
7. Shaking - Shaking the body or limbs.
8. Environment exploration Moving and turning inspect the environment.
Displayed behaviors and initial action labels were annotated on-site in consultation with neuro- physiologists to determine the monkeys' expressed behaviors. The refined ethogram labels were subsequendy produced by the recording operator and reviewed during an initial labeling phase com- prising more than 150 behavioral categories, together with an expert who works with these monkeys on daily basis. The categorization of actions required approximately 18 hours.
While these labels serve as description of the dataset and macaques display social behavior in laboratory environments (Franch et al. 2024), the recorded frequencies do not generally represent captive or wild-life monkey behavior. First, the videas were cut to show variety of different and actions, and secondly, the monkeys' behavior could have been affected by the filming conditions. Moreover, the recorded male individuals were between 5 and 7 years old and can therefore capture the behavior of juveniles or females only to a limited extent.
A.4 DETECTION AND POSE ESTIMATION
The image data train both models comprises 3,712 images, where these frames include multiple viewpoints from poses of up to three interacting monkeys. Keyframes were selected based on distinct initial action categories and maximize pose variability, while preserving balanced distribution of individuals. The detection confusion matrix and precision- recall curve are displayed in Figure 6 showing good recognition performance across individuals.
We further report COCO results on the validation set based on default settings for training in the well-established toolbox from (Contributors) 2020 Yu al. 202T) for pose estimation, and (Jocher) et al. 2023) for detection.
Table 6: COCO evaluation metrics for HRNet-w48 at trained epoch 180. The model was trained with an 80%/10%/10% rain/validation/test split. AP and AR denote average precision and average recall, respectively. The notation x' indicates evaluation at an loU threshold of x (e.g.. AP@0.5 average precision at IoU 0.5), while 'all IoU' refers to averaging across thresholds from 0.5 to 0.95 in steps of 0.05.
Metric
Score
AP (all IoU)
0.9268
AP @ 0.5
0.9900
AP @ 0.75
0.9664
AR (all IoU)
0,9355
AR @ 0.5
0.9918
AR @ 0.75
0.9696
20

---

Published as a conference paper at ICLR 2026
Confust Natrix Normalized
035
C.N4
212
backgrund
Trup
(a) Detection confusion matrix.
Precision-Recall Curve
1.0
0.933
0.
0.985
0.8
0.994
0.995
0.959
T 0.983
all classes 0.976 mAP@0.5
Precision
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Recall
(b) Precision-reche curve.
Detection performance illustrated through the confusion matrix (top) and preci- sion- non-recall curve (bottom), demonstrating robust recognition across individuals.
Figure
21

---

Published as a conference paper at ICLR 2026
Table YOLOv8I model validation results epoch 49. The model was trained with 5% validation split for early stopping. mAP @ 0.5 indicates mean average precision at IoU = 0.5, while mAP @ 0.5:0.95 is the mean average precision averaged over IoU thresholds from 0.5 0.95 in steps of 0.05.
Metric
Value
Precision (P)
0.9526
Recall (R)
0.9561
mAP @ 0.5
0.9762
mAP @ 0.5:0.95
0.8538
1: Right eye
2: eye
3: Nose
Neck
Right shoulder
0: Right elbow
Right
B: Right hand tip
9: Left shoulder
elbow
11: Left wrat
13:
14: Right kinee
Right
15: Right foot Up
Left krew
19: Left foot tip
Figure 7: Predicted 2D keypoints of an exemplary frame.
A.5 KRYPOINT ANNOTATION AND PROCESSING CONTROT. STEPS
The 3D labeling procedure was carried out by two other annotators. To support keypoint anno- tation using our multi-camera calibration, once at least two views of keypoint were labeled, we triangulated its 3D position using the direct linear transform (see also Appendix [B.5) and projected the estimate into the remaining unlabeled views. In addition, for every annotated keypoint, epipolar lines were displayed in the remaining views, including cases with fewer than two annotated views, to guide the annotators toward geometrically consistent locations. This multi-view assistance substan- tially accelerated the annotation process. Annotators could refine existing labels add additional keypoint annotations as needed. Before proceeding to the next body landmark, we required the mean reprojection error across all labeled views to remain below 5 pixels to ensure geometric consistency. Both annotators independently checked for keypoint swaps, after which the video acquisition spe- cialist reviewed and refined the annotations when necessary. Each monkey body required 15 and 20 minutes of annotation time per annotator and an additional 10 minutes for review, totaling 140 hours of labeling effort for the 306 monkey bodies.
The reprojection errors across all 20 labeled monkey keypoints are shown in Table These errors remain low, largely because the annotation interface supports zooming and produces geometrically consistent keypoints. To estimate the expected displacement between true and estimated 3D key- points produced by the annotation tool, we sampled the entire table area and a volume up to 60 em above it, resulting in 36 known 3D locations. After perturbing the 2D projections of these points, we computed 3D errors between triangulated (perturbed) annotations and their ground-truth positions. The estimated 3D locations remain highly accurate even when triangulated from only two annotated views under the reprojection threshold of 5 pixels (see Table 99. This scenario, or the more extreme case of 10 pixels, represents conservative lower bound annotation quality, since in practice also multiple viewpoints were annotated and all 2D keypoint projections from the cameras were visible to the annotators.
22

---

Published as a conference paper at ICLR 2026
Table Reprojection error statistics in pixels (px) of the 20 body keypoints in the 3D Labeling Tool across the 306 labeled monkey bodies.
Keypoint index
Keypoint name
Mean
Std
Right eye
0.431
0.753
Left eye
0.341
0.502
3
Nose
0.369
0,500
Neck
0,980
1.252
5
Right shoulder
0.937
1.280
Right elbow
0,956
1.234
7
Right wrist
0.769
0.998
Right hand tip
0.330
0,531
Left shoulder
0.884
1.140
10
Left elbow
0.991
1.321
11
Left wrist
0.791
1.132
12
Left hand tip
0.277
0.460
13
Hip
1.027
1.419
14
Right knee
0.976
1.312
Right ankle
0.776
1.074
16
Right foot tip
0.327
0.544
Left knee
0.977
1.120
18
Left ankle
0.796
1.009
19
Left foot tip
0.320
0.523
20
Tail
0.707
1.086
Table 9; Mean = std of the 3D Euclidean error (mm) of estimated 3D ground-truth points under different annotation noise levels and number of annotated cameras. Projected ground-truth points per annotation view were shifted randomly, and independently in (x. image dimensions for [-or, 0, a) pixels (px) before triangulation. Columns indicate the annotation noise σ; rows indicate the number of annotation cameras used for triangulation.
# Cameras
1px
2px
5px
10px
2
1.440 1.636
2.990 + 5.374
7.329 +:2 8.318
15.506 31.656
3
1.007 + 0.578
2.011 + 1.219
5.015 + 3.048
10.045 = 5.768
0.818 0.427
1.638 + 0.841
4.087 2.126
8.202 4.216
23

---

Published as a conference paper at ICLR 2026
Bounding boxes were derived from the extremal 2D keypoints closest to the image boundaries in each camera view. To ensure full spatial coverage each monkey, we expanded the associated triangulated 3D keypoints by safety margin of 10 em along the plane orthogonal to the camera before projecting them back to 2D. This allowed us to retrieve bounding box labels from keypoint annotations.
We also applied a set of quality checks to labels generated by YOLOv8. Specifically, only individ- uals known be present in the scene (according our multi-view recording documentation) were considered, allowing us to filter out erroneously detected identities. For each individual, only the highest-confidence detection was retained. Missing or duplicate detections were flagged erro- neous, and the corresponding camera view was excluded for that frame in surface model fitting.
During mesh optimization, we additionally weight each camera viewpoint by its detection confi- dence. Furthermore, we also weight the contribution of individual keypoints using the confidence scores provided by HRRNet-W48
For the action recognition benchmark, we excluded actions for which less than 95% of timepoints could be linked to surface reconstructions all individuals the scene. This filtering inherently scene- and video-dependent, since a monkey may only momentarily engage with another individual while otherwise remaining beneath the table, preventing the generation of labels and, consequently, surface reconstructions. In such cases, the optimization would fail, resulting in non-convergent pose reconstructions, i.c., frames in which the estimated pose parameters contain invalid (NaN) values.
Dataset
BigMaQ (N-763)
Mag00 (N=511)
14.
sequence length [s]
12
HN0-00000 do 000
0 000
00
10-
d0o0s dR00000
aa0 000000000000
0000
8-
Locormotion
Object Interaction
Social Interaation
Others
Figure 8: Box plots showing the distribution action sequence lengths for BigMa and Big- MaQ500 across the four behavioral categories. The filtering step does not substantially affect the action length distribution in the action recognition benchmark.
B MESH OPTIMIZATION DETAILS
3D mesh tracking aims to recover the articulated motion and surface deformation of an individ- ual across time using parametric mesh driven by an underlying skeleton. Each mesh vertex is associated with set of skinning weights W that specify how it moves under joint rotations 0 these weights are typically created during rigging of a high-poly template model (shown below) by an artist and can be automatically initialized and manually refined in commercial software such as Autodesk Maya or ZBrush. For computational efficiency, we derive a low-poly mesh from our high-poly template, preserving the rig and skinning weights, which allows optimization while retaining the option to reconstruct higher-detail surfaces when needed (e.g.. for neurophysiological stimulus creation). During tracking, subject-specifie shape adaptation op-
perform fast
24

---

Published as a conference paper at ICLR 2026
tionally be performed beforehand using manual high-accuracy 3D keypoint annotations together with silhouette information to better fit individuals not well represented by an existing shape space, such SMAL, after which the adapted model is used to fit full video sequences.
Figure 9: Side view of the high-poly mesh.
B. ADAPTABLE MODEL PARAMETERS
Let us consider the extended version of equation that includes parameters to adapt the vertices and joints for different individuals
V P - R (BS J(a),, W) +
To adapt the mesh different subject skeletal structures, we include learnable bone lengths or in the function J ((x), ) shifting the initial joint locations Jn if needed as described in Badger et al. (2020); Wang et al. (2021b). The posed joints Jp are then determined by the relative body, and global pose transformations bone lengths, we introduce learnable offsets g for each vertex template mesh to yield shifted vertices denoted by
with posed vertices Vp. To account for differences in body shape beyond
V V A
As the number of such offsets scales with the number of vertices in the mesh, we divide them into groups of symmetrical and unique vertices. This helps further reduce the parameter space for finding subject-specific mesh offsets. We therefore include sparse mapping
E Nv x3795 with 333955x3 where symmetrical vertices share weights, but the direction of the vector is flipped. In the low-poly version, these offsets reduce to ÊtP E |R1305x3, As we neither allow fingers to be posed in pose space nor their bone lengths to vary, we also exclude these vertices from & thereby further decreasing the number of learnable parameters. Consequently, only 27 joints participate in posture estimation. We denote this restricted number of joints for posing as NJn-
B.2 Loss FUNCTIONS
Keypoint Reprojection Loss.
DG defines the weighted mean squared error of 2D projected joint locations of the posed avatar J along with corresponding image keypoints P. for given camera
L (0) =
(
wk
(J$) - 1
Σ
WE
1=1
where II transforms the 3D points from global space into the camera's image plane. The weights Wk manage the contribution of individual joints, where emphasize the importance of the hip, and tail keypoints. In practice, we extend these fixed joint weights adaptively by the confidence scores provided by the 2D keypoint estimation model.
25

---

Published as a conference paper at ICLR 2026
Silhouette Reprojection Loss. Let S denote the silhouette renderer and write the rendered mask as S [0, 1] x and the ground-trune truth mask S(r) E {0,1) ( x
S( = S Vr, F).
The per-cameen careea silhouette loss normalizes by image size and is accumulated only for views with sufficiently small keypoint reprojection error
1
| s if (pppp)) <
Okp:
(+)
Ii 0,
otherwise.
This follows Rüeeg et al. (2022) with the addition of the dimension-basee normalization to avoid resolution-dependent scaling.
Pose and Bone Constraints. We penalize poses that strongly deviate from the template four- legged standing pose favor more realistic poses using the error term
NaR
(p( EN' We
Σ wol 0x112- k=1
This suppresses uncontrolled joint rotations of the reduced of NJR joints of the kinematic tree, such as those of the hand, which are not sufficiently constrained by the set of keypoints. Similar the keypoint reprojection loss Eyp we also impose per-joint weights to specifically allow more flexibility in tail joints.
As in Badger et al. (2020). we penalize individual bone lengths outside pre-defined limits (a'mnn Ommax) by La(ax, (os) max(0, O;; o- -m max Ommin a0;) for individual bones. We calculate the entire bone length loss as the sum of the squared individual error terms Ls(ox) Σ, (a
Shape Refinement. We further adapt the surface shape the model for individual differences apart from bone lengths Therefore, define smoothness loss of neighboring vertex offsets of the entire mesh by
Ny Σ
Lum (E)
Σ II(AE):: - (AE),.112
1=1
where vj is vertex within the neighborhood. A u;. With Oi.., we denote the i-th row of matrix, so that (A.).. and (AE);, represent the respectively learned i-th and j-th vertex offsets.
B.3 ANGULAR VELOCITY
Let ((m) E 3 denote the axis-angle vector of joint j at time step n, with
in)
(n) = ||0 122
|
The per-joint angular velocity (Bauchau & Trainelli 2003 is approximated by finite differences as
(m)
A (n) m(m) + sin (0)) I+ cos ())))) (u Auf"))
At
where
A A = 0f (n+1) (n)
Au (n) (at1) (n) Uj u
and [u]x denotes the 3 skew-symmetrri matrix.
26

---

Published as a conference paper at ICLR 2026
B.4 OPTIMIZATION SCHEME
To optimize the articulate mesh model, we use Adam (Kingma & Ba 2017) with adaptive learning epochs and parameters. Before optimizing O initialize R, by Procrustes transforma- tion to speed up the alignment and prevent weird joint rotations that can occur when simultaneously optimizing and
Individual Shape Adaptation. To create for each monkey subject-specific mesh model, adapt model parameters a stage-wise manner. First we align the poses to match the manual label data. Then, adapt the mesh for multiple views and keyframes of the same monkey, followed by coloring the resulting surface. An overview of the resulting meshes is given in Figure I0 and the ranges of the adapted parameters across individuals is further quantified in Table TO Afterwards, we use these surface models to derive the monkey's individual behavior over time. Individual loss weights A are given below.
N
Figure 10: An overview of the individual mesh adaptations for all eight monkeys in B BigMaQ.
Table 10: mean, std).
Summary statisties for adapted parameters in individual shape adaptation (min, max,
Parameter
Min
Max
Mean
Std
Global Scale 7
0.791
1.001
0.908
0.060
Bone Lengths
0.833
1.179
0.973
0.124
Vertex Offsets g
-0,030
0.039
0.000
0.005
Vertex Colors
26.854
254.747
164.602
35.398
Dynamic Pose Fitting. We detect the cameras be used in optimization either by switching minimal set of cameras in a greedy manner, or by a random selection of available cameras per frame. The former would select the cameras for the longest non-interrupted sequence of frames, and switch only to another camera if further labels can not be provided. This setting was used generate the poses in Bi g which increases temporal smoothness compared random selection. For comparisons with other methods, we selected actions that provide continuous stream of six cameras, since dynamic camera changes are not supported in MAMMAL.
27

---

Published as a conference paper at ICLR 2026
typical of typer-parameter for shape and dynamic pose estimation used for the Big BigMaQ dataset. The first three columns are involved in individual shape fitting, whereas the last column refers the stage of dynamic pose fitting.
Table 11:
Pose
Mesh
Color
Time
XP
3
0
0
3
100
100
0
Asm
0
5
0
0
Akp
2
1
0
1
Asal
1500
100
0
50
AC
0
0
0
NT
0
0
10
kpp
10
Epochs
1200
150
30
350
B.5 3D ERROR METRICS
To quantify the alignment of the tracked surface with the keypoints, we measure the mean per-joint position error (MPJPE) between the triangulated 3D keypoints P and the corresponding posed joint locations J Jp
(PJPE(Jp.P))
JJ
P*112.
P*
NK
When reporting MPJPE for an entire action sequence, we compute the error per frame and then average these values across time.
To evaluate temporal smoothness of the posed keypoints over an action of length T, we follow al. 2023) and use the mean per-joint temporal deviation (MPJTD)
T
NK
MPJTD(]>) - NK
Σ
(n.k)
J (n+1,k)
||2
n=
k=l
which corresponds to the mean velocity of posed joints across time.
We triangulate keypoints using the direct linear transform with random sample consensus (RANSAC) following KKrashhhuk et al.] (2021). For each marker and timestep in the predicted keypoint trajectories, randomly select up to five camera views generate 3D point proposals. Among these proposals, we retain the point P with the lowest mean reprojection error across the selected cameras. When triangulating annotator-provide keypoints in the 3D labeling tool, all available annotations due to the substantially lower computational cost.
B.6 ABLATIONS
For dynamic pose estimation, we ablate different X's to give an intuition about their influence in the optimization procedure of an entire action of 123 frames or ~ 3 in Figure [T and the associated Ta- b2e Keypoints are an integral part of the optimization procedure, and without the model remains relatively fixed (see MPJTD value below) at wrong locations. More specifically, the silhouettes are not sufficient to align the mesh with multi-camera data. However, the silhouettes on reconstruction quality. This is apparent since without temporal loss the IoU scores remain rela- tively high even though the extremities can end up in an unnatural sequence of joint rotations. The Xp pose prior particularly helpful for preventing large pose deviations from the template during static shape estimation. In dynamic pose estimation, removing this term does not affect the quality metrics for this example. Nevertheless, generally retain it due its negligible computational
have an impact
28

---

Published as a conference paper at ICLR 2026
cost and the additional robustness it provides, even though the temporal alignment already fulfills similar role in correcting rotational artifacts.
We further evaluate whether incorporating bounding box and keypoint confidence values improves surface fitting, and how the number available viewpoints affects dynamic pose reconstruction, as summarized in Table 13 Excluding confidence values consistently degrades surface fitting quality across all metrics. Likewise, reducing the number of viewpoints lowers accuracy in every measure, with the most pronounced impact observed in the mean per-joint position errors.
aa) Full model.
(b) Akp
(c) Aua
(d) Ar
Figure 11: Surface fitting results when disabling relevant À entries in dynamic pose estimation. The red outline indicates that this particular camera was not involved in surface fitting, ensuring an unbiased different viewpoint evaluate reconstruction quality.
Table 12: Ablation experiment on weighting factors À in dynamic optimization: mean error metrics for the entire time sequence and across cameras involved in the surface reconstruction. Arrows indicate whether lower or higher values better.
of the action shown in Figure 11 IoU scores averaged over time
Ablation Setting
MPJPE 1 (mm)
MPITD (mm/frame)
IoU f
None (full model)
16.479
9.076
0.855
Aup 0. A
210.081
1.844
0.375
16.593
9.344
0.812
AT = 0.
17.082
10,620
0.853
29

---

Published as a conference paper at ICLR 2026
Table 13: Ablation experiment on confidence-valuee integration and viewpoint reduction in dynamic optimization: mean error metrics for the entire time sequence of the action shown in Figure [1 IoU scores are averaged over time and across cameras the full model (Newss whether lower or higher values are better.
6). Arrows indicate
Ablation Setting
MPJPE + (mm)
MPJTD (mm/frame)
IoU f
None (full model)
16.479
9.076
0.855
w/o conf. values
18.676
9.756
0.847
Nviews 5.
19.052
9.971
0.848
Nviews = 4.
29.322
11.085
0.822
Nviews 3.
26.286
11.138
0.825
Nviews 2.
36.053
11.902
0.801
B.7 FAILURE CASES
If the mesh optimization fails, it usually due to the quality of the labels used in alignment. While the SAM silhouettes are generally of good quality (see also the Supplementary Video), some frames can contain silhouettes that include background features, combined or partial individuals. For this reason, we reduced the silhouette contribution in the time-optimization stage (see Table [T]. Incorrect detections, and consequently incorrect keypoint predictions, can degrade the fitting quality, as illustrated Figure (first row, third column). Although such errors would need to persist across viewpoints and multiple frames due to the temporal consistency constraint, establishing detection agreement across views during camera selection could help mitigate these issues.
In otherwise well-tracked sequence of an individual, conflicting input signals for particular frame can cause the temporal loss to lock in an incorrect orientation, especially when strong empha- sis is placed extremities, hands, and feet (second row). This failure may be mitigated by initial per-frame optimization without temporal constraints, or by increasing the weight of the pose prior.
When animals leave the capture volume or make contact with the glass, the estimated keypoints can become distorted or squashed, in part because such cases are underrepresented in the pose estimator's 's training data (third row). Unlike the previous cases, this type of failure is tied to the capturing setup and cannot be easily resolved without altering the enclosure or recording conditions.
30

---

Published as a conference paper at ICLR 2026
Figure 12: Failure cases of a frame in action. The first and third column shows the rendered mesh on top of an image of the particular scene, including bounding box and keypoints labels, where the green frame outline indicates the camera's participation in optimization. The associated SAM silhouette is shown in the second and fourth column, respectively. Rows show results for different individuals.
C ACTION RECOGNITION DETAILS
Implementation. All sequences were padded to the maximum action length, with padding masks provided to a transformer consisting of layers, each with 8 heads and model dimension of 256. Appearance and pose features are concatenated and fused through linear layer before being passed to the transformer. Each modality is first encoded by a two-layer multilayer perceptron (MLP), fol- lowing (Rajasegaran al. 2023). For models using spatial embeddings (patch tokens), an additional multi-head attention module is applied in the visual stream before the two-layer MIP. For the pose- only and visual-only baselines, the disabled modality is replaced with zero-fcature vector, ensuring that the model processes only a single stream while keeping the overall architecture unchanged. All models were trained for 20 epochs with 70%/10%//0%% stratified train/validation/test split tions using Adam W (Loshchilov & Hutter 2019) and EarlyStopping. All individuals and action categories present in all splits. Viewpoints of the same action training runs solely based on 3D information, where the same input would be provided for multiple camera views, we report 5-fold cross validation results. In cases where visual information is used, we report mean and standard deviation results of three independent training runs. For based training. 0 is mapped to rotation matrices.
shared across splits. For
D COMPUTATIONAL REQUIREMENTS
Action tracking, label generation, and surface fitting was executed on a single NVIDIA Geforce RTX 3090 GPU. A single optimization step in surface fitting takes 0.019 resulting in an effective fer-fr runtime of 6.65 s for our default 350-epoch setting.
Inference of single video for behavior recognition requires from 525.82-729.0S M FLOPs, assum- ing pre-computed visual and pose features. Of this, the pose stream accounts for approximately 30 M FLOPs. Training the feature-procesten heads for the full 20 epochs 10 Hz (for both the visual
31

---

Published as a conference paper at ICLR 2026
and pose streams) takes 3-4 minutes for ViT-base-cls and up to 120 minutes for VideoPrism-base features on the BigMaQQ00 data split.
32


Reviews:
[{"review_id": "zaVDImZF4a", "reviewer": "Reviewer_nfdd", "summary": "This paper presents BigMac3D, a large multi-view video dataset of rhesus macaques with dense surface-based 3D pose/shape reconstructions and action labels (an ethogram). The recordings come from 16 calibrated cameras in a neuroscience laboratory and cover >750 action scenes and a derived action-recognition benchmark (BigMac500) of $~\\sim$8k labeled videos. The authors build subject-specific textured avatars by adapting a high-quality macaque template mesh and introduce processing/optimization improvements (symmetric time loss, cropped differential rendering, integrated texture) to make large-scale surface reconstruction tractable.", "questions": "Refer to questions", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Impactful dataset: fills an important gap for non-human primate research by linking dense 3D surface reconstructions with behavior labels.\n\nPractical pipeline: believable and reproducible-in-principle mesh-fitting and rendering pipeline that scales to hundreds of scenes.\n\nDemonstrated utility: empirical evidence that 3D pose/shape descriptors can improve action recognition across multiple visual backbones.\n\nRich annotation set: identities, segmentation masks, 2D keypoints, calibrated views, per-frame poses and action labels.", "weaknesses": "Ethical / animal welfare documentation (major). The recordings come from a neuroscientific laboratory and involve captive macaques. I cannot find any explicit statement of oversight, animal-use committee (e.g., IACUC or institutional equivalent) approval, or details of welfare protocols. The paper says recordings were “not induced by humans except feeding,” but ethical approval and animal care protocols must be stated explicitly. \n\nStatistical significance and variance. Report standard deviations / confidence intervals and number of seeds for all recognition results (Tables 3-4) and any other ML experiments. If training is deterministic (unlikely), explain why; otherwise run multiple seeds and report variability. Also provide basic compute footprints (GPU types, training time per model, FLOPs if possible) so the community can compare methods fairly.\n\nAnnotation reliability. How were action labels assigned? How many annotators per clip? Multi-label or single-label? What was the inter-annotator agreement (kappa / % agreement)? Actions like “aggression,” “dominance display,” or “anxiety” can be subjective—please document.\n\nDataset bias & generalization. The dataset contains eight male macaques in one lab. Discuss how sex, age, enclosure, and captive vs. wild behavior may affect generality. Are there plans to extend diversity (females, juveniles, different environments)?\n\nPrivacy / safety considerations. While this is animal data, the authors should discuss potential misuse (e.g., tracking in the wild or exploiting behavior data) and state any restrictions on dataset use if applicable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0Pcvg5Fjnh", "reviewer": "Reviewer_Maxm", "summary": "The paper presents BigMac3D, a large-scale 3D dataset of macaques that bridges visual appearance and 3D pose representations for action recognition. The dataset comprises over 750 multi-view scenes recorded with 16 synchronized cameras and annotated with 3D skeletal joint rotations, segmentation masks, 2D keypoints, and ethogram-aligned behavioral labels.\nTo generate these annotations, the authors design a markerless motion capture pipeline that integrates surface-based mesh tracking with subject-specific textured avatars, enabling precise reconstruction of individual body shapes and motions. They further introduce BigMac500, a curated benchmark subset for evaluating pose-informed action recognition, demonstrating that incorporating 3D pose descriptors leads to significant gains in mean average precision (mAP) across diverse visual backbones.", "questions": "### **Questions for the authors**\n\n* I am not fully clear on what using “Real” rather than “Synthetic” 3D data concretely enables. Could you clarify the specific advantages it brings for model quality, bias reduction, or downstream generalization?\n* How were the action labels produced in practice? Please describe who annotated them, the protocol, the ethogram mapping procedure, inter-annotator agreement, and any quality control. I could not find these details in the paper or the cited appendix.\n* Can you provide evidence that the benchmark gains transfer to in-the-wild data? For example, a cross-domain test, few-shot adaptation, or evaluation on an external macaque or primate dataset.\n* How well does the model personalize to individual monkeys? Concretely, how morphable is the 3D template, and what quantitative measures do you have on shape fitting accuracy across individuals?\n* Do you quantify annotation uncertainty for the intermediate labels (segmentation, keypoints, identities) and analyze how these errors propagate to the mesh fitting and final pose vectors?\n* How robust is the method to partial occlusion or missing views? Please report performance as a function of visible views, occlusion level, and view dropout, and describe any mechanisms that was used for handling this.\n\n### **Summary comment**\n\nThe paper is strong and promising, but for a dataset and pipeline of this complexity, the current qualitative evidence is not sufficient. A set of targeted quantitative analyses would make the contribution much more convincing: ablations for each loss term, uncertainty propagation from 2D annotations to 3D fits, view-drop robustness, transfer to in-the-wild scenarios, and individual-specific morphability metrics. If provided, these results could substantially strengthen the work and its impact.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "### **Strengths**\n\n* The paper introduces a large-scale dataset that integrates detailed 3D pose–shape representations with action labels for macaques, addressing an important gap in current non-human primate research.\n* The proposed annotation and reconstruction pipeline is innovative, combining markerless motion capture, subject-specific mesh tracking, and photometric texturing in a scalable way.\n* The BigMac500 benchmark is a valuable contribution, demonstrating quantifiable improvements in action recognition performance when incorporating 3D pose descriptors.\n* The annotation process **appears** well designed and yields annotations of good quality.\n* The paper is simple and well written.\n* The research problem is both interesting and challenging.\n* The scale and diversity of the dataset is appealing.", "weaknesses": "### **Main Weaknesses**\n\nThe paper currently lacks sufficient detail about the dataset annotation pipeline. A clear, visual overview of the entire annotation process would be highly valuable—ideally presented as a dedicated figure (either in the main paper or, at minimum, in the appendix).\n\nSecondly, the loss functions introduced for optimizing the annotations are not supported by ablation studies. Without these, it remains unclear which losses meaningfully contribute to the final annotation quality, and to what extent.\n\nMoreover, there is **no quantitative evaluation of the annotation quality** beyond Table 2, which only reports IoU scores. This metric assesses 2D mask alignment but provides limited insight into the accuracy of the 3D surface models. It would be important to include additional metrics that reflect 3D fidelity and realism—such as *PSNR*, or *KL divergence*—as well as a measure of temporal consistency across frames. Since multiple losses are designed to optimize different parameters, it would be critical to demonstrate, both quantitatively and qualitatively, how these losses affect the reliability of the learned representations.\n\nFinally, while the reported IoU of 0.844 is promising, it requires further clarification. Where do the remaining percentage points of error originate? A visualization comparing predicted and pseudo-ground-truth masks would help interpret this discrepancy—distinguishing between errors due to poor mesh fitting (undesirable), inaccurate pose estimation (undesirable), imperfect 2D masks (potentially acceptable), or minor border inaccuracies (acceptable).\n\n### **Other Weaknesses**\n\n* The paper provides no information about computational requirements. Details on runtime, GPU hardware, and annotation throughput should be included.\n* The dataset was collected in controlled laboratory conditions, which may limit generalization to in-the-wild scenarios.\n* The pose baseline used for prediction is not described. The paper should clarify the model architecture, input format, and training procedure.\n* Loss ablations should be provided wherever feasible .", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mRknxIP57V", "reviewer": "Reviewer_Q757", "summary": "This paper introduces BigMac3D, a large-scale dataset of over 750 multi-view video scenes capturing rhesus macaques performing diverse individual and social behaviors. Using 16 synchronized calibrated cameras, the authors reconstruct 3D surface-based pose and shape representations through a markerless motion-capture pipeline with skeletal rigs (115 joints) and subject-specific meshes. The dataset provides pose vectors, segmentation masks, keypoints, bounding boxes, and ethogram-aligned action labels, forming the first large-scale benchmark coupling 3D surface pose with action recognition in non-human primates. A derived benchmark, BigMac500, evaluates how these pose descriptors improve action classification performance when fused with various vision backbones (ResNet50, ViT, DINOv2, VideoPrism), achieving consistent mAP gains (≈+8–12%)", "questions": "Could the authors provide more details on how the dataset is organized and validated? In particular, it would be helpful to clarify how the training and testing splits are defined for BigMac3D and BigMac500, and whether subjects are disjoint across splits to avoid identity memorization. Additionally, more explanation on how 3D pose accuracy and annotation confidence were assessed would strengthen the paper’s reliability. Finally, reporting key dataset statistics such as frame sampling rules, camera ID distributions, and sequence lengths would improve the dataset’s transparency and reproducibility.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1.The dataset is unprecedented in scale and annotation richness for macaques, combining high-fidelity 3D surface tracking with ethogram-based behavioral labels across multi-individual interactions\n\n2.The work builds on recent multi-view surface tracking advances (MAMMAL, AniMer+) but contributes practical improvements, such astemporal regularization, cropped differential rendering, and texture fitting, to enable efficient processing of thousands of sequences.\n\n3.Comparisons against MAMMAL and AniMer+ confirm clear reconstruction gains (IoU = 0.84 vs 0.71/0.59). The action-recognition experiments further demonstrate that incorporating 3D pose vectors significantly boosts mAP across all tested encoders.\n\n4.The dataset addresses a genuine need in neuroscience and ethology by bridging vision models with interpretable 3D motion cues, promoting reproducible behavioral analyses for NHP research.", "weaknesses": "1.The paper omits several critical details necessary for reproducibility, such as frame selection criteria, train/test split definitions, and subject-level separation (e.g., whether BigMac500 avoids identity leakage across splits). Although the authors describe a 70/10/20 split for the BigMac500 action recognition subset, this rule applies only to that benchmark, not the full BigMac3D dataset. There is no explicit statement on whether subjects are disjoint across splits. Furthermore, while the dataset uses a 16-camera synchronized capture setup, no visualization of camera IDs, baseline distances, or view diversity is provided in any figure or table, making it difficult to assess spatial coverage or view redundancy. These omissions weaken the dataset’s clarity and reproducibility.\n\n2.While the authors claim “high-quality surface reconstruction,” the paper provides no quantitative 3D validation, such as per-joint reconstruction error or ground-truth comparison. Qualitative figures (e.g., Fig. 3) indeed show visually strong fits, but the absence of uncertainty estimates, human-verified subsets, or error histograms leaves the actual accuracy unquantified. This weakens confidence in downstream claims about 3D pose reliability and consistency across subjects.\n\n3.Related to the first issue, the lack of clarity on subject-level isolation across training and testing sets raises potential concerns about identity memorization in the reported benchmarks. Explicitly ensuring disjoint subjects would help establish a fair evaluation protocol for behavior recognition tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J2DRUN6X9a", "reviewer": "Reviewer_6cL8", "summary": "The authors present BigMac3D, a large dataset derived from hundreds of scenes of interacting rhesus macaques in a laboratory setting. Each scene is accompanied by 3D pose descriptions in the form of keypoints and textured surface meshes; per-view 2D keypoints and segmentation masks; and per-scene action labels. The authors then provide baseline action recognition results using a variety of pose and video descriptors, and demonstrate model improvements when high-quality pose descriptors are added to video descriptors.", "questions": "Section A.4 describes the performance of the detection and pose estimation pipelines, but it is not clear from the reported metrics how less-than-perfect detections/poses influence mesh optimization. Furthermore, it is conceivable that these numbers will be highly dependent on the behavioral category, so it might be helpful to break them down this way. Section B.5 addresses this briefly, but I feel there should be much more space allocated to these issues.\n\nAre there any post-processing steps used for the detection and pose estimation steps? For exmaple, multi-view (in)consistency across views could help flag detection errors.\n\nThere is no quantification of the segmentation masks from SAM2. Have the authors verified the performance of this model in any way? Has any post-processing been applied to clean up the segmentation masks? Presumably the predicted silhouettes would be helpful here? Is there a reason to include the SAM2 segmentation masks and not just the silhouettes output by the mesh-tracking stage?\n\nIt is not clear what the high-poly template model is used for.\n\nHow are skinning weights W chosen?\n\nI am not super familiar with 3D mesh-tracking; it would be useful for that section to start with a few-sentence overview of the process before diving into specifics. For example, the appendix mentions this briefly, but it's not clear that the first step is to perform individual shape adaptation using the manual annotations, and then this information is used for the videos - this is not clear in the main text. Some other confusions I have:\n- low- vs high-poly meshes, when are they used/not used\n- SAM2 masks - are these used at all in the optimization process? If not, are they ever used to validate pieces of the optimization process?\n- when are ground-truth keypoint annotations used in the pipeline, versus keypoints predicted by the pose estimation model?\n\nThe loss function in Eq. 2 contains a large number of hyperparameters. These values are given in Table 7 but there are no details on how these are chosen, or how sensitive the different hyperparameter values are.\n\nTable 2: it is unclear what this actual metric is. The text says \"IoU of mesh fits over time\" - what is \"ground truth\" here? Is it IoU of meshes between time t and t+1? Table 2 legend says \"first frame of the respective action\", which seems contradictory. Please clarify.\n\nL377: how is \"reliable pose reconstruction\" defined?\n\nL411: the Kolesnikov reference should be Dosovitskiy\n\nThe conclusions from the action recognition results section are't entirely fair. The pose information is (presumably) sampled at 40 Hz, while the video information is sampled at 10 Hz. This confounds any direct comparison between the two. The more proper way to conduct this experiment to make the claims the authors want would be to downsample to pose data to 10 Hz or upsample the video data to 40 Hz. It is true that the former will lead to overall lower performance, and the latter to increased compute, and the hybrid results presented in the current submission could provide a useful approach for trading off accuracy and compute. But one of the aforementioned control experiments should be done.\n\nRegarding the effects of pose representation, Blau et al. (https://arxiv.org/abs/2407.16727) looked at the incorporation of time into the pose representation and found a large performance improvement across multiple datasets. BigMac3D cool dataset to test these findings across a wider variety of spatial representations, if they are so inclined.\n\nIt's not actually clear what the difference between BigMac3D and BigMac500 are - BigMac3D has 750 \"scenes\", while BigMac500 has 8k \"videos\". Is one a subset of the other? Does BigMac500 not have segmentation masks, 2D keypoints, etc.? In general, a table with more exact details about what data is included in each dataset would help clarify this confusion.\n\nDo the authors plan to publicly release the manual keypoint annotations? These could be very valuable for developers of 3D/multi-view pose estimation models.\n\nIt would be useful to provide train/test split information for the action recognition part so that new users can compare other models.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 4, "strengths": "BigMac3D presents a highly valuable resource to the behavioral quantification community. The dataset is very complete in terms of videos, keypoints, meshes, masks, etc., as well as the behavior annotations. This allows various types of methods developers to utilize this dataset.\n\nThe authors have also provided action recognition performance for a variety of baseline models, which is useful for methods developers looking to benchmark their models on this dataset.", "weaknesses": "The manuscript as written lacks clarity in various ways. The distinction between BigMac3D and BigMac500 isn't obvious. The types and amount of data related to each dataset is also not clear. The actual pipeline for collecting the raw data and creating the derived quantities (3D labels, meshes, etc.) is never clearly laid out. Addressing these issues would vastly improve the mansuscript.\n\nThe manuscript also lacks enough details about the quality of the dataset. The metrics in Tables 5+6 are a start but these numbers should, for example, be shared by behavior category. Example videos overlaid with keypoints and/or bounding boxes and/or segmentation masks would provide better intution for dataset quality. It is common to perform some kind of post-processing on the outputs of various models (detectors/pose estimators/segmenters) to clean up obvious errors; either this was not done or it was not described. These sorts of quality control steps are crucial before releasing such a large dataset to the community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)