CoMotion: Concurrent Multi-person 3D Motion
Authors: Alejandro Newell
Peiyun Hu
Lahav Lipson
Stephan Richter
Vladlen Koltun
Venue: ICLR 2025 Poster (2025)
Keywords: human pose estimation
3d human pose
tracking
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Online multi-person 3D pose estimation and tracking in video with recurrent cross-attention

Abstract: We introduce an approach for detecting and tracking detailed 3D poses of multiple people from a single monocular camera stream. Our system maintains temporally coherent predictions in crowded scenes filled with difficult poses and occlusions. Our model performs both strong per-frame detection and a learned pose update to track people from frame to frame. Rather than match detections across time, poses are updated directly from a new input image, which enables online tracking through occlusion. We train on numerous image and video datasets leveraging pseudo-labeled annotations to produce a model that matches state-of-the-art systems in 3D pose estimation accuracy while being faster and more accurate in tracking multiple people through time.

Published as a conference paper at ICLR 2025
COMOTION: CONCURRENT MULTI-PERSON 3D MOTION
Alejandro Newell
Peiyun Hu
Lahav Lipson
Stephan R. Richter
Vladlen Koltun
Apple
ABSTRACT
We introduce approach for detecting and tracking detailed 3D poses of multiple people from a single monocular camera stream. Our system maintains temporally coherent predictions in crowded scenes filled with difficult poses and occlusions. Our model performs both strong per-frame detection and learned pose update to track people from frame to frame. Rather than match detections across time, poses are updated directly from a new input image, which enables online tracking through occlusion. We train on numerous image and video datasets leveraging pseudo- labeled annotations to produce model that matches state-of-the-an systems in 3D pose estimation accuracy while being faster and more accurate in tracking multiple people through time.
INTRODUCTION
Some of the most exciting applications of computing require understanding where people and how they move. A single monocular camera can support a variety of human-centric applications if coupled with system that can model the motion of all humans in visual field. For the broadest range of applications, we are interested in systems that (a) see multiple people, (b) estimate their poses 3D, (c) track their poses through time monocular video stream, even through occlusion, and (d) do so online, in a streaming fashion, processing each frame and updating everyone's 's estimated 3D poses without peeking into the future.
For such system to work in the wild, it must handle cluttered scenes that are crowded with people. As people move about, the model must keep track of who who from frame to frame in order to produce a coherent 3D motion estimate. This can be difficult as people occlude each other, pass behind objects, and step out of view. Maintaining accurate 3D pose estimates online particularly challenging, the model cannot leverage future context to fill in the gap for missing observation. Instead, the model must forecast to the best of its ability and snap to the correct state as soon as the person is visible again.
Many existing approaches to pose tracking follow two-stage -assect-and-asocciate paradigm (Insa futdinov et al. 2017 Xiao et al. 2018, Doering et al. 2022 Rajasegaran al.) 2022). The idea is to run a state-of- -thee-rrt pose estimation system for each frame, and then link poses across frames using cues such as proximity and appearance. As with many two-stage approaches, the second-stage pose estimates and tracks can be undermined by flaws in the first stage. If detection is missed in the first stage, some other mechanism must be introduced update the pose. For example, one option is to fill in any missing poses offline using the context of the full track (Goel et al. 2023). However, this solution not suitable for online applications that deal with live video.
We present CoMotion, fundamentally different manner. Following the tracking by attention paradigm (Meinhardt al. 2022), we do not link independent per-frame detections rather, we train a recurrent model that maintains a set tracked 3D poses and updates them when a new frame arrives. To update the poses, CoMotion directly ingests the pixels of the new frame. The model uses whatever image evidence is available to update the poses of all people in the scene simultancously - even those who may not be currently visible. One advantage of this approach is that CoMotion can learn utilize subtle pixel cues. A pair of feet poking out may be insufficient to trigger an independent detection, but can still be quite informative if the model has been following that person's 's movement.
aadeo-based approach that performs Trame-to-frame pose updates in

---

Published as a conference paper at ICLR 2025
Input
CoMotion
4DHumans
Input
CoMotion
4DHumans
Figure CoMotion tracks 3D poses online from monocular RGB video. Rather than detect new poses in each frame and associate them existing tracks, CoMotion updates tracks directly from incoming image features. As result, CoMotion keeps track of distinct individuals as they overlap in the camera frame (top) and occlude each other (bottom). Arrows highlight some points of interest.
Data a challenge with such a video-based approach, for both training and evaluation. No dataset exists that provides ground-truth 3D poses of groups of people moving in diverse, cluttered real- world scenes. As result, is rare to find video-based 3D multi- pose methods that support in-the-wild videos with large numbers of people. Comparable methods evaluate tracking on limited settings with a handful of people (Reddy et al. 2021 Sun et al. (023). Only top-down methods report tracking results on more challenging real- world videos (Rajasegaran et al. 2022).
We find that we can achieve excellent performance by training on heterogeneous mixture of datasets. Each dataset offers complementary form supervision some are single-image, some are video, some are synthetic, and some provide partial ground truth. In particular, we leverage pseudo- lseudo-labeling to supervise our system on challenging real-world video (Andriluka et al. 2018 [Sun et al.) (022) that would be exceedingly difficult annotate otherwise. We unaware of any prior work this that trains on a comparable heterogenous mix of datasets. In particular, there are no multi-person 3D pose methods that supervise temporally unrolled predictions complex videos such those provided by Pose Track (Andriluka et al. 2018 Doering et al. 2022).
We evaluate pose estimation performance and multi-person tracking using standard datasets for each task. CoMotion exhibits ffte-of-the-ar pose estimation performance, while providing better and faster tracking with fewer temporal artifacts such as jitter and dropped poses. Figure 1 illustrates this
2

---

Published as a conference paper at ICLR 2025
combination of 3D pose estimation accuracy and tracking stability. On PoseTrrac221 (Doering et al., 2022), CoMotion improves MOTA by 14% and IDFI by 12% relative to the prior state of the art. At the same time, CoMotion is order of magnitude faster than the prior state of the art.
2 RELATED WORK
Many approaches to pose estimation and tracking have been explored. Some detect poses only in a single image, some operate video but only handle single person, and some track multiple people in video but don't 't estimate their poses. Few take on the holistic setting of predicting and tracking the poses of multiple people in video, let alone in streaming (online) manner. For accessible overview of the different perspectives, we classify relevant work along two dimensions, namely the input modality (single image or video) and the handling of multiple people.
Single-person, single-frame 3D pose. Many approaches to 3D human pose estimation predict a pose from single image that is cropped to the target person (Bogo et al. 2016; Kanazawa al. 2018 Pavlakos et al., 2018). Poses are commonly represented by the SMPL model (Loper et al.) 2015). which defines a mesh from joint angles and shape parameters. The SMPL parameters can be regressed directly from an input image (Gocl et al., (2023). Another common strategy is to employ feedback loop that alternates between predicting SMPL parameters and observing the reprojection to image
space to inform further updates (Zhang et al. [0211a] 2023a c Wang & Daniilidis 2023). Much
progress in this setting can be attributed to scaling up datasets, model sizes, and training times (Goel et al. 2023 Khirodkar et al. 2024 Sárándi & Pons-Moll 2024).
Single-person pose from video. This setting assumes that person has already been detected and tracked, such that it can be presented to the model through sequence of cropped frames bounding boxes. A line of work operates on pre-extracted 2D keypoints (Zheng et al., 2021 Zhang et al. 2022: Tang et al. 2023 Zhao et al. 2023) with transformers because the sparse input signal enables modeling of large temporal windows, Alternatively, recurrent networks enable online aggregation of features between frames. VIBE (Kocabas et al., 2020) and PMCE (You et al., 2023) use GRU, and RoHM (Zhang et al. 2024) uses a denoising diffusion model to decode subsequent frames. Our approach also uses GRU to run inference on video. However, where these prior methods rely on an external tracking solution and can only perform inference on single subject, our method does its own tracking to handle multiple subjects in parallel.
Multi-person, single-frame pose. A straightforward approach to predicting poses of multiple people is detecting their individual bounding boxes and processing those via a Lingle-person pose estimator (Goel et al. 2023 Fang et al. [017 Paaannreouu et al. 2017: Huang et al. (017). However, this approach scales poorly to crowded scenes. More recently, several works extract a dense feature map from the input image and perform cross-attention on set of query tokens regressed from ResNet or VIT applied to the same image (Liu et al. 2023 Shi et al., 2022; Baradel et al. 2024). In these approaches, the costly dense feature extractor is only run once per image. We also apply cross- attention, but our model produces queries in each timestep conditioned on past history and must also attend to people who may not be visible in the current frame (e.g., because they temporarily move out of the camera's 's field of view), which would not be relevant in the single-image setting.
Multi- pose from video. A key challenge in multi- pose from video is associating new observations with pre-existing tracks. An early approach was to estimate a graph of pose keypoints with intra-/inter-fran edge weights and then estimate connected components by solving a minimum-cost multi-cut problem (Insafutdinov et al. (017). Better results were later obtained using a track-and-detecd paradigm, where independent per-framme frame detections are grouped into tracks using visual keypoint features and image location (Girdhar et al., 2018) as well as 2D motion (Xiao et al. 2018). More recently, PHALP (Rajasegaran et al. 2022) adopted this mechanism for 3D pose rather than 2D, and also used the predicted 3D poses as an additional feature for grouping. 4D Humans (Goel et al. 2023) went further with a more general-purpose purpose pose representation (Loper et al. 2015) and new architecture. These approaches achieve excellent results modern benchmarks but scale poorly on videos with many subjects, since they run a pose estimator independently on each cropped detection. In contrast to these tracking-by Merkiingby-detettii -ydtecctioo methods, CoMotion estimates poses and localizes existing tracks in parallel, reasoning over all poses simultancously. In addition to a more holistic treatment of the scene, this approach is an order of magnitude faster.
3

---

Published as a conference paper at ICLR 2025
features
Image encoder
Detection moduue
Image Features
Pose update module
5D Poses Hiddon ststes
Pose state encoder
Cross-attention
Frame
Transformer
/eaturee
New
New track
decoder
decoder
Pose update
Pose Matching
GRU
Trocks Framet: 1-
Updated tracks
Tracks Frame
3D Poses
Hiddon stanos
Figure 2: Overview. CoMotion estimates 3D poses for all people in a frame. An image encoder produces image features Ft, which are passed through the detection module identify potential new tracks. In parallel, the pose update module attends to fst to update the existing tracks from the previous timestep. Both outputs are compared each other to decide whether instantiate or remove any tracks. If a detection is flagged as a new track, it is passed through the update module before being added to the final output tracks for the current frame. The inset details the pose update module.
Unlike byeiing-b--ettcttonn, tracking- by-tttttttttt (Meinhardt et al. 2022, Sun al. 2020 detects new objects and updates existing tracks jointly by performing cross-attention between an image and set of learned per-track and new-object query tokens. While many of these works condition the query tokens only on the previous few frames (Zeng et al. (022)), MeMOTR maintains a long-term memory of per-track query tokens and gradually updates them based on new observations (Gao & paradigm in this work, but find that there are many challenges to instantiating this paradigm for multi-person person 3D pose tracking, including the nature of available training data. Bounding-boo box-based methods can get away with training on pairs of frames (Meinhardt er al. 2022), but we find that it critical unroll on longer video sequences during training to learn to track detailed 3D poses accurately.
Wang (023). 3). We follow the baockin-bb--ttentiin
3 COMOTION
Preliminaries. Given a sequence of monocular RGB images (I'.P...., I'. ['} Rhxwxx3 we aim to produce 3D pose estimates for each person in the scene at each time-step L conditioned only on the past and the present (no peeking into the future). Each prediction should associated with particular identity such that we know all poses belonging to given individual. Predictions provided as complete trajectories from the beginning to the end of track. Our system does not model tracks through long occlusions nor does it perform re-identification to link tracks.
We follow standard practice (Bogo et al., 2016 Kanazawa et al. 2018)) and parameterize each person with SMPL (Loper al. 2015). which consists of translation term y R3, joint angles R? (including a global orientation), and shape parameters 3 E 10
All estimates are made in the camera coordinate frame. We do not explicitly model any changes to the camera pose or intrinsics (e.g., due to camera motion or zooming). During inference, the system accepts as input an intrinsics matrix K. All output 3D estimates will project correctly back into the image according to this provided matrix using the pinhole camera model. If no ground truth intrinsics available, we fall back to reasonable defaults. To increase robustness against incorrect intrinsies, we augment K during training allowing to make predictions on in-the-will videos by providing either generic default setting for К or using the correct intrinsics when they are available.
Overview. CoMotion is a complete tracking stack which consists of logic for starting and ending tracks, and means to update tracks from frame to frame. Figure provides an overview of the architecture and logic. Given the current frame, the detection module detects 3D poses to serve as candidates for new tracks. The pose update module adjusts the poses and hidden states of all existing tracks. Both modules operate directly image features FL produced by image encoder in our case standard ConvNextV2 model (Woo et al. 2023). We compare detections to existing tracks in order to determine if any new tracks should be created and to identify which (if any) existing tracks are stale. We describe each component in the following paragraphs, and refer to the appendix for additional details.
4

---

Published as a conference paper at ICLR 2025
Detection module. Our detection module is single-shot (Liu et al. 2016) and consists of detection head that applies several consecutive downsampling and ConvNeXt blocks (Woo et al.) 2023) to the given image features F*, yielding low-resolution feature pyramid. At each level of the pyramid, 1x1 convolution decodes candidate detections. Instead of multiple anchor boxes as a traditional bounding-hox approach, our detector produces multiple candidate poses each spatial location, each consisting of SMPL parameters and a confidence value. After pruning candidate poses via non-maximum suppression, we complement poses with hidden states to form detected tracks for the current frame. More formally, we define track X = (r, 8, B, h) for each detection, where h is a prro-initialized hidden state vector.
Pose update module. The pose update module refines a set of tracks given image features Ft, Let X U X be the union of all existing tracks X for persons i at timestep t. We can then write the update to poses and hidden states of existing tracks as X update( (F X!-1).
The inset on the right of Fig. 2 iilluttrates the high-level architecture of the update module. We start by encoding X into set of tokens. Specifically, from each track's i SMPL. parameters, we compute 2D projected keypoint locations, then pass the raw SMPL parameters. the 2D keypoints, and the current hidden states through an MLP to produce initial track tokens. Given these tokens, we use cross-attention to query image features fe attending to the entire image for information about how each pose has changed from the previous timestep. We perform further processing with several transformer layers, from which we decode a new SMPL pose and updated hidden state for each track. We use GRU for the recurrent hidden state update. For more architecture details, please refer to Appendix A.3
The model must learn which information in the current frame is relevant for updating each track. This allows refining poses based on partial or indirect observations that may not be enough to trigger a detection. Furthermore, if visual information is available that pertains occlusion), the model is still responsible for updating the pose and position of the person.
given track (i.e., during
Track management. We use a simple set of heuristics to determine when delete or instantiate tracks. To guide these heuristics, we measure the similarity between tracks with modified version of Object Keypoint Similarity (OKS) from COCO (Lin et al. 2014). OKS compares the distance between pairs of keypoints and returns value between 0 and 1, with I indicating perfectly overlapping poses. This is a useful signal as it distinguishes cases where a coarser similarity signal like bounding box intersection-over-une
(IOU) would be insufficient. Specifically, we can differentiate two people who occupy the same space in the image but have differing poses (such as a pair of dancers clasped together). For details on modifications to the OKS calculation see Appendix A.3.4
Instantiation. From all detections X = -U Xj. we identify new tracks those with insufficient
similarity to existing tracks (i.e., with max; (OKS(X) X')) < 0.2). We initialize these tracks by refining them via the pose update module which makes slight adjustments to the pose and yields a nonzero initial hidden state. The resulting new tracks are then added to X
Deletion. For each track X,, we maintain an exponential moving average of matched OKS scores sf = a max,(OKS(XX X!)) + (1 - a) sf with a = 0.2. If this average drops below 15, we delete the track. We also delete any track that has existed for fewer than 4 frames and registers an OKS below 0.15 (without the moving average) as this often indicates a spurious detection.
Another important check is whether two tracks overlap for too long. Specifically, if pair of tracks Xx) maintains an OKS above 0.6 for more than 20 consecutive frames (two-thirds of a second), delete the track with worse alignment to recent detections indicated by a lower s'. While this heuristic is effective in flagging tracks that have collapsed to the same pose, it may on occasion prematurely end occluded tracks that the network is handling correctly.
Overall, these heuristics serve as simple, quick checks for managing the discrete set of tracks X at no additional computational cost while letting the network do the more difficult task of modeling how people move across time. The specific thresholds and values provided serve to provide a rough sense of how we use the system in practice with reasonable defaults, but these are not magic numbers which need to be adhered to perfectly. A nice feature of CoMotion is that these values can be easily tuned to target different downstream settings, e.g., some cases may be helpful to be more forgiving before eliminating uncertain or poorly matched tracks.
5

---

Published as a conference paper at ICLR 2025
4 TRAINING
4.1 DATASETS
We follow 4D Humans and train on a mixture of diverse pscudo-labeled image datasets to increase robustness on in-the-wild images (Goel et al. 2023). Specifically, we train on Insta Vnstittriety (Kanazawal et al. 2 COCO (Lin et al. (2014)) and MPII (Andriluka et al. 2014). As we found the original pseudo-labels provided by 4D Humans to be too sparse for training our detection module, relabeled the datasets with NLF, a recent state-of-the-art single-person 3D pose model (Sárándi & Pons-Moll 2024). Further discussion of this relabeling process can be found in Appendix A.4
In addition, we train on videos with ground truth tracks from Pose Track (Andriluka et al. 2018) and DanceTrack (Sun et al. 2022). PoscTrack only provides 2D keypoint annotations while DanceTrack only offers bounding boxes, so we pseudo-label the videos with NLF. While this yields 3D annotations for challenging real-world video sequences that would be impossible to obtain otherwise, we note one drawback. The predicted labels are unreliable for people in close proximity, so we flag and ignore these samples during training.
We further train on large set of sequences with perfect synthetic 3D ground truth. Specifically, we include BEDLAM (Black et al., 2023), which consists of scenes with many people with sampled motions sourced from AMASS (Mahmood et al.) 2019), and WHAC-A-MOLE (Yin et al.. 2024). which features clips of dancing couples.
4.2 CURRICULUM
We follow a three-stage curriculum to train CoMotion.
Stage pretrains the image encoder and detection module on large batches of single images, namely the pseudo-labeled Insta Variety, COCO, and MPII datasets and BEDLAM. We match candidate de- tections to ground-truth annotations and supervise the output SMPL terms and detection confidences.
To optimize the model's s SMPL predictions, we employ the following loss functions: an -osss to minimize 2D projection error, an L1 -loss for the root-normalized 3D joint position error, and an C2-loss for the difference in SMPL joint angles. We supervise the betas on samples from BEDLAM with -loss,, but refrain from applying a loss on the betas from the eseudo-labeled samples.
We apply a binary cross-entropy loss to supervise the output confidence term sampling a random subset of unmatched detections to serve negatives. Additionally, we use a keypoint heatmap loss (Cao et al. 2017) as an additional auxiliary training signal. The model is trained for 400K iterations 32 ATO0 GPUs taking approximately 3 days. By the end of this stage, we have a strong single frame multi-person 3D pose estimation system.
Stage 2 freezes the image encoder and detection module and exclusively trains the pose update module on multiple frames. Our training procedure is simpler than the tracking setting deployed test time as we do not actively manage tracks during training. That is, no point do add or remove tracks as we unroll through a clip. This is a practical detail to support training on minibatches. Instead, on the first frame we match detections to a subset of ground-truth annotations and unroll those specific samples through time.
In this stage, we train on short, 8-frame video clips from BEDLAM and THACC--MOLL as well as the pseudo-labeled PoseTrack and DanceTrack data. We also synthesize videos by panning and zooming on single images from Insta Variety. We train for 200k iterations, which take 3 more days. We supervise the updated poses at each timestep applying the same SMPL losses described in Stage
Stage extends training of the pose update module longer video sequences. We sample sequences of length 96 from DanceTrack and length 32 from WHAC-A-MOLL Samples from such long sequences are particularly slow to train so include mix of shorter 8-frame clips from PoseTrack and images from Vastetariety. To reduce GPU memory consumption, we enable gradient checkpointing. The model is fine-tuned for 50K iterations over 1.5 days.
Overall, this training curriculum allows us to efficiently train a system to handle video by front- loading the initial visual feature learning on single images before the slower video training stage.

---

Published as a conference paper at ICLR 2025
Table 1: Tracking evaluation. Performance of CoMotion and several baselines on PoseTrack21. We report results using the official evaluation code (top) as well as results after fixing bug in the evaluation code that caused the 'ignore' regions to be handled incorrectly (bottom, marked with t).
PoseTrrac21
Method
MOTA
IDFI
IDP
IDR
TRMOT Wang ct al.
2020
47.2
57.3
70.0
46.6
FairMOT (Zhang ct al
2021b
56.3
63.2
81.0
51.8
CorrTrrack + RelD (Doering el al.
2022
52.0
66.5
72,4
61.4
Tracktor++ (Bergmann al. 2019
59.5
69.3
76.4
63.5
CoMotion (ours)
67.6
77.9
83.4
73.0
4DHumans Goel et al. 2023
56.7
70.9
87.1
59.7
CoMotion (ours)
71.4
79.5
87.1
73.0
Also, we can fit much larger batches during video training since we do not have to calculate gradients for or update the weights of the image encoder. And even though most training on very short clips, the model behaves well when unrolled over long temporal sequences spanning minutes.
5 EXPERIMENTS
No standard benchmark evaluates the full combination characteristics that CoMotion was designed for: tracking 3D poses of multiple people through video. For quantitative comparisons prior work, we evaluate CoMotion on subset of its capabilities at a time, "projecting" it to existing benchmarks that focus either on tracking stability (without evaluating 3D pose accuracy) or pose estimation on images (without evaluating tracking). We use established benchmarks for 2D and 3D pose estimation and, separately, tracking, and adhere established evaluation protocols and metrics.
Tracking. To evaluate tracking across crowded sequences with interesting poses, we turn to Pose- Track21 (Doering et al. 2022 We use the evaluation code provided by Doering et al. (022) to compute standard tracking metries. We report Multi-Object Tracking Accuracy (MOTA), an aggregate score that penalizes missed detections, false positives, and ID switches (100 is perfect score). Other metrics such IDFI and ID precision and recall (IDP and IDR) quantify how well the network preserves a single tracked identity per person. The results are listed in (top).
We run our full stack on a target input video from beginning to end. Frames are padded and resized to 512x5 12, and we unroll frames at this fixed resolution (at no point do we perform any cropping to individuals). CoMotion substantially outperforms prior work, improving MOTA by 14% and IDFI by 12%.
In the course of performing this evaluation and analyzing the results, we found a bug in the Pose- Track21 I evaluation code. PoseTrrac211 provides partial annotations accompanied by 'ignore' regions that mark parts of images in which ground-truth data is missing. The 'ignore' regions crucial for evaluation because they signify that tracks predicted in these regions should not be regarded as false positives they may be correct, but may not align with ground-truth annotations because annotations in these regions are known to be missing. The bug pertains to the handling of these 'ignore' regions and caused many detections in the 'ignore' regions to be incorrectly regarded as false positives. Table [ bottom) reports results after this bug is fixed. (Here we can only benchmark CoMotion and method that is reproduced in our environment, since it's not possible to carry numbers over from prior literature.) We will release our fix to Pose'Track21 and recommend that future work adopt the corrected evaluation code. More details can be found in Appendix A.1
The most competitive baseline that also performs high-quality 3D pose estimation is 4D Humans (Goel cet al. 2023). The paper Goel et al. (2023) reports results on PoseTrackI8 (Andriluka et al. 2018) 8) but not on the more accurately and completely annotated PoseTrack21. Unfortunately, due to the drastically incomplete annotations Pose PoseTrack18 (which motivated the creation of Pose (osekTrck21) 1), we observe that tracking evaluation on PoseTrack 18 can be misleading. Many sequences in Track18 only have annotations for a fraction of the people in the scene, and models are penalized for correctly detecting and tracking people who lack annotations. See Sec. A. for examples.
7

---

Published as a conference paper at ICLR 2025
Table 2: PoseTTrack18 vs. PoseTTrack21. The annotations in PoseTrack18 are drastically incomplete, penalizing methods that correctly detect and track people in the scene. Indeed, this inspired the creation of PoseTrack21 (Doering et al. 2022), which provides more complete annotations and was released direct replacement of Pose Track18 (same images, more complete annotations). We provide results on PoseTrrack15 for backward compatibility with Goel et al. (202)). but strongly
recommend that all future work adopt Track2 instead. (* We rerun the authors" code in order to report performance on Pose FracTr.c))..
PoscTrack18
PoseTrack21
Method
HOTA
IDs
MOTAT
MOTA
IDF1
IDPt
IDR
FPI
FNG
FPS
4DHumans (Goel et al. 20033
57.8
382
61.4
4DHumans (reproduced)
58.0
349
61.8
56.7
70.9
87.1
59.7
7817
50652
0.51
CoMotion strict
58.2
232
59.9
61.8
74.0
89.1
63.3
6086
45664
CoMotion
54.9
344
51.3
71.4
79.5
87.1
73.0
8115
30394
5.68
Since 4D Humans does not report results on PoseTTrac21,, we rerun their method and reproduce their original predictions (which match the numbers reported in their paper on their PoseTrrak 18 evaluation). While it performs well on the incomplete ToseTrack18 setting, we observe many missed detections which result in lower MOTA on PoseTrack21 where the full annotations We dig further into this in Table 2 observing that we can match the numbers of 4D Humans in the original DosekTack111 18 setting with CoMotion by applying stricter threshold on detection confidences to dismiss a larger number of (correct) detections.
available.
When evaluated on PoseTrack21 we observe that our 'strict' model is better than 4D Humans across the board. When we replace the strict threshold with our default, the system is heavily penalized in PoseTrack18, since many correct tracks are regarded as false positives, lowering MOTA from 59,9 to 51.3. But when these same predictions are evaluated on the more complete annotations provided by Prsc2rack21, MOTA increases from 61.8 to 71.4.
Pose estimation. 3D pose estimation is typically assessed in an 'oracle' single-person setting where the bounding box of the target individual known in advance. For fair comparison to prior work using standard pose estimation metrics, we run CoMotion on individual frames and tight bounding- box crops while also reporting performance in the much more difficult setting where the full image is provided as input without oracle conditioning signal.
We evaluate both 2D and 3D metries, since 2D pose datasets offer more diverse in-the-wild data with challenging poses. Specifically, report Percentage of Correct Keypoints (PCK) COCO (Lin) cct al. 2014) and PoseTrack11 18 (Andriluka et al. 2018). and Mean Per-Joint Position Error (MPJPE) on 3DPW (von Marcard al., 2018). PCK calculates the percentage of estimates whose distance to the ground truth falls under a given threshold, while MPJPE is the mean distance between 3D points after centering around the pelvis. PA-MPJPE performs additional Procrustes Alignment step first. The results are summarized in Table 3 We follow the same setup as prior work, with the same bounding boxes, cropping, and resizing of each subject. Note that no video information is used here. As opposed to other methods, our model outputs multiple detected poses given an input image; we return the detection whose IOU is highest with the provided bounding box.
On oracle-cropped inputs, we observe similar pose estimation performance to 4D Humans, particularly on Pose ToaeKrack, and much stronger performance on 3DPW. We attribute the improvement on 3DPW to the updated pseudo-labels from NLF (Sárándi & Pons-Moll 2024). Unfortunately, the quality of the pseudo-labes
from NLF is worse on close-ups of people. which affects the performance of our model on such images. This is particularly pronounced on COCO, which contains much higher percentage of close-ups than Track (which generally consists of images featuring full figures).
To assess the quality of CoMotion's 's pose estimates in more practical setting, we further evaluate it on full images (bottom rows of Tab. 3). Since this setting no longer provides an explicit signal about person's 's location and size within the image, we would expect considerable drop in performance.
Surprisingly, the effect on pose accuracy is modest. We find that despite tackling the much more complex task of detecting multiple people and estimating their poses simultaneously, CoMotion estimates poses at a similar level of accuracy to a ttate-of-the-art top-down single-person method on oracle-crops.
8

---

Published as conference paper at ICLR 2025
Table 3: Pose estimation. Normalized PCK accuracy on projected 2D keypoints at varying thresholds on the COCO and PrseTTrack datasets, alongside MPJPE of 3D keypoints on the 3DPW dataset. We highlight that our model performs similarly when provided the full image as input rather than an oracle-resized crop around target person. See text for analysis.
COCO
PoseTrack
3DW
Method
PCKn@00051
PCKn@0.11
PCKn@0.057
PCKn@0.11
MPIPE/
PA MPIPEJ
PyMAF
(Zhang et al 202 a
0.68
0.86
0.77
0.92
92.8
58.9
Laaa 20744
0.64
PARE Kocala
2021
0.72
0,91
0.79
0.93
82.0
50,9
oracle
PyMAF-X
Zhang el al
2025a
0.79
0.93
0.85
0.95
78.0
47.1
HMR 2.0a
Goel et
2023
0.79
0.95
0.97
70.0
HMR 2.0b
(Gocl
0.86
0,96
0.90
0.98
81.3
54.3
CuMotion
0.79
0.93
0.90
0.97
63.6
36.1
CoMotion
0.79
0.92
0.88
0.96
60.0
37.3
only
Input
CoMotion
4D Humans
Figure 3: We compare predictions made by CoMotion and 4D Humans unrolled through time on sample from PoseTTrack Due making independent predictions per frame, we observe that 4D Humans occasionally makes abrupt changes to the estimated pose (see green track on the right).
By default, we consider the standard CoMotion output to be the result of running both the detection and pose update modules. For completeness, we also report accuracy when using the outputs of the detection module only (last row of Tab. 3}. While the additional update step slightly improves poses, we find that the detection head is already quite accurate and can be used standalone for single image multi-person pose estimation.
Analysis. CoMotion substantially outperforms the of the on multi-person tracking, while also yielding state-of-the-ar 3D pose estimates. At the same time, CoMotion is an order of magnitude faster than 4DHumans, the strongest comparable system in the literature (Goel al. (003). Beyond the numerical results, there are qualitative differences in the behavior of CoMotion and 4DHumans. On challenging or truncated poses, 4DHumans sometimes exhibits high-frequency, abrupt changes as it jumps between possible interpretations of given track from frame frame. In contrast, CoMotion more temporally coherent due to its recurrent predictions across time (Figure
5.1 CONTROLLED EXPERIMENTS
We conduct several controlled experiments on architecture and dataset decisions. The results indicate that dropping the GRU modestly hurts performance, while removing the hidden state altogether is worse. On Pose Track. ID switches are about 15% higher with no hidden state and pose accuracy drops by couple points.

---

Published as a conference paper at ICLR 2025
We also find that the third stage of the training curriculum does not substantially affect overall pose performance but leads to a notable improvement in tracking metrics. An interesting detail here is that the baseline after stage two of training is already quite strong. In large part this is probably due the fact that occlusions that require reasoning over longer temporal windows are rare in our validation setting. That said, the reduction in ID switches after finetuning on longer sequences does suggest some meaningful change in model behavior. For tables and a more thorough review of these results we refer the reader Appendix A.2
6 CONCLUSION
CoMotion performs joint multi-person 3D pose tracking from monocular video. It operates online, processing each incoming frame in a streaming fashion. By unrolling predictions across time and allowing the network to jointly attend to all pose tracks and the full image context, CoMotion maintains accurate and temporally stable tracking and robust inference through occlusion.
There is still lot of room for CoMotion improve. One remaining failure mode is tracks that collapse together into a single identity. There are also occasional surprising identity switches where the network abruptly shifts a track a different person. It is clear that the model lacks a notion of physicality to ground tracks in realistic ways.
Many aspects of the system would benefit from scaling, including model size, training time, and input resolution. A significant impediment to progress is the lack of high-quality video training data. We observe some initial positive results from adding lebulel forms of PoseTrack and DanceTrrack to this work, but this would benefit from further investigation. In particular, this first attempt at pseudo-laheling misses out on annotations when people are in close proximity, and this is arguably an area that would be most important to supervise well for this class of model.
Another issue is the train-test discrepancy experienced by CoMotion. The model exposed to fixed short clips with a fixed number of people during training but is then run in a setting where it can be unrolled for hundreds of frames in which tracks are constantly being added and deleted. This would be nontrivial implement a proper training stack, but there would be substantial benefits to doing so. One opportunity would be to train a differentiable track management module that would replace the track management heuristics used in this paper.
Another promising direction to also model camera motion. An emerging body of work models 3D poses in consistent world coordinate frame (Shin et al. 2024 Wang al. 2024 Kocabas et al.. 2024 Yin et al. 2024). We believe that decoupling the camera from the motion of people in the scene can increase robustness to extreme camera motion. Furthermore, CoMotion does not currently reason about absolute scale, thus 3D poses are not grounded in a shared extrinsic world frame. This is most salient with children, who are not modeled well by the SMPL parameterization and are placed further into the distance as larger adult-sized humans.
CoMotion provides clean high-perfeen performing starting point for exploring these and other ideas. We will release the implementation and the trained model upon publication.
Acknowledgements. We thank Benoit Landry and Yurong You for early contributions to the project.
REFERENCES
Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2D human pose estimation: New benchmark and state of the art analysis. In CVPR, 2014.
Mykhaylo Andriluka, Umar lqbal, Eldar Insafutdinov, Leonid Pishchulin, Anton Milan, Juergen Gall, and Bernt Schiele. Prsckrack: benchmark for human pose estimation and tracking. In CVPR, 2018.
Fabien Baradel, Matthieu Armando, Salma Galaaoui, Romain Brégier, Philippe Weinzaepfel, Grégory Rogez, and Thomas Lucas. Multi-HMR: Multi-person whole-body human mesh recovery in single shot. In ECCV, 2024.
Philipp Bergmann, Tim Meinhardt, and Laura Taixe. Tracking without bells and whistles. In ICCV, 2019.
10

---

Published as a conference paper at ICLR 2025
Michael Black, Priyanka Patel, Joachim Tesch, and Jinlong Yang. BEDLAM: A synthetic dataset of bodies exhibiting detailed lifelike animated motion. In CVPR, 2023.
Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black. Keep SMPL: Automatic estimation 3D human pose and shape from single image. In ECCV, 2016.
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2D pose estimation using part affinity fields. CVPR, 2017.
Andreas Doering, Di Chen, Shanshan Zhang, Bernt Schiele, and Juergen Gall. PoscTrack21: A dataset for person search, multi-object tracking and multi-person pose tracking. In CVPR, 2022.
Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu. RMPE: Regional multi-person pose estimation. In ICCV, 2017.
Ruopeng Gao and Limin Wang. MeMOTR: Long-term memory-augmenteed transformer for multi- object tracking. In ICCV, 2023.
Rohit Girdhar, Georgia Gkioxari, Lorenzo Torresani, Manohar Paluri, and Du Tran. Detect-and-track: Efficient pose estimation videos. CVPR, 2018
Shubham Goel, Georgios Pavlakos, Jathushan Rajasegaran, Angjoo Kanazawa, and Jitendra Malik.
Humans in 4D: Reconstructing and tracking humans with transformers. In ICCV, 2023.
Shaoli Huang, Mingming Gong. and Dacheng Tao. coarse-fine network for keypoint localization. In ICCV, 2017.
Eldar Insafutdinov, Mykhaylo Andriluka, Leonid Pishchulin, Siyu Tang, Evgeny Levinkov, Bjoern Andres, and Bernt Schiele. ArtTrack: Articulated multi-person tracking in the wild. CVPR, 2017.
Angjoo Kanazawa, Michael Black, David W Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In CVPR, 2018.
Angjoo Kanazawa, Jason Y Zhang, Panna Felsen, and Jitendra Malik. Learning 3D human dynamics from video. In CVPR, 2019.
Rawal Khirodkar, Aayush Bansal, Lingni Ma, Richard Newcombe, Minh Vo, and Kris Kitani. Ego-humans: An ego-centric 3D multi-human benchmark. ICCV, 2023.
Rawal Khirodkar, Timur Bagautdinov, Julieta Martinez, Su Zhaoen, Austin James, Peter Selednik, Stuart Anderson, and Shunsuke Saito. Sapiens: Foundation for human vision models. In ECCV, 2024.
Muhammed Kocabas, Nikos Athanasiou, and Michael J Black. VIBE: Video inference for human body pose and shape estimation. In CVPR, 2020.
Muhammed Kocabas, Chun-Hao Huang, Otmar Hilliges, and Michael J Black. PARE: Part attention regressor for 3D human body estimation. In ICCV, 2021.
Muhammed Kocabas, Yuan, Pavlo Molchanov, Yunrong Guo, Michael J Black, Otmar Hilliges, Jan Kautz, and Umar lqbal. PACE: Human and camera motion estimation from in-the-wild videos. In 3DV, 2024.
Zhihao Li, Jianzhunng Liu, Zhensong Zhang, Songcen Xu, and Youliang Yan. CLIFF: Carrying location information in full frames into human pose and shape estimation. In ECCV, 2022.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona. Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
Huan Liu, Qiang Chen, Zichang Tan, diang-Janng Liu, Jian Wang, Xiangbo Su, Xiaolong Li, Kun Yao, Junyu Han, Errui Ding, et al. Group pose: simple baseline for end-to-end multi multi-person pose estimation. In ICCV, 2023.
11

---

Published as a conference paper at ICLR 2025
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. SSD: Single shot multibox detector. In ECCV, 2016.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. SMPL: A skinned multi-person linear model. TOG, 34(6), 2015.
Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. AMASS: Archive of motion capture as surface shapes. In ICCV, 2019.
Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixe, and Christoph Feichtenhofer. TrackFormer: Multi-object tracking with transformers. In CVPR, 2022.
George Papandreou, Tyler Zhu, Nori Kanazawa, Alexander Toshev, Jonathan Tompson, Chris Bregler, and Kevin Murphy. Towards accurate multi-person pose estimation in the wild. In CVPR, 2017.
Georgios Pavlakos, Luyang Zhu, Xiaowei Zhou, and Kostas Danilidis. Learning to estimate 3D human pose and shape from a single color image. In CVPR, 2018.
Jathushan Rajasegaran, Georgios Pavlakos, Angjoo Kanazawa, and Jitendra Malik, Tracking people by predicting 3DD appearance, location & pose. In CVPR, 2022.
N Dinesh Reddy, Laurent Guigues, Leonid Pishchulin, Jayan Eledath, and Srinivasa G Narasimhan. Pesse'Track, End-to-end learnable multi-person articulated 3D pose tracking. In CVPR, 2021.
István Sárándi and Gerard Pons-Moll. Neural localizer fields for continuous 3D human pose and shape estimation. NeurlPS, 2024.
Dahu Shi, Xing Wei, Liangqi Li, Ye Ren, and Wenming Tan. End-to-end multi-person pose estimation with transformers. In CVPR, 2022.
Soyong Shin, Juyong Kim, Eni Halilaj, and Michael Black. Wham: Reconstructing world- grounded humans with accurate 3D motion. In CVPR, 2024.
Peize Sun, Yi Jiang, Rufeng Zhang, Enze Xie, Jinkum Cao, Xinting Hu, Tao Kong, Zehuan Yuan, Changhu Wang, and Ping Luo. TransTrack: Multiple-ohject tracking with transformer. arXiv:2012. 15560, 2020.
Peize Sun, Jinkun Cao, Yi Jiang, Zehuan Yuan, Song Bai, Kris Kitani, and Ping Luo. DanceTrack: Multi-object tracking in uniform appearance and diverse motion. CVPR, 2022.
Yu Sun, Qian Bao, Wu Liu, Tao Mei, and Michael Black. TRACE: 5D temporal regression of avatars with dynamic cameras in 3D environments. In CVPR, 2023.
Zhenhua Tang, Zhaofan Qiu, Yanbin Hao, Richang Hong, and Ting Yao. 3D human pose estimation with spatio-temporal criss-cross attention. In CVPR, 2023.
Timo von Marcard, Roberto Henschel, Michael J Black, Bodo Rosenhahn, and Gerard Pons-Moll. Recovering accurate 3D human pose in the wild using IMUs and moving camera. In ECCV, 2018.
Yufu Wang and Kostas Daniilidis. ReFit: Recurrent fitting network for 3D human recovery. In ICCV, 2023.
Yufu Wang, Ziyun Wang, Lingjie Liu, and Kostas Daniiidii.. TRAM: Global trajectory and motion of 3D humans from in-the-wild videos. In ECCV, 2024.
Zhongdao Wang, Liang Zheng. Yixuan Liu, Yali Li, and Shengjin Wang. Towards real-time multi- object tracking. In ECCV, 2020.
Ross Wightman. Pytorch image models.
pytorch- iww-iiiaaeeaae models 2019.
Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. ConvNeXt V2: Co-designing and scaling ConvNets with masked autoencoders. In CVPR, 2023.
12

---

Published as a conference paper at ICLR 2025
Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimation and tracking. In ECCV, 2018.
Wanqi Yin, Zhongang Cai, Ruisi Wang, Fanzhou Wang, Chen Wei, Haiyi Mei, Weiye Xiao, Zhitao Yang. Qingping Sun, Atsushi Yamashita, et al. WHAC: World-grounded humans and cameras. In ECCV, 2024.
Yingxuan You, Hong Liu, Ti Wang, Wenhao Li, Runwei Ding, and Xia Li. Co-evolution of pose and mesh for 3D human body estimation from video. In ICCV, 2023.
Fangao Zeng, Bin Dong. Yuang Zhang, Tiancai Wang, Xiangyu Zhang, and Yichen Wei. MOTR: End-to-end muleip tracking with transformer. In ECCV, 2022.
Hongwen Zhang, Yating Tian, Xinchi Zhou, Wanli Ouyang, Yebin Liu, Limin Wang, and Zhenan Sun. PyMAF: 3D human pose and shape regression with pyramidal mesh alignment feedback loop. In ICCV, 202
Hongwen Zhang, Yating Tian, Yuxiang Zhang, Mengcheng Li, Liang An, Zhenan Sun, and Yebin Liu. PyMAF-X: Towards well-aligned full-body model regression from monocular images. 7PAMI, 45 (10), 2023a.
Jinlu Zhang, Zhigang Tu, Jianyu Yang. Yujin Chen, and Junsong Yuan. MixSTE: Seq2seq mixed ppatio-temporal encoder for 3D human pose estimation in video. In CVPR, 2022.
Juze Zhang, Ye Shi, Yuexin Ma, Lan Xu, Jingyi Yu, and Jingya Wang. IKOL: Inverse kinematics optimization layer for 3D human pose and shape estimation via Gauss-Newton differentiation. In AAAL, 2023b.
Siwei Zhang, Bharat Lal Bhatnagar, Yuanlu Xu, Alexander Winkler, Petr Kadlecek, Siyu Tang, and Federica Bogo. Rohm: Robust human motion reconstruction via diffusion. In CVPR, 2024.
Yi Zhang, Pengliang Ji, Angtian Wang, Jieru Mei, Adam Kortylewski, and Alan Yuille. 3D-aware neural body fitting for occlusion robust 3D human pose estimation. ICCV, 2023c.
Yifu Zhang, Chunyu Wang, Xinggang Wang. Wenjun Zeng, and Wenyu Liu. FairMOT: On the fairness of detection and re-identification multiple object tracking. IJCV, 129(11), 2021b.
Qitao Zhao, Ce Zheng, Mengyuan Liu, Pichao Wang, and Chen Chen. PoseFormerV2: Exploring frequency domain for efficient and robust 3D human pose estimation. In CVPR, 2023.
Ce Zheng. Sijie Zhu, Matias Mendieta, Taojiannan Yang. Chen Chen, and Zhengming Ding. 3D human pose estimation with spatial and temporal transformers. In ICCV, 2021.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 TRACKING EVALUATION DETAILS
Impur
Herves
Prediction
Figure 4: Incorrect handling of missing annotations in Prse Due to incomplete annota- tions in PoseTrack18, 88, predicted tracks may be incorrectly regarded "false positives". We show representative samples where annotations are green and "false positives" are red.
Figure 5: Incorrect handling of missing annotations in PoseTrack 21. PoseTrack21 addresses the incompleteness of Pose Track 18 annotations by providing 'ignore' regions to accompany the annotated tracks. For the frame on the left, the center image illustrates the annotation of the person in the center (shown in green) and polygon defining the 'ignore' region in blue. The right image shows predicted tracks red, which are still penalized as false positives by the Pose'rrac221 evaluation code despite being contained in the 'ignore region'. This is a bug that fix.
Tracking evaluation metrics, such as MOTA, are sensitive to false positives. A MOTA score lies in (-oo, 100], where 100 is perfect score; we observe on the scene illustrated in Figure if false positives are not ignored, our method garners a MOTA of around -1500.
Figure 4 visualizes the problematic handling of missing annotations in PoseTrack1 Due missing annotations, correctly predicted tracks are labeled as false positives, which may produce misleading results.
To address this problem, Doering et al. (2022) introduced PoseTrack21 with more comprehensive annotations. 1 provides keypoints for many more people and marks regions with missing annotations through "ignore regions".. Unfortunately, the provided evaluation code contains a bug that still allows for incorrect handling missing annotations. Figure visualizes the behavior PoseTTrckk? evaluation. Akin to PoseTrackk 8, predicted boxes in the ignore region are treated as false positives.
The issue has to do with the calculation determining whether not to ignore predicted box. The evaluation code looks at the IOU between the ignore region and the box and discards any box with an IOU greater than 0.1. Unfortunately, when the ignore region is particularly large, most boxes do not intersect a large percentage of the region's 's area. The IOUs in the example above vary between 0.01 and 0.07 not enough to be discarded.
We make a one-line change to the Pose 1rscTrack21 evaluation code for our "fixed" evaluation: instead of IOU, we calculate the percentage of the bounding box that overlaps with the ignore region. This
14

---

Published as a conference paper at ICLR 2025
Table 4: Hidden state ablation. From our full system (first row) we remove either the GRU (second row) or the hidden state altogether (third row).
Track
EgoHumans
3DPW (val)
Hidden
GRU
2D@0.05f
MOTA
IDF1
IDs
3D@0.11
MPJPEL
3D@0.11
MPJPEL
v
74,9
70.9
78.6
491
77.4
83.8
90.4
61.3
73.0
69.8
76.9
506
76.3
83.8
90.5
62.0
72.6
70.1
77.2
559
72.7
89.2
90.4
61.2
results in the expected evaluation behavior on scenes with ignore regions. We adjust the discard threshold from 0.1 to 0.3, so this means if 30% of bounding box overlaps into an ignore region will be discarded.
There a few minor quirks that still remain in the annotations and evaluation. In some clips, ignore regions appear and disappear, and occasionally completely overlap with ground-truth annotations. We observe in the evaluation code by the PoseTTrack21 authors that there is mechanism to recover predictions that are matched to ground- truth tracks that have been filtered out by the ignore region. Perhaps this is to address the samples that have been both marked as ground-truth boxes and covered in ignore regions. Overall, these present modest amount of corner cases and do not adversely affect the evaluation the same way as the ignore region IOU calculation.
We would like to acknowledge that benchmarking and annotation work are seriously undervalued by the community and we appreciate the incredible work that went into producing both versions of PoseTrack The above notes are just to provide clarity what goes into calculating these metrics and the subtle ways in which unexpected and undesired behavior can slip through.
A.2 CONTROLLED EXPERIMENTS
We perform additional experiments to investigate the impact of major design decisions on the final performance of the proposed approach. We report results the PoseTrack validation set, on the 3DPW validation set, and on a subset of clips from EgoHumans (Khrrodkar et al. 2023).
With EgoHumans, we do not follow official evaluation setting but create our evaluation by sampling some of the exocentric camera clips from the Lego, Tagging, and Fencing scenes. We find EgoHumans interesting for multiple people. Critically, there are annotations through occlusion because data was collected in multi-view setting. Most other evaluation datasets do not have ground truth for occluded and truncated samples.
of the few datasets with 3D annotations and complete annotations
All evaluation done across video in the multi-person setting with full images resized and padded to 512x512. 12. For the purpose of this evaluation we do not manage tracks. On the first frame, we choose detections that match closest the corresponding ground truth annotations and then unroll the update step without any further modifications to the set of tracks. The evaluation assumes that each unrolled track will match the annotations corresponding to the person assigned from the first frame. On Pose Track, we unroll 16 frames, and on 3DPW and EgoHumans we unroll 48 frames.
One note, for these ablations, the 2D PCK @ 0.05 on PoseTrack calculated differently from the results using the HMR evaluation code reported in Tables The normalization factor is more strict and thus not directly comparable. We use this metric only in the context of ablations comparing different versions of our model.
Hidden state. We assess the importance of the recurrent hidden features through an ablation study. We report detailed metrics in Table The first row represents full approach, the second row drops the GRU but preserves a learned hidden state that is passed through time, and the third row drops the hidden state altogether. Performance is slightly worse without the GRU, and even worse without any hidden state at all.
Without a hidden state, the only information passed across consecutive frames the previous SMPL poses. For the vast majority tracking situations this is sufficient to know which pose corresponds to which person, but can lead failures in the face of occlusions and similar poses in similar positions.
15

---

Published as a conference paper at ICLR 2025
Table 5: Ablation study on the impact of Pose Track and DanceTrack.
PoseTrack
EgoHumams
3DPW (val)
PoscTrack
DanceTrrac
2D@00.01
MOTAT
IDFIt
IDs]
3D@0.11
MPIPE,
3D@0.1f
MPJPE]
68.4
75.6
576
76.5
84.2
90.0
72.4
69.8
75.6
703
77.5
84.7
90.6
74.9
70.9
78.6
491
77.4
83.8
90.4
61.3
Table 6: Impact of additional training stage on longer sequences.
PoseTrrack
EgoHumans
3DPW (val)
Stage
Stage 3
2D@0.05f
MOTAT
IDFI
IDs]
3D@0.17
MPIPE,
3D@0.11
MPJPEJ
74.9
70.9
78.6
491
77.4
83.8
90.4
61.3
74.6
71.4
382
77.9
90.1
61.9
We observe that without a hidden state, overall pose accuracy on videos from EgoHumans and PoseTrack drops, accompanied by notable decrease in tracking performance. Removing the hidden features increases ID) switches by almost 15%. In contrast, 3DPW performance is largely unchanged which makes sense as the videos in the validation set consist mostly of one person, sometimes two, so tricky tracking situations do not arise in this context.
Pscudolabeled video datasets. We report results Table 5 after removing either PoseTrack or DanceTTrach our two real world datasets that were pseudolabeled for video training to see the role, any, they have on performance. These results are calculated after the stage training on 8 frame sequences, not the additional longer sequence finetuning. We observe modest effect on simpler scenes like 3DPW and EgoHumans that only consist of up to four people with static cameras. A more measurable drop in performance occurs Pose'Track if either dataset is excluded, particularly in the tracking results (note the number of ID switches). PoseTrrack itself seems to have a larger impact which would make sense is in domain for its own validation set while PanceTrack consists of fairly different video distribution.
We examine how much performance improves with additional training stage longer sequences. After stage 2 the model has only ever been trained on clips up to 8 frames in length. In some ways, is impressive that does well it does having never been exposed to sequences longer than quarter of second. The pose accuracy does not change much with additional finetuning. but there are notable gains in tracking performance (Table Cp.
Curriculum.
CoMotion Accuracy vs Assumed Pinhole "fx"
03500
PCKn @ 0.05 wrists
0.6
0.4
Seen During Training
80
0.2
0.0
10
10 10
10
10'
10
10"
Figure 6: Impact of the assumed focal length. To investigate how the focal length of the intrinsics model on PoseTrack videos (for which do not have ground
matrix affects performance truth camera calibration) and report 2D PCK accuracy. We adjust the assumed focal length and observe that the network's 2D keypoint accuracy is consistent long as remain within realm of values which correspond to what one would typically find with most camera bardware apart from extremely wide-angle options such as a fish-eye lens. In the above figure, fx (the x-axis) normalized by the image width.
16

---

Published as a conference paper at ICLR 2025
Sensitivity to provided camera intrinsies. To assess the sensitivity of model the input camera focal length, we evaluated CoMotion on Pose Track videos (for which we do not have ground truth camera calibration) at various focal lengths. Figure visualizes the results. We find that the network's 2D keypoint accuracy remains consistently high across a wide range inputs.
O
A.2.1 RUNTIME
We measure the runtimes of CoMotion and baselines on the PoseTTrack21 validation and report results in Figure 7 All measurements were made the same hardware using the code provided by the authors each method. We measure the time to run the complete tracking stack unrolled across all PoseTrack validation videos. We find that CoMotion significantly outperforms prior work. Specifically, CoMotion is approximately 1.4x faster than PARE (176ms vs 258ms) and 12x faster than 4D Humans (I (176ms vs 2163116666 2163ms) average.
3000
2500
2000
Millsccoods srtsscoodd--er-ffann ces-per-framm frame
1500
1000
500
CoMotion (ours)
PARE
PyMAF
PyMAF-X
4D Humans
Figure Comparing the fer-frame runtime CoMotion with prior work on the PoseTrack21 validation set. All measurements authors. CoMotion is significantly faster than prior work. Specifically. CoMotion is approximately 1. 1.4x faster than PARE (176ms 258ms) and 12x faster than 4D Humans (176ms 2163ms) average.
made V100 GPU using the code released by the respective
17

---

Published as a conference paper at ICLR 2025
def get image i features (image)
$ Run ConvNextV2 stages
X convnext. (image)
feat_pyramid []
for stage convnext.stage
x sx)ge(x)
# Linearly project image features from each stage feat_pyramid. (ccnv(x))
Upsample low-res features (using einops notation)
upsample_pattter = .... (c h2 w2) h и -> ... c (h h2) (W w2) feat_pyramid rearrange (upsample_pattemn, h2-2, w2-2) feat_pyramid[3] rearrange (upsample_patterm, h2-4, w2-4)
Fuse into single tenser at 1/8th input resolution image_feature features = sum (wwattffeeaattttte features Layer_nemm (image featuress___
return image_feature
Figure 8: Pseudocode for the image encoder.
A.3 ADDITIONAL МETHOD DETAILS
A.3.1 IMAGE ENCODER
We use the ConyNextV2 WWoo et al. 2023 implementation provided in the timm library (Wightman) 2019). As with most convolutional backbones, features are extracted in consecutive stages to progressively lower resolutions. There are four stages yielding features at 1/4th, 1/8th, 1/16th, and 1/32nd of the input image resolution. As simple strategy fuse early and late features, we linearly project the outputs from each stage to a feature tensor at 1/8th the input resolution. For example, 2x2 convolution with stride 2 used on the highest resolution features, while a 1x convolution is applied to the lower resolution feature maps with an output of either 4 or 16 times the number of feature channels which can be reshaped into an upsampled tensor. The resulting features are added together to produce a feature tensor Fe for subsequent stages. We refer to Figure for PyTorch-like pseudocode outlining these steps.
We find performance improves if we add some signal about intrinsics to the features Ft. We apply rotary positional embeddings to fraction of the channels to encode the u and v values in pixel space and the corresponding and y values in world coordinate space. This yields modest bump in keypoint accuracy.
A.3.2 DETECTION STEP
Architecture. We perform detection using a single-shot detection approach (Liu et al. 2016). We depict the architecture in Fig. 9 Given the output features 1e from the image encoder, we define a detection head that applies several consecutive downsampling and ConvNeXt blocks (Woo et al.) 2023) to yield a low-resolution feature pyramid. At each feature scale we apply 1x1 convolutions to decode candidate detections. The model produces multiple candidate poses at each spatial location. The number of poses per location a hyperparameter, and we found 4 to provide good balance of performance and compute overhead.
The model outputs over thousand candidate detections for an input $ 512x512 image before non- maximum suppression. Specifically, the feature pyramid is pooled down 32x, 64x. and 128x from the input image resolution, so for 511x5122 input image, output detections are produced on 16,16, 8x8, and 4x4 grids. With 4 output poses per position in each grid, the total pool of candidate detections is 1,344 for given image.
Output representation. For each detection, the network produces SMPL parameters (translation, root orientation, body pose, betas) as well as a confidence term. For the body pose, rather than produce joint angles directly, the network predicts latent embedding that is passed to separate
18

---

Published as a conference paper at ICLR 2025
Image Shoode
Colection Hoad
Cendoo
Recrape
F-r cire
Suges
and
Now May SApSPERR0T
Rertapa
Phoragg
Figure 9: Architecture of the image encoder and detection step. We visualize the core modules involved in producing encoded image features and per-frame detections. The image encoder produces a multi-resolution feature pyramid using the intermediate activations from ConvNeXtV2, ranging from 3/8 to 3/61 the input resolution. The features' channel dimensions are then expanded with 1x1 convolutions such that they can all be reshaped to ğ resolution and summed. The image encoder also applies positional encodings derived from the uncalibrated pixel coordinates. Next, the detection head produces a large set of candidate SMPL parameters and associated confidences, derived from feature maps at three spatial resolutions (g) 16 ). We apply non non-max maximum suppression return the final set of detections.
MLP decoder that maps to the output axis-angle pose representation. This decoder is shared in the update step as well.
We take measures to ensure the model's predictions project to the image correctly given the provided input intrinsics matrix K. An output on the grid at row and column j is placed at corresponding base pixel position (u,, v;), the model predicts and offsets in pixel space (du, du). The depth is predicted in log-space (Etog) and is scaled by the focal length fo and a hyperparameter 2de fault- We lower the default depth at each level of the pyramid that at coarser spatial resolutions, candidates are initialized closer to the camera. The SMPI. translation term is then computed as follows:
ú + du
(1)
V = Uú +
du
(2)
de fault
* fa exp (Elog)
fz
Cu
u
7
0
fg
Cy
0
In summary, we have the following network outputs:
y (translation):
Zoog and pixel x, y offsets (du, do) applied relative the output pixel position. Together these terms are used to derive which is the translation camera coordinate frame.
0 (pose): The SMPL pose angles are the concatenation of the 3DOF root orientation, and body pose joint angles which are decoded from an MLP applied the output 256 dimensional pose embedding from the detection head.
3 (shape): 10-dimensional SMPL betas 3 that determine body shape and proportions.
confidence: Output confidence of the detection.
If particular detection is chosen to serve new track instance we define a track state X (7, 0, B, h) from these outputs. The value à refers to an additional latent hidden state for the track, this is not predicted by the detection head and instead is initialized with a vector of all zeros.
Supervision. is is nontrivial to define a predetermined strategy to assign ground truth annotations to particular candidate output at specific spatial location. Instead, we flatten and concatenate all candidate detections and assign them to the ground truth detections with Hungarian matching. We compute the OKS between the 2D projected keypoints of detections and ground truth annotations and adjust the score by the confidence of the detection so that confident detections are more likely to be matched and have loss applied to them.
19

---

Published as a conference paper at ICLR 2025
def cross Image key and value shape
eross_attent attention (image_ key, image_value, tokens)
B: batch_size, H W: height x width, C: feature_dim
Input tokens shape
B: batch_size, N: num_pe M: num nun_tokens, C: feature_din
# Perform cross attention union of all tokens for all people token_to_querry (tokens)
9. rearrang n m b (n m) e")
px pe__ebab = F.scaled_dot___ ptt.ott_rrducctttttettttee
9, image_key, image_value
px se_feeeback.. reeebbac ("b (n e tokens tokens + post attenticn mip (px feedback)
c*)
# Additional transformer with attention across people tokens. rearrange tokens cros_pople_ (tokens)
(b m) n c*)
Additional transformer with attention per-person tokens. rearrange (o m) n -> (b n) c*)
tokens per_person_attention attenticnnntttee (tckens))
tokens. rearrange (p n) m b -omme
return tckens
def update (K, image_features, smpl_params, hidden)
Encode tokens
pred 2d get 2d projection (K, (mplmpsraas)
tokens encode 2d to toketss (pred 2d)
tokens encode
tckens (smpl params)
tokens eacoe__hidden_tottekeee (hidden)
Prepare image keys and values image_tokens cat feature, positional_encodingl,, 1)) image_tckens rearrange (*b (*b h b (h w) c*)
image_key innea (image_tcken) image_ value = linear lamaag__ccknnn
Apply cross-attention
tokens cross_attention saaaee_key, image_value, tokens)
tokens cross_attention (image_keg) key, image_value, tokens)
Decode outputs
hidden gru gru_updated (tokens, hidden)
smpl_params decode _ecode_update www.te (takens, smp1 params, hidden)
return smpl_params, hidden
Figure 10: Pseudocode for the update step.
For the matched detections we apply losses to the 2D reprojection of their 2D keypoints as well as SMPL losses on the joint angles, root-centered 3D keypoints, and in the case of BEDLAM to the ground-trutt -tund betas. We also apply a binary cross entropy loss to the confidence term, using the matched samples as positives and a random subset of the unmatched samples negatives. At test time, we do standard non-man non-maximum suppression using OKS instead bounding box IOU to get the final set of detections.
20

---

Published as a conference paper at ICLR 2025
A.3.3 UPDATE STEP
The update step attends to the latest input image to provide new poses for all tracked people in the scene. To perform the update, first we encode the previous state X into a set of tokens RRNx M where N the number of tracked people, M the number of tokens used to represent an individual track, and D is the feature dimension. We can set M to any value to control the memory and compute overhead of the update step, and for all experiments we set M = 24, D 512. To get a, we take the current SMPI. parameters and compute 2D projected keypoint locations, then pass the raw SMPI. parameters, the 2D keypoints, and the current hidden state for that person through an MLP to produce M tokens, batching this operation and concatenating to produce the full N M set.
We then attend to image tokens which are produced by flattening the image features F and concate- nating them with sinusoidal positional embeddings. We apply linear layers to get per-pixel key and value embeddings which are attended to with queries from the per per-person tokens z. The output of the cross-attention operation gets added residually to z, and we apply additional transformer layers for further processing. We then pool across the M tokens for each person and apply an MLP to predict the new output poses and hidden states. This entire process outlined in pseudo-code provided in Figure 10
The goal of the above steps is to have flexible way to operate on an unordered discrete set of tracks and efficiently attend to dense per-pixel image information without explicitly defining what information might be relevant for which tracks. We use standard attention as the building block upon which we exchange information between image features and the tracks themselves yielding a clean and straightforward update to all poses.
Output representation. The output follows the same output pattern of the detection head. The breakdown of outputs is follows:
y (translation): We output ( (Ztog: du, de) and follow the same calculation for y as in the detection stage (Sec. A.3.2 In this case, the pixel offsets (du, dv) are added to the projected
location of T/-1-
9 (pose): The network predicts a new root orientation and pose embedding that is passed through the same MLP decoder as in the detection step. If we instead ask the network to learn residual updates to either the root orientation or pose embedding, the network struggles when transitioning between rapid rotations or rare and seldom seen poses.
3 (shape): We do not update the SMPL betas at all. When allowed to do so, the network would sometimes opt to make person larger smaller rather than physically moving them towards or away from the camera. Results were more sensible and stable with a fixed body shape across time.
A.3.4 MODIFIED OKS
At various stages of our pipeline, we need to measure the similarity between two poses (e.g.. matching detections to ground truth annotations, matching detections to existing tracks). We perform these comparisons on their projected 2D keypoint locations. We choose 2D keypoints rather than SMPL pose parameters or other 3D cues for compatibility with datasets where only 2D keypoints are available. We follow the COCO OKS ("Object Keypoint Similarity") calculation found at #keypoints
cocodat aset. www.et.orgg/ee wwtpoooggaaeee.oooo http://atteeet/eeoooiitt -evan which returns a normalized value from to even with variable number of valid keypoints:
Σ. U. exp(-df/25"p)
OKS
In the above equation d, is the Euclidean distance between the pair of keypoints corresponding to the vth body part, while v; is binary value indicating whether that part is valid. The standard deviation of the Gaussian is defined by the object scale s and K4. predefined constant to adjust the deviation per-keypoint, We make some slight modifications the above calculation to better suit our needs:
E (1 1.-2//22 d/220k22-
OKS oors
(6)
Σ, V/
21

---

Published as a conference paper at ICLR 2025
Figure 11: Comparison of pseudolabeled annotations. Left: input image. Middle: original pseudolabels provided by Goel et al. (2023). Right: the new pseudolabels produced by running NLF (Sárándi & Pons- Pons-Moll, 2024). We observe better head and foot correspondence with the new outputs (top) and more comprehensive annotations across people in the scene (bottom).
We use single constant K instead of per-keypoint values, and replace the Gaussian with a Cauchy distribution which has heavier tail. This is important we often compare points that are far from each other and need nonzero signal at those larger distances. Finally, another important detail is that we do not have ground-truth scale & as would be used in the COCO evaluation, so we calculate scale term the fly based the projected 2D bone lengths of each set of keypoints:
[p: [P:-Ppll Pyll 4i
U2 U1
(.,))L
Limbs are defined by reference of pairs of adjacent keypoints and default limb lengths calculate the distance between the two endpoints of the limb p; and Pj and compare this to the reference distance lyy yielding ratio of the size of the current projected limb compared our baseline size. We take the max over all ratios to account for any limbs that are foreshortened. The largest value typically correlates well with the perceived size of the person in the image. While only a rough approximation, this signal provides more reliable indicator of the size of person opposed to alternatives based on bounding box size. When comparing two sets of keypoints we will use the larger of the two scales, in the case where are comparing to ground-truth annotation use the scale of the annotation.
A.4 PSEUDOLABELED DATA
Training on large-scale in-the-wild pseudolabeled data is crucial for achieving accurate 3D pose estimates on the most challenging poses (Goel al. 2023). But pseudolabeling imperfect. For example, many of the samples in the data made available by the authors of 4D Humans has odd behavior when people are truncated which includes raised hands and feet at the image boundaries. Models trained on this data inherit these quirks. More importantly, we use this data for training our model perform detection, but the pseudolabeled data is quite sparse. our model would learn to selectively ignore people in the scene which proved difficult overcome with adjustments to losses and other training decisions.
A recent strong 3D pose model was released referred to NLF (Neural Localizer Fields) (áárándi PonssNoe Moll 2024) which offered an opportunity to produce new pseudolabels. We ran this model off-the-shelf the full set of Insta Variety, COCO, and MPII images. With strong detector, the images are much more comprehensively annotated than before and the quality of poses improved. In particular, there is better alignment of the head and feet the image (see Figure 1T).
22

---

Published as a conference paper at ICLR 2025
We also investigated the use of pseudolabels two video datasets: PoseTrack and DanceTrack. We leverage the existing track annotations provided by these datasets, and run the model conditioned on the provided dataset bounding boxes. This offers some of the only in-the- challenging multi- person 3D video data. Unfortunately there is one issue which difficult to overcome, which that bounding boxes poor conditioning signals when two people in close proximity. The output of the model by Sáránnd & Pons-Moll (2024) will occasionally be heavily distorted or more typically, will be collapsed to a single person. This is one of the areas where it is most important get the supervision right when training CoMotion. Pose comes with 2D annotations that allowed us to catch some of these situations automatically, and we further develop heuristic for flagging these cases and setting the annotations invalid on DanceTrack.
An additional failure mode of the NLF model is that we occasionally run into odd outputs with truncated people. Fortunately, the model output includes reasonably well calibrated uncertainty terms which we use to automatically filter out bad samples. Even so, our system does inherit bit of this behavior, so we find that after training, the results of CoMotion close ups and truncated figures does not align quite as well as figures that are fully in frame.
Another motivation for pseudolabeling all of the real image and video datasets with this model was to provide a consistent output annotation format. We experimented with training on mixed annotation formats, but often into issues as the keypoints used to annotate 2D datasets like COCO and Pose Track do not align perfectly with the corresponding SMPL. keypoints. The differences can be subtle, but the mismatch in positioning of the hips and shoulders can lead to the network fitting poses in odd ways in order to minimize the 2D reprojection loss used during training.
This does have effect when comes benchmarking across these datasets. The model trained on this new data achieves a significant improvement in 3DPW numbers (Table 7 Looking qualitatively the outputs of models trained on these respective datasets (Figure 12). the result after training on the original pseudolabels are not bad, but the NLF-based pscudolabels align much better to the expectations of the 3DPW annotations.
Table 7: Pseudolabeled data ablation. Comparing 3DPW train performance between model trained with the original pseudolabels from Goel et al. (202) and on the new pseudolabels from Sárándi & Pons-Moll (2024).
3D 101
MPJPEЧ
Original pseudolabels
76.8
73.8
New pseudolabels
90.4
54.9
To summarize, we identified several key areas that motivated the switch to new pseudolabels:
The original pseudolabels are sparse, often only accounting for one or two people in a crowded image which adversely affected detection performance.
The new pseudolabels better match people's 's heads and feet.
The new pseudolabels improve performance on rare and very difficult poses.
The new pseudolabels allow us to train on both image and video datasets with a uniform output format and not worry about mismatches in annotation styles.
As trade-off we unfortunately regressed on closer shots of people, but this is something that could be addressed easily in the future with better data and is not fundamental limitation of our method.
23

---

Published as a conference paper at ICLR 2025
Figure 12: Comparison of model predictions when trained with differing pseudolabels. Left: input image. Middle: detections trained on original pseudolabels provided by Goel et al. (2023). Right: detections trained on new pseudolabels produced by running NLF (Sárándi & Pons-Moll 2024). The effects are subtle, but the body fitting is improved and when tested in-the-wild we notice better accuracy on extreme poses like those seen in breakdancing and gymnastics clips. The one notable regression is on closeups, illustrated in the bottom row.
24


Reviews:
[{"review_id": "CTmpocBxGY", "reviewer": "Reviewer_ZfVY", "summary": "This paper addresses a challenging problem of multi-person 3D pose tracking. Compared to tracking-by-detection pipelines, this paper extends the tracking-by-attention pipelines from Multi-Object Tracking to Multi-person pose tracking, which detect new humans and update existing tracks by learned tokens. Due to the lack of complete training data, this paper explores the mixed datasets with different forms of supervision. The experimental results demonstrate superior performance over other state-of-the-art methods on several benchmarks.", "questions": "I really appreciate authors' hard work, which gives fast and accurate results on the benchmarks. However, I am doubtful of this paper's novelty and insight and I give borderline reject accordingly.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper considers a new pipeline to address multi-person video-based tracking and pose estimation. \n2. The paper is well-written and easy to understand. The qualitative results clearly show the advance of the proposed method. \n3. This method is very fast compared to other methods and achieve good results in several benchmarks.", "weaknesses": "1. This paper is more like a technical report rather than research paper. The contributions are hard to recognise and not clearly highlighted. This makes this paper less insightful. \n2. To make the contributions clearer, i would recommend giving some preliminaries about the tracking-by-attention pipelines applied in other applications, e.g., MeMOTR, to distinguish what’s new in your method and what has been done.\n3. One important claim in your paper in CoMotion can utilise subtle pixel cues in Line52-53, which is not elaborated is supported in the following text and experiments.\n4. The compared methods in tab.1 seems not very recent, what’s the comparison between CoMotion and more advanced MOT methods?\n5. Some minor issues: \n- Metrics can be introduced. If the space is not enough, you can include them in the appendix.\n- I would recommend give short descriptions about ablation studies instead of checking them directly in appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hqxp2H4DED", "reviewer": "Reviewer_MAjj", "summary": "This paper addresses the problem of 3D human pose and shape estimation and cross-frame tracking. The proposed method estimates 3D poses for all the persons in a frame and then updates the tracking poses and bounding boxes. A novel pose update module with GRU is introduced to update the tracking results and store the hidden information not present in the SMPL parameters. The proposed method is benchmarked on both bounding box tracking and pose estimation tasks and demonstrated better performance than the previous method. The comparisons on per-frame runtime also demonstrated its much more efficient design compared to the previous method.", "questions": "1. The videos show unstable pose estimation for static humans, which should be addressed better in the pose tracking method than in the single-frame human pose estimation method. What is the cause of such instability, as the model has information about previous estimations and a learned hidden state? \n2. The authors mentioned using a 'strict' threshold in evaluating pose track metrics. Could the authors elaborate more on the choice of threshold and its relationship with the tracking method's precision and recalls? \n3. Will the proposed method be open-sourced? There is only limited information on the model architecture and hyper-parameters. It will be hard to reproduce the proposed method if the source code and checkpoint are not available. \n4. The proposed method has worse results compared to CLIFF in Table 3. What's the possible cause of it? \n5. Dense object tracking has been much improved by papers like CoTracker. How will the proposed method perform when compared to the method Cotracker? It will be interesting to see the performance difference between a 3D-aware object-specific method performed compared to a generic method.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method estimates the tracking and pose simultaneously, which could take advantage of prior information on humans.\n2. The design choice of hidden state and GRU is sound. Keeping tracking alive seems to be a valid approach to address the missing people in a shorter period. \n3. The evaluation results on PoseTrack benchmarks are impressive, validating the proposed method's design choice.", "weaknesses": "Overall, the paper is of good quality, but there are still some weaknesses that the reviewer would like to discuss here and in the following questions. \n1. The implementation details and model architecture are unclear. The model architecture of both the pose estimation module and the pose update module is unclear. It's also unclear how the detection of bounding boxes is achieved. \n2. Although the hidden state and GRU are sound approaches, the experiments do not rigorously evaluate their effectiveness. \n3. The proposed model takes advantage of multiple training datasets, although it's not uncommon in the community of human pose estimation, but part of the improvements in the tracking results might be attributed to this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v0E8LKX3Lv", "reviewer": "Reviewer_VXUb", "summary": "CoMotion introduces a complex system for multiple person tracking from a monocular camera, based on the tracking-by-attention paradigm. To overcome the challenge of limited data for pose estimation the authors train the model in multiple stages and add various datasets to gradually teach the model to track 3D persons across time.", "questions": "L318: what is the “off-the-shelf MAE”?\n\nI wonder if the authors have thought about utilizing a generative model as pose prior, ie. Following ScoreHMR. The system could then be understood as a Recursive Bayesian Filter, which would provide a nice mathematical foundation for the tracking algorithm.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The method solve a highly relevant problem, tracking multiple people from a monocular camera. The method performs well on tracking and pose estimation benchmarks and the authors identified and fixed a bug in existing benchmarks, which is very helpful to the research community.", "weaknesses": "The method makes use of various datasets and human pose modalities, such as SMPL, pseudo-SMPL and COCO. Could the authors describe how they handle different pose skeleton modalities, i.e. how do they train their SMPL pose estimation model with COCO skeleton supervision?\n\nLL292-299: it is interesting that the authors do not add tracks during training: have the authors conducted an analysis of the training data to determine how many tracks appear / disappear on average? I would assume that this could potentially be very detrimental to the learning, as the model might learn to “ignore” certain people (i.e. if they were occluded before, or if they are at the edge of the image). Have the authors done any analysis of those kind of edge cases?\n\nWhat do the authors mean by “[we] do not expect the shape of the mesh to correspond to the person’s physical body” (LL207-208)? The shape does not just model the mesh surface but also influences the bone lengths, so the skeleton is affected by it as well. Could the authors clarify this?\n\nWhat do the authors mean by “the intrinsic matrix [does not] necessarily reflects the actual calibration of the camera“? Especially, as the authors claim in the next sentence that their method is robust to any provided intrinsic camera! Such a claim should be ablated, i.e. compare the system predictions with K provided and without K provided.\n\nCan the authors elaborate on what they mean by “strict” in Table 2?\n\nFigure 2 is not self-containing as it does not describe all elements of the image, i.e.e method part (A) is missing.\n\nTypo in L205-206: the parameters $t$, $\\beta$ and $\\theta$ are missing from the SMPL parameter discretion. \n\nLL324-325: the authors should provide more details on how they apply the regularization for the beta optimization.\n\n— suggestions — (will not influence my rating)\n\nThe authors could utilize the CHI3D [a] as an additional benchmark as it comes with well-tracked 3D poses (SMPL-X) at higher quality than 3DPW and with more challenging human-human interactions.\n\n\n[a] Fieraru, Mihai, et al. \"Three-dimensional reconstruction of human interactions.\" CVPR 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zIZcei1PqA", "reviewer": "Reviewer_5TGo", "summary": "This paper describes a method to simultaneously track and regress coherent 3D human motion from videos with occlusion scenes. The key innovation is to update the 3D pose of multiple tracking targets from sequential images in a frame-by-frame manner. The baseline method for  comparison is 4DHuman, which tracks people via artificially associating the 3D human pose, location & appearance based on pre-defined matching rules. While the proposed method follows a tracking-by-attention way, which detects objects and updates existing tracklets via cross-attention between image and tracklet tokens. Query tokens of existing tracklets are firstly updated with current image tokens. The updated 3D human poses are matched with 2D bouding box detection of current frame. The unmatched detection will be initialized with new tracks and get updated to generate new 3D pose. Evalutation results on PoseTrack21 testified the effectiveness of the prospoed method. Compared with 4DHumans, the MOTA, IDF1, and IDR get significantly improved.", "questions": "See weakness part.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Investigating the tracking-by-attension way of mutl-person 3D human motion tracking in crowded scenes is very challenging and interesting. The results of this paper is quite encouraging and exciting.   \n2. Updating the tracklet state with assistance of bouding box detection is reasonable. Design of 3D human pose update module is great and seems to be very effective. Subtle 3-stage training process is brilliant.  \n3. The experiment details are sincerely and clearly explained.", "weaknesses": "1. The tracking experiment is mostly performed on PoseTrack dataset only. Maybe some challenging occlusion scenes in 3D benchmarks, such as MuPoTS-3D and 3DPW could testify the performance of the proposed method in a more comprehensive way.\n\n2. The way of showing qualitative videos is inconvenient for reader to watch. One video with more attractive demos / comparisons would help a lot.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)