CoMotion: Concurrent Multi-person 3D Motion

Authors: Alejandro Newell, Peiyun Hu, Lahav Lipson, Stephan Richter, Vladlen Koltun
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: human pose estimation, 3d human pose, tracking

Abstract:
We introduce an approach for detecting and tracking detailed 3D poses of multiple people from a single monocular camera stream. Our system maintains temporally coherent predictions in crowded scenes filled with difficult poses and occlusions. Our model performs both strong per-frame detection and a learned pose update to track people from frame to frame. Rather than match detections across time, poses are updated directly from a new input image, which enables online tracking through occlusion. We train on numerous image and video datasets leveraging pseudo-labeled annotations to produce a model that matches state-of-the-art systems in 3D pose estimation accuracy while being faster and more accurate in tracking multiple people through time.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730653366645

thread_type:
official_review

<thread_object>
creator: reviewer_5tgo

<content>
summary:
This paper describes a method to simultaneously track and regress coherent 3D human motion from videos with occlusion scenes. The key innovation is to update the 3D pose of multiple tracking targets from sequential images in a frame-by-frame manner. The baseline method for  comparison is 4DHuman, which tracks people via artificially associating the 3D human pose, location & appearance based on pre-defined matching rules. While the proposed method follows a tracking-by-attention way, which detects objects and updates existing tracklets via cross-attention between image and tracklet tokens. Query tokens of existing tracklets are firstly updated with current image tokens. The updated 3D human poses are matched with 2D bouding box detection of current frame. The unmatched detection will be initialized with new tracks and get updated to generate new 3D pose. Evalutation results on PoseTrack21 testified the effectiveness of the prospoed method. Compared with 4DHumans, the MOTA, IDF1, and IDR get significantly improved.

strengths:
1. Investigating the tracking-by-attension way of mutl-person 3D human motion tracking in crowded scenes is very challenging and interesting. The results of this paper is quite encouraging and exciting.   
2. Updating the tracklet state with assistance of bouding box detection is reasonable. Design of 3D human pose update module is great and seems to be very effective. Subtle 3-stage training process is brilliant.  
3. The experiment details are sincerely and clearly explained.

weaknesses:
1. The tracking experiment is mostly performed on PoseTrack dataset only. Maybe some challenging occlusion scenes in 3D benchmarks, such as MuPoTS-3D and 3DPW could testify the performance of the proposed method in a more comprehensive way.

2. The way of showing qualitative videos is inconvenient for reader to watch. One video with more attractive demos / comparisons would help a lot.

questions:
See weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730673653648

thread_type:
official_review

<thread_object>
creator: reviewer_vxub

<content>
summary:
CoMotion introduces a complex system for multiple person tracking from a monocular camera, based on the tracking-by-attention paradigm. To overcome the challenge of limited data for pose estimation the authors train the model in multiple stages and add various datasets to gradually teach the model to track 3D persons across time.

strengths:
The method solve a highly relevant problem, tracking multiple people from a monocular camera. The method performs well on tracking and pose estimation benchmarks and the authors identified and fixed a bug in existing benchmarks, which is very helpful to the research community.

weaknesses:
The method makes use of various datasets and human pose modalities, such as SMPL, pseudo-SMPL and COCO. Could the authors describe how they handle different pose skeleton modalities, i.e. how do they train their SMPL pose estimation model with COCO skeleton supervision?

LL292-299: it is interesting that the authors do not add tracks during training: have the authors conducted an analysis of the training data to determine how many tracks appear / disappear on average? I would assume that this could potentially be very detrimental to the learning, as the model might learn to “ignore” certain people (i.e. if they were occluded before, or if they are at the edge of the image). Have the authors done any analysis of those kind of edge cases?

What do the authors mean by “[we] do not expect the shape of the mesh to correspond to the person’s physical body” (LL207-208)? The shape does not just model the mesh surface but also influences the bone lengths, so the skeleton is affected by it as well. Could the authors clarify this?

What do the authors mean by “the intrinsic matrix [does not] necessarily reflects the actual calibration of the camera“? Especially, as the authors claim in the next sentence that their method is robust to any provided intrinsic camera! Such a claim should be ablated, i.e. compare the system predictions with K provided and without K provided.

Can the authors elaborate on what they mean by “strict” in Table 2?

Figure 2 is not self-containing as it does not describe all elements of the image, i.e.e method part (A) is missing.

Typo in L205-206: the parameters $t$, $\beta$ and $\theta$ are missing from the SMPL parameter discretion. 

LL324-325: the authors should provide more details on how they apply the regularization for the beta optimization.

— suggestions — (will not influence my rating)

The authors could utilize the CHI3D [a] as an additional benchmark as it comes with well-tracked 3D poses (SMPL-X) at higher quality than 3DPW and with more challenging human-human interactions.


[a] Fieraru, Mihai, et al. "Three-dimensional reconstruction of human interactions." CVPR 2020.

questions:
L318: what is the “off-the-shelf MAE”?

I wonder if the authors have thought about utilizing a generative model as pose prior, ie. Following ScoreHMR. The system could then be understood as a Recursive Bayesian Filter, which would provide a nice mathematical foundation for the tracking algorithm.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730686025355

thread_type:
official_review

<thread_object>
creator: reviewer_majj

<content>
summary:
This paper addresses the problem of 3D human pose and shape estimation and cross-frame tracking. The proposed method estimates 3D poses for all the persons in a frame and then updates the tracking poses and bounding boxes. A novel pose update module with GRU is introduced to update the tracking results and store the hidden information not present in the SMPL parameters. The proposed method is benchmarked on both bounding box tracking and pose estimation tasks and demonstrated better performance than the previous method. The comparisons on per-frame runtime also demonstrated its much more efficient design compared to the previous method.

strengths:
1. The proposed method estimates the tracking and pose simultaneously, which could take advantage of prior information on humans.
2. The design choice of hidden state and GRU is sound. Keeping tracking alive seems to be a valid approach to address the missing people in a shorter period. 
3. The evaluation results on PoseTrack benchmarks are impressive, validating the proposed method's design choice.

weaknesses:
Overall, the paper is of good quality, but there are still some weaknesses that the reviewer would like to discuss here and in the following questions. 
1. The implementation details and model architecture are unclear. The model architecture of both the pose estimation module and the pose update module is unclear. It's also unclear how the detection of bounding boxes is achieved. 
2. Although the hidden state and GRU are sound approaches, the experiments do not rigorously evaluate their effectiveness. 
3. The proposed model takes advantage of multiple training datasets, although it's not uncommon in the community of human pose estimation, but part of the improvements in the tracking results might be attributed to this.

questions:
1. The videos show unstable pose estimation for static humans, which should be addressed better in the pose tracking method than in the single-frame human pose estimation method. What is the cause of such instability, as the model has information about previous estimations and a learned hidden state? 
2. The authors mentioned using a 'strict' threshold in evaluating pose track metrics. Could the authors elaborate more on the choice of threshold and its relationship with the tracking method's precision and recalls? 
3. Will the proposed method be open-sourced? There is only limited information on the model architecture and hyper-parameters. It will be hard to reproduce the proposed method if the source code and checkpoint are not available. 
4. The proposed method has worse results compared to CLIFF in Table 3. What's the possible cause of it? 
5. Dense object tracking has been much improved by papers like CoTracker. How will the proposed method perform when compared to the method Cotracker? It will be interesting to see the performance difference between a 3D-aware object-specific method performed compared to a generic method.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730745038362

thread_type:
official_review

<thread_object>
creator: reviewer_zfvy

<content>
summary:
This paper addresses a challenging problem of multi-person 3D pose tracking. Compared to tracking-by-detection pipelines, this paper extends the tracking-by-attention pipelines from Multi-Object Tracking to Multi-person pose tracking, which detect new humans and update existing tracks by learned tokens. Due to the lack of complete training data, this paper explores the mixed datasets with different forms of supervision. The experimental results demonstrate superior performance over other state-of-the-art methods on several benchmarks.

strengths:
1. This paper considers a new pipeline to address multi-person video-based tracking and pose estimation. 
2. The paper is well-written and easy to understand. The qualitative results clearly show the advance of the proposed method. 
3. This method is very fast compared to other methods and achieve good results in several benchmarks.

weaknesses:
1. This paper is more like a technical report rather than research paper. The contributions are hard to recognise and not clearly highlighted. This makes this paper less insightful. 
2. To make the contributions clearer, i would recommend giving some preliminaries about the tracking-by-attention pipelines applied in other applications, e.g., MeMOTR, to distinguish what’s new in your method and what has been done.
3. One important claim in your paper in CoMotion can utilise subtle pixel cues in Line52-53, which is not elaborated is supported in the following text and experiments.
4. The compared methods in tab.1 seems not very recent, what’s the comparison between CoMotion and more advanced MOT methods?
5. Some minor issues: 
- Metrics can be introduced. If the space is not enough, you can include them in the appendix.
- I would recommend give short descriptions about ablation studies instead of checking them directly in appendix.

questions:
I really appreciate authors' hard work, which gives fast and accurate results on the benchmarks. However, I am doubtful of this paper's novelty and insight and I give borderline reject accordingly.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732785928283

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank all reviewers for their valuable feedback and respond to concerns and questions directly after each review.
</content>
</thread_object>

</thread>


Metareview:
metareview:
## Summary
This paper presents a multi-person 3D pose tracking method that extends tracking-by-attention pipelines, detecting new humans and updating existing tracks using learned tokens. The method uses mixed datasets and different supervision forms, demonstrating superior performance in various benchmarks. The method also introduces a novel pose update module with GRU, improving performance and efficiency in bounding box tracking and pose estimation tasks.

## Strengths
* The method solves a relevant problem of tracking multiple people from a monocular camera; tracking-by-attension method in crowded scenes, leveraging prior human information. 
* 3D human pose update module design is effective and fast and achieves good results in several benchmarks.
* Design choice of hidden state and GRU is sound. 

## Weaknesses
* The tracking experiment is primarily conducted on the PoseTrack dataset.
* Preliminaries about tracking-by-attention pipelines applied in other applications, such as MeMOTR, could help distinguish new contributions.
* The comparison between CoMotion and more advanced MOT methods is not very recent.
* The proposed model takes advantage of multiple training datasets, which could contribute to improvements in tracking results. 
* The authors mentioned using a'strict' threshold in evaluating pose track metrics, which could be elaborated on.
* The proposed method has worse results compared to CLIFF, which could be due to various factors.
* The authors do not add tracks during training, which could potentially be detrimental to learning.
* The authors do not expect the shape of the mesh to correspond to the person’s physical body. 
* The authors should provide more details on how they apply the regularization for the beta optimization.
* The authors could utilize the CHI3D [a] as an additional benchmark due to its well-tracked 3D poses and more challenging human-human interactions.

##Conclusion
Although there are a few weaknesses, the paper tackles an interesting problem and proposes a well defined solution evaluated on several benchmarks. Based on the reviews and author's feedback, the paper should be accepted but should include all the points raised by the reviewers.