Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a method for tracking multiple people in monocular videos using 3D information (3D appearance, 3D pose, and 3D location) by extending the HMA model. The paper shows the effectiveness of 3D information especially the 3D appearance in some benchmark datasets. The proposed method also shows better performance than the other state-of-the-art algorithms. 
The main contribution of the paper is the use of 3D representation in improving the people tracking performance in videos of monocular cameras. The work is well written and structured. A potential place for improvement is the simple association algorithm (Algorithm 1), which may be inadequate in some complex scenes with lots of occlusions or people wearing similar clothes. 
main_review: Summary:
This paper presents a method for tracking people in monocular videos using 3D representations. The main idea is to extend the popular Human Mesh and Appearance (HMA) architecture to include appearance information of people on their 3D geometry representations (3D pose and location). 
This extended HMA is named HMAR (R stands for “Recovery”). HMAR model recovers the appearance of the person in the form of the texture map, which is then transformed to a compact embedding using an auto-encoder. In addition, 3D pose embedding and the 3D location of the key points are encoded in the HMAR model. Furthermore, a transformer is used to create a encode global information across space and time (video frames). Finally, the paper uses a simple association scheme to associate the 3D representations across frames using the Hungarian algorithm. 
The authors conducted experiments to test two aspects of the proposed approach: 1) 3D representations vs 2D representations; 2) the effectiveness of the three components in the 3D representations. The results show that using 3D representations performs consistently better than their counter 2D representations. The ablation study shows that 3D appearance is the most important component. In addition, the proposed approach is compared with other state-of-the-art tracking methods (i.e., Trackformer, Tracktor, Alphapose, and PoseFlow) and shows better performance, especially in the IDs score. 

Originality: 
I agree with the authors that 3D representation is more effective than 2D representations in solving the multiple people tracking problem, especially in scenes with lots of occlusions. There exists some work that uses 3D representations to tracking multiple people in videos shoot by monocular cameras. For example, Brau [1] uses 3D to represent people and simultaneously estimate the camera parameters while inferring the 3D locations of people in 3D in monocular videos. Therefore, I think the author is not the first one that uses the 3D representation to solve tracking problems with monocular cameras. However, this paper uses a much more sophisticated 3D model than the one used in [1] and generates more 3D information of the tracked person (e.g., the key points of the pose). 

Clarity: 
This paper is well-written and structured. 

Suggestions: 
In the result videos, when people get occulted, their tracks disappeared (probably because there are no bounding boxes detected). If the same person appears shortly, they will be identified as the same person by the association algorithm. The missing part could be interpolated for the missing detection part. The advantage of using 3D representation is that they can naturally explain the 2D occlusion phenomena. 
limitations_and_societal_impact: The author adequately addressed the limitations and potential negative societal impact of their work. d
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper presents an approach for multi-people tracking in videos using 3D body representations based on the SMPL model and their associated texture maps. Given an input video, the 2D bounding boxes are assumed to be given and the paper focuses on estimating the tracklets of each person, i.e. solving the temporal association of the bounding boxes. 
For this purpose, a first module building upon HMR [Kanazawa 2018] is used to extract the SMPL body parameters for each bounding box. Additionally, the 3D location and appearance of the bounding box is also regressed, the latter using UV texture images. 
Then, temporal aggregation of features is performed using a Transformer model. Finally, the transformed features per bounding box are associated to existing tracklets using the Hungarian algorithm.
The main contribution and novelty of this approach is in injecting 3D body representations into the tracking pipeline, jointly with the UV map appearance. 
The approach is evaluated on three datasets, Posetrack, MuPoTs and AVA, and demonstrates improved performance w.r.t. state-of-the-art of 2D tracking methods.

main_review: ********ORIGINALITY********

As mentioned above, the main contribution of this work is in encoding person information using its 3D body representation and the associated appearance UV map. This allows reasoning about the 3D spatial position of the people, which helps to solve situations with occlusions. 

While introducing 3D reasoning into the tracking pipeline is a very promising research line and this improves the most recent state-of-the-art on 2D multi-people tracking (Trackformer [27] and Tracktor [4]) the paper has a number of limitations:

1) The method relies on images with a relatively high resolution of the people. Otherwise, the 2D-to-3D regression would be very unreliable, affecting the overall performance of the approach. In contrast, Trackformer and Tracktor can handle situations with smaller resolution of the bounding boxes.

The evaluation section is not convincing. Concretely:

2) The proposed approach is evaluated on three datasets, Posetrack, MuPots and AVA, and only compared to Trackfomer, Tracktor, Alphapose and PoseFlow. However, many other approaches perform better on these datasets. For instance, in the Posetrack dataset, the work of Wang et al. 2020 “Combining detection and tracking for human pose estimation in videos” reports a MOTA score > 64, both in PoseTrack2017 and PoseTrack2018 test sets. This is significantly higher than the MOTA scores obtained by the submitted approach and Trackformer/Tracktor.  Keytrack [Snower 2019] and FlowTrack [Xiao 2018] also report better results on Posetrack than this submission.

3) MuPots is intended for 3D tracking and standard metrics used to evaluate methods on this dataset are PCK and MPJPE. In order to position the current submission within the state-of-the-art, the authors should compute these metrics and compare with the most recent works. The best results on MuPots are reported by [Cheng et al. 2021] “Graph and Temporal Convolutional Networks for 3D Multi-person Pose Estimation in Monocular Videos”.  A direct comparision with this approach should be done.

In summary, without clearly positioning the current submission within SOTA the reviewer does not believe this paper can be accepted. The proposed methodology is interesting, but the authors focus their comparison on Trackformer and Tracktor, which are designed to track people in more crowded scenes, like those of the MOT17 and MOT20 datasets. 


********CLARITY******** 
The paper in general reads very well, although there are details of the method that remain unclear. For instance: 

1) What is the maximum number of people that can be tracked. How does  this this number compared to Trackformer and Tracktor?

2) The regression of the UV maps does not seem correct. As shown in Figure 1, a UV map should contain black regions, as the flattening of the body is not continuous. Also, the different parts of the body (head, torso, legs ...)  should be clearly visible. However, the regressed UV maps (Fig.2 or Fig.3) show a continuous pattern in which the different body parts cannot be distinguished. 

********SIGNIFICANCE********

While the approach shows improvement w.r.t. Trackformer and Tracktor in the chosen datasets, the fact that other recent papers have not been considered makes it difficult to really assess the significance of the proposed method.

limitations_and_societal_impact: The authors shortly mention some limitations and societal impact in the “Concluding Remarks” section.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work introduces a method for monocular multi-person full-body tracking. The method itself is an extension of an existing approach HMR that predicts body meshes from monocular images. 
The key idea of this work is to rely on a view- independent appearance representation based on UV texture mapping: one of the model's branches predict a 2D field that is used to sample input image to produce the texture, and is trained to reconstruct the input images. A transformer network is then used to propagate information temporally among similar object hypothesis, and the resulting features are then associated via hungarian matching. Experimental evaluation is conducted on multiple real-world datasets.
main_review: 
### Writing quality / clarity
**+** The paper is generally clear and well-written, apart from some minor issues (see Misc below).

### Novelty / significance

**+** The general idea of using a UV texture as a view-independent way to represent human appearance makes a lot of sense. It is not particularly novel, but it seems like this work is one of the first ones to exploit it for multi-person tracking. 

**-** (minor) It is not strictly correct to say that these UV images are pose-invariant though, as they certainly contain a lot of pose-dependent information (even if they were complete). 

**+/-** Using transformer-like architectures for multi-object tracking to enable learnable associations is also a reasonable design choice, albeit not very novel (TransTrack, Trackformer). 

**-** One concern about the method is that it assumes that detections are provided as input, and does not really provide a way to improve those based on resulting tracking. One of the reasons to do tracking in the first place is to be more robust to the quality of the detections. This seems like a major limitation, and it would be great to hear authors' comments on this. 


### Evaluation

**+** Experimental evaluation indicates that the method outperforms several recent baselines in terms of MOTA and other metrics. 

**+/-** Authors provide extensive visual results on in-the-wild sequences, and tracking seems to robust compared to the presented baselines. It does appear that the method is not particularly robust to occlusions, which might be related to the heavy reliance of the method on output of the detections.

**+** Ablation studies are quite thorough, and do indicate that most of the proposed components are necessary to achieve good performance.

**-** It seems strange that authors do not provide comparison to XNect, considering that the model is publicly available. While it is indeed the case that that method provides an alternative way to track individuals (with part association), it does seem quite robust, and still can be used for multi-human tracking. 

**+/-** When looking at the resulting tracking videos (and in particular failure modes), it seems a bit strange that for people with significantly different appearance (and body sizes) there are identity switches (e.g. around 03:30 of the video). Would be great to have an intuition on why this happens. 

**-** There are no numbers on running time / computational complexity analysis, which is important for the given application and that some of the competing approaches are real-time.

### Misc
* 2048 seems like an incredibly large number to represent a human pose. SMPL has ~80 parameters, what is the reasoning behind using more than an order of magnitude more? 
* L4: this serves
* L12: that assign
* L24-L25: * There has been *a lot* of work on tracking people in 3D (more often in multi-view settings). Might make sense to take this into account.
* It might be helpful to visualize sample UV textures to get a feeling of how representative is the appearance information.
* Does this method also work in online settings? Judging from Figure 5, the algorithm is also taking into account "future" predictions. 
limitations_and_societal_impact: Authors do discuss some of the relevant limitations and societal impact. 
One issue that might be worth mentioning is the scalability of the attention module for large number of identities.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The submission presents a tracking approach based on 3D representations (i.e., 3D appearance, pose, and location) for tracking multiple people in the video. The paper claims that 3D representations are more effective than 2D representations for tracking humans, which is also validated in several datasets including Posetrack, MuPoTs, and AVA.
main_review: The paper proposed an approach using 3D Representations for Tracking, which is based on a pipeline for human mesh and appearance recovery and spatio-temporal feature aggregation. Though the recovery of mesh and appearance is not new in literature, jointly using 3D appearance and pose for tracking is somewhat interesting.

+ Overall, this manuscript is well-written and includes technical contributions for tracking multiple people in 3D space.

+ State-of-the-art results in quantitive and qualitative aspects.

There are a few concerns regarding this paper:

- In Sec. 3.2, the global feature outputted from the backbone network is used as the pose representation, how about using the shape and pose parameters of the SMPL model? Though it also makes sense to use deep features as the representation for end-to-end learning, using the SMPL parameters as the pose representation should be also an alternative solution.

- The paper introduces a model for human mesh and appearance recovery, however, there are no quantitive results on the performance of mesh or appearance recovery. It seems that the spatio-temporal feature aggregation can also improve the mesh reconstruction in the video. Adding quantitive results should further strengthen the paper.

- There are also existing approaches related to appearance recovery and reconstruction under the shot changes in videos. It is recommended to including discussions with these papers.

TexturePose: Supervising Human Mesh Estimation with Texture Consistency

Human Mesh Recovery from Multiple Shots
limitations_and_societal_impact: The proposed approach can better handle the shot changes in videos, but the results are not so good under the occlusions and overlap between multiple people (see videos in the supplementary material). The statement "particularly suitable for challenging cases involving person-person occlusion" in L274 seems somewhat over-claimed.
review_title: null

Meta-review: This paper is somewhere in between traditional tracking, person re-identification and 3D human modeling. As claimed by the authors, the use of 3D representations in a tracking context has rarely been done before (mostly due to computational constraints). Since the method requires high-res input (or at least large humans), it may not be suited for any tracking application, which makes comparison against some state-of-the-art methods harder. The paper is accepted, but authors are expected to clearly mention and discuss limitations of the proposed method (e.g. include computational costs mentioned in rebuttal) and why some standard tracking benchmarks like the MOTchallenge were not considered. 
Decision: Accept (Poster)