Tracking People with 3D Representations
Authors: Jathushan Rajasegaran
Georgios Pavlakos
Angjoo Kanazawa
Jitendra Malik
Venue: NeurIPS 2021 Poster (2021)
Keywords: Tracking
3D
Humans
Transformers

Abstract: We present a novel approach for tracking multiple people in video. Unlike past approaches which employ 2D representations, we focus on using 3D representations of people, located in three-dimensional space. To this end, we develop a method, Human Mesh and Appearance Recovery (HMAR) which in addition to extracting the 3D geometry of the person as a SMPL mesh, also extracts appearance as a texture map on the triangles of the mesh. This serves as a 3D representation for appearance that is robust to viewpoint and pose changes. Given a video clip, we first detect bounding boxes corresponding to people, and for each one, we extract 3D appearance, pose, and location information using HMAR. These embedding vectors are then sent to a transformer, which performs spatio-temporal aggregation of the representations over the duration of the sequence. The similarity of the resulting representations is used to solve for associations that assigns each person to a tracklet. We evaluate our approach on the Posetrack, MuPoTs and AVA datasets.  We find that 3D representations are more effective than 2D representations for tracking in these settings, and we obtain state-of-the-art performance. Code and results are available at: https://brjathu.github.io/T3DP.

Tracking People with 3D Representations
Jathushan Rajasegaran, Georgios Pavlakos, Angjoo Kanazawa, Jitendra Malik UC Berkeley
Abstract
We present novel approach for tracking multiple people in video. Unlike past approaches which employ 2D representations, we focus using 3D representations people, located in three-dimensional space. To this end, we develop a method, Human Mesh and Appearance Recovery (HMAR) which in addition to extracting the 3D geometry of the person as a SMPL mesh, also extracts appearance as a texture map the triangles of the mesh. This serves as a 3D representation for appearance that is robust to viewpoint and pose changes. Given a video clip, we first detect bounding boxes corresponding people, and for each one, extract 3D appearance, pose, and location information using HMAR. These embedding vectors then sent to transformer, which performs spatio-temporal aggregation of the representations over the duration of the sequence. The similarity of the resulting representations used solve for associations that assigns each person to a tracklet. We evaluate our approach on the Posetrack, MuPo'Ts and AVA datasets. We find that 3D representations tracking in these settings, and we obtain state-of- toefthheartt performance. Code and results are available at: hhttps:/bbjjee athu. wws.tt..gttti.... github. 1o/T3DP
more effective than 2D representations for
Introduction
Humans are diree-dimensional beings, who live and move in a three-dimensional space. However, monocular people tracking algorithms from the computer vision community, e.g., 4 12 28 36
41,46) 46) are typically based on 2D) representations of people in 2D space. This made a lot of sense in the 1990s when people tracking first came to the fore as (1) the application context was often video surveillance, where people
low resolutions (2) there were not reliable algorithms for lifting people from 2D 3D. But today neither of these limitations hold. Applications of video analysis can use HD video (1080p or better), and we have excellent technology e.g.. (17) for 3Dfying people.
Figure ! iillstrates why tracking people might be easier in 3D than 2D. Even when two people overlap and occlude each other in 2D, their spatial location is typically separable in 3D. While 2D appearance features are sensitive viewpoint and body pose, 3D appearance features are less affected by viewpoint and pose changes. These properties enable robust tracking through people-to-peope occlusion and even shot changes, common phenomena in edited media and online videos.
Tracking people in 3D also opens up many downstream tasks such as predicting 3D human motion from video 21). predicting their behavior 48), and imitating human behavior from video (31). These tasks involve videos of people other than pedestrians, such as videos of people performing sports, dance, daily activities (TO), even edited media such as movies TV shows 1 30).
Our input is monocular video with detected bounding boxes. On each bounding box. we develop an approach that extracts both 3D geometry and appearance information of the person. Specifically, we extend HMR (17), a method that predicts 3D human mesh from monocular images, to also extract the appearance information of the person by predicting a flow field that estimates the texture map, which is a viewpoint and pose invariant space. We refer to this model as Human Mesh and Appearance Recovery (HMAR). From HMAR output, extract 3D appearance, 3D pose, joints,
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

3D location more separable than 2D location
appearance invariant to viewpoint and pose
2D representation
3D representation
2D representation
3D representation
Figure The benefits of tracking people in 3D. Left: Using the 3D location of people makes easier to separate them than using their 2D location or 2D bounding boxes. Right: The appearance of people, when computed in a 3D representation (texture map) is less affected by changes in viewpoint and pose.
and the estimate of their 3D location as separate features. We then aggregate this per-bounding box 3D information across time and space via a transformer (38) that takes these separate 3D cues as input. The transformer acts as a pemtoolt diffusion mechanism that can propagate information across similar features by means of attention. Finally, associate the diffused features across time to assign identities to each bounding box, which produces the final tracklets. The overview of our approach is shown in Figure 2
We evaluate our algorithm on three different datasets: Pose D,, MuPots 27 and AVA (3). AVA is particularly interesting dataset as it is derived from movies, which include shot changes. These exhibit much greater variety behavior than videos in the traditional tracking challenges such as MOT. In addition, people are larger (in terms of number of image pixels), which is essential for the 3D approach to work. We find the following: 1. 3D representations are more effective than 2D representations, 2. Ablation studies show that using 3D information in both appearance and location is helpful, 3. We outperform current state of the art on these datasets.
2 Related work
Tracking The tracking literature is vast and we refer readers to 7 for comprehensive summary.
Tracking may be designed for generic categories such as any bounding boxes (52), here we focus on methods that track humans. These approaches mostly depend on 2D location or keypoint features (12) 36.41 411 and sometimes 2D appearance (4, 28 43 46).. Most modern-day tracking approaches focus on evaluating their performance on the MOT benchmark 77. which mainly focus high-density pedestrian scenes observed in outdoor or indoor settings. Under this setting, detection plays a key role and many methods jointly solve for both detection and association (4 28). In this work, we are interested in the effectiveness of 3D representations for tracking and thus assume that detection bounding boxes are provided, which we associate through our representations. Furthermore, we are interested in tracking people in every day Internet videos where people occupy larger portion of the scene, performing activities such as sports or dancing, and even edited media such as movies.
There are methods that incorporate 3D information in tracking, however these approaches assume multiple input cameras (23 49 3D point cloud observation from lidar data (40). Here we focus on the setting where the input is a monocular video. This setting has also been addressed in the past, e.g., (5), however, here leverage much more expressive and fine-grained 3D information, including detailed representations for appearance and 3D joint locations, which help to improve the overall tracking performance.
Monocular 3D human reconstruction There is long history of methods that predict 3D human structure motion from monocular images videos 66 14]. Recently, there has been rapid progress in this area due to the emergence of statistical models of human bodies such as SMPL (25 that provide a low dimensional parameterization of a deformable 3D mesh of human bodies. Human
Mesh Recovery (HMR) method (I7) and follow-up works, e.g., (3) (2). employ convolutional neural networks to predict the parameters of an existing body model from a single image. Extending this result in the temporal dimension, HMMR (19) and VIBE (2 reconstruct mesh output for a single person given a series of tracked frames or tracklets as the input. Extensions to movie data have also been investigated in (30). Other works focus on multiple people (16) 37 47) and regress the

---

Input: Video with bbox detections
3D Feature Extraction
Spatio- Temporal Aggregation
Output: Tracklets
5D Pose
Transformer
Association
Appesrarcc
Figure 2: Overview of our framework. The input to our pipeline consists of video sequence with bounding box detections. For each bounding box. use HMAR model extract 3D representations corresponding to the person's 3D appearance, pose, and location. The 3D representations for bounding boxes are then processed jointly by a transformer which aggregates information across space and time. Finally, an online association step assigns identity to each detected human, which constitute the tracklets.
parameters corresponding to multiple people on the same input frame. All of these methods rely on detected bounding boxes or tracklets in the case of temporal domain. As such, these approaches rely heavily on existing tracking algorithms to extract long tracklets associated to the same individual. In this work we explore how the predicted 3D human information may be used to empower tracking in exchange. There are two recent works that jointly solve tracking and 3D pose estimation of multiple people from monocular video (26) 33). While these approaches focus the task of 3D pose estimation, we focus on the problem of identity association and study the effectiveness of 3D vs 2D representation and compare extensively against state-of-the-arr 2D tracking methods.
Related to our approach are recent methods that extract the texture of person along with their geometry (29 39 (5).. The first recovers the texture only for the visible areas of the body, and uses improve the 3D reconstruction, while the other
use identity labels to learn an encoder that predicts complete 3D appearance expressed as UV images. In this work we explore how UV images can be used for associating people across time for tracking.
Method
In order to assess the effectiveness of 3D information in tracking, we propose simple tracking framework that assumes as input a monocular video and set of detected bounding boxes. We then explore how associate these bounding boxes across time using various cues.
Our tracking algorithm consists of two main modules: our proposed HMAR model, which encodes humans into rich embedding space, and a transformer model for learning associations between detected humans across multiple frames. The HMAR model learns to predict appearance, pose, and keypoints. These are essential cues for tracking, and each of them carries complementary information. For example, when person is temporarily occluded, the appearance important to establish its identity after re-appearance, while when many people share similar clothing in video, pose and location become the primary cues for tracking. Given this rich embedding of a person, we need to learn associations between different human identities that each person can be matched in the upcoming frames. We train a modified version of a transformer (38) to learn these associations across multiple human identities.
3.1 Human Mesh and Appearance Recovery (HMAR)
Our starting point for 3D feature extraction the popular HMR architecture (17.. Given a bounding box of a person, HMR regresses the 3D pose and shape parameters of the person. While the 3D pose
3

---

provides rich geometric information about the person, it throws away the appearance information, which is essential for tracking. To augment this, we extend HMR such that it can also recover the 3D appearance of the person by means of texture image, which is a space that is viewpoint and pose invariant. We call this method HMAR, which stands for Human Mesh and Appearance Recovery.
In order to recover the texture image, we create skip connections from the spatial features of the Resnet-50 backbone used in HMR and connect them to new appearance head that maintains the spatial alignment of features through a series of convolution and bilinear upsampling layers (Figure B) Please see the supplemental for details of the architecture. The appearance head takes these spatial features over multiple scales as input and is trained to predict the texture of the person, using the canonical SMPL texture mapping (25). This is done in the form of appearance or texture flow in the spirit of (18) 5T). Specifically, we regress a texture flow F that has the same dimensions as the texture map. Each value of this flow indicates the coordinates (æ, y) of the input image that the corresponding texel is sampled from. Eventually, the final texture map Tex G(T is produced by bilinear sampling G of the input image T. The predicted texture map then used to render the textured 3D mesh. The model is trained to minimize the reconstruction loss after rendering the texture and geometry on the foreground pixels using LI and perceptual loss.
3.2 3D Representations for tracking
The tracking performance is greatly affected by the cues and the type of representation used for association. Our embeddings are computed as natural outputs of the HMAR model as follows:
Appearance: HMAR recovers the appearance of the person in the form of the texture image L For tracking, we need to create a compact embedding for appearance, so we employ an auto-encoder to further encode this appearance. The auto-encoder takes the texture image as input and is trained to reconstruct it by going through a low dimensional bottleneck. This bottleneck is represented by a 51 512-dimensional vector a, which is used as the representation of the appearance in our pipeline.
3D Pose: For the pose representation, we follow previous work 19 48) and use the embedding 22048 of HMR as representation of the pose of the person.
3D Location: Given an image with dimensions [W, H). HMAR is applied on square bounding box with center [cz, Cyl and dimensions [b, b). HMAR also predicts the local camera parameters [s, fay tyl (scale and 2D translation) for the input bounding box. By assuming constant focal length f for the image, we can recover the approximate 3D global translation of the person, which localizes the mesh in the view coordinate 3D space corresponding to the full image. This translation is expressed as:
T = tyr
20,_", by - W
26,
2
2b
Since we do not know the actual focal length, this is only an approximation of the metric location, but we can still recover consistent relative 3D positions of the people over a single video up to scale.
For a more detailed description of the 3D location, we also extract the 3D keypoints of the person that are directly regressed from the recovered mesh and we position them in the view coordinate 3D space by adding the translation T to them. This representation is strictly more informative than the 21) bounding box information, both because is in 3D, and because it provides localized information about the pose of the person than a single bounding box. We also concatenate the temporal information to the 3D keypoints in the spirit of (38), which leads to space-time representation E 90
3.3 Spatio-temporal Feature Aggregation
Our HMAR model encodes the 3D appearance, pose, and location information of each bounding box. While this gives us a rich 3D embedding, this representation only captures information from a fingle-frame. In order to incorporate the spatio-temporal information of the surrounding bounding boxes, we employ a modified transformer model to aggregate global information across space and time. This transformer essentially performs anisotropic diffusion (32) of the representation across all bounding boxes, where the similarity is captured by attention.
Specifically, the input to transformer is the combination of the three different representations obtained from HMAR corresponding to 3D appearance, pose and location, h aa"', ,"',"","" , e 22250x1 The transformer architecture consists of 3 blocks of self-attention and feed-forward layers. We use

---

Backbone
HMAR
Outputs
Canter cropped
input mage
Max Pool
Encodor
Appearance
Locaton
Figure Human Mesh and Appearance Recovery (HMAR): Our HMAR model learns to predict 3D pose, UV-image and set of keypoints relative to whole image from a single person center cropped image. For the pose, we use the HMR model (17), which gives us appearance head to HMR, which learns to predict flow-field. When the input image is sampled using the flow-field, produces UV-image corresponding the appearance of the person the image. The UV-image then encoded into appearance vector (a). Finally, we extract the location of 15 3D keypoints (s) from the 3D mesh. These keypoints represent the position of human the 3D space. Therefore, using our HMAR model we can get pose, appearance, and location information of the person.
pose vector (p). We add an additional
Figure HMAR results. each example we show from to right: (a) Input image, (b) Predicted mesh, (c) predicted texture map, (d) and (c) textured mesh from frontal and side view.
single head attention inside the self-attention layer. During training, we feed number of frames, each with maximum number of P people on it. Therefore, for a single clip. the transformer could T P number of humans.
We use a modified transformer that treats each of the three attrihutes of the h separately. Specifically, inside the self-attention layer, appearance, pose and keypoints have their value, query, and key vectors.
f, W app
opp
a.
L pose
W pose P.
toc
(2)
Here, Wo R512x512 pose 2048 and W E 90000 the transformation matrices for the value vectors of each attribute in h vector. Similarly, we can get 1,4 aposes qood and [Kepp, K loc] vectors by learning projections W Wht.. for alt E [pose, loc]. With this setting, we can find attentions for each attribute separately. Finally, the total attention A E RTPXPP taken weighted sum over various attentions (see also F)
Doit (k )T
A (0
Σ
Bast sof tmax
(3)
Vdimate
attE[uyp.pose.kk
Here, Papp: Spose and Sloc are of hyper- prper-pprameters.. While we use equal values for all of our experiments for simplicity, this design allows for selective attention to different attributes. For example, if we use [0, 1, 0] for 9app= 8pose and Btoe respectively, the appearance and location embeddings are accumulated based only on the attention of the pose embedding. This could be useful for extreme cases, such as when where all people wear the same uniform. Finally, the output of the self-attention layer f is calculated as the weighted sum of the total attention.
h
44
Σ AA..0.E
(Ve'd'
Vposer V
L'
We use the same strategy for following feed-forward layers and normalization layers, by treating each attribute separately as Eq After these human vectors h have been processed by the transformer, the output vector t will accumulate information across multiple people over space-time. However, ideally we want 4. to attend only the persons with same identities. For example, h should attend H if and only if i' are the same person in different frames. If this happens, 4 and h i will

---

stsctitinn lager"
Trensformer
1h
RIMAR
Querne
Pranre
Promett
Figure Spatio-Tene Temporal Aggregation with Transformer: For frames a video, we obtain set of bounding boxes from an off-the-shelf human detector (34). We use HMAR to extract representations of 3D appearance, pose, and location from each bounding box, whose concatenation serves as token for each bounding box. Since this information only extracted from single image, send the tokens through transformer. which acts a diffuusion mechanism to propagate information across space and time. Our modified transformer computes attention on each embedding separately, which then gets merged together for a final attention map.
be the same or at least close to each other in some similarity metric d d(h ().. We enforce this during training by minimizing the RelD (Re-Identífication) loss.
LR =
Σ 1.2...2....
d(n ( max m d(' 1 h
(5)
Here, h and h are the embedding vectors for the same person in frame t, t' respectively, while h and h 3' are the embedding vectors for two different people in frames t, h acts as an anchor for the contrastive loss. We use margin m 10 for the negative samples and L2 distance for the distance metric d.
3.4 Association
Given the 3D representations that have been processed over space-time, the remaining task is to assign identities each bounding box. We employ a simple association scheme demonstrate the effectiveness these representations. Specifically, the output vectors from the transformer h for all i [P] at frame t + need to be associated to the existing tracklets up to t to new tracklet no match exists. We use a simple tracking algorithm that takes one frame at a time and assigns track-ids based on the similarities between the past tracklets and the new detections.
First, we start with frame 1, and assign unique identities for each person. This initiates set of tracklets T = (Trpt for all people in the first frame. Every tracklet is also associated with the corresponding human embeddings h Then we move on to frame 2 and calculate the affinity matrix between the detected humans D = 2 in frame 2, and all active tracklets T= [Trr We use the Hungarian algorithm assign detected humans to the tracklets, using the minimum distance between the last 20 embeddings vectors associated with the tracklet and the embeddings vector of the detected human. This process continues for all frames sequentially. In this process, some of the tracklets might be missed and some detections might not be assigned. In some cases, there will be new humans added the scene, and some people will be occluded leave the scene. For the first case, we assign a new tracklet to the new person if there are matches between old tracklets. For the latter case, we keep the old tracklets alive, even if there are no matches for few number of frames. We remove the tracklet if no new person is assigned to the tracklet continuously for 20 frames. A sketch of this tracking algorithm is shown Algorithm
Experiments
To evaluate the proposed approach, we first evaluate the efficacy of 3D representations over 2D representations, then provide ablations for the different components of our system and finally compare against fate-of-theean tracking methods. We experiment on three datasets: PoseTrack W.

---

Algorithm Tracking Algorithm
1: procedure ONLINE TRACKING
2: Require: h for all t E [7], E [P], total number of frames 7, maximum number of people
per frame P.
3: T T [T, (1), Vi E [1,2 .|. PI] b Initiate tracklets with detections from the first frame.
4: for t 3,. T) do
5:
Create a detected human set. Calculate Affinity A.j f (in d(D,, Try)) for D; E D and Tr, E T
D (h+, vi (1, PI}
6:
7:
M, D. + Assignment(A)
Hungarian algorithm to assign detections tracklets. assign detections to tracks. Increase the age of unmatched tracks by one.
8:
T (Tr, (h*)) (age) 0, v(i,j E EM}
9:
T (Try(age)+ =1, v() TT)
10:
T (Tr, (h*), v(i) Du, j IT +
Add new tracks for detections.
Kill the tracks with age 24.
return Tracklets T
MuPoTS (27). and AVA (3), which span large variety of challenging sequences, including sports, daily activities and storytelling in movies. Since our approach requires mid-to-high resolution images of people, we do not evaluate the MOTChallenge datasets (7), because of their focus on low resolution humanss. However, we expect that as the technology for 3D human reconstruction is improving, e.g., (44), our approach will be applicable in these settings well. In all cases, we rely on detections from an off-the-shelf detection system (34), and we solve for the association, assigning identity labels each bounding box using the proposed approach. As a result, the metries employ also focus on identity tracking on the bounding box level. Specifically, we report Identity switches (IDs), or the number of times recovered trajectory switches from one ground truth tracklet to another. We also report the standard Multi- Objlet-Ojjett Tracking Accuracy (MOTA) (20) which summarizes errors from false positives. false negatives and ID switches. Finally, we also report the ID Fl score (IDFL) (35) which emphasizes the preservation of the track identity over the entire sequence.
Tracking people in edited media requires robustmess against shot changes. For this evaluation use AVA, which is dataset of movies with actors annotated with identities. We use the test split from 30, which consists of sequences that contain shot changes. In this setting we only evaluate on tracklets that are persistent across shot changes, so to avoid being influenced by detection, we only report IDs and IDFI on this dataset.
4.1 Implementation details
To train HMAR, we use a pretrained HMR model as the starting point. On top of the Resnet-50 backbone that HMR uses, we add additional layers from which predict the texture flow predict the texture map. We train this method using images from COCO (24), MPII (2) and Human3.6N (I5). We train the appearance head for roughly 500k iterations with a learning rate of 0.0001 and a batch size of 16 images while keeping the pose head frozen. Please see supplemental for more details. After training HMAR, We train an auto-encoder on the predicted texture map, which results in an encoded appearance embedding a. We train this using the same datasets as HMAR, and training lasts for about 100k iterations, with learning rate of 0.0001. Finally, for training the transformer, we use the training set of Track (I) as this process requires videos with multiple people with labeled IDs for training. Training lasts for 10k iterations with learning rate of 0.001. All experiments are conducted on single RTX 2080 Ti.
4.2
Quantitative evaluation
4.2.1 3D vs 2D
We first evaluate the effectiveness of 3D representations over 2D representations. To this end, we train a simpler version of our system that only uses one cue and compare with 2D and 3D versions of these cues. The full results these experiments are presented in Table [ We experiment with two popular 2D appearance features used in AlphaPose (9] (5)) and Tracktor (4), which are convolutional features trained with Re-ID losses. We compare this against the proposed 3D appearance i.e., the encoded texture map. As can see at the first three rows of Table I our 3D-based representation
7

---

outperforms the popular 2D-based representations, except on AVA where we perform on par with Tracktor. Regarding the performance of pose/location cues, we compure 3D keypoints with the corresponding 21) keypoints (coming from the projection of the 3D keypoints on the image plane). Again, the 3D>-baedd cue outperform the 2D-based counterpart across the board.
Poseirack MOTA
MuPoT' MOTA
AVA
Embedding
2D/3D
IDs.
IDFIt
IDs]
IDFL
IDs]
IDF1
Rc-ID (OSNet)
50
2D
892
63.1
77.2
88
62.7
81.2
454
59.2
P&L App.
Re-ID (Tracktor) 4
2D
848
63.2
78.0
62.8
79.8
242
64.6
Texture map (Ours)
3D
783
63.4
77.3
62.9
82.2
246
64.2
pose & location
2D
1872
59.8
74.2
62.2
80.8
575
56.6
3D pose & location
3D
1563
61.6
75.4
52
62.8
80.1
280
64.2
Table Effect of 3D Information: assess the effect 3D information, we train our model using only one embedding that corresponds to either appearance or position/location and compare between the 2D and
3D representations. Using representations based on 3D information performs consistently better than the corresponding 2D- representations.
4.2.2 Relative importance of different components
Having established the effectiveness of the 3D representation for tracking, we continue with a careful ablation of the components of our system. We report the performance of the pipeline by individually removing each of the 3D cues (i.e.. appearance, pose and 3D location). We also remove the spatio- temporal aggregation of the transformer for ablation. The results for the different versions, as well as the full system are presented in T
These ablations show that 3D appearance is of the most salient cue across the board. On PoseTTrack and MuPoTs 3D location is more important than 3D pose (which is partially subsumed by location since that is based on keypoints). However on AVA, because of shot changes, the two most important cues are appearance and 3D pose, because across a shot boundary 3D pose is preserved while location is not. The transformer consistently adds to the overall robustness of the method. Note that Pose'Track training data is limited (roughly 800 videos each less than a minute), particularly compared to the usual scale of the datasets that transformers are usually trained on, e.g. 300M images for JFT (8. Potentially using larger training data could improve the performance even more.
Pose Track MOTA
MuPoT's MOTA
AVA
Method
IDs]
IDFIt
IDs]
IDF1
IDs]
IDFI
w/o appearance
1783
55.2
73.7
151
64.5
78.1
739
48.7
w/o 3D pose
711
55.6
72.2
43
65.3
82.1
355
65.1
w/o 3D location
1315
54.3
69.7
59
65.3
82.8
273
61.9
w/o transformer
685
49.6
64.5
41
59.1
73.7
260
63.4
Full system
655
55.8
73.4
38
62.1
79.1
240
61.3
Table 2: Ablation of the components of our xystem: We train our system by removing each cue at time as well as the transformer for the ablative study. We find that appearance is salient cue across the board, 3D location is more important than 3D pose in Track and MuPoT's, but in AVA due to shot changes 3D pose (which is view invariant) is more uscful than location. Transformer consistently improves the performance.
4.2.3 Comparison with the state-of-the-art
Lastly, we compare our approach with the state-of-the-art tracking methods. For this comparison, we use four recent popular methods for tracking: Trackformer (28). Tracktor (4). Alphapose ( and PoseFlow (4)). We run these methods out-of-the- on all benchmarks and report their performance on identity tracking. For more clear comparison, we provide results with two versions of our approach, one without using the transformer (Ours-v1), and one with the spatio-tempore aggregation of the transformer (Ours-v2). We do this, because our transformer trained on PoseTrack, but some of the above baselines are not using PoseTrackk for training 44 28), so the Ours-vl baseline is more fair choice for a direct comparison. The full results are presented in Table 3 The two versions of approach perform consistently better than previous work across the board. In Figure 6 we can
8

---

Poso Track
MuPoTS
AVA
Input
Ours
Trackformrr AlphaPose
Figure 6: Qualitative results. We show samples from the three datasets we experiment on, PoscTrack (., MuPoTs (27) and AVA (3)) We provide results our method, AlphaPose (9) and Trackformer (28) More video results are available in: https::// /tbe::///jjaattt github. io/T3DP
also see qualitatively the benefits of our approach compared to these baselines. More specifically, the design choices of our approach make it particularly suitable for challenging cases involving person- person occlusion, or shot changes which are common in edited media (AVA dataset), because our 3D representations improve identity association, when comes linking IDs after the person-person occlusions or shot changes.
Posetrack MOTA
MuPoTs MOTA
AVA
Method
IDs
IDFI
IDs
IDF1
IDs
IDFI
Trackformer
28
1263
33.7
64.0
43
24.9
62.7
716
40.9
Tracktor 4
702
42.4
65.2
53
51.5
70.9
289
46.8
Ours-vl
685
49.6
64.5
41
59.1
73.7
260
63.4
AlphaPose 9
2220
36.9
66.9
117
37.8
67.6
939
41.9
FlowPose 42
1047
15.4
64.2
49
21.4
67.1
452
52.9
Ours-v2
655
55.8
73.4
62.1
79.1
61.3
Table Comparison with state-of-the-art tracking methods: We provide results for two versions of approach. Ours-vl does not include the transformer, while Ours-v2 applies the transformer the spatio- temporal aggregation. Since our transformer is trained on PoseTrack, we provide these two versions for a more fair comparison methods that do not train this dataset (i.e., (8)).
Concluding Remarks
We have demonstrated the value of using 3D representations for tracking people in a wide variety of videos. The proposed approach is simple and modular, and can casily incorporate extra representations such as face embeddings, which may be useful for movies. Our technique relies on having enough resolution each person to enable lifting 3D from 2D. Our approach is also susceptible bad detections, since 3D features computed on non-human regions lead to spurious 3D representations. There are many applications of tracking people in video that have positive societal impact, such as home monitoring for elder care, design of architectural spaces, scientific analysis of child development and more. However, such technology also be misused for mass-surveillance.
Acknowledgements: This work was supported by ONR MURI (N00014- 4-1-67) as well as BAIR and BDD sponsors.

---

References
[1] Mykhaylo Andriluka, Umar Igbal, Eldar Insafutdinov, Leonid Pishchulin, Anton Milan, Juergen Gall, and
Bernt Schiele. PoseTrack: A benchmark for human pose estimation and tracking. In CVPR, 2018.
[2]
Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2D human pose estimation: New benchmark and state of the art analysis. In CVPR, 2014.
[3] Anurag Arnab, Carl Doersch, and Andrew Zisserman. Exploiting temporal context for 3D human pose estimation in the wild. In CVPR, 2019.
[4] Philipp Bergmann, Tim Meinhardt, and Laura Leal-Taixe. Tracking without bells and whistles. In ICCV, 2019.
[5] Ernesto Brau, Jinyan Guan, Kyle Simek. Luca Del Pero, Colin Reimer Dawson, and Kobus Barnard. Bayesian 3D tracking from monocular video. In CVPR, 2013.
[6] Christoph Bregler and Jitendra Malik. Tracking people with twists and exponential maps. In CVPR, 1998.
[7] Patrick Dendorfer, Aljosa Osep. Anton Milan, Konrad Schindler, Daniel Cremers, Ian Reid, Stefan Roth, and Laura Leal- Taixé. MOTChallenge: A benchmark for single-camera multiple target tracking. IJCV, 129(4))845-881, 2021.
[8] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer. Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR,
2021.
[9] Hao-Shu Fang. Shuqin Xie, Yu- Yu-Wing Tai, and Cewu Lu. RMPE: Regional multi-person pose estimation.
In ICCV, 2017.
[10] David Fouhey, Wei-cheng Kuo, Alexei Efros, and Jitendra Malik. From lifestyle VLOGG to everyday interactions. CVPR, 2018.
[11] Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. Recurrent network models for human dynamics. In ICCV, 2015.
[12] Rohit Girdhar, Georgia Gkioxari, Lorenzo Torresani, Manohar Paluri, and Du Tran. Detect-and and-track: Efficient pose estimation in videos. In CVPR, 2018.
[13] Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan. George Toderici, Susanna Ricco, Rahul Sukthankar, Cordelia Schmid, and Jitendra Malik. AVA: A video dataset of spatio-temporally localized atomic visual actions. In CVPR, 2018.
[14] Peng Guan, Alexander Weiss, Alexandru O Balan, and Michael J Black. Estimating human shape and posc from a single image. In ICCV, 2009.
[15] Catalin loneseu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6M, Large scale datasets and predictive methods for 3D human sensing in natural environments. PAMI, 36(7):1325 1139, [16] Wen Jiang, Nikos Kolotouros, Georgios Pavlakos, Xiaowei Zhou, and Kostas Danilidiss. Coherent reconstruction multiple humans from single image. CVPR, 2020.
[17] Angjoo Kanazawa, Michael J Black, David Jacobs, and Jitendra Malik. End-to-end recovery of human
shape and pose. In CVPR, 2018
[18] Angjoo Kanazawa, Shubham Tulsiani, Alexei A Efros, and Jitendra Malik. Learning category-specific mesh reconstruction from image collections. In ECCV, 2018.
[19] Angjoo Kanazawa, Jason Y Zhang, Panna Felsen, and Jitendra Malik. Learning 3D human dynamics from video. In CVPR, 2019.
[20] Rangachar Kasturi, Dmitry Goldgof, Padmanabhan Soundararajan, Vasant Manohar, John Garofolo, Rachel Bowers, Matthew Boonstra, Valentina Korzhova, and Jing Zhang. Framework for performance evaluation of face, text, and vehicle detection and tracking in video: Data, metrics, and protocol. PAMI, (122):313333 2008.
[21] Muhammed Kocabas, Nikos Athanasiou, and Michael J Black. VIBE: Video inference for human body pose and shape estimation. In CVPR, 2020.
[22] Nikos Kolotouros, Georgios Pavlakos, Michael J Black, and Kostas Daniiidiis, Learning to reconstruct 3D human pose and shape via model-fitting in the loop. In ICCV, 2019.
[23] Oh-Hun Kwon, Julian Tanke, and Juergen Gall. Recursive bayesian filtering for multiple human pose tracking from multiple cameras. In ACCV, 2020.
[24] Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár,
and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
[25] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. SMPL: A
skinned multi-person linear model. ACM Transactions Graphics (TOG), 34(6):1-16. 2015.
[26] Dushyant Mehta, Oleksandr Sotnychenko, Franziska Mueller, Weipeng Xu, Mohamed Elgharib, Pascal Fua, Hans-Peter Seidel, Helge Rhodin, Gerard Pons-Moll, and Christian Theobalt. XNect: Real-time multi-person 3D motion capture with single RGB camera. ACM Transactions on Graphics (TOG), 39(4):22-1, 2020.
[27] Dushyant Mehta, Oleksandr Sotnychenko, Franziska Mueller, Weipeng Xu, Srinath Sridhar, Gerard Pons-
Moll, and Christian Theobalt. Single-shot multi-person 3D pose estimation from monocular RGB. In 3DV, 2018.
[28] Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixe, and Christoph Feichtenhofer. TrackFormer: Multi-
object tracking with transformers. arXiv preprint arXiv:2101.02700 2021.
[29] Georgios Pavlakos, Nikos Kolotouros, and Kostas Danilidiss. TexturePose: Supervising human mesh
estimation with texture consistency. In ICCV, 2019.
10

---

[30] Georgios Pavlakos, Jitendra Malik, and Angjoo Kanazawa. Human mesh recovery from multiple shots.
arXiv preprint (0Xi:20022098822 2020.
[31] Xue Bin Peng, Angjoo Kanazawa, Jitendra Malik, Picter Abbeel, and Sergey Levine. Sfv: Reinforcement learning of physical skills from videos. ACM Transactions on Graphics (TOG), 37(6): 1-14, 2018.
[32] Pietro Perona and Jitendra Malik. Scale-space and edge detection using anisotropic diffusion. PAMI,
12(():629-633 1990.
[33] N Dinesh Reddy, Laurent Guigues, Leonid Pischulin, Jayan Eledath, and Srinivasa Narasimhan. Track:
End-to-end learnalle multi-person articulated 3D pose tracking. In CVPR, 2021.
[34] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R- CNN: towards real-time object detection
with region proposal networks. PAMI, 39()): 1149, 2016.
[35] Ergys Ristani, Francesco Solera, Roger Zou, Rita Cucchiara, and Carlo Tomasi. Performance measures
and data set for multi-target, multi-camera tracking. In ECCV, 2016.
[36] Michael Snower, Asim Kadav, Farley Lai, and Hans Peter Graf. 15 keypoints is all you need. In CVPR, 2020.
[37] Yu Sun, Qian Bao, Wu Liu, Yili Fu, Michael J Black, and Tao Mei. Monocular, one-stage, regression of multiple 3D people. In ICCV, 2021.
[38] Ashish Vaswani. Noam Shazeer. Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
[39] Jian Wang, Yunshan Zhong. Yachun Li, Chi Zhang, and Yichen Wei. Re-identification supervised texture generation, In CVPR, 2019.
[40] Xinshuo Weng. Yongxin Wang, Yunze Man, and Kris M Kitani. GNN3DMOT: Graph neural network for
3D multi-object tracking with 2D-3D multi-feature learning, In CVPR, 2020.
[41] Bin Xiao, Haiping Wu, and Yichen Wei. Simple haselines for human pose estimation and tracking. In
ECCV, 2018.
[42] Yuliang Xiu, Jicfeng Li, Haoyu Wang, Yinghong Fang, and Cowu Lu. Pose Flow: Efficient online pose
tracking. 2018.
[43] Jiarui Xu, Yue Cao, Zheng Zhang, and Han Hu. Spatial-temporal relation networks for multi-object tracking. In ICCV, 2019.
[44] Xiangyu Xu, Hao Chen, Francese Moreno-Noguer, László A Jeni, and Fernando De Torre. 3D human
shape and pose from a single low-resolution image with self- supervisedd learning. In ECCV, 2020.
[45] Xiangyu Xu, Hao Chen, Francesc Moreno-Noguer, Laszlo A Jeni, and Fernando De la Torre. 3D human
pose, shape and texture from low-resolution images and videos. PAMI, 2021.
[46] Yihong Xu, Aljosa Osep, Yutong Ban, Radu Horaud, Laura Leal-Taixé, and Xavier Alameda-Pineda. How
to train your deep multi-object tracker. In CVPR, 2020.
[47] Andrei Zanfir, Elisabeta Marinoiu, Mihai Zanfir, Alin-lonut Popa, and Cristian Sminchisescu. Deep
network for the integrated 3D sensing of multiple people natural images. NeurlPS, 2018.
[48] Jason Y Zhang, Panna Felsen, Angjoo Kanazawa, and Jitendra Malik. Predicting 3D human dynamics from video. In /CCV, 2019.
[49] Yuxiang Zhang. Liang An, Tao Yu, Xiu Li, Kun Li, and Yebin Liu. 4D association graph for realtime multi-person person motion capture using multiple video cameras. CVPR, 2020.
[50] Kaiyang Zhou, Yongxin Yang, Andrea Cavallaro, and Tao Xiang. Omni-scale feature learning for person re-identification. ICCV, 2019.
[51] Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexci Efros. View synthesis by xengaracce flow, ECCV, 2016.
[52] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Tracking objects as points. In ECCV, 2020.


Reviews:
[{"review_id": "hPLP5Nda0AP", "reviewer": "Reviewer_5QdY", "summary": "This paper presents a method for tracking multiple people in monocular videos using 3D information (3D appearance, 3D pose, and 3D location) by extending the HMA model. The paper shows the effectiveness of 3D information especially the 3D appearance in some benchmark datasets. The proposed method also shows better performance than the other state-of-the-art algorithms. \nThe main contribution of the paper is the use of 3D representation in improving the people tracking performance in videos of monocular cameras. The work is well written and structured. A potential place for improvement is the simple association algorithm (Algorithm 1), which may be inadequate in some complex scenes with lots of occlusions or people wearing similar clothes. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Summary:\nThis paper presents a method for tracking people in monocular videos using 3D representations. The main idea is to extend the popular Human Mesh and Appearance (HMA) architecture to include appearance information of people on their 3D geometry representations (3D pose and location). \nThis extended HMA is named HMAR (R stands for “Recovery”). HMAR model recovers the appearance of the person in the form of the texture map, which is then transformed to a compact embedding using an auto-encoder. In addition, 3D pose embedding and the 3D location of the key points are encoded in the HMAR model. Furthermore, a transformer is used to create a encode global information across space and time (video frames). Finally, the paper uses a simple association scheme to associate the 3D representations across frames using the Hungarian algorithm. \nThe authors conducted experiments to test two aspects of the proposed approach: 1) 3D representations vs 2D representations; 2) the effectiveness of the three components in the 3D representations. The results show that using 3D representations performs consistently better than their counter 2D representations. The ablation study shows that 3D appearance is the most important component. In addition, the proposed approach is compared with other state-of-the-art tracking methods (i.e., Trackformer, Tracktor, Alphapose, and PoseFlow) and shows better performance, especially in the IDs score. \n\nOriginality: \nI agree with the authors that 3D representation is more effective than 2D representations in solving the multiple people tracking problem, especially in scenes with lots of occlusions. There exists some work that uses 3D representations to tracking multiple people in videos shoot by monocular cameras. For example, Brau [1] uses 3D to represent people and simultaneously estimate the camera parameters while inferring the 3D locations of people in 3D in monocular videos. Therefore, I think the author is not the first one that uses the 3D representation to solve tracking problems with monocular cameras. However, this paper uses a much more sophisticated 3D model than the one used in [1] and generates more 3D information of the tracked person (e.g., the key points of the pose). \n\nClarity: \nThis paper is well-written and structured. \n\nSuggestions: \nIn the result videos, when people get occulted, their tracks disappeared (probably because there are no bounding boxes detected). If the same person appears shortly, they will be identified as the same person by the association algorithm. The missing part could be interpolated for the missing detection part. The advantage of using 3D representation is that they can naturally explain the 2D occlusion phenomena. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The author adequately addressed the limitations and potential negative societal impact of their work. d", "review_text": "", "extra_scores": {"ethical_concerns": "None that I'm aware of. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "YpNzFZo5B6Z", "reviewer": "Reviewer_4pGJ", "summary": "This paper presents an approach for multi-people tracking in videos using 3D body representations based on the SMPL model and their associated texture maps. Given an input video, the 2D bounding boxes are assumed to be given and the paper focuses on estimating the tracklets of each person, i.e. solving the temporal association of the bounding boxes. \nFor this purpose, a first module building upon HMR [Kanazawa 2018] is used to extract the SMPL body parameters for each bounding box. Additionally, the 3D location and appearance of the bounding box is also regressed, the latter using UV texture images. \nThen, temporal aggregation of features is performed using a Transformer model. Finally, the transformed features per bounding box are associated to existing tracklets using the Hungarian algorithm.\nThe main contribution and novelty of this approach is in injecting 3D body representations into the tracking pipeline, jointly with the UV map appearance. \nThe approach is evaluated on three datasets, Posetrack, MuPoTs and AVA, and demonstrates improved performance w.r.t. state-of-the-art of 2D tracking methods.\n", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "********ORIGINALITY********\n\nAs mentioned above, the main contribution of this work is in encoding person information using its 3D body representation and the associated appearance UV map. This allows reasoning about the 3D spatial position of the people, which helps to solve situations with occlusions. \n\nWhile introducing 3D reasoning into the tracking pipeline is a very promising research line and this improves the most recent state-of-the-art on 2D multi-people tracking (Trackformer [27] and Tracktor [4]) the paper has a number of limitations:\n\n1) The method relies on images with a relatively high resolution of the people. Otherwise, the 2D-to-3D regression would be very unreliable, affecting the overall performance of the approach. In contrast, Trackformer and Tracktor can handle situations with smaller resolution of the bounding boxes.\n\nThe evaluation section is not convincing. Concretely:\n\n2) The proposed approach is evaluated on three datasets, Posetrack, MuPots and AVA, and only compared to Trackfomer, Tracktor, Alphapose and PoseFlow. However, many other approaches perform better on these datasets. For instance, in the Posetrack dataset, the work of Wang et al. 2020 “Combining detection and tracking for human pose estimation in videos” reports a MOTA score > 64, both in PoseTrack2017 and PoseTrack2018 test sets. This is significantly higher than the MOTA scores obtained by the submitted approach and Trackformer/Tracktor.  Keytrack [Snower 2019] and FlowTrack [Xiao 2018] also report better results on Posetrack than this submission.\n\n3) MuPots is intended for 3D tracking and standard metrics used to evaluate methods on this dataset are PCK and MPJPE. In order to position the current submission within the state-of-the-art, the authors should compute these metrics and compare with the most recent works. The best results on MuPots are reported by [Cheng et al. 2021] “Graph and Temporal Convolutional Networks for 3D Multi-person Pose Estimation in Monocular Videos”.  A direct comparision with this approach should be done.\n\nIn summary, without clearly positioning the current submission within SOTA the reviewer does not believe this paper can be accepted. The proposed methodology is interesting, but the authors focus their comparison on Trackformer and Tracktor, which are designed to track people in more crowded scenes, like those of the MOT17 and MOT20 datasets. \n\n\n********CLARITY******** \nThe paper in general reads very well, although there are details of the method that remain unclear. For instance: \n\n1) What is the maximum number of people that can be tracked. How does  this this number compared to Trackformer and Tracktor?\n\n2) The regression of the UV maps does not seem correct. As shown in Figure 1, a UV map should contain black regions, as the flattening of the body is not continuous. Also, the different parts of the body (head, torso, legs ...)  should be clearly visible. However, the regressed UV maps (Fig.2 or Fig.3) show a continuous pattern in which the different body parts cannot be distinguished. \n\n********SIGNIFICANCE********\n\nWhile the approach shows improvement w.r.t. Trackformer and Tracktor in the chosen datasets, the fact that other recent papers have not been considered makes it difficult to really assess the significance of the proposed method.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors shortly mention some limitations and societal impact in the “Concluding Remarks” section. \n", "review_text": "", "extra_scores": {"ethical_concerns": "The authors also briefly discuss ethical issues in the “Concluding Remarks” section.\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "OjSNy3tjLu3", "reviewer": "Reviewer_VGK2", "summary": "The submission presents a tracking approach based on 3D representations (i.e., 3D appearance, pose, and location) for tracking multiple people in the video. The paper claims that 3D representations are more effective than 2D representations for tracking humans, which is also validated in several datasets including Posetrack, MuPoTs, and AVA.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposed an approach using 3D Representations for Tracking, which is based on a pipeline for human mesh and appearance recovery and spatio-temporal feature aggregation. Though the recovery of mesh and appearance is not new in literature, jointly using 3D appearance and pose for tracking is somewhat interesting.\n\n+ Overall, this manuscript is well-written and includes technical contributions for tracking multiple people in 3D space.\n\n+ State-of-the-art results in quantitive and qualitative aspects.\n\nThere are a few concerns regarding this paper:\n\n- In Sec. 3.2, the global feature outputted from the backbone network is used as the pose representation, how about using the shape and pose parameters of the SMPL model? Though it also makes sense to use deep features as the representation for end-to-end learning, using the SMPL parameters as the pose representation should be also an alternative solution.\n\n- The paper introduces a model for human mesh and appearance recovery, however, there are no quantitive results on the performance of mesh or appearance recovery. It seems that the spatio-temporal feature aggregation can also improve the mesh reconstruction in the video. Adding quantitive results should further strengthen the paper.\n\n- There are also existing approaches related to appearance recovery and reconstruction under the shot changes in videos. It is recommended to including discussions with these papers.\n\nTexturePose: Supervising Human Mesh Estimation with Texture Consistency\n\nHuman Mesh Recovery from Multiple Shots", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The proposed approach can better handle the shot changes in videos, but the results are not so good under the occlusions and overlap between multiple people (see videos in the supplementary material). The statement \"particularly suitable for challenging cases involving person-person occlusion\" in L274 seems somewhat over-claimed.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "F-MEF7MUu9X", "reviewer": "Reviewer_tVEj", "summary": "This work introduces a method for monocular multi-person full-body tracking. The method itself is an extension of an existing approach HMR that predicts body meshes from monocular images. \nThe key idea of this work is to rely on a view- independent appearance representation based on UV texture mapping: one of the model's branches predict a 2D field that is used to sample input image to produce the texture, and is trained to reconstruct the input images. A transformer network is then used to propagate information temporally among similar object hypothesis, and the resulting features are then associated via hungarian matching. Experimental evaluation is conducted on multiple real-world datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n### Writing quality / clarity\n**+** The paper is generally clear and well-written, apart from some minor issues (see Misc below).\n\n### Novelty / significance\n\n**+** The general idea of using a UV texture as a view-independent way to represent human appearance makes a lot of sense. It is not particularly novel, but it seems like this work is one of the first ones to exploit it for multi-person tracking. \n\n**-** (minor) It is not strictly correct to say that these UV images are pose-invariant though, as they certainly contain a lot of pose-dependent information (even if they were complete). \n\n**+/-** Using transformer-like architectures for multi-object tracking to enable learnable associations is also a reasonable design choice, albeit not very novel (TransTrack, Trackformer). \n\n**-** One concern about the method is that it assumes that detections are provided as input, and does not really provide a way to improve those based on resulting tracking. One of the reasons to do tracking in the first place is to be more robust to the quality of the detections. This seems like a major limitation, and it would be great to hear authors' comments on this. \n\n\n### Evaluation\n\n**+** Experimental evaluation indicates that the method outperforms several recent baselines in terms of MOTA and other metrics. \n\n**+/-** Authors provide extensive visual results on in-the-wild sequences, and tracking seems to robust compared to the presented baselines. It does appear that the method is not particularly robust to occlusions, which might be related to the heavy reliance of the method on output of the detections.\n\n**+** Ablation studies are quite thorough, and do indicate that most of the proposed components are necessary to achieve good performance.\n\n**-** It seems strange that authors do not provide comparison to XNect, considering that the model is publicly available. While it is indeed the case that that method provides an alternative way to track individuals (with part association), it does seem quite robust, and still can be used for multi-human tracking. \n\n**+/-** When looking at the resulting tracking videos (and in particular failure modes), it seems a bit strange that for people with significantly different appearance (and body sizes) there are identity switches (e.g. around 03:30 of the video). Would be great to have an intuition on why this happens. \n\n**-** There are no numbers on running time / computational complexity analysis, which is important for the given application and that some of the competing approaches are real-time.\n\n### Misc\n* 2048 seems like an incredibly large number to represent a human pose. SMPL has ~80 parameters, what is the reasoning behind using more than an order of magnitude more? \n* L4: this serves\n* L12: that assign\n* L24-L25: * There has been *a lot* of work on tracking people in 3D (more often in multi-view settings). Might make sense to take this into account.\n* It might be helpful to visualize sample UV textures to get a feeling of how representative is the appearance information.\n* Does this method also work in online settings? Judging from Figure 5, the algorithm is also taking into account \"future\" predictions. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors do discuss some of the relevant limitations and societal impact. \nOne issue that might be worth mentioning is the scalability of the attention module for large number of identities. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is somewhere in between traditional tracking, person re-identification and 3D human modeling. As claimed by the authors, the use of 3D representations in a tracking context has rarely been done before (mostly due to computational constraints). Since the method requires high-res input (or at least large humans), it may not be suited for any tracking application, which makes comparison against some state-of-the-art methods harder. The paper is accepted, but authors are expected to clearly mention and discuss limitations of the proposed method (e.g. include computational costs mentioned in rebuttal) and why some standard tracking benchmarks like the MOTchallenge were not considered. 
Decision: Accept (Poster)