Tracking People with 3D Representations

1 Introduction . Humans are three-dimensional beings , who live and move in a three-dimensional space . However , monocular people tracking algorithms from the computer vision community , e.g. , ( 4 ; 7 ; 12 ; 28 ; 36 ; 41 ; 46 ) are typically based on 2D representations of people in 2D space . This made a lot of sense in the 1990s when people tracking first came to the fore as ( 1 ) the application context was often video surveillance , where people are at low resolutions ( 2 ) there were not reliable algorithms for lifting people from 2D to 3D . But today neither of these limitations hold . Applications of video analysis can use HD video ( 1080p or better ) , and we have excellent technology e.g. , ( 17 ) for 3Dfying people . Figure 1 illustrates why tracking people might be easier in 3D than in 2D . Even when two people overlap and occlude each other in 2D , their spatial location is typically separable in 3D . While 2D appearance features are sensitive to viewpoint and body pose , 3D appearance features are less affected by viewpoint and pose changes . These properties enable robust tracking through people-to-people occlusion and even shot changes , a common phenomena in edited media and online videos . Tracking people in 3D also opens up many downstream tasks such as predicting 3D human motion from video ( 18 ; 21 ) , predicting their behavior ( 11 ; 48 ) , and imitating human behavior from video ( 31 ) . These tasks involve videos of people other than pedestrians , such as videos of people performing sports , dance , daily activities ( 10 ) , or even edited media such as movies or TV shows ( 13 ; 30 ) . Our input is a monocular video with detected bounding boxes . On each bounding box , we develop an approach that extracts both 3D geometry and appearance information of the person . Specifically , we extend HMR ( 17 ) , a method that predicts 3D human mesh from monocular images , to also extract the appearance information of the person by predicting a flow field that estimates the texture map , which is a viewpoint and pose invariant space . We refer to this model as Human Mesh and Appearance Recovery ( HMAR ) . From HMAR output , we extract 3D appearance , 3D pose , joints , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . z 2D tracking 3D tracking 3D location is more separable than 2D location 3D Appearance and Pose is invariant to viewpoint 3D representation2D representation z 2D tracking 3D tracking 3D location is more separable than 2D location 3D Appearance and Pose is invariant to viewpoint 3D representation2D representation and the estimate of their 3D location as separate features . We then aggregate this per-bounding box 3D information across time and space via a transformer ( 38 ) that takes these separate 3D cues as input . The transformer acts as a spatio-temporal diffusion mechanism that can propagate information across similar features by means of attention . Finally , we associate the diffused features across time to assign identities to each bounding box , which produces the final tracklets . The overview of our approach is shown in Figure 2 . We evaluate our algorithm on three different datasets : PoseTrack ( 1 ) , MuPots ( 27 ) and AVA ( 13 ) . AVA is a particularly interesting dataset as it is derived from movies , which include shot changes . These exhibit much greater variety of behavior than videos in the traditional tracking challenges such as MOT . In addition , people are larger ( in terms of number of image pixels ) , which is essential for the 3D approach to work . We find the following : 1 . 3D representations are more effective than 2D representations , 2 . Ablation studies show that using 3D information in both appearance and location is helpful , 3 . We outperform current state of the art on these datasets . 2 Related work . Tracking The tracking literature is vast and we refer readers to ( 7 ) for a comprehensive summary . Tracking may be designed for generic categories such as any bounding boxes ( 52 ) , here we focus on methods that track humans . These approaches mostly depend on 2D location or keypoint features ( 12 ; 36 ; 41 ) and sometimes 2D appearance ( 4 ; 28 ; 43 ; 46 ) . Most modern-day tracking approaches focus on evaluating their performance on the MOT benchmark ( 7 ) , which mainly focus on high-density pedestrian scenes observed in outdoor or indoor settings . Under this setting , detection plays a key role and many methods jointly solve for both detection and association ( 4 ; 28 ) . In this work , we are interested in the effectiveness of 3D representations for tracking and thus assume that detection bounding boxes are provided , which we associate through our representations . Furthermore , we are interested in tracking people in every day Internet videos where people occupy larger portion of the scene , performing activities such as sports or dancing , and even edited media such as movies . There are methods that incorporate 3D information in tracking , however these approaches assume multiple input cameras ( 23 ; 49 ) or 3D point cloud observation from lidar data ( 40 ) . Here we focus on the setting where the input is a monocular video . This setting has also been addressed in the past , e.g. , ( 5 ) , however , here we leverage much more expressive and fine-grained 3D information , including detailed representations for appearance and 3D joint locations , which help to improve the overall tracking performance . Monocular 3D human reconstruction There is long history of methods that predict 3D human structure or motion from monocular images or videos ( 6 ; 14 ) . Recently , there has been rapid progress in this area due to the emergence of statistical models of human bodies such as SMPL ( 25 ) that provide a low dimensional parameterization of a deformable 3D mesh of human bodies . Human Mesh Recovery ( HMR ) method ( 17 ) and follow-up works , e.g. , ( 3 ; 22 ) , employ convolutional neural networks to predict the parameters of an existing body model from a single image . Extending this result in the temporal dimension , HMMR ( 19 ) and VIBE ( 21 ) reconstruct mesh output for a single person given a series of tracked frames or tracklets as the input . Extensions to movie data have also been investigated in ( 30 ) . Other works focus on multiple people ( 16 ; 37 ; 47 ) and regress the Output : TrackletsInput : Video withbbox detections 3D Feature Extraction Spatio-Temporal Aggregation AssociationTransformer ! Appearance 3D Pose 3D Location Output : TrackletsInput : Video withbbox detections 3D Feature Extraction Spatio-Temporal Aggregation parameters corresponding to multiple people on the same input frame . All of these methods rely on detected bounding boxes or tracklets in the case of temporal domain . As such , these approaches rely heavily on existing tracking algorithms to extract long tracklets associated to the same individual . In this work we explore how the predicted 3D human information may be used to empower tracking in exchange . There are two recent works that jointly solve tracking and 3D pose estimation of multiple people from monocular video ( 26 ; 33 ) . While these approaches focus on the task of 3D pose estimation , we focus on the problem of identity association and study the effectiveness of 3D vs 2D representation and compare extensively against state-of-the-art 2D tracking methods . Related to our approach are recent methods that extract the texture of a person along with their geometry ( 29 ; 39 ; 45 ) . The first recovers the texture only for the visible areas of the body , and uses it to improve the 3D reconstruction , while the other two use identity labels to learn an encoder that predicts complete 3D appearance expressed as UV images . In this work we explore how UV images can be used for associating people across time for tracking . 3 Method . In order to assess the effectiveness of 3D information in tracking , we propose a simple tracking framework that assumes as input a monocular video and a set of detected bounding boxes . We then explore how to associate these bounding boxes across time using various cues . Our tracking algorithm consists of two main modules : our proposed HMAR model , which encodes humans into a rich embedding space , and a transformer model for learning associations between detected humans across multiple frames . The HMAR model learns to predict appearance , pose , and keypoints . These are essential cues for tracking , and each of them carries complementary information . For example , when a person is temporarily occluded , the appearance is important to establish its identity after re-appearance , while when many people share similar clothing in a video , pose and location become the primary cues for tracking . Given this rich embedding of a person , we need to learn associations between different human identities so that each person can be matched in the upcoming frames . We train a modified version of a transformer ( 38 ) to learn these associations across multiple human identities . 3.1 Human Mesh and Appearance Recovery ( HMAR ) . Our starting point for 3D feature extraction is the popular HMR architecture ( 17 ) . Given a bounding box of a person , HMR regresses the 3D pose and shape parameters of the person . While the 3D pose provides rich geometric information about the person , it throws away the appearance information , which is essential for tracking . To augment this , we extend HMR such that it can also recover the 3D appearance of the person by means of a texture image , which is a space that is viewpoint and pose invariant . We call this method HMAR , which stands for Human Mesh and Appearance Recovery . In order to recover the texture image , we create skip connections from the spatial features of the Resnet-50 backbone used in HMR and connect them to a new appearance head that maintains the spatial alignment of features through a series of convolution and bilinear upsampling layers ( Figure 3 ) . Please see the supplemental for details of the architecture . The appearance head takes these spatial features over multiple scales as input and is trained to predict the texture of the person , using the canonical SMPL texture mapping ( 25 ) . This is done in the form of appearance or texture flow in the spirit of ( 18 ; 51 ) . Specifically , we regress a texture flow F that has the same dimensions as the texture map . Each value of this flow indicates the coordinates ( x , y ) of the input image that the corresponding texel is sampled from . Eventually , the final texture map Itex = G ( I , F ) is produced by bilinear sampling G of the input image I . The predicted texture map is then used to render the textured 3D mesh . The model is trained to minimize the reconstruction loss after rendering the texture and geometry on the foreground pixels using L1 and perceptual loss . 3.2 3D Representations for tracking . The tracking performance is greatly affected by the cues and the type of representation used for association . Our embeddings are computed as natural outputs of the HMAR model as follows : Appearance : HMAR recovers the appearance of the person in the form of the texture image Itex . For tracking , we need to create a compact embedding for appearance , so we employ an auto-encoder to further encode this appearance . The auto-encoder takes the texture image as input and is trained to reconstruct it by going through a low dimensional bottleneck . This bottleneck is represented by a 512-dimensional vector ā , which is used as the representation of the appearance in our pipeline . 3D Pose : For the pose representation , we follow previous work ( 19 ; 48 ) and use the embedding p̄ ∈ R2048 of HMR as a representation of the pose of the person . 3D Location : Given an image with dimensions [ W , H ] , HMAR is applied on a square bounding box with center [ cx , cy ] and dimensions [ b , b ] . HMAR also predicts the local camera parameters [ s , tx , ty ] ( scale and 2D translation ) for the input bounding box . By assuming a constant focal length f for the image , we can recover the approximate 3D global translation of the person , which localizes the mesh in the view coordinate 3D space corresponding to the full image . This translation is expressed as : T = [ tx + 2cx−W sb , ty + 2cy−H sb , 2f sb ] . ( 1 ) Since we do not know the actual focal length , this is only an approximation of the metric location , but we can still recover consistent relative 3D positions of the people over a single video up to scale . For a more detailed description of the 3D location , we also extract the 3D keypoints of the person that are directly regressed from the recovered mesh , and we position them in the view coordinate 3D space by adding the translation T to them . This representation is strictly more informative than the 2D bounding box information , both because it is in 3D , and because it provides a localized information about the pose of the person than a single bounding box . We also concatenate the temporal information to the 3D keypoints in the spirit of ( 38 ) , which leads to space-time representation s̄ ∈ R90 .

Summary: This paper presents an approach for multi-people tracking in videos using 3D body representations based on the SMPL model and their associated texture maps. Given an input video, the 2D bounding boxes are assumed to be given and the paper focuses on estimating the tracklets of each person, i.e. solving the temporal association of the bounding boxes.  For this purpose, a first module building upon HMR [Kanazawa 2018] is used to extract the SMPL body parameters for each bounding box. Additionally, the 3D location and appearance of the bounding box is also regressed, the latter using UV texture images.  Then, temporal aggregation of features is performed using a Transformer model. Finally, the transformed features per bounding box are associated to existing tracklets using the Hungarian algorithm. The main contribution and novelty of this approach is in injecting 3D body representations into the tracking pipeline, jointly with the UV map appearance.  The approach is evaluated on three datasets, Posetrack, MuPoTs and AVA, and demonstrates improved performance w.r.t. state-of-the-art of 2D tracking methods.