Coarse-to-fine Animal Pose and Shape Estimation

1 Introduction . Animals play an important role in our everyday life and the study of animals has many potential applications in zoology , farming and ecology . Although great success has been achieved for modeling humans , progress for the animals counterpart is relatively slow mainly due to the lack of labeled data . It is almost impossible to collect large scale 3D pose data for animals , which prevents the direct application of many existing human pose estimation techniques to animals . Some works solve this problem by training their model with synthetic data [ 4 , 13 , 33 ] . However , generating realistic images is challenging , and more importantly , the limited pose and shape variations of synthetic data may result in performance drop when applied to real images . Other works avoid the requirement of 3D ground truth by making use of 2D weak supervision . For example , [ 2 , 12 ] render the estimated mesh or project the 3D keypoints onto the 2D image space , and then compute the losses based on ground truth silhouettes or 2D keypoints . This setting is more pragmatic since 2D animal annotations are more accessible . In view of this , we focus on learning 3D animal pose and shape with only 2D weak supervision in this work . 1https : //github.com/chaneyddtt/Coarse-to-fine-3D-Animal 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Reconstruction of the 3D mesh from a single RGB image with only 2D supervision is challenging because of the articulated structure of animals . Most existing works [ 4 , 2 ] rely on the parametric statistical body shape model SMAL [ 35 ] to reconstruct animal poses and shapes . Specifically , the low dimensional parameterization of the SMAL model makes it easier for deep learning to learn the high dimensional 3D mesh with 3,889 vertices for high-quality reconstruction . However , the shape space of the SMAL model is learned from 41 scans of toy animals and thus lacks pose and shape variations . This limits the representation capacity of the model and results in poor fittings of the estimated shapes to the 2D observations when applied to in-the-wild animal images , as illustrated in Figure 1 . Both 2D keypoints ( the left front leg in the first example and right ear in the second example ) and silhouettes rendered from the coarse estimations do not match well with the ground truth annotations ( rendered and ground truth silhouettes are denoted with green and red regions , and the yellow region is the overlap between them ) . To mitigate this problem , we design a two-stage approach to conduct a coarse-to-fine reconstruction . In the coarse estimation stage , we regress the shape and pose parameters of the SMAL model from an RGB image and recover a coarse mesh from these parameters . The coarse mesh is then used as a starting point by a MRGCN to predict a per-vertex deformation in the refinement stage . This combination of SMAL-based and vertex-based representations enable our network to benefit from both parametric and non-parametric representations . Some refined examples are shown in Figure 1 , where both keypoint locations and overall shapes are improved . To facilitate the non-parametric representation in the refinement stage , we consider a mesh as a graph structure and use a GCN [ 15 ] to encode the mesh topology ( i.e. , the edge and face information ) . Existing animal mesh reconstruction estimation works [ 2 , 12 , 33 ] rely on a global image feature to recover 3D meshes . However , we observe that the global feature fails to capture local geometry information that is critical to recover shape details . To solve this problem , we introduce a local feature extractor to retrieve a per-vertex feature and use it together with the global feature as the input of each node in the MRGCN . The combination of the vertex-level local feature and the image-level global feature helps to capture both overall structure and detailed shape information . Moreover , the representation power of traditional GCNs is limited by the small receptive field since each node is updated by aggregating features of neighboring nodes . To mitigate this , we design our MRGCN as an encoder-decode structure . Specifically , the downsampling and upsampling operations in the encoder-decoder structure enables a hierarchical representation of the mesh data , where features of different mesh resolutions can be exploited for mesh refinement . Our main contributions can be summarized as follows : 1 ) We propose a coarse-to-fine approach that combines parametric and non-parametric representations for animal pose and shape estimation . 2 ) We design an encoder-decoder structured GCN for mesh refinement , where both image-level global feature and vertex-level local feature are combined to capture overall structure and detailed shape information . 3 ) We achieve state-of-the-art results for animal pose and shape estimation on three animal datasets . 2 Related work . We review works on pose and shape estimation for both human and animals since they share similar ideas . We mainly discuss approaches that reconstruct 3D meshes which is the focus of this work . Human pose and shape estimation We first review model-based human pose estimation . This line of works are based on a parametric model of human body , such as SMPL [ 20 ] or SCAPE [ 1 ] , to reconstruct 3D human meshes . Bogo et al . [ 5 ] first propose a fully automatic approach SMPLify , which fits the SMPL model to 2D locations estimated by an off-the-shelf 2D keypoint detector . The shapes estimated by SMPLify are highly underconstrained although strong priors are added during the fitting process . To improve this , subsequent works incorporate more information during optimization , such as silhouettes [ 19 ] or multi-view videos [ 10 ] . The drawback of the optimizationbased approaches is that the iterative optimization process is quite slow and tends to fail when the 2D keypoints or silhouettes are noisy . On the other hand , regression-based approaches directly regress the body model parameters from an input image , which is more computationally efficient than optimization-based approaches . Kanazawa et al . [ 11 ] introduce an end-to-end framework for recovering 3D mesh of a human body by using unpaired 2D keypoints and 3D scans . Pavlakos et al . [ 23 ] propose to employ a differentiable renderer to project the generated 3D meshes back to 2D image space such that the network can be trained with 2D supervision . Other intermediate representations such as semantic segmentation map [ 22 ] or IUV map [ 31 ] are also introduced in order to improve the robustness . Recently , Kolotouros et al . [ 16 ] propose SPIN , where the optimization process is incorporated into the network training such that strong model-based supervision can be exploited . It has been observed that directly regressing model parameters is quite challenging because of the representation issues of 3D rotation [ 28 , 17 , 7 , 32 , 8 ] . Hence , some works focus on model-free based human pose and shape estimation , where they do not directly regress the model parameters . Varol et al . [ 28 ] proposed BodyNet , which estimates volumetric representation of 3D human with a Voxel-CNN . Kolotouros et al . [ 17 ] design a graph convolutional human mesh regression model to estimate vertex locations from a global image feature . Choi et al . [ 7 ] propose a graph convolutional system that recovers 3D human mesh from 2D human pose . We also apply GCN in our approach to refine the initial coarse estimation . However , we do not rely on 3D supervision as done in [ 17 , 7 ] , and our mesh refinement GCN has a different architecture . Our work is also related to PIFu [ 25 ] in terms of making use of local features to better capture shape details . Instead of using a complete non-parametric representation as in [ 25 ] , we combine the vertex-based representation with the SMAL-based representation such that our network can be weakly supervised with 2D annotations . Animal pose and shape estimation Animal pose estimation is less explored compared to human pose estimation mainly because of the lack of large scale datasets . It is almost impossible to collect thousands of 3D scan of animals as has been done in SMPL [ 20 ] . The first parametric animal model SMAL is recently proposed by Zuffi et al . [ 35 ] , which is learned from 41 scans of animal toys . The SMAL model is further improved in [ 34 ] by fitting the model to multiple images such that it can generalize to unseen animals . In the view of the lack of animal data , synthetic datasets are generated in [ 33 , 13 , 4 ] to train their networks . However , there is domain gap between the synthetic and real images in terms of appearance , pose and shape variations . Although the domain shift problem can be mitigated by using a depth map [ 13 ] or silhouette [ 4 ] as the input , these information are not always available in practice . More recently , Biggs et al . [ 2 ] collect 2D keypoint and silhouette annotations for a large scale dog dataset and train their network with those 2D supervision . The authors observe that the SMAL model failed to capture the large shape variations of different dog breeds , and propose to incorporate an EM algorithm into training to update the shape priors . However , it is not guaranteed that the priors are valid shapes since they are computed from the network predictions . We also work on the dog category following [ 2 ] in this work , and we overcome the limited representation capacity of the SMAL model by combining it with a vertex-based representation . 3 Our method . We propose a two-stage approach for coarse-to-fine animal pose and shape estimation . The overall architecture is illustrated in Figure 2 . The coarse estimation stage is based on the SMAL model , where we directly regress the pose , shape and translation parameters from the input image . We concurrently estimate the camera parameter to project the estimated 3D meshes onto the image space for the network supervision with the 2D annotations . Due to the limited representation capacity of the SMAL model , the coarse estimations may not fit well to the 2D evidences . We mitigate this problem by further refinement with a MRGCN in the second stage . The input feature for each node in the MRGCN consists of both image-level global feature and vertex-level local feature retrieved by a local feature extractor . The MRGCN is designed as an encoder-decoder structure , where the mesh downsampling and upsampling operations enable a hierarchical representation of the mesh data . Similar to the coarse estimation stage , the mesh refinement stage is also supervised with the 2D annotations . 3.1 Model-based coarse estimation . We use a parametric representation based on the SMAL model in the first stage . The SMAL model M ( β , θ , γ ) is a function of shape β , pose θ and translation γ . The shape parameter β is a vector of the coefficients of a PCA shape space learned from scans of animal toys . The pose parameter θ ∈ RN×3 is the relative rotation of joints expressed with Rodrigues vectors , where we use N = 35 joints in the kinematic tree following [ 33 , 2 ] . The translation γ is a global translation applied to the root joint . Given a set of model parameters Θ = { β , θ , γ } , the SMAL model returns 3D mesh vertices V = M ( β , θ , γ ) , where V ∈ RC×3 and C = 3 , 889 is the number of animal mesh vertices . The body joints of the model is defined as a linear combination of the mesh vertices J ∈ RN×3 = W×V , where W is a linear regressor provided with the SMAL model . As illustrated in the coarse estimation stage of Figure 2 , we first extract a global feature hg from the input image with an encoder . The global feature is fed into separate multi-layer perceptron ( MLP ) heads to predict the SMAL model parameters Θ′ = { β′ , θ′ , γ′ } and the camera parameter f . Following [ 33 , 2 ] , we use independent branches for the translation in the camera frame γ′xy and depth γ′z . The 3D meshes and joints can then be computed from the estimation parameters as : Vc = M ( β′ , θ′ , γ′ ) , J3D = W × Vc . ( 1 ) We further project the predicted 3D joints onto 2D image space with the estimated camera parameter and supervise them with the ground truth 2D keypoints : Lkp1 = ∥J2D −Π ( J3D , f ) ∥2 . ( 2 ) J2D and Π ( . , . ) denote the ground truth 2D keypoints and the camera projection function , respectively . Existing works apply the L1 [ 33 ] or L2 distance [ 2 , 23 ] to compute the loss for silhouettes . As illustrated in Figure 3 , we observe that the model trained with L1 or L2 loss tends to generate a 3D mesh that has a foreground region ( green region ) larger than that of the ground truth silhouette ( red region ) after rendering . We analyze that this may be due to the imbalance between foreground and background pixels in the training images . Note that we crop each image according to a bounding box during preprocessing . This causes the foreground object to occupy a larger area of an image and thus an imbalance of more foreground than background pixels . The imbalance problem may cause the network to estimate a mesh with larger size or larger focal length such that there will be more foreground pixels in the rendered silhouette . To solve this problem , we adopt the Tversky loss [ 26 ] from the semantic segmentation task : T ( P , G ; α , β ) = |PG| ( |PG|+ α|P\G|+ β|G\P | ) , ( 3 ) where |PG| denotes foreground pixels present in both the predicted binary mask P and the ground truth mask G. |P\G| denotes background pixels that are predicted as foreground and |G\P | vice versa , i.e . false positive and false negative predictions . α and β are hyperparamters to control the weights for false positive and false negative predictions . By setting α to a value larger than β , the Tversky loss penalizes more on false positive predictions , which corresponds to the foreground pixels that are not present in the ground truth silhouettes . Our silhouette loss can thus be represented as : Lsilh1 = 1− T ( S , R ( Vc , f ) , α , β ) , ( 4 ) where S represents the ground truth silhouette and R represents the mesh renderer , The network may predict unrealistic shapes and poses when trained with 2D supervision alone . In view of this , we encourage the predicted model parameters to be close to a prior distribution by computing the Mahalanobis distance between them : Lβ = ( β′ − µβ ) ⊤Σ−1β ( β ′ − µβ ) , Lθ = ( θ′ − µθ ) ⊤Σ−1θ ( θ ′ − µθ ) . ( 5 ) The prior distributions of the shape and pose are defined by the corresponding means { µβ , µθ } and variances { Σβ , Σθ } estimated from the SMAL training data . We also apply the pose limit prior [ 35 ] , denoted as Llim , to enforce valid pose estimations . Finally , the overall loss function for the coarse estimation stage is computed as : Lst1 = λkp1Lkp1 + λsilh1Lsilh1 + λβLβ + λθLθ + λlimLlim , ( 6 ) where λkp1 , λsilh1 , λβ , λθ and λlim are the weights to balance different loss terms .

Summary: This paper presents a coarse-to-fine pipeline to recover animal pose and shape. They first predict SMAL parameters and then use an MRGCN to refine surface vertices to achieve better image alignment. Their MRGCN follows an encoder-decoder structure, which is the main difference against previous GCN based methods. Their method achieves state of the art results on StanfordExtra, Animal Pose and BADJA datasets.