Coarse-to-fine Animal Pose and Shape Estimation

1 Introduction . Animals play an important role in our everyday life and the study of animals has many potential applications in zoology , farming and ecology . Although great success has been achieved for modeling humans , progress for the animals counterpart is relatively slow mainly due to the lack of labeled data . It is almost impossible to collect large scale 3D pose data for animals , which prevents the direct application of many existing human pose estimation techniques to animals . Some works solve this problem by training their model with synthetic data [ 4 , 13 , 33 ] . However , generating realistic images is challenging , and more importantly , the limited pose and shape variations of synthetic data may result in performance drop when applied to real images . Other works avoid the requirement of 3D ground truth by making use of 2D weak supervision . For example , [ 2 , 12 ] render the estimated mesh or project the 3D keypoints onto the 2D image space , and then compute the losses based on ground truth silhouettes or 2D keypoints . This setting is more pragmatic since 2D animal annotations are more accessible . In view of this , we focus on learning 3D animal pose and shape with only 2D weak supervision in this work . 1https : //github.com/chaneyddtt/Coarse-to-fine-3D-Animal 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Reconstruction of the 3D mesh from a single RGB image with only 2D supervision is challenging because of the articulated structure of animals . Most existing works [ 4 , 2 ] rely on the parametric statistical body shape model SMAL [ 35 ] to reconstruct animal poses and shapes . Specifically , the low dimensional parameterization of the SMAL model makes it easier for deep learning to learn the high dimensional 3D mesh with 3,889 vertices for high-quality reconstruction . However , the shape space of the SMAL model is learned from 41 scans of toy animals and thus lacks pose and shape variations . This limits the representation capacity of the model and results in poor fittings of the estimated shapes to the 2D observations when applied to in-the-wild animal images , as illustrated in Figure 1 . Both 2D keypoints ( the left front leg in the first example and right ear in the second example ) and silhouettes rendered from the coarse estimations do not match well with the ground truth annotations ( rendered and ground truth silhouettes are denoted with green and red regions , and the yellow region is the overlap between them ) . To mitigate this problem , we design a two-stage approach to conduct a coarse-to-fine reconstruction . In the coarse estimation stage , we regress the shape and pose parameters of the SMAL model from an RGB image and recover a coarse mesh from these parameters . The coarse mesh is then used as a starting point by a MRGCN to predict a per-vertex deformation in the refinement stage . This combination of SMAL-based and vertex-based representations enable our network to benefit from both parametric and non-parametric representations . Some refined examples are shown in Figure 1 , where both keypoint locations and overall shapes are improved . To facilitate the non-parametric representation in the refinement stage , we consider a mesh as a graph structure and use a GCN [ 15 ] to encode the mesh topology ( i.e. , the edge and face information ) . Existing animal mesh reconstruction estimation works [ 2 , 12 , 33 ] rely on a global image feature to recover 3D meshes . However , we observe that the global feature fails to capture local geometry information that is critical to recover shape details . To solve this problem , we introduce a local feature extractor to retrieve a per-vertex feature and use it together with the global feature as the input of each node in the MRGCN . The combination of the vertex-level local feature and the image-level global feature helps to capture both overall structure and detailed shape information . Moreover , the representation power of traditional GCNs is limited by the small receptive field since each node is updated by aggregating features of neighboring nodes . To mitigate this , we design our MRGCN as an encoder-decode structure . Specifically , the downsampling and upsampling operations in the encoder-decoder structure enables a hierarchical representation of the mesh data , where features of different mesh resolutions can be exploited for mesh refinement . Our main contributions can be summarized as follows : 1 ) We propose a coarse-to-fine approach that combines parametric and non-parametric representations for animal pose and shape estimation . 2 ) We design an encoder-decoder structured GCN for mesh refinement , where both image-level global feature and vertex-level local feature are combined to capture overall structure and detailed shape information . 3 ) We achieve state-of-the-art results for animal pose and shape estimation on three animal datasets . 2 Related work . We review works on pose and shape estimation for both human and animals since they share similar ideas . We mainly discuss approaches that reconstruct 3D meshes which is the focus of this work . Human pose and shape estimation We first review model-based human pose estimation . This line of works are based on a parametric model of human body , such as SMPL [ 20 ] or SCAPE [ 1 ] , to reconstruct 3D human meshes . Bogo et al . [ 5 ] first propose a fully automatic approach SMPLify , which fits the SMPL model to 2D locations estimated by an off-the-shelf 2D keypoint detector . The shapes estimated by SMPLify are highly underconstrained although strong priors are added during the fitting process . To improve this , subsequent works incorporate more information during optimization , such as silhouettes [ 19 ] or multi-view videos [ 10 ] . The drawback of the optimizationbased approaches is that the iterative optimization process is quite slow and tends to fail when the 2D keypoints or silhouettes are noisy . On the other hand , regression-based approaches directly regress the body model parameters from an input image , which is more computationally efficient than optimization-based approaches . Kanazawa et al . [ 11 ] introduce an end-to-end framework for recovering 3D mesh of a human body by using unpaired 2D keypoints and 3D scans . Pavlakos et al . [ 23 ] propose to employ a differentiable renderer to project the generated 3D meshes back to 2D image space such that the network can be trained with 2D supervision . Other intermediate representations such as semantic segmentation map [ 22 ] or IUV map [ 31 ] are also introduced in order to improve the robustness . Recently , Kolotouros et al . [ 16 ] propose SPIN , where the optimization process is incorporated into the network training such that strong model-based supervision can be exploited . It has been observed that directly regressing model parameters is quite challenging because of the representation issues of 3D rotation [ 28 , 17 , 7 , 32 , 8 ] . Hence , some works focus on model-free based human pose and shape estimation , where they do not directly regress the model parameters . Varol et al . [ 28 ] proposed BodyNet , which estimates volumetric representation of 3D human with a Voxel-CNN . Kolotouros et al . [ 17 ] design a graph convolutional human mesh regression model to estimate vertex locations from a global image feature . Choi et al . [ 7 ] propose a graph convolutional system that recovers 3D human mesh from 2D human pose . We also apply GCN in our approach to refine the initial coarse estimation . However , we do not rely on 3D supervision as done in [ 17 , 7 ] , and our mesh refinement GCN has a different architecture . Our work is also related to PIFu [ 25 ] in terms of making use of local features to better capture shape details . Instead of using a complete non-parametric representation as in [ 25 ] , we combine the vertex-based representation with the SMAL-based representation such that our network can be weakly supervised with 2D annotations . Animal pose and shape estimation Animal pose estimation is less explored compared to human pose estimation mainly because of the lack of large scale datasets . It is almost impossible to collect thousands of 3D scan of animals as has been done in SMPL [ 20 ] . The first parametric animal model SMAL is recently proposed by Zuffi et al . [ 35 ] , which is learned from 41 scans of animal toys . The SMAL model is further improved in [ 34 ] by fitting the model to multiple images such that it can generalize to unseen animals . In the view of the lack of animal data , synthetic datasets are generated in [ 33 , 13 , 4 ] to train their networks . However , there is domain gap between the synthetic and real images in terms of appearance , pose and shape variations . Although the domain shift problem can be mitigated by using a depth map [ 13 ] or silhouette [ 4 ] as the input , these information are not always available in practice . More recently , Biggs et al . [ 2 ] collect 2D keypoint and silhouette annotations for a large scale dog dataset and train their network with those 2D supervision . The authors observe that the SMAL model failed to capture the large shape variations of different dog breeds , and propose to incorporate an EM algorithm into training to update the shape priors . However , it is not guaranteed that the priors are valid shapes since they are computed from the network predictions . We also work on the dog category following [ 2 ] in this work , and we overcome the limited representation capacity of the SMAL model by combining it with a vertex-based representation . 3 Our method . We propose a two-stage approach for coarse-to-fine animal pose and shape estimation . The overall architecture is illustrated in Figure 2 . The coarse estimation stage is based on the SMAL model , where we directly regress the pose , shape and translation parameters from the input image . We concurrently estimate the camera parameter to project the estimated 3D meshes onto the image space for the network supervision with the 2D annotations . Due to the limited representation capacity of the SMAL model , the coarse estimations may not fit well to the 2D evidences . We mitigate this problem by further refinement with a MRGCN in the second stage . The input feature for each node in the MRGCN consists of both image-level global feature and vertex-level local feature retrieved by a local feature extractor . The MRGCN is designed as an encoder-decoder structure , where the mesh downsampling and upsampling operations enable a hierarchical representation of the mesh data . Similar to the coarse estimation stage , the mesh refinement stage is also supervised with the 2D annotations . 3.1 Model-based coarse estimation . We use a parametric representation based on the SMAL model in the first stage . The SMAL model M ( β , θ , γ ) is a function of shape β , pose θ and translation γ . The shape parameter β is a vector of the coefficients of a PCA shape space learned from scans of animal toys . The pose parameter θ ∈ RN×3 is the relative rotation of joints expressed with Rodrigues vectors , where we use N = 35 joints in the kinematic tree following [ 33 , 2 ] . The translation γ is a global translation applied to the root joint . Given a set of model parameters Θ = { β , θ , γ } , the SMAL model returns 3D mesh vertices V = M ( β , θ , γ ) , where V ∈ RC×3 and C = 3 , 889 is the number of animal mesh vertices . The body joints of the model is defined as a linear combination of the mesh vertices J ∈ RN×3 = W×V , where W is a linear regressor provided with the SMAL model . As illustrated in the coarse estimation stage of Figure 2 , we first extract a global feature hg from the input image with an encoder . The global feature is fed into separate multi-layer perceptron ( MLP ) heads to predict the SMAL model parameters Θ′ = { β′ , θ′ , γ′ } and the camera parameter f . Following [ 33 , 2 ] , we use independent branches for the translation in the camera frame γ′xy and depth γ′z . The 3D meshes and joints can then be computed from the estimation parameters as : Vc = M ( β′ , θ′ , γ′ ) , J3D = W × Vc . ( 1 ) We further project the predicted 3D joints onto 2D image space with the estimated camera parameter and supervise them with the ground truth 2D keypoints : Lkp1 = ∥J2D −Π ( J3D , f ) ∥2 . ( 2 ) J2D and Π ( . , . ) denote the ground truth 2D keypoints and the camera projection function , respectively . Existing works apply the L1 [ 33 ] or L2 distance [ 2 , 23 ] to compute the loss for silhouettes . As illustrated in Figure 3 , we observe that the model trained with L1 or L2 loss tends to generate a 3D mesh that has a foreground region ( green region ) larger than that of the ground truth silhouette ( red region ) after rendering . We analyze that this may be due to the imbalance between foreground and background pixels in the training images . Note that we crop each image according to a bounding box during preprocessing . This causes the foreground object to occupy a larger area of an image and thus an imbalance of more foreground than background pixels . The imbalance problem may cause the network to estimate a mesh with larger size or larger focal length such that there will be more foreground pixels in the rendered silhouette . To solve this problem , we adopt the Tversky loss [ 26 ] from the semantic segmentation task : T ( P , G ; α , β ) = |PG| ( |PG|+ α|P\G|+ β|G\P | ) , ( 3 ) where |PG| denotes foreground pixels present in both the predicted binary mask P and the ground truth mask G. |P\G| denotes background pixels that are predicted as foreground and |G\P | vice versa , i.e . false positive and false negative predictions . α and β are hyperparamters to control the weights for false positive and false negative predictions . By setting α to a value larger than β , the Tversky loss penalizes more on false positive predictions , which corresponds to the foreground pixels that are not present in the ground truth silhouettes . Our silhouette loss can thus be represented as : Lsilh1 = 1− T ( S , R ( Vc , f ) , α , β ) , ( 4 ) where S represents the ground truth silhouette and R represents the mesh renderer , The network may predict unrealistic shapes and poses when trained with 2D supervision alone . In view of this , we encourage the predicted model parameters to be close to a prior distribution by computing the Mahalanobis distance between them : Lβ = ( β′ − µβ ) ⊤Σ−1β ( β ′ − µβ ) , Lθ = ( θ′ − µθ ) ⊤Σ−1θ ( θ ′ − µθ ) . ( 5 ) The prior distributions of the shape and pose are defined by the corresponding means { µβ , µθ } and variances { Σβ , Σθ } estimated from the SMAL training data . We also apply the pose limit prior [ 35 ] , denoted as Llim , to enforce valid pose estimations . Finally , the overall loss function for the coarse estimation stage is computed as : Lst1 = λkp1Lkp1 + λsilh1Lsilh1 + λβLβ + λθLθ + λlimLlim , ( 6 ) where λkp1 , λsilh1 , λβ , λθ and λlim are the weights to balance different loss terms .

Summary: This paper studies the problem of joint animal pose and shape reconstruction from a single image. This work is largely based on the literature [Ref 2] but adopts a coarse-to-fine mechanism for joint pose and shape learning.  The method extracts global embedding as before in the first stage. In the second stage, this work leverages a local feature extractor to provide pixel-aligned local features for mesh refinement. Specifically, the extracted local features are given as input to the U-Net to refine meshes at a different resolution. Experimental comparisons with existing work [Ref 2, 3, 32] have been conducted on the StanfordExtra dataset and BADJA dataset.