Coarse-to-fine Animal Pose and Shape Estimation
Authors: Chen Li
Gim Hee Lee
Venue: NeurIPS 2021 Poster (2021)
Keywords: 3D animal pose and shape estimation
coarse-to-fine approach
weakly supervised

Abstract: Most existing animal pose and shape estimation approaches reconstruct animal meshes with a parametric SMAL model. This is because the low-dimensional pose and shape parameters of the SMAL model makes it easier for deep networks to learn the high-dimensional animal meshes. However, the SMAL model is learned from scans of toy animals with limited pose and shape variations, and thus may not be able to represent highly varying real animals well. This may result in poor fittings of the estimated meshes to the 2D evidences, e.g. 2D keypoints or silhouettes.  To mitigate this problem, we propose a coarse-to-fine approach to reconstruct 3D animal mesh from a single image. The coarse estimation stage first estimates the pose, shape and translation parameters of the SMAL model. The estimated meshes are then used as a starting point by a graph convolutional network (GCN) to predict a per-vertex deformation in the refinement stage. This combination of SMAL-based and vertex-based representations benefits from both parametric and non-parametric representations. We design our mesh refinement GCN (MRGCN) as an encoder-decoder structure with hierarchical feature representations to overcome the limited receptive field of traditional GCNs. Moreover, we observe that the global image feature used by existing animal mesh reconstruction works is unable to capture detailed shape information for mesh refinement. We thus introduce a local feature extractor to retrieve a vertex-level feature and use it together with the global feature as the input of the MRGCN. We test our approach on the StanfordExtra dataset and achieve state-of-the-art results. Furthermore, we test the generalization capacity of our approach on the Animal Pose and BADJA datasets. Our code is available at the project website.

Coarse-to-fine Animal Pose and Shape Estimation
Gim Hee Lee Department of Computer Science National University of Singapore [lic, gimhee -eemmeomp...........o.....oo.... nus edu.
Chen Li
Abstract
Most existing animal pose and shape estimation approaches reconstruct animal meshes with parametric SMAL model. This is because the low-dimensional pose and shape parameters of the SMAL model makes it easier for deep networks learn the -igh-dimensioaal animal meshes. However, the SMAL model is learned from scans of toy animals with limited pose and shape variations, and thus may not be able to represent highly varying real animals well. This may result in poor fittings the estimated meshes to the 2D evidences, e.g. 2D keypoints or silhouettes. To mitigate this problem, we propose coarse-to-fine approach to reconstruct 3D animal mesh from a single image. The coarse estimation stage first estimates the pose, shape and translation parameters of the SMAL model. The estimated meshes are then used as starting point by a graph convolutional network (GCN) to predict per-vertex deformation in the refinement stage. This combination of SMAL-based and vertex-base based representations benefits from both parametric and non-pan non-parametric representations. We design mesh refinement GCN (MRGCN) decoder structure with hierarchical feature representations to overcome the limited receptive field of traditional GCNs. Moreover, observe that the global image feature used by existing animal mesh reconstruction works is unable to capture detailed shape information for mesh refinement. We thus introduce local feature extractor to retrieve a vertex-level feature and use it together with the global feature the input of the MRGCN. We test our approach the StanfordExtra dataset and achieve state-of-the of-th----heeartttt results. Furthermore, we test the generalization capacity of our approach on the Animal Pose and BADJA datasets. Our code is available at the project website
encoder-
Introduction
Animals play important role in our everyday life and the study of animals has many potential applications in zoology, farming and ecology. Although great success has been achieved for modeling humans, progress for the animals counterpart is relatively slow mainly due to the lack of labeled data. almost impossible to collect large scale 3D pose data for animals, which prevents the direct application of many existing human pose estimation techniques to animals. Some works solve this problem by training their model with synthetic data [4, 13, 33]. However, generating realistic images challenging, and more importantly, the limited pose and shape variations of synthetic data may result in performance drop when applied to real images. Other works avoid the requirement of 3D ground truth by making or project the 3D keypoints onto the 2D image space, and then compute the losses based on ground truth silhouettes or 2D keypoints. This setting is more pragmatic since 2D animal annotations more accessible. In view of this, we focus on learning 3D animal pose and shape with only 2D weak supervision in this work.
of 2D weak supervision. For example, [2, 12] render the estimated mesh
fine-o----ii--- Animal
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Figure 1: Some results of our coarse-to-fin tonene approach. The first column is the input RGB images. The second to fourth columns are the estimated meshes in the camera view, the projected 2D keypoints and silhouettes from the coarse estimations. The fifth column to seventh columns the estimated meshes, the projected 2D keypoints and silhouettes from the refined estimations. Note that The red and green regions in the fourth and seventh columns represent the ground truth and rendered silhouettes, respectively, and the yellow region is the overlap between them.
Reconstruction the 3D mesh from a single RGB image with only 2D supervision is challenging because of the articulated structure of animals. Most existing works [4, 2] rely on the parametric statistical body shape model SMAL [35] to reconstruct animal poses and shapes. Specifically, the low dimensional parameterization of the SMAL model makes it easier for deep learning to learn the high dimensional 3D mesh with 3,889 vertices for high-quality reconstruction. However, the shape space of the SMAL model is learned from 41 scans of toy animals and thus lacks pose and shape variations. This limits the representation capacity of the model and results in poor fittings of the estimated shapes the 2D observations when applied in-the-wild animal images, illustrated in Figure Both 2D keypoints (the left front leg in the first example and right ear in the second example) and silhouettes rendered from the coarse estimations do not match well with the ground truth annotations (rendered and ground truth silhouettes are denoted with green and red regions, and the yellow region is the overlap between them). To mitigate this problem, we design two-stage approach to conduct coarse-to- srve-to-fing reconstruction. In the coarse estimation stage, we regress the shape and pose parameters of the SMAL model from an RGB image and recover a coarse mesh from these parameters. The coarse mesh is then used as starting point by MRGCN to predict per-vertex deformation in the refinement stage. This combination of SMAL-based and vertex -based representations enable our network to benefit from both parametric and non-parametric representations. Some refined examples are shown Figure I, where both keypoint locations and overall shapes are improved.
To facilitate the non-parametric representation in the refinement stage, we consider mesh as graph structure and use a GCN [15] to encode the mesh topology (i.e., the edge and face information). Existing animal mesh reconstruction estimation works [2, 12, 33] rely on global image feature to recover 3D meshes. However, we observe that the global feature fails capture local geometry information that is critical to recover shape details. To solve this problem, we introduce a local feature extractor to retrieve a per-vertex feature and use together with the global feature as the input of each node in the MRGCN. The combination of the vertex-level local feature and the image-level global feature helps to capture both overall structure and detailed shape information. Moreover, the representation power of traditional GCNs is limited by the small receptive field since each node is updated by aggregating features of neighboring nodes. To mitigate this, we design our MRGCN encoder-decode structure. Specifically, the downsampling and upsampling operations in the encoder-decoder structure enables a hierarchical representation of the mesh data, where features of different mesh resolutions can be exploited for mesh refinement.
Our main contributions can be summarized as follows: 1) We propose coarse-to to-fie approach that combines parametric and ponaamametiic representations for animal pose and shape estimation. 2) We design an encoder-decoder structured GCN for mesh refinement, where both image-level global feature and vertex-level local feature are combined to capture overall structure and detailed shape information. 3) We achieve state-of-the-arr results for animal pose and shape estimation on three animal datasets.

---

2
Related work
We review works on pose and shape estimation for both human and animals since they share similar ideas. We mainly discuss approaches that reconstruct 3D meshes which is the focus of this work.
Human pose and shape estimation We first review model-based human pose estimation. This line of works are based on a parametric model of human body, such as SMPL [20] or SCAPE []], to reconstruct 3D human meshes. Bogo el al.[5] first propose fully automatic approach SMPLify, which fits the SMPL model to 2D locations estimated by an off-the-shelff 2D keypoint detector. The shapes estimated by SMPLify are highly underconstrained although strong priors are added during the fitting process. To improve this, subsequent works incorporate more information during optimization, such as silhouettes [19] or multi-view videos [10]. The drawback of the optimization- based approaches is that the iterative optimization process is quite slow and tends to fail when the 2D keypoints or silhouettes are noisy. On the other hand, regression- bereesion-bbsed approaches directly regress the body model parameters from an input image, which is more computationally efficient than optimization-basee
approaches. Kanazawa el al.[11] introduce an end-to-end framework for recovering 3D mesh of a human body by using unpaired 2D) keypoints and 3D scans. Pavlakos et al.[23] propose to employ differentiable renderer to project the generated 3D) meshes back to 2D image space such that the network can be trained with 2D supervision. Other intermediate representations such as semantic segmentation map [22] or IUV map [31] are also introduced in order to improve the robustness. Recently, Kolotouros et al.[16] propose SPIN, where the optimization process is incorporated into the network training such that strong model- supervision can be exploited.
It has been observed that directly regressing model parameters is quite challenging because of the representation issues of 3D rotation [28, 17, 7, 32, 8J. Hence, some works focus on model-free based human pose and shape estimation, where they do not directly regress the model parameters. Varol et al|[28] proposed BodyNet, which estimates volumetric representation of 3D human with a VoxeI-CNN. Kolotouros et al.[17] design a graph convolutional human mesh regression model to estimate vertex locations from global image feature. Choi et al.[7] propose a graph convolutional system that recovers 3D human mesh from 2D human pose. We also apply GCN in our approach to refine the initial coarse estimation. However, we do not rely on 3D supervision
done in [17, 7], and our mesh refinement GCN has a different architecture. Our work is also related to PIFu [25] in terms of making use of local features to better capture shape details. Instead of using complete con-maaaettiic representation as in [25]. we combine the vertex- representation with the SMAL-based representation such that our network can be weakly supervised with 2D annotations.
Animal pose and shape estimation Animal pose estimation is less explored compared to human pose estimation mainly because of the lack of large scale datasets. It is almost impossible to collect thousands of 3D scan of animals as has been done in SMPL [20]. The first parametric animal model SMAL is recently proposed by Zuffi et al.[3]], which is learned from 41 scans of animal toys. The SMAL model is further improved in [34] by fitting the model to multiple images such that it can generalize unseen animals. In the view of the lack of animal data, synthetic datasets are generated in [33, 13, 4] to train their networks. However, there is domain gap between the synthetic and real images in terms of appearance. pose and shape variations. Although the domain shift problem can be mitigated by using a depth map [13] or silhouette[[4 [4] as the input, these information are not always available in practice. More recently, Biggs et al.[2] collect 2D keypoint and silhouette annotations for a large scale dog dataset and train their network with those 2D supervision. The authors observe that the SMAL model failed to capture the large shape variations of different dog breeds, and propose to incorporate an EM algorithm into training to update the shape priors, However, it is not guaranteed that the priors are valid shapes since they are computed from the network predictions. We also work on the dog category following [2] in this work, and we overcome the limited representation capacity of the SMAL model by combining it with a vertex-based representation.
3 Our method
We propose a two-stage approach for coarse-to-fine animal pose and shape estimation. The overall architecture is illustrated Figure 2. The coarse estimation stage is based on the SMAL model, where we directly regress the pose, shape and translation parameters from the input image. We
3

---

Model parameter
Encoder
parameter
Mi (4,)64)
projection
downsamplingg
Local feature extractor
2D supervision
Coarse estimation stage
Mesh refinement stage
Figure 2: Overall architecture of our network. Our network consists a coarse estimation stage and mesh refinement stage. The SMAL model parameters and camera parameter are regressed from the input image in the first stage for coarse reconstruction. This coarse mesh is further refined by an encoder-decoder structured GCN in the second stage, where the initial node feature consists of global feature, local feature and the initial 3D coordinate.
concurrently estimate the camera parameter to project the estimated 3D meshes onto the image space for the network supervision with the 2D annotations. Due to the limited representation capacity of the SMAL model, the coarse estimations may not fit well to the 2D evidences. We mitigate this problem by further refinement with MRGCN in the second stage. The input feature for each node in the MRGCN consists of both image-level global feature and vertex-level local feature retrieved by a local feature extractor. The MRGCN designed as an encoder-decoder structure, where the mesh downsampling and upsampling operations enable hierarchical representation of the mesh data. Similar to the coarse estimation stage, the mesh refinement stage is also supervised with the 2D annotations.
3.1 Model-based coarse estimation
use a parametric representation based on the SMAL model first stage. The SMAL model ((3, 7, is function of shape 3, pose 0 and translation y. The shape parameter 3 is vector of the coefficients 0 E IR Nx3 is the relative rotation of joints expressed with Rodrigues vectors, where
PCA shape space learned from scans animal toys. The pose parameter
N 35 joints in the kinematic tree following [33, 2]. The translation is a global translation applied to the joint. Given set of model parameters θ {B, 0, 7) the SMAL model returns 3D mesh vertices V V=M(3, 0, 7), where V E RCx3 and = 3, 889 the number of animal mesh vertices. The body joints of the model is defined linear combination of the mesh vertices J R WxV, where W is a linear regressor provided with the SMAL model.
As illustrated in the coarse estimation stage of Figure 2. first extract global feature hg from input image with an encoder. The global feature is fed into separate multi-layer perceptron (MLP) heads to predict the SMAL model parameters O' { 0', n'} and the camera parameter f. Following [33, 2], we use independent branches for the translation in the camera frame Ya and depth 7/5 The 3D meshes and joints can then be computed from the estimation parameters as:
V. MM33, 0', 8',)),
JED W V.
(1)
We further project the predicted 3D joints onto 2D image space with the estimated camera parameter and supervise them with the ground truth 2D keypoints:
App = 125 [[Ja,,
(2)
Jap and ([...) denote the ground truth 2D keypoints and the camera projection function, respectively.
Existing works apply the [33] or La distance [2, 23] to compute the loss for silhouettes. As illustrated in Figure 3, we observe that the model trained with L, or L-2 loss tends to generate a 3D mesh that has foreground region (green region) larger than that of the ground truth silhouette (red region) after rendering. We analyze that this may be due to the imbalance between foreground and

---

background pixels in the training images. Note that we crop each image according to bounding box during preprocessing. This causes the foreground object occupy a larger area of an image and thus an imbalance of more foreground than background pixels. The imbalance problem may cause the network to estimate a mesh with larger size or larger focal length such that there will be more foreground pixels in the rendered silhouette. To solve this problem, we adopt the Tversky loss [26] from the semantic segmentation task:
PG
a.a, (IPG) + al P())
(3)
where PG] denotes foreground pixels present in both the predicted binary mask P and the ground truth mask G. [P denotes back- ground pixels that are predicted as foreground and |G\P] vice versa, i.e. false positive and false negative predictions. paramters control the weights for false posi- tive and false negative predictions. By setting a to value larger than 3, the Tversky loss penal- izes more on false positive predictions, which corresponds to the foreground pixels that are not present the ground truth silhouettes. Our silhouette loss can thus be represented as:
and 3 are hyper-
Figure 3: Illustration of the rendered meshes oc- cupying more pixels than 2D observation when applying L loss for silhouette.
Lath = 1 1-5, a),
where represents the ground truth silhouette and R. represents the mesh renderer, The network may predict unrealistic shapes and poses when trained with 2D supervision alone. In view of this, we encourage the predicted model parameters to be close to prior distribution by computing the Mahalanobis distance between them:
(5) The prior distributions of the shape and pose are defined by the corresponding means (ug, (D) and variances (Ea, Σe} estimated from the SMAL training data. We also apply the pose limit prior [35], denoted as Liim. enforce valid pose estimations. Finally, the overall loss function for the coarse estimation stage is computed as:
L3 ( (2)) 1 1a). Lo (6' nO) 10' (so)-
Cat Expt + Aalnt + AgLa + do + Aum where Akpl. Asiht, Ag. 0g and Aum are the weights balance different loss terms,
3.2 Model-free mesh refinement
As illustrated in the coarse estimations in Figure 1, the 3D meshes estimated from the first stage may not fit well to the 2D observations due to the limited representation capability of the SMAL model. We thus use the coarse estimation V. as a starting point and further refine with our MRGCN. The GCN is able to encode the mesh topology and has been used in previous works [17, 7] for human pose and shape estimation. However, our MRGCN differs from [17, 7] several aspects: 1) We incorporate a vertex-level local feature into the feature embedding of each node; 2) We design our MRGCN as an encoder-decoder structure with skip connections to exploit features of different scales; 3) We supervise our MRGCN with only 2D weak supervision.
Combination of global and local features. The initial feature of each node in [17, is either combination of the global image feature and the initial 3D coordinate or a combination of the 3I) coordinate and the corresponding 2D joint location. In contrast, we introduce a vertex-level local feature to capture more detailed shape information. This is inspired by previous works [30, 29] with the exceptions that do not need ground truth 3D annotations or camera parameter. The local feature extraction process is illustrated in the left bottom part of Figure 2. We first project each mesh vertex of (I;; V/s 2) E V. onto 2D image space with the estimate camera parameter p: I[ f) and then retrieve the features at p: from all feature maps of the encoder. These features are concatenated and used as our local feature h. Finally, the input feature for each vertex is a combination of the global feature, the local feature and the initial 3D coordinates hς [hg. b, 24, Di, 2.). This combination of both global and local features helps to capture the overall structure and detailed shape information.
5

---

Encoder-decoder structured GCN. We define the graph on animal mesh as G = (V, where V represents the mesh vertices and A {0, 1,cxC represents an adjacency matrix defining edges of animal mesh. We adopt the graph convolutional layer = AX W proposed in [15], where A is symmetrically normalized from A, X is the input feature and W represents the weight matrix. In the original GCN, features of each node are updated by averaging over features of neighboring nodes. This results in small receptive field and limits the representation capacity. To mitigate this problem, we design our MRGCN as an encoder-decoder architecture to effectively exploit features of different mesh resolutions. Specifically, the graph encoder consists of consecutive graph convolutional layers and downsampling layers, and the decoder is a symmetry of the encoder architecture except that the downsampling layers replaced with upsampling layers. Following [24], the mesh downsampling is performed by using a transformation V = Q.V, where Qa E {0, 1 } represents the transformation matrix, and k are the number of mesh vertices before and after downsampling. The downsampling transformation matrix is obtained by contracting vertex pairs iteratively based on quadric matrices. We show an illustration of the downsampling process in the mesh downsampling block of Figure 2. Similarly, the upsampling is performed by Vi2 = Qu Vá. Furthermore, inspired by the stacked hourglass network [21], we add skip connections between the encoder and decoder layers to preserve the spatial information at each resolution.
Weakly supervised mesh refinement. Given the coarse estimation from the first stage, our MRGCN predicts a per-vertex deformation based on the input feature:
Ar: F(h)). where h = [bg. h1, ni, Vi. =x]-
The final refined mesh is obtained by:
(8) Similar to the first stage, we project the mesh vertices and 3D keypoints onto 2D image space with the estimated camera parameter f. and compute the 2D keypoints loss Lxp2 and silhouette loss Lame in the same way as Eq. (2) and Eq. (4). Note that we do not refine the camera parameter in the second stage as we postulate that the camera parameter relies more on the global features, e.g. the global scale. The shape and pose prior in the first stage can not be used we directly regressing the vertex deformation. Without any constraint, the network is susceptible to large deformations in its predictions in order to align perfectly with the 2D evidences. This may result in unrealistic output meshes. In view of this problem, we add a Laplacian regularizer to prevent large deformations such that only fine-grained details are added upon the coarse estimation. Specifically, we encourage the Laplacian coordinates of the refined mesh to be the same as the coarse mesh by enforcing:
V Ve + AV, where AV = [Av1, Ar2 Avc].
Lapp = Σ 1l 5ufll?, where dv, =
1
Σ Vj- jEN(1)
d.
of and of represent the vertex in the refined and coarse meshes, respectively. du; is the Laplacian coordinate of vs which is the difference between the original coordinate and the center of its neighbors in the mesh. Finally, the training loss for the refinement stage can be represented as:
Laz Akp + Caih2 + Atap Cmp:
(10)
where Akp2, Aaina and Aaa are the weights for corresponding losses.
4 Experiments
Network details. The structure of the encoder in the first stage is similar to [33, 2]. We extract the features of an input image of size 224 x 224 x with a Resnet50 [9] module, which is followed by convolutional layer with group normalization and leaky ReL.U. We further add two fully connected layers to obtain the global feature. This global feature is fed into different MLP branches to estimate the SMAL model parameters θ and the camera parameter f. All intermediate feature maps of the encoder are resized and concatenated for the local feature extraction of the refinement stage. The encoder of our MRGCN consists of consecutive graph convolutional blocks and downsampling layers, where the mesh graph is downsampled by factor of 4 each time. As in [17], the graph convolutional block is similar to the Bottleneck residual block in [9], where the 1 X 1 convolutions are replaced by per-vertex fully connected layers and the Batch Normalization is replaced by Group Normalization. Similarly, the decoder consists of consecutive graph convolutional blocks and upsampling layers to reconstruct 3D mesh in the original resolution. The skip connections are added between the encoder and decoder layers with concatenation operation.

---

Table 1: IOU and @ scores for the StanfordExtra dataset. 'GT" and 'Pred' represent whether ground truth or predicted keypointss silhouettes are used during test, and '-' represents that the corresponding information is not required. 'Ours-coarse' and 'Ours' represent the results of the coarse and refined estimations, respectively. (Best results in bold)
Method
Keypoints
Silhouette
IOU
PCK@0.1.
Avg
Legs
Tail
Ears
Face
3D-M [35]
Pred
Pred
69.9
69.7
68.3
68.0
57.8
93.7
3D-M
GT
GT
71.0
75.6
74.2
89.5
60.7
98.6
3D-M
GT
Pred
70.7
75.5
74.1
88.1
60.2
98.7
3D-M
Pred
GT
70.5
70.3
69.0
69.4
58.5
94.0
CGAS [4]
CGAS
Pred
63.5
28.6
30.7
34.5
25.9
24.1
CGAS
CGAS
GT
64.2
28.2
30.1
33.4
26.3
24.5
WLDO [2]
74.2
78.8
76.4
63.9
78.1
92.1
72.5
77.0
75.9
55.3
76.1
89.8
Ours
81.6
83.4
81.9
63.7
84.4
94.4
Training details. We train our network in three stages. We first train the coarse estimation part with all losses in Lui except the silhouette loss Lailll for 200 epochs. We did not include the pose limit prior Ltim this stage as find the network tends to estimate invalid poses only when apply the silhouette loss. Subsequently, the mesh refinement part is trained with the keypoints loss Lipa for 10 epochs with the coarse estimation part kept frozen. Finally, we train the whole network with both Lai and La2 for 200 epochs. We do not include the silhouette loss in the first two stages because it has been observed [4, 2] that the silhouette loss can lead the network to unsatisfactory local minima if applied too early, Our model is implemented with Pytorch, and we use the Adam optimizer with learning rate of 10 in the first two stages and of 10 10-5 in the third stage. The whole training process takes around 30 hours on RTX2080Ti GPU.
4.1 Datasets and evaluation metrics.
We train our model the StanfordExtra dataset [2], and test the StanfordExtra, the Animal Pose [6] and the benchmark animal dataset of joint annotations (BADJA) [4] datasets.
StanfordExtra dataset. The StanfordExtra dataset is a newly introduced large-scale dog dataset with 2D keypoint and silhouette annotations. The images are taken from an existing dataset for fine-grained image categorization [14], which consists of 20,580 images of dogs taken in the wild and covers 120 dog breeds. Images with too few keypoints within the field-of-view are removed. Each dog-breed divided into training and testing split with a ratio of 1.
Animal Pose and BADJA datasets. The Animal Pose dataset is recently introduced for 2D animal pose estimation, and the BADIA dataset is derived from the existing DAVIS video segmentation dataset. We use these two datasets to test the generalization capacity of our network.
Evaluation metrics. Following previous works [33, 2). we use two evaluation metries: 1) Percent- age of Correct Keypoints (PCK), and 2) intersection-over of joints that are within a normalized distance to the ground truth locations, and we use it to evaluate the quality of the estimated pose. We normalize the joint distance based on the square root of 2D silhouette area and use a threshold of 0.15, which we denote as PCK @0. 15. Following [2], we also evaluate PCK @0.15 different body parts, including legs, tail, ears and face. The IOU measures the overlap between the rendered silhouettes and the ground truth annotations, and we use it to evaluate the quality of the estimated 3D shapes.
(IOU). PCK computes the percentage
4.2 Results on the StanfordExtra dataset
We show the results of our approach and compare with existing works in Table We take the results of the existing works from [2]. 3D-M [35] is an optimization- based approach that fits the SMAL model the 2D keypoints and silhouette by minimizing an energy term. The optimization is run for
each image, which means 2D keypoints and silhouettes are needed during test. We list the results of 3D-M when ground truth (GT) or predicted (Pred) keypoints and silhouette are used. CGAS [4] first predicts 2D keypoints from silhouette with their proposed genetic algorithm, denoted as CGAS,
7

---

Table 2: IOU and PCK @0.15 scores for the Animal Pose dataset. (Best results in bold)
Method
Keypoints
Silhouette
IOU
PCK @0.15
Avg
Legs
Tail
Ears
Face
3D-M [35]
Pred
Pred
64.9
59.2
55.7
56.9
61.3
86.7
WLDO [2]
67.5
67.6
60.4
62.7
86.0
86.7
Ours-coarse
67.5
62.0
57.1
45.1
75.8
78.9
Ours
75.7
67.8
62.2
45.1
86.6
87.8
and then run an optimization step to match the 2D evidences. Both WLDO [2] and our approach are regression-based, hence do not need 2D keypoints or silhouette during test. We show PCK @0.15 and IOU for both coarse (ours-coarse) and refined (ours) estimaations. As shown in Table 1, our approach outperforms the optimization-base
approaches [35, 4] in terms of both IOU and average PCK @ 00.15 even when ground truth 2D annotations are used. We also outperform the regression- based approach [2] for all metrics, with a relative improvement of 10.0% for IOU and 5.8% for average PCK @0.15. The fact that the refined estimation has higher scores than the coarse estimation demonstrates the effectiveness of our coarse-to-fin strategy. This is also evident from the qualitative results in Row 1-3 of Figure 4, where both keypoint locations (e.g. keypoints on the right rear legs in the first row and on the face in the second row) and silhouettes are improved (more yellow pixels and less red or green pixels) after the refinement step.
4.3 Results on the Animal Pose and BADJA datasets
To test the generalization capacity, we directly apply our model trained on StanfordExtra to the Animal Pose and BADJA datasets. The results for the Animal Pose dataset are shown in Table 2. We take the performance of 3D-M and WLDO from [2]. Note that the results of 3D-M is obtained by using the predicted keypoints and silhouettes for fair comparison. We can see that approach outperforms the state-of oatte-of-thearr approach [2] for all metrics, especially for the IOU score. This demonstrates the importance of the refinement step in improving the coarse estimations since the architecture of our coarse estimation stage is similar to WLDO except that we do not apply the EM updates. We show the qualitative results this dataset in Row 4-6 of Figure 4.
We also evaluate on the BADJA dataset and compare with the state-of-the-art WLDO in Table The results of WLDO are obtained by rumning the publicly available checkpoint. Note that there is large domain gap between the BADJA and the StanfordExtra datasets in terms of animal pose and camera viewpoint. Most images in the BADIA dataset are taken from a video where a dog is running, and sometimes the viewpoint is from the back. For the StanfordExtra dataset, most images are taken from front view and the dogs are in a standing, siting or prone pose. We show some examples of the BADJA dataset in Row 7-9 of Figure 4 for illustration. We see from Table that our approach still obtains reasonable scores given the domain gap between these two datasets. improving over WLDO by 10.8% for IOU and 11.3% for average PCK 05. We also notice that the performance gap between the coarse and fine estimations is larger than that of the other two datasets. This may be attributed to the domain shift of the input image, which causes the coarse estimation to be inaccurate. Nonetheless, this inaccuracy can be reduced after refinement since the refinement step does not directly rely on the input image. We can observe from the qualitative results (Row 7-8) that keypoinnss on the legs aligned much better after refinement. This also demonstrates the advantage of our coarse- to fine approach in the presence of domain gap. We show failure cases in the last two rows of Figure 4, where the camera looking at the back of the animal or there is severe occlusion.
Table 3: IOU and 10.15 for the BADIA dataset. Method
IOU
PCK @ @0.15
Avg
Legs
Tail
Ears
Face
WLDO [2]
65.0
48.6
40.4
78.2
55.2
76.5
coarse
59.6
42.5
33.7
57.5
63.4
79.2
Ours
72.0
54.1
47.6
76.1
66.2
74.4
4.4 Ablation study
Table shows the results the ablation studies on the StandfordExtra dataset. We remove one component from our full pipeline (Full) at each time to evaluate the corresponding contribution.
8

---

Figure 4: Qualitative results on the StanfordExtra, Animal Pose and BADJA datasets. The first column is the input images, the second to fourth columns and fifth to seventh columns are the coarse and refined estimations, respectively. The last column is the 3D mesh in another view.
We first evaluate our coarse-to-fine strategy by removing the MRGCN (-MR), which corre- sponds to the results of 'Ours-coarse' in the previous tables. We then the contribution of the local features by removing it from the input of the MRGCN (-LF). This gives similar results with the coarse estimation, which means that the global feature alone not able to improve the initial estimations, Our MRGCN is designed
Table Ablation study for each component.
Method IOU
PCK @0.15
Avg
Legs
Tail
Ears
Face
Full
81.6
83.4
81.9
63.7
84.4
94,4
-MR
72.5
77.0
75.9
55.3
76.1
89.8
-LF
73.3
76.9
76.1
57.0
75.1
89.1
-ED
79.4
80.2
79.2
59.3
79.9
91.5
-TL
81.1
82.5
81.0
65.2
82.9
92.8

---

an encoder-decoder structure. We verify this design by replacing the structure with a traditional GCN (-ED). where the mesh vertices are processed in the same resolution across all layers. We also evaluate the contribution of the Tversky loss by replacing it with commonly used T1 loss (-TL.. We can see that the performance drops when each part is removed from the full pipeline. This verifies the contribution of each component.
5 Conclusion
We introduce coarse-to-fine approach for 3D animal pose and shape estimation. Our two-stage approach is combination of SMAL-based and vertex-based representations that benefits from both parametric and non-parametric representations. We design the mesh refinement GCN as an encoder-decoder structure, where the input feature of each node consists of both image-level and vertex-level features. This combination global and local features helps to capture overall structure and detailed shape information. Our approach outperforms both optimization- -ased and regression- based approaches on the StanfordExtra dataset. Results on the Animal Pose and BADJA datasets verify the generalization capacity of our approach unseen datasets.
Limitations and potential negative societal impacts
There still exist challenging cases not handled by our work, such as scenarios where the camera looks at the back of the animal with severe occlusions. A naive adoption of our approach to monitor animal health may result in potential negative consequences, i.e. the inaccurate shape reconstruction of the animals in this scenario can provide misleading information for the farmer animal breeder. Nonetheless, the problem can be mitigated by collecting animal images with more variations of camera views and animal poses by synthesizing novel images with existing datasets [18]. Moreover, an alternative solution for the occlusion problem is to model the ambiguity by predicting multiple hypotheses [3] by predicting a probability distribution over the SMAL parameters [27].
Acknowledgement
This research is supported in part by the National Research Foundation, Singapore under its AI Singapore Program (AISG Award No: AISG2-RP-2020-012 and the Tier grant M0ET2EP20120- 0011 from the Singapore Ministry of Education.
References
[1] Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. Scape: shape completion and animation of people. In ACM SIGGRAPH 2005 Papers, pages 408-416. 2005.
[2] Benjamin Biggs, Oliver Boyne, James Charles, Andrew Fitzgibbon, and Roberto Cipolla. Who left the dogs out?: 3D animal reconstruction with expectation maximization in the loop. In ECCV, 2020.
[3] Benjamin Biges, Sébastien Ehrhadt, Hanbyul Joo, Benjamin Graham, Andrea Vedaldi, and David Novotny. 3d multi-bodies: Fitting sets of plausible 3d human models to ambiguous image data. arXiv preprint arXiv:2011 1.09800 2020.
[4] Benjamin Biggs Thomas Roddick, Andrew Fitzgibbon, and Roberto Cipolla. Creatures great and SMAL: Recovering the shape and motion of animals from video. In ACCV, 2018
[5] Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black. Keep it smpl: Automatic estimation of 3d human pose and shape from a single image. In European conference compuder vision, pages 561-578. Springer, 2016.
[6] Jinkun Cao, Hongyang Tang, Hao-Shu Fang. Xiaoyong Shen, Cewu Lu, and Yu- Yu-Wing Tai. Cross-domain adaptation for animal pose estimation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 9498-9507, 2019.
[7] Hongsuk Choi, Gyeongsik Moon, and Kyoung Mu Lee. Pose2mesh: Graph convolutional network for 3d human pose and mesh recovery from a 2d human pose. In European Conference on Computer Vision, pages 789-787. Springer, 2020.
10

---

[8] Valentin Gabeur. Sébasiébatien Franco, Xavier Martin, Cordelia Schmid, and Gregory Rogez. Moulding, humans: con-parametric 3d human shape estimation from single images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2232-2241, 2019.
[9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[10] Yinghao Huang, Federica Bogo, Christoph Lassner, Angjoo Kanazawa, Peter V Gehler, Javier Romero, Ijaz Akhter, and Michael Black. Towards accurate marker-less human shape and pose estimation over time. In 2017 international conference on 3D vision (3DV), pages 421-430 IEEE, 2017.
[11] Angjoo Kanazawa, Michael Black, David W Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7122-71 71
[12] Angjoo Kanazawa, Shubham Tulsiami, Alexci A. Efros, and Jitendra Malik. Learning category-specifie mesh reconstruction from image collections. In ECCV, 2018.
[13] Sinead Kearney, Wenbin Li, Martin Parsons, Kwang In Kim, and Darren Cosker. Rgbd-dog: Predicting canine pose from rghd sensors. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[14] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine-graie fine-grained image categorization: Stanford dogs. In Proc. CVPR Workshop on Fine-Grained Visual Categorization (FGVC), volume 2. Citeseer, 2011.
[15] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017.
[16] Nikos Kolotouros, Georgios Pavlakos, Michacl J Black, and Kostas Daniilidis. Learning reconstruct 3d human pose and shape via model-fitting in the loop. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2252-2261, 2019.
[17] Nikos Kolotouros, Georgios Pavlakos, and Kostas Daniilidis. Convolutional mesh regression for single- image human shape reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4501-4510, 2019.
[18] Jogendra Nath Kundu, Siddharth Seth, Varun Jampani, Mugalodi Rakesh, R Venkatesh Babu, and Anirban Chakraborty. Self-supervised 3d human pose estimation via part guided novel image synthesis. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6152-6162, 2020.
[19] Christoph Lassner, Javier Romero, Martin Kiefel, Federica Bogo, Michael J Black, and Peter V Gehler. Unite the people: Closing the loop between 3d and 2d human representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6050-6059, 2017.
[20] Matthew Loper, Naurcen Mahmood, Javier Romero, Gerard Pons-Moli and Michael Black. Smpl: A skinned multi-person linear model. ACM transactions on graphics (TOG), 34(6):1-16, 2015.
[21] Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In European conference computer vision, pages 483-499. Springer, 2016
[22] Mohamed Omran, Christoph Lassner, Gerard Pons-Moll, Peter Gehler, and Bernt Schiele. Neural body fitting: Unifying deep lcarning and model based human pose and shape estimation. In 2018 international conference on 3D vision (3DV), pages 484-494. IEEE, 2018
[23] Georgios Pavlakos, Luyang Zhu Xiaowei Zhou, and Kostas Daniilidis. Learning to estimate 3d human pose and shape from single color image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 459-468, 2018
[24] Anurag Ranjan, Timo Bolkart, Soubhik Sanyal, and Michael J Black. Generating 3d faces using convolu- tional mesh autocncoders. In Proceedings of the European Conference on Computer Vision (ECCV), pages 704-720, 2018
[25] Shunsuke Saito, Zeng Huang. Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2304 2314, 2019.

---

[26] Seyed Sadegh Mohseni Salehi, Deniz Erdogmus, and Ali Gholipour. Tversky loss function for image segmentation using 3d fully convolutional deep networks. In International workshop machine learning in medical imaging, pages 379-387. Springer, 2017.
[27] Akash Sengupta, Ignas Budvytis, and Roberto Cipolla. Probabilistic 3d human shape and pose estimation multiple unconstrained images the wild. Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16094-16104. 2021.
[28] Gul Varol, Duygu Ceylan, Bryan Russell, Jimei Yang, Ersin Yumer, Ivan Laptev, and Cordelia Schmid. Bodynet: Volumetric inference 3d human body shapes. In Proceedings of the European Conference on Computer Vision (ECCV), pages 20 36, 2018.
[29] Nanyang Wang, Yinda Zhang. Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings of the European Conference on Computer Vision (ECCV), pages 52 67, 2018.
[30] Weiyue Wang. Xu Qiangeng, Duygu Ceylan, Radomir Mech, and Ulrich Neumann. Dism: Deep implicit surface network for high-quality single-view 3d reconstruction. In NeurlPS, 2019.
[31] Yuanlu Xu, Song-Chur Zhu, and Tony Tung. Denserac: Joint 3d pose and shape estimation by dense mader-and-compare In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 7760 7770, 2019.
[32] Wang Zeng, Wanli Ouyang, Ping Luo, Wentao Liu, and Xiaogang Wang. 3d human mesh regression with dense correspondence. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7054-7063, 2020.
[33] Silvia Zuffi, Angjoo Kanazawa, Tanya Berger-Wood and Michael Black. Three-d safari: Learning to estimate zebra pose, shape, and texture from images" in the wild". In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 5359-5368, 2019.
[34] Silvia Zuffi, Angjoo Kanazawa, and Michael Black. Lions and tigers and bears: Capturing non non-right rirrriiid, 3d, articulated shape from images. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 3955-3963, 2018.
[35] Silvia Zuffi, Angjoo Kanazawa, David Jacobs, and Michael J. Black, 3D menagerie: Modeling the 3D shape and pose of animals. In Proceedings IEEE Conference Computer Vision and Pattern Recognition (CVPR) 2017, pages 5524-5532. IEEE, July 2017.
12


Reviews:
[{"review_id": "uhymaw9jViJ", "reviewer": "Reviewer_xmMe", "summary": "- Paper focuses on 3D dog reconstruction from a single monocular image\n- A coarse 3D dog estimate is generated using an existing pipeline (WLDO, SMALST), although with Tversky loss added to prevent oversized estimates (a known problem in prior work). \n- The paper introduces an encoder-decoder graph convolutional network which incorporates vertex-level local image features for applying free-form refinements to the the vertices of this coarse estimate.\n- These are supervised using only weak 2D supervision and are shown effective for modelling dog shapes (which are very diverse).\n- Ablation study + results on StanfordExtra, BADJA and AnimalPose show state-of-the-art performance.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n- Paper presents an effective approach to 3D dog reconstruction. This is a hard task due to shape diversity in the category, and approaches must rely on weak 2D supervision since 3D annotations are unavailable. Additionally the 3D morphable model (SMAL) is of low detail, having been constructed from scanned artist toys rather than real subjects. \n- I'm not fully convinced of the novelty of all parts, but the combination of ideas (refining coarse prediction with encoder-decoder GCN that incorporates local vertex information) is novel and particularly suitable for animal reconstruction. The contribution of these components are supported by results, although only using 2D metrics.\n- Results appear of high quality, and are shown on diverse subjects with a range of challenging poses. However, results are shown only from the front view.\n- Tversky loss from semantic segmentation task is proposed to avoid oversized SMAL estimates -- a problem in previous work.\n- Paper is well written and easy to follow. Code was provided at time of submission.\n\n**Weaknesses**\n- The novelty appears limited. [Fully Convolutional Mesh Autoencoder using Efficient Spatially Varying Kernels, Zhou et al.](https://arxiv.org/pdf/2006.04325.pdf) seem to describe an encoder-decoder graph network with up+down sampling layers for SMPL (see Figure 2 in the link). [Pixel2Mesh, Wang et al.](https://arxiv.org/abs/1804.01654) has demonstrated using local features in a GCN and [Convolutional Mesh Regression, Kolotouros et al. CVPR 2019](https://arxiv.org/abs/1905.03244) has demonstrated using a GCNs as a refinement network. Additionally, [Lions and Tigers and Bears; Zuffi et al](https://files.is.tue.mpg.de/black/papers/zuffiCVPR2018.pdf) show that using free-form vertex deformation (albeit via energy minimization) improves the accuracy of animal shape fitting. This seems to leave the novelty of this paper the combination of these techniques, their application to single-image monocular 3D dog reconstruction and supervising these architectures with weak 2D supervision only.   \n\n- The authors compare their method on numerous 2D datasets (StanfordExtra, BADJA, AnimalPose) demonstrating quant/qual improvements over state of the art approaches. However, evaluating a 3D reconstruction using 2D PCK/IOU metrics, particularly rendered from the camera viewpoint is not ideal. For this reason, it is a shame the paper does not evaluate on 3D metrics which are available in RGBD-Dog, Kearney et al. CVPR 2020. \n\n- The paper contains only figures showing the reconstructed mesh from the camera's viewing direction. This can be misleading and causes some doubt when evaluating qualitative results. Reversed views of the mesh are needed.\n\n- L158-163. Claim that oversized SMAL estimates caused by image foreground occupying a large area of the image after the cropping + this imbalance causes the network to estimate a mesh with larger size or larger focal length. I don't follow the logic here. The silhouette error is minimized if the predicted mesh matches the silhouette exactly. Why would preprocessing cause difficulties? However, I agree that the Tversky loss which penalizes false positive predictions, would resolve systematic issue of oversized predictions.\n\n- L35 - Claim \"we generate synthetic data for all animals, especially endangered species?\" This appears untrue; the SMALST paper generates synthetic data on the Grey's zebra (an endangered zebra species) and obtains accurate results. \n\n- Paper title is 'ANIMAL pose and shape estimation' but results are only shown on dogs. Unless results are shown on other categories, this perhaps should be 'Dog Pose and Shape estimation'. \n\n**Minor**\n- The authors refine the vertices using graph convolutions. However, another implementation (albeit to refine parameters) shown in [Human Mesh Recovery; Kanazawa et al., CVPR 2018] is to use [recurrent steps](https://github.com/nkolot/SPIN/blob/master/models/hmr.py#L140). A discussion on the differences would be useful and interesting.\n- Paper shows a failure case which they explain is due to the camera looking at the back of the animal. This is a hard case, and the authors put it down to a lack of views in the training data. However, I wonder if this could be better handled in modelling the ambiguity in 3D reconstructions. Perhaps work like [Probabilistic 3D Human Shape and Pose Estimation from Multiple\nUnconstrained Images in the Wild; Sengupta et al] and [3D Multibodies; Biggs et al. NeurIPS 2020] would help? \n- The paper proposes the Laplacian regularizer to constrain the free-form vertex deformation step. This differs from ARAP used in [Lions \\& Tigers \\& Bears, Zuffi et al, CVPR 2018]. Could the authors explain the choice here?\n- Is your training/test split the same as published in the WLDO Github repository?\n\n- Some typos e.g. L144  `\"global feature is FED\", L158 \"we RECKON\" (informal language), L266 - capitalize WE. Possibly others, have not checked exhaustively. \n\n**Review score**\n\nThis paper demonstrates an effective approach for a challenging 3D reconstruction task, focusing on an articulated category with limited training data and not a huge amount of prior work. I'm not convinced of the novelty of any of the individual components (encoder-decoder GCN, free-form refinement, local vertex features, Tversky loss all seem to be prior art). However I believe the combination of these techniques to be non-trivial, and they are shown to be effective in modelling dog shapes (which are much more diverse than humans and have only 2D supervision).\n\nThe experimental section would benefit from having side-views and evaluation on a 3D dataset (e.g. RGBD-Dog) which is now available. However, previous animal papers have not conducted a 3D evaluation so I don't think this is enough to reject this paper.\n\nAll things considered, I believe this paper is interesting and pushes the field of 3D animal reconstruction forward. I am on the fence, but overall I consider this paper just about worthy of acceptance to NeurIPS.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations indicated above. Societal impact is discussed briefly in conclusions.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hrs"}, "extra_text": {}}, {"review_id": "H2NLflVG0G-", "reviewer": "Reviewer_HKW6", "summary": "This paper presents a coarse-to-fine pipeline to recover animal pose and shape. They first predict SMAL parameters and then use an MRGCN to refine surface vertices to achieve better image alignment. Their MRGCN follows an encoder-decoder structure, which is the main difference against previous GCN based methods. Their method achieves state of the art results on StanfordExtra, Animal Pose and BADJA datasets. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Originality \nAs far as I know, coarse-to-fine pipeline is very common in human pose and shape estimation area, and GCN is well investigated for human body or hand prediction. A refinement stage could certainly reduce the reconstruction error. So, the overall novelty of this paper seems low. \nHowever, it is the first time for me to see an encoder-decoder GCN structure, and the authors prove its ability in their ablation study.\n\n## Quality \nThis paper is easy to read except some spelling mistake (e.g. L.144, feed -> fed). They do a lot of experiments to show the effectiveness of each part of their pipeline. Their quantitative results achieve the best.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "However, I feel like the qualitative results are not satisfactory, see my comments below.  \n\n## Other Questions \n1.\tDoes this paper only estimate pose and shape of dogs? Although this paper claims ‘animal’ pose and shape estimation, I can see only dogs in Fig.1 and Fig.4. The training dataset StanfordExtra contains only dogs. I do not know their performance on other animals. \n2.\tIn Fig.4, these dogs have very different body shapes and surface geometry, however, I find the shape results in Column 5 are quite similar. As a comparison, qualitative results in WLDO (Fig.1 of their paper) demonstrate richer shape variability. I think this is a weakness of qualitative results.\n3.\tI would like to see an alternative view visualization of the models in Fig.4. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "ADtVVf4ejvK", "reviewer": "Reviewer_gzXJ", "summary": "This paper presents a method for estimating the pose and shape of animals from images. The paper addresses limitations of previous approaches that cause misalignments and 2D keypoint locations. The main idea is to initialize using a parametric animal model and subsequently use a graph convolution network (GCN) to optimize for vertex positions to better fit shape segmentation. Experimental results how improved performance.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I would like to begin by listing several things I like with the paper.\n\n- The problem addressed is important and gaining interest in the community.\n- The paper presents several interesting ideas and identifies the key problem with using parametric models for image-based estimation tasks. The problems stated are not only a problem for animals but also for human parametric body models and estimation for human shape/pose from images.\n- The paper is well written and experiments appear to be comprehensive and convincing.\n\nIn the following, I will focus on comments, questions, issues and suggestions to improve the paper (in no particular order).\n\n- In figure 1, it is not immediately clear that the red and green masks denote false negatives and false positives. It would help to make this clear in the caption/legend.\n\n- Related work: There's quite a bit of previous work that has recognized the issues that the paper has identified, i.e., using 3D models for image-based estimation results in misalignments at object boundaries because the 3D models do not model the image formation process.  Here are a couple of examples.\n\nFor humans, PiFU suggests tying 3D models to image pixels. This mitigates the misalignment problem.\n\nPIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization, Saito et al.\n\nMore recent work proposes to model cameras explicitly (new work that appeared only recently).\n\nCAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields, Niemeyer et al.\n\nThe paper also makes it sound like this problem is unique to animal reconstruction, but in it's really a large problem and points to important limitations of existing parametric models.\n\nFurthermore, there is rich literature on convolutions on meshes which would be worth discussing. For instance, see:\n\nMeshCNN: A Network with an Edge, Hanocka et al.\n\n- The proposed \"coarse\" fitting method makes sense. For the mesh refinement, I would liked to see a bit more intuition for *why* the proposed GCN method is well suited? Why not some other method like predicting per-vertex correctives (as is common for faces) using a simpler neural net?\n\n- My biggest issue with the paper is that it claims to refine both shape and keypoints, but very little details are provided for how the keypoints are refined. In figure 4, it is clear that the keypoints are improved, but *how*? There is a vague reference in lines 220, but other than that I see no other information.\n\n- Line 221 says \"estimated camera parameter f\". Where is this estimated?\n\nMisc.\n\n- Line 137: \"verctors\" --> \"vectors\"\n\n\nOverall, I believe the paper moves the state of the art forward in the field of animal shape/pose estimation, and maybe even has some lessons for human shape/pose estimation. There are some exposition and reasoning issues which I would encourage the authors to address in the rebuttal.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "While limitations and broader impacts are discussed, these are limited to 2-3 lines. I would suggest expanding discussion of limitations, particularly discussing failure in figure 4, last row. Adding bolded subheadings would also make these sections more prominent.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "-l3guhCmoJU", "reviewer": "Reviewer_nqoz", "summary": "This paper studies the problem of joint animal pose and shape reconstruction from a single image. This work is largely based on the literature [Ref 2] but adopts a coarse-to-fine mechanism for joint pose and shape learning.  The method extracts global embedding as before in the first stage. In the second stage, this work leverages a local feature extractor to provide pixel-aligned local features for mesh refinement. Specifically, the extracted local features are given as input to the U-Net to refine meshes at a different resolution. Experimental comparisons with existing work [Ref 2, 3, 32] have been conducted on the StanfordExtra dataset and BADJA dataset.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* [S1] The proposed coarse-to-fine design makes sense and the paper is generally clearly written with a good amount of technical details and experimental setups.\n* [S2] The proposed method achieves superior IOU and PCK@0.15 performances compared to the previous work. The qualitative results in Figure 4 look promising.\n\nWeaknesses:\n* [W1] While the 2D projections from the input camera view look decent, it is unclear if the actual 3D shape looks reasonable or not. The reviewer would expect to see the rotated shape from another angle as did in [Ref 2] (see Figure 5 and Figure 6 of [Ref 2]). It becomes difficult to evaluate the significance of the results without such information.\n* [W2] Reviewer understands that the proposed method is regression-based, but would like two see if the test-time optimization could further push the limit. In addition, when running test-time optimization with keypoints available, is this method still superior to [Ref 2]? If so, how much performance gain comes from the coarse-to-fine refinement design? It would be good to have such discussions in the rebuttal.\n* [W3] The reviewer would like to know if the second stage method is very sensitive to the first stage result. In other words, how does the first stage reconstruction affect the final performance? It would be great to provide some ablation studies on poor fitting on the camera parameter and/or SMAL model parameter.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "N / A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received 4 positive final ratings: 7, 6, 6, 6.
On the positive side, the reviewers appreciated importance of the problem, an overall meaningful approach and strong empirical performance. The main remaining concern was around novelty of individual components, but at the end the reviewers agreed that the combination of those is sufficiently novel and effective in the given setting. The remaining concerns were mostly around clarity, gaps in related works, and somewhat limited evaluation (dogs only). Some of those were addressed in the rebuttal, as acknowledged by the reviewers.
The final recommendation is to accept as a poster. The authors are highly encouraged to incorporate all feedback from the reviewers in the camera ready version of the manuscript.
Decision: Accept (Poster)