To The Point: Correspondence-driven monocular 3D category reconstruction
Authors: Filippos Kokkinos
Iasonas Kokkinos
Venue: NeurIPS 2021 Poster (2021)
Keywords: 3D reconstruction
3D meshes
non-rigid
structure from template
structure from motion
learnable optimization

Abstract: We present To The Point (TTP), a method for reconstructing 3D objects from a single image using 2D to 3D correspondences given only foreground masks, a category specific template and optionally sparse keypoints for supervision.  We recover a 3D shape from a 2D image by first regressing the 2D positions corresponding to the 3D template vertices and then jointly estimating a rigid camera transform and non-rigid template deformation that optimally explain the 2D positions through the 3D shape projection. By relying on correspondences we use a simple per-sample optimization problem to replace CNN-based regression of camera pose and non-rigid deformation and thereby obtain substantially more accurate 3D reconstructions. We treat this optimization as a differentiable layer and train the whole system in an end-to-end manner using geometry-driven losses. We report systematic quantitative improvements on multiple categories and provide qualitative results comprising diverse shape, poses and texture prediction examples.

To The Point: Correspondence-driven monocular 3D category reconstruction
Filippos Kokkinos
Iasonas Kokkinos
Department of Computer Science University College London filippos. kokkinos@ucl. ac uk
Department of Computer Science University College London, Snap Inc. i kookkkkknsssee .uc..ucll... wcl.ac.e. ac.uk
Abstract
We present To The Point (TTP), a method for reconstructing 3D objects from single image using 2D to 3D correspondences learned from weak supervision. We recover a 3D shape from a 2D image by first regressing the 2D positions corresponding the 3D template vertices and then jointly estimating rigid camera transform and non-rigid template deformation that optimally explain the 2D positions through the 3D shape projection. By relying on 3D-2D correspondences we use a simple per-sample optimization problem to replace ENN-based regression camera pose and non-rigid deformation and thereby obtain substantially more accurate 3D reconstructions. We treat this optimization as a differentiable layer and train the whole system in an end-to-end manner. We report systematic quantitative improvements on multiple categories and provide qualitative results comprising diverse shape, pose and texture prediction examples. Project website: https: / //fkokkinos github.iinn io/to_ the_point/.
Introduction
Monocular 3D reconstruction of general categories is task that humans perform with ease, yet remains challenging for computer vision due to its inherently ill-posed nature: the observed 2D image is the result of confluence of multiple sources of variation, including non-rigid intra-category shape variation, rigid transforms due to camera pose, as well as appearance variation. CNNs can easily leam to discard appearance variation, yet the treatment of the geometrie sources of variability remains elusive. Even though seroonlly-uueeviin for human reconstruction [1], for general categories need to rely weaker forms of supervision as well as self-supervision stemming from the know-how of computer vision.
upproaches have delivered compelling results e.g.
3D vision has traditionally relied on correspondences to recover both rigid scenes from 2D images for the Structure-froot from-Motion (SFM) problem [2, 3, 4] as well as the more challenging problem of recovering Non-Rigid structure from 2D point tracks (NR-SFM) [5, 6, 7, 8]. In all those problems 3D reconstruction accomplished by minimizing the reprojection error between the 3D positions the inferred 3D scene and their 2D image correspondences. While these solutions have been developed for the (potentially deformable) single-instance case, the idea relying on correspondences to supervise monocular 3D reconstruction has transpired in recent deep learning works.
CNN-driven monocular 3D category reconstruction [9, 10, 11, 12] has largely relied on self- supervision for 3D recovery expressed in terms of correspondence- based loss terms. For instance the geometric cycle loss terms [13, 14, 15] are explicitly phrased terms of correspondence established from UV maps while the texture-driven loss terms of [9, 13, 12, 16, 11] are implicitly relying on pixel correspondence. The common ground such loss terms is that if the 3D shape is predicted correctly, it should project to the image in way that is consistent with the 2D obser- vations, as measured in a pixel-by-pine sense. These correspondence terms are typically used in
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

tandem with explicit geometric priors such as 3D symmetry [10, 9], predefined camera viewpoint ranges [14, 15, 10], or predefined object scales [14, 15, 10, 12] in order tackle the ill-posed nature of the problem and the presence of multiple local minima in the associated learning problem.
Local minima however emerge even in the simpler single-instance case of NR-SFM, while highly sophisticated optimization schemes have been introduced to address them, e.g. [17]. Current CNN- based approaches seem to ignore this problem and further exacerbate it by delegating the solution of 3D reconstruction to back-propagation with SGD: separate network heads are tasked with regressing the camera pose and non-rigid deformation given an image and are trained in an end-to-end manner, aiming to minimize the correspondence-drin
losses. We argue that this is making optimization harder: network training aims at simultaneously establishing the association between images and rigid and non-rigid pose parameters as well as solving the 3D reconstruction problem in terms of these parameters. Each of these problems hard enough in isolation and putting them together makes the optimization even harder.
This challenge is reflected in the complicated numerical schemes currently used to mitigate local minima; for instance [10, 12] use multiple camera hypotheses during both training and testing. The number of hypotheses can range from 8 up to 40 for single reconstruction and the hypotheses have to be accompanied with a probabilistic method to select the most accurate pose either predicted by an MLP [14] or using heuristic loss-based weighting schemes [10]. Another example of brittle
optimization. even when keypoint supervision is available, are the works of [9, 18] where in a first stage SFM/NR- -FM is used to get the camera pose right based on keypoint supervision, which is then followed by optimization with image- losses to recover mesh. This challenge has been observed also in the grrongl--uupervise case of human pose estimation, and the use of per-sample numerical optimization [19] was shown to improve performance in [20, 21, 22, 23, 24].
In this work we deviate from the current practice of using a CNN to regress camera and mesh deformation estimates. Instead, during both training and testing we solve a per-sample optimization problem that explicitly aims at providing a 3D reconstruction that projects "To The Point" (TTP). We take as input the 2D coordinates corresponding to the 3D vertices of mesh and recover the 3D vertex positions by optimizing with respect to the rigid and non-rigid pose parameters through differentiable optimization [25, 26]. We obtain the 2D points required by our layer by only relying on mask annotations and optionally small number of 2D semantic keypoint annotations, as well as seff-supervision coming from the 3D reprojection loss. We jointly learn the 2D) point regression and the 3D modes of shape variability through end-to-end optimization, while treating the per- per-instance rigid and non-rigid pose parameters as latent variables that are optimized on-the-fly, per sample.
We claim that predicting the correspondences is not only sufficient, but also more appropriate for driving monocular 3D reconstruction: it spares us from the use of any additional geometric priors results while only relying on a single camera hypothesis. We evaluate our approach on 3D shape. pose and texture reconstruction on four objects categories using real- world datasets CUB [27] and PASCAL3D+ [28]. We demonstrate competitive 3D reconstruction quality to previous state-of-the-art methods and our ablation study confirms the importance of the supersuue losses we employ.
and also yields state-of-the-
2 Related Work
Monocular 3D reconstruction Recent works on this problem [29, 9, 14, 15, 16, 12, 30] have relied on varying forms of supervision. Earlier approaches [9, 29] treat the problem of 3D reconstruction from single images using known masks and manually labelled keypoints from single viewpoint image collections. Recent works [10, 16, 14] have removed the need for keypoints but introduced multiple viewpoint and deformation hypotheses accompanied with probabilistic method to select the most accurate pose. [13, 31, 32] jointly recover cameras and non-rigid 3D meshes with single hypothesis- based networks, but limit themselves to simpler, almost planar categories like faces, or exploit symmetry priors, limiting their broader applicability.
Closer to our work is Canonical Surface Mapping (CSM) and Articulated Canonical Surface Mapping (ACSM) [14, 15] where the 3D representation produced in the form of rigid or articulated template using a DD- cycle-consistency loss.
2

---

Correspondenaes
Camera and Deformation Estimation
Iteration 1
Iteration N
R,t
Heprocction
t
Deformed Mesh
Textured Mesh
Template T
Figure 1: Overview of our method: Given an image use network <p to regress the 2D positions corresponding to the 3D vertices of template; we then use a differentiable optimization method to compute the rigid (camera) and non-rigid (mesh) pose: in every iteration we refine our camera and mesh pose estimate to minimize the reprojection error between u and the reprojected mesh (visualized top of the input image). The end result is the monocular 3D reconstruction of the observed object, comprising the object's deformed shape, camera pose and texture.
Non-rigid structure from motion (NR-SIM) The aim of NR-SMM is the recovery of the 3D shape and accompanying camera pose given only 2D landmarks without any explicit 3D supervision [7, 8, 33, 34j. Lately, several deep learning [13, 35, 36] methods have been proposed that surpassed the performance of traditional methods while being considerably faster. All of the aforementioned methods employ different priors to tackle the under-constrained problem NR-SMM. The priors are embedded into the methods using low-rank subspaces [37, 7, 17], spatio-temporal domains [34, 38), cquivariance constraints [35] or sparse basis coefficients using Ll constraints [36, 39, 40].
Learnable Optimization: Common methods for incorporating optimization as layers in deep neural networks include implicit function differentiation [25, 26, 41, 42, 43] and optimization unrolling [44, 45, 36]; we refer to [26, 25] for survey. In 3D reconstruction recent works address the challenge of incorporating RANSAC in an end-to-end trainable pipeline for camera pose estimation based on the Perspective-n- Poion (PnP) problem, such as differentiable blind PnP [42, 43] or DSAC [46]. Unlike these works, we do not have to address the combinatorial nature of correspondence, but rather focus on regressing the 2D image positions of 3D template with fixed number of vertices.
3 To-The-Point Monocular 3D Mesh Reconstruction
We start in Sec. 3.1 by introducing the 2D quantities predicted by our network, we then present our differentiable camera and mesh optimization layer in Sec. 3.2, and finally present the losses driving our end-to-end training in Sec. 3.4.
3.1 Predicting 3D to 2D Correspondences
Our method assumes that the template of our object category can be described in 3D in terms of N points. As shown in Fig. given an image, we use CNN, de. to regress the 2D coordinates IR Nxx corresponding to these N 3D points. We also predict visibility vector where every 2 E [0, 1] indicates whether the 2D 3D correspondence is occluded in the image. Recognition of occluded points allows for accurate camera pose estimation by eliminating the influence of noisy predicted points belonging to the non-visible areas of the object.
3.2 Estimation of Pose and Deformation
Our aim is to estimate the camera pose and object deformation using only the predicted 2D points the visibility vector and the template mesh T. We first introduce our assumptions about the rigid and non-rigid part of the shape and then turn to the resulting optimization problem.
3

---

Firstly, as in 19, 10, 14, 16, 11], we model the 3D-to-2D projection through weak perspective [2]. This involves 2 x 3 3D-to-2D "scaled orthographic" projection matrix of the following form:
C
0
where the scaling factor accounts for global scaling due to depth variation; given a set of 2D image points this is set to their standard deviation, yielding invariance of the pose parameters to similarity transforms of the 2D and 3D coordinate frames. The rigid pose parameters comprise a rotation R. and translation that account for viewing 3D object V from a given camera position. The parametric estimates for the 2D projections of 3D object can thus be obtained as follows:
à iR, t) C(RV + t)
where V is 3 x N, is x N, and we overload notation for + assuming that t is replicated N times.
the modeling of non-rigid categories. For this we rely on the deformable template paradigm [47] commonly used also in the NR-SFM literature [7, 8], and obtain shape estimate V by adding offsets Av Bc to template shape T, yielding V = Av + T. Combined with Eq. 2, we have the following parametric estimate of the 2D positions:
Having covered the rigid object case,
ú,C, R, C(R[-CCRTTTT + Bc) tt)
which is bilinear in R, t and c.
So far we do not deviate substantially from recent works [9, 10, 16, 11] in terms of modelling: these also rely the morphable model paradigm [47] and regress shape update Av and rotation matrix through network heads. A minor modelling difference is that we have a low-rank model for Bc, while their shape update is driven by a nigh-dimmnsional latent vector. Our main difference that rather than delegating to the network the task of predicting the 'right' values of R. t, V, we directly optimize for them through lightweight and differentiable optimization scheme by exploiting our regressed 2D correspondences. This 'optimizes out' these parameters and ensures our 3D inference will project as accurately as possible to the 2D points, rather than delegating the optimization to backprop and the network heads.
approach aims at minimizing the following re-projection error between the predicted 2D points u and the 2D point estimates deliver by the parametric, 3D>based prediction:
Our
N
IR, t, = Σ v4 Iu ú; (, (C, + +
(4)
where we weigh the discrepancy between the two quantities by the regressed visibility, ensuring that the reconstruction process is robust to occluded points. We also add regularization weight on the expansion coefficients to avoid instabilities in the first stages of training, when the basis B still unknown and can lead to prematurely committing to large arbitrary deformations.
To minimize the loss term in Eq. (4), use alternating optimization scheme and perform separate updates for camera pose estimation and mesh deformation:
R = arg min (R, t, e'), subject R E SO(() R2t
(5)
a++1
arg min 1 (R' ())c)
where at each step we use some of the previously estimated quantities (denoted by hat) as fixed and optimize with respect with the remaining ones to update their estimates.
Starting from the optimization in Eq. 5. we satisfy the constraint that R E SO(3) by using the angle- axis representation of the rotation R, such that R, exp(r)x where [:]x is the skew symmetric operator The underlying non-linear problem is solved using the L-BFGS optimizer [48] and when t >> 1 is initialized with the estimate the previous iteration. To backpropagate through L-BFGS we use the implicit function theorem as described in [42, 43].
Given the rotation matrix R. we solve Eq 6 in closed form follows:
A+1
- X.. (S2 XΩ + +71) '2

---

Each row of 5 E 22N* is defined as S2 CR 1-/-11 B; and X E 22Nx2N is a diagonal matrix containing the visibility vector V. Finally, the vector Y E R2N is defined as Y u, CR++IT, Ct For clarity we provide the analytic derivation of the closed form solution in the supplementary material. We also backpropagate through matrix inversion, meaning that the chaining of the update steps can be treated as differentiable layer and be used in tandem with end-to-end training.
Even though the only quantities regressed by our CNN are the 2D positions and associated visibilities, the basis B that represents the shape variability of our category in Eq.3 is a parameter of this layer, randomly initialized, and is estimated through back prok-agattioon. As shown in the supplemental material, the basis elements learned this way can be intuitively understood, while our experimental results indicate that they suffice for the accurate recovery of intricate mesh deformations.
3.3 Texture
The final part of monocular 3D reconstruction the estimation of the texture of the reconstructed 3D shape. The texture of the object is sampled from the input image utilizing the predicted 2D points closely resembling the sampling-based texturing method of CMR [9]. Unlike CMR, we do not predict uv locations to sample pixel values for the image with a dedicated learnable regressor, but use the predicted 2D points u that drive our whole 3D reconstruction process. For this we use sampling- baaedd texture approach where the face color is computed by interpolating the u coordinates and then sampling from the texture map, ie the input image in our case. Any losses applied on the estimated texture back- procppropagtte information to the predicted correspondences allowing to use appearance image cues in addition to foreground masks to enable accurate correspondence predictions.
3.4 Loss terms
To train our approach incorporate different losses focusing on pose estimation, texture prediction as well as mesh regularization.
Texture Loss compares the rendered textured image I and the image appearance in terms of the perceptual similarity metric of [49] after masking by the silhouette S:
Lpisel = dist (ioS,
We also apply the loss on the symmetric texture predictions by using a bilateral symmetric viewpoint and average the two viewpoints. The soft symmetry constraint ensures that the texture of the non-visible side is still inline with the visible side. This constraint has been employed in prior works [9, 16], however, unlike the proposed method it commonly applied in a hard-coded manner by symmetrizing the texture across an axis.
Points Chamfer Distance enforces points to lie inside and cover the silhouette of the depicted object [29, 15]. In order to formulate our loss term we define Crmask as the Chamfer distance field of the binary mask of silhouette correspondences of an instance to lie inside its silhouette. This can be achieved by penalizing the points projected outside the instance mask by their distance from the silhouette. Silhouette coverage enforces the predicted points u, to fully cover the mask of the depicted object and allows us to predict better camera poses and mesh deformations.
Silhouette consistency simply enforces the predicted 2D
LChamferr
Σ Cmaakk(n) (u.) +
Σ min PES
I1 - -112
u.
silbouette consistency
silhonette coverage
Region similarity loss compares the object support computed from the mesh by a differentiable renderer [50] to instance segmentations S provided either by manual annotations or pretrained CNNs using their absolute distance:
Linask
Σ IsS - frender (V,, n.)]-
Cycle and Visibility loss Similarly to CSM [14] we use cycle loss between the regressed 2D correspondences and the projected 3D points to ensure that regressed points, that form neigh- borhoods in the template shape, remain close in the image space. The cycle loss is defined as
5

---

E, - Furthemore, visibility of correspondences aids the camera pose esti- mation in weakly supervised cases. The visibility loss encourages the predictor Φy to encode the visible area of the mesh in an image by enforcing the predicted visibilities to be similar to those of the rendered z-buffer vat
Leycle =
Liis = Σ V fl
Equivariance Loss The point regressor should be robust to the pose variations. For each training image, draw a random spatial transform (-)- from a predefined parameter range. We use random affine transformations (scale, rotation, and shifting) for spatial transforms as well as vertical flipping (-). The detailed transform parameters are present in the supplementary material. We pass both the input image and transformed image = T, (I) through the Φ9 network and obtain the corresponding predictions and u'. For vertical flipping retrieve two pose estimations R and R' for 7 and the flipped image (T). We compute the equivariance loss as follows:
1 Legy = Elluf (u,)ll + arccos 2 (Tr(T, - 1).
(Optional) Keypoint reprojection loss While we are primarily interested training without any manual annotations, approach can be extended to leverage arbitrary number of high-level semantic keypoints. This is achieved by setting manually the 3D keypoints on the template mesh and encoding them matrix K acting the mesh. The structure of K entails that each K, is fixed vector that regresses the i-th 3D semantic keypoint from the mesh. Given the 2D annotations for an image I and camera keypoint reprojection loss is formed between the groundtruth annotation and the projected 3D) points:
Tap = Σ - (K,
siigiid-a--possiblle (ARAP) constraint Without any mesh deformation regularization, the pre- dicted mesh deformation will lead to arbitrary deformations exhibiting spikes and other anomalies. As such, we use the s--rigi--a--possibl (ARAP) [51] constraint as a loss function similar to [11]. The predieted shape V is locally rigid transformation from the predicted base shape T by:
Lamp
T. V)
Σ t= JEN()
Σ
wis (I v²) - R4 (T' T
N
where (ii) represents the neighboring vertices of a vertex i, Wsj the cotangent weights and R2 the best approximating rotation matrix, as described in [51]. Beyond mesh regularization, the same loss is applied on each basis component that leads to smooth and locally rigid components.
Even with ARAP there are cases where the network will squeeze the non-visible side of the recon- structed object. This erroneous deformation not penalized by the ARAP loss, long is locally rigid, and causes the method to predict flattened meshes. We further apply an la constraint to the deformations penalize the method retain the original volume of the template.
Experiments
Datasets We present extensive ablation results and comparisons on bird reconstruction, as well as quantitative results on three more object categories (planes, cars, motorbikes). For birds
the CUB [27] dataset for training and testing on birds which contains 6000 images. The train/val/test split we use for training and report is that of [9]. For the rest of the objects we use the Pascal3D+ dataset [28] and the associated pre-defined training and validation sets. Similarly to [9], we use
both PASCAL VOC and Imagenet images train models and use Mask-RCNN [52] to obtain foreground masks for the ImageNet subset. For templates, we use identical to those of CSM [14] for CUB dataset and for PASCAL33+ we select one of the available CAD models for each object.
Evaluation Metrics We evaluate our model on the CUB dataset [27] and report both the mean Intersection over Union (mloU) and keypoint reprojection accuracy (PCK) following CMR [9].

---

Table 1: Evaluation of TTP performance on the CUB [27] dataset. We report mean and standard deviation (in parantheses, where applicable) of 2D mloU and keypoint re- re-projection accuracy (PCK) along with related supervision signals for recent monocular 3D reconstruction methods.
Rigid
Non-Rigid
Camera Priors
Camera Hypotheses
keypoints
mloU t
PCK 1
mloU 1
PCK f
CMR [9]
0.703
81.2
(AYCSM [15]
0.622
68.5
0.705
72.4
ACMR [1]]
1
0,708
85.5
TTP (ours)
V
1
0.656 (0.002)
70.0 (0.53)
0.760 (0.004)
93.4 (0.14)
(AYCSM [15]
/
8
0.625
50.9
0.693
46.8
(A)CSM [15]
1
0.637 (0.004)
39.0 (1.07)
0.684 (0.011)
44.5 (1.21)
TTP (ours)
1
0.652 (0.008)
48.7 (0.66)
0.752 (0.003)
50.9 (0.43)
Table 2: Performance of TTP method through iterations for pose and deformation estimation. We achieve the best results with more iterations at inference, but even a single iteration suffices for competitive scores.
Number of Iterations
2
3
4
mloU
PCK
mloU
PCK
mloU
PCK
mloU
PCK
TTP w/ KP
0.732
92.5
0.755
93.3
0.758
93.3
0.758
93.4
TTP w/o KP
0.746
51.4
0.752
51.1
0.752
51.1
0.752
51.1
For Pascal3D+ we report canonical 3D mean Intersection over Union metric which measures the 3D overlap between the groundtruth and predicted deformed mesh; in order to compute the overlap, both meshes are voxelized using 32 grid size before computing the 3D mloU as in [10, 9, 53, 29].
Network Architecture Following prior work [9], we use ResNet18 encoder map image to a latent feature map E I 256 The position regressor is fully connected layer having as input the flattened feature map z and outputs the regressed 2D positions u E R x and their respective visibility v E RR xx, The number of basis components is set to = 16 and the number of iterations for the camera and deformation estimation is four.
Network Training To train the 3D reconstruction model we first warm up the model without applying any deformation for 100 epochs. This warm-up process allows the model to find the best pose possible given the rigid template using available cues like masks, texture and optional keypoints. We then train the full 3D reconstruction network with deformation enabled and all available cues for another 100 epochs. All training details can be found in the supplementary material. All experiments were run on single RTX 2080 Ti GPU.
4.1 Quantitative Results
Evaluation on CUB In Table we evaluate TTP on the CUB dataset and report the average and standard deviation of 5 experiments with different seeds; the rigid part of the results table indicates the performance of models that do not use a deformable component, while the non-rigid part amounts to the more challenging problem of estimating both camera and mesh deformations.
We observe that our method outperforms the baseline models on both reported metrics, i.e. mean IoU and keypoint re- projection accuracy, while requiring no camera priors. When using 2D keypoint supervision (upper part of the table) our method achieves the best results, outperforming the closest baseline by almost 8 accuracy points. For the case where no 2D keypoints are used the only published result in the literature is the ACSM approach of [15] which relies on camera hypotheses during both training and testing, while also using manual annotation of part-based rigs to bootstrap the deformation model. Our work outperforms this baseline while relying on a single camera hypothesis and without requiring any manual mesh annotation.
7

---

Table 3: Ablation Study We ablate the self-supervised losses used train TTP for 3D reconstruction for the case of training both with and without keypoint supervision. We also study the impact of the number of basis elements.
(b) Ablation on number of basis components.
(a) Ablation losses.
With KP
Without KP
With KP
Without KP
Basis
mloU f
PCK
mIoU f
PCK f
mloU 1
PCK
mloU t
PCK
TTP
0.765
93.6
0.749
50.9
rigid
0.657
70.7
0.646
48.4
TTP-
pisel
0.752
92.7
0.667
49.2
0.726
88.2
0.72
47.9
TTP- Lvis
0.75
92.3
0.74
0.745
90.6
0.733
50.4
TTP- Logil
0.751
92.5
0.71
28.2
16
0.765
93.6
0.749
50.9
32
0.771
93.7
0.748
49.8
64
0.775
94.2
0.752
49.8
Table PASCAL3D+ evaluation. We provide numerical score of TTP with and without keypoint supervision during training. We observe that even without keypoint supervision TTP is competitive with the other methods which, except for UCMR, require keypoints.
CSDM [29]
DRC [53] UCMR [10]
CMR [9]
TTP w/ KP
TTP w/o KP
aeroplane car
0.4
0.42
0.468
0.488 0.67
0.45
0.6
0.67
0.646
0.64
0.665
We posit that using multiple cameras in [15] aims at mitigating the local minima in network training and optimization. We have therefore rerun the system of [15] with a single camera and five different optimization seeds and observed further gap in performance compared our work, well larger variance in the reconstruction accuracy compared to that of our work for the non-rigid case, suggesting potentially higher chance of getting stuck in local minima.
Ablation study We ablate various terms in our learning objective and report the mloU and the semantic keypoint reprojection (PCK) metrics. In particular examine the impact of removing any of the utilized losses in Table 3a. When using keypoint supervision the differences in performance small. However in the absence of keypoints, the method struggles to align the template with the depicted object when we remove the visibility loss. While mloU remains high, PCK score decreases substantially meaning that the pose and deformation of the template cover the foreground mask when rendered but not from a proper viewpoint of the object. Similar performance drop occurs without the equivariance loss since the method produces pose estimates biased towards one vertical direction. Finally, removing the texture loss causes mloU performance to drop significantly.
In Table 3b we study how performance changes as a function of the number of basis elements. Increasing the number of components tends to increase performance but up to 16 elements for both set of experiments. When training with semantic keypoints increasing the number of basis components further improves performance, however since the same does not apply to the mask-only case we have set K 16 for all of our experiments.
Finally, key aspect of TTP the iterative pose and deformation estimation process. Table we provide the mloU and PCK scores for every iteration for two experiments trained with and without keypoint supervision. Multiple iterations have to be executed to get the best performance, however TTP's performance with single iteration still outperforms prior work for both metrics.
We are complementing these quantitative results with qualitative results in Figure 2 where we show that we can correctly deform the template mesh to produce highly accurate 3D reconstructions.
Evaluation on Pascal3D+ While primary evaluation is on the CUB dataset,
supplementary experiments on the cars, airplanes and motorcycle categories of PASCAL3D+ dataset. For cars and aeroplanes
provide comparisons against CMR [9]. UCMR [10]. volumetric prediction network [53] and a fitting based method [29]. Three of the methods use segmentation masks, cameras and keypoints for supervision except UCMR that does not require keypoints.
We train our method with and without keypoint supervision and provide our 3D mloU results in Table 4. We observe that our method performs considerably better than competing methods even when
8

---

CMR
UCMR
Ours-u
Ours-Mesh
Ours-Texture
New view
Figure 2: Bird reconstructions For each input image we provide the results of CMR [9] and UCMR [15] alongside with our method. We visualize the input image, predictions from prior works and TTP's predicted correspondence (u), mesh reconstruction and textured mesh from two viewpoints. We observe that we better capture texture details, deformation and pose estimation.
Mesh
Texture
New view
Mesh
Texture
New view
Figure 3: Pascal3D+ results We show predictions of our TTP method for test set images. For cach inpul image we visualize the 3D shape from the predicted camera, the textured shape and new view.
keypoint supervision is not utilized. Beyond the 3D mloU metric we also provide the PCK scores of our method for the cars dataset and compare against the only available reported method [14]. Our approach trained with and without keypoints results in 74.9 mloU and 45.7 PCK scores while CSM achieves 51.2 and 40.0 respectively. The difference is signiticant in both cases, especially in the keypoint-free methods where predicting correct camera pose is challenging due the weak supervision setup. We provide qualitative results of TTP Figure and the supplementary material.
4.2 Failure Cases
We visualize some failure cases of the proposed method in Figure 4. Common failure cases are related to the inability to predict good camera pose and 21) points.

---

Figure 4: Failure Cases: We visualize some failure modes our method. The columns present the input image, the predicted 2D points, and 3D reconstruction with and without texture.
Table 5: Run-time analysis milliseconds of various self-supervised 3D methods. All benchmarks
20 times using images of size 256x256 and report the average times.
Device
Iters
CNN (msec)
Optimization (msec)
Total (msec)
CMR
GPU
5.11
N/A
5.11
CSM
GPU
150.73
N/A
150.73
ACSM
GPU
191.46
N/A
191.46
4.02
33.44
37.46
TTP
GPU
4.02
125.14
129.16
18.52
33.44
51.96
TTP
CPU
4
18.52
125.14
143.67
4.3 Run-time Analysis
In Table 5 we provide a run time analysis our and prior methods on the task of self-supervised 3D reconstruction. The analysis was performed on a machine with NVidia RTX 2080Ti, an Intel Xeon W-2255 and 32 GBs of RAM. We report average of 20 runs for the reconstruction of 256x256 image. To ensure that the benchmark is fair for all methods, we compute the execution time between the moment an image given input network up to the moment where the network predicts the mesh, the camera pose and the texture; the implementations of the methods used for comparison are the publicly available ones from the original authors. As reported in Table TTP is faster than ACSM by considerable margin even the CPU. CMR is the fastest method of all due to its simplicity, however, requires keypoint supervision and has substantially lower results as indicate in Table 1.
We note that our implementation relies on PyTorch and we have used PyTorch's python-based LBFGS optimizer for convenience and autograd for Jacobian computation; understandably for AR applications the LBFGS timing can become substantially faster in C or custom CUDA kernels and with explicitly coded Jacobian matrix computation.
5
Discussion
We have proposed a method to reconstruct 3D meshes, poses and textures generic objects in the wild without any direct supervision. We learn unsupervised correspondences between 2D image locations and 3D template vertices and use them to compute the camera pose and deformation of the object. Even though our CNN architecture predicts substantially fewer outputs compared e.g. to [9] where all of the 3D vertices and the camera are directly regressed by the network, we deliver substantially better results. We attribute this to the use of a direct optimization scheme to optimize the 3D reconstruction problem both during training and testing. The resulting optimization problem particularly lightweight, meaning that can be used interactive applications, e.g. Augmented Reality, while our results indicate that even single step of the optimization suffices for accurate mesh recover. In future work we aim to extend our approach to cover categories with diverse topologies (c.g. chairs) as well as exploit video-based supervision [12] to further improve accuracy.
10

---

References
[1] R. A. Güler, N. Neverova, and L. Kokkinos, "Densepose: Dense human pose estimation in the wild," in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7297-7306, 2018. 1
[2] R. I. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision. Cambridge University Press,
second cd., 2004, 1,4
[3] S. Agarwal, Y. Furukawa, N. Snavely, I. Simon, B. Curless, S. M. Seitz, and R. Szeliski, "Building rome in day." Communications of the ACM, vol. 54, no. 10, pp. 105-112, 2011. 1
[4] J. L. Schönberger and J.-M. Frahm, Cerructurefrommmootee and Pattern Recognition (CVPR), 2016. 1
revisited," in Conference on Computer Vision
[5] R. Yu, C. Russell, N. D. F. Campbell, and L. Agapito, "Direct, dense, and deformable: Template-based non-rigid 3d reconstruction from rgb video," in Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015. 1
[6] I. Akhter, Y. Sheikh, S. Khan, and T. Kanade, "Nonrigid structure from motion in trajectory space." in Advances Neural Information Processing Systems (D. Koller, D. Schuurmans, Y. Bengio, and L., Bottou, eds.), vol. 21, Curran Associates, Inc., 2009. 1
[7] L. Torresani, A. Hertzmann, and C. Bregler, "Learning non-rigid 3D shape from 2D motion," Advances in neural information processing systems, 2003. 1, 3,4
[8] M. Paladini, A. Del Bue, M. Stosic, M. Dodig, J. Xavier, and L. Agapito, "Factorization for non-rigid and articulated structure using metric projections." in 2009 JEEE Conference on Computer Vision and Pattern Recognition, pp. 2898-2905, IEEE, 2009. 1,
[9] A. Kanazawa, Tulsiani, A. A. Efros, and J. Malik, "Learning category-specifie mesh reconstruction from image collections," in ECCV, 2018. 1, 4, 5, 6, 7, 8, 9, 10, 14, 16
[10] S. Goel. A. Kanazawa, and J. Malik, "Shape and viewpoints without keypoints." in ECCV, 2020. 1, 4.
7, 8. 14, 15, 17
[11] X. Li, S. Liu, S. De Mello, K. Kim, X. Wang, M.-H. Yang, and J. Kautz, "Online adaptation for consistent mesh reconstruction in the wild," in NeurlPS, 2020. 1,4, 6,
[12] F. Kokkinos and Kokkinos, "Learning monocular reconstruction articulated categories from motion," "
in CVPR, 2021. 1,2, 10
[13] M. Sahasrabudhe, Z. Shu, E. Bartrum, R. Alp Guler, D. Samaras, and I. Kokkinos, "Lifting autoencoders: Unsupervised learning of a dissa-disent 3d morphable model using deep non-rigid structure from motion," " in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, Oct 2019. 1, 2,
[14] N. Kulkarni, A. Gupta, and S. Tulsiani, "Canonical Surface Mapping via Geometric Cycle Consistency." International Conference on Computer Vision (ICCV), 2019. 5,
[15] N. Kulkarni, A. Gupta, D. F. Fouhey, and S. Tulsiani, AAriiullttion-wwar canonical surface mapping," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 452-461,
2020. 1, 2, 4, 5, 7, 8, 9, 16
[16] X. Li, S. Liu, K. Kim, S. De Mello, V. Jampani, M.-H. Yang, and J. Kautz, "Self-supervieed Single-view 3D Reconstruction via Semantic Consistency." in ECCV, 2020. 2,
[17] Y. Dai, H. Li, and M. He, "A simple prior-free method for non-rigid
from-mmtion factorization,"
International Journal of Computer Vision, vol. 107, no. 2, pp. 101-122, 2014. 2,3
[18] D. Novotny. R. Shapovalov, and A. Vedaldi, "Canonical 3D Deformer Maps: Unifying parametric and non-parametric methods for dense weakly-superrised category reconstruction." " in NeurlPS, 2020. 2
[19] F. Bogo, A. Kanazawa, C. Lassner, P. Gehler, J. Romero, and M. J. Black, "Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image," in Computer Vision - ECCV 2016, Lecture
Notes in Computer Science, Springer International Publishing, Oct. 2016. 2
[20] R. A. Guler and Kokkinos, "Holopose: Holistic human reconstruction in-the-wild," Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2019. 2
[21] N. Kolotouros, G. Pavlakos, M. J. Black, and K. Danillidis, "Learning reconstruct 3d human pose and shape via model-fitting in the loop." in Proceedings of the IEEE International Conference on Computer Vision, 2019.
[22] H. Joo, N. Neverova, and A. Vedaldi, "Exemplar fine-tuning for 3d human model fitting towards in-the-wild 3d human pose estimation." " 2020. 2
[23] D. Kulon, R. A. Guler, I. Kokkinos, M. M. Bronstein, and S. Zafeiriou, supervied mesh- convolutional hand reconstruction in the wild." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.

---

[24] F. Mueller, M. Davis, F. Bernard, O. Solnychenko, M. Verschoor, M. A. Otaduy, D. Casas, and C. Theobalt, "Real-time Pose and Shape Reconstruction of Two Interacting Hands With Single Depth Camera," ACM Transactions on Graphics (TOG), vol. 38, no. 4, 2019.
[25] A. Agrawal, B. Amos, S. Barratt, S. Boyd, S. Diamond, and Z. Kolter, "Differentiable convex optimization layers," Advances in Neural Information Processing Systems, 2019. 2,
[26] S. Gould, R. Hartley, and D. Campbell, "Deep declarative networks: new hope." tech. rep., Australian National University (arXiv:1909 (9999.9944)...... Sep 2019. 2,3
[27] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie, "The Caltech- UCCD Birds-200-2011 Dataset,"
Tech. Rep. CNS-TR-201 1-001, California Institute of Technology, 2011, 2, 6,
[28] Y. Xiang, R. Mottaghi, and S. Savarese, "Beyond pascal: A benchmark for 3d object detection in the wild,"
in IEEE Winter Conference on Applications of Computer Vision (WACV), 2014. 6
[29] A. Kar, S. Tulsiani, J. Carreira, and J. Malik, (Category-speciife object reconstruction from single image," in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1966-1974, 2015.
2, S, 7,8
[30] T. Cashman and A. W. Fitzgibbon, "What shape are dolphins? building 3d morphable models from 2d images." " IEFE transactions on pattern analysis and machine intelligence, vol. 35, no. 1, pp. 232-244, 2012. 2
[31] S. Jenni and P. Favaro, "Self-supervised multi-view synchronization learning for 3d pose estimation." in Proceedings of the Asian Conference on Computer Vision, 2020. 2
[32] S. Wu, C. Rupprecht, and A. Vedaldi, "Unsupervised learning probably symmetric deformable 3D objects from images in the wild," in Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition (CVPR), 2020. 2
[33] R. Garg, A. Roussos, and L. Agapito, "Dense variational reconstruction of non-rigid surfaces from monocular video," in IEEE Conference on Computer Vision and Pattern Recognition, 2013. 3
[34] K. Fragkiadaki. H. Hu, and J. Shi, "Pose from flow and flow from pose." in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 2059-2006,, 2013. 3
[35] D. Novotny, N. Ravi, B. Graham, N. Neverova, and A. Vedaldi, "C3dpo: Canonical 3d pose networks for non-rigid structure from motion," in Proceedings of the IEEE International Conference on Computer Vision, 2019. 3
[36] C. Kong and S. Lucey, "Deep non-rigid structure from motion," in Proceedings of the IEEF/CVF Interna- tional Conference on Computer Vision, pp. 1558-15677 2019. 3
[37] K. Fragkiadaki, M. Salas, P. A. Arbelaez, and J. Malik, "Grouping-based low-rank trajectory completion and 3d reconstruction." in NIPS, vol. 2, p. 7, Citeseer, 2014. 3
[38] H.-Y. Y. Tung, H.-W. Tung, E. Yumer, and K. Fragkiadaki, "Self-supervised learning of motion capture," in Advances in Neural Information Processing Systems 30 (1. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, eds.), pp. 5236 5246, Curran Associates, Inc., 2017.
[39] X. Zhou, M. Zhu, S. Leonardos. and K. Daniilidis. "Sparse representation for 3d shape estimation: A convex relaxation approach." IEEE transactions on pattern analysis and machine intelligence, vol. 39, no. 8, pp. 1648-1661, 2016.
[40] X. Zhou, M. Zhu, S. Leonardos, K. G. Derpanis, and K. Daniilidis, "Sparseness meets deepness: 3d human pose estimation from monocular video." " in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4966 4975, 2016. 3
[41] S. Bai, J. Z. Kolter, and V. Koltun, "Deep equilibrium models," in Advances in Neural Information Processing Systems (H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bue, E. Fox, and R. Garnett, eds.), vol. 32, Curran Associates, Inc., 2019. 3
[42] D. Campbell", I.. Liu", and S. Gould, "Solving the blind perspective-n-point problem end-to-end with robust differentiable geometric optimization," in ECCV, 2020. equal contribution. 3,
[43] B. Chen, A. Parra, J. Cao. N. and T.-J. Chin, "End-to-end learnable geometric vision by backpropagating pup optimization," in CVPR, 2020. 3.4
[44] M. Andrychowicz, M. Denil, Gómez, M. W. Hoffmman, D. Pfau, T. Schaul, Shillingford, and N. de Fre- itas, "Learning to learn by gradient descent by gradient descent," in Advances in Neural Information Processing Systems (D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, eds.), vol. 29, Curran
Associates, Inc., 2016. 3
[45] F. Kokkinos and S. Lefkimmiatis, "Iterative joint image demosaicking and denoising using residual denoising network." IEEE Transactions on Image Processing, vol. 28, no. 8, pp. 4177-4188 2019. 3
12

---

[46] E. Brachmann, A. Krull, S. Nowozin, J. Shotton, F. Michel, S. Gumhold, and C. Rother, "Dsac- differentiable ransac for camera localization," in Proceedings of the ILEE Conference Computer Vision and Puttern Recognition, pp. 6684-6692, 2017.
[47] V. Blanz and T. Vetter, "A morphable model for the synthesis of 3D faces," in Proceedings of the 26th annual conference Computer graphics and interactive techniques, 1999. 4
[48] D. C. Liu and J. Nocedal, "On the limited memory bigs method for large scale optimization." Mathematical programmiing, vol. 45, no. 1, pp. 503-528, 1989. 4
R. Zhang, Isola, A. Efros, Shechtman, and Wang, "The unreasonable effectiveness of deep features as perceptual metrie," in CVPR, 2018.
[50] N. Ravi, J. Reizenstein, D. Novotuy, T. Gordon. W.-Y. Lo, Johnson, and G. Gkioxari, "Accelerating 3d deep learning with pytorch3d," arXiv:2007.08800) 2020. 5
[51] O. Sorkine and M. Alexa, "As-siiidaa--posiillcc surface modeling," in Symposium on Geometry processing, vol. 4, pp. 109-116, 2007. 6
[52] He, Gkioxari, P. Dollár, R. Girshick, "Mask r-cnn," Proceedings of the IEEE international
conference on computer vision, pp. 2961-2969, 2017. 6
[53] S. Tulsiani, Zhou, A. Efros, and Malik, "Multi-view supervision for single-view reconstruction via
differentiable ray consistency." Computer Vision and Puttern Regognition (CVPR), 2017. 7,
13


Reviews:
[{"review_id": "rbvzn0JqtCR", "reviewer": "Reviewer_t6xY", "summary": "This paper tackles the problem of learning single-image 3D object reconstruction from single-view collections. The proposed method couples a correspondence prediction network with an iterative optimization procedure that solves shape and pose based on the predicted correspondences, starting with a category template shape.\nThe claim is that, with this iterative optimization procedure in the loop, it alleviates additional supervision for camera viewpoint such as keypoints, or multiple manual hypotheses.\n\nThe results show that this method is able to achieve comparable reconstruction accuracy without keypoints compared to previous methods that require keypoints. Overall, this paper presents an interesting approach of incorporating iterative optimization in a shape learning framework, but the method also has its limitations and the evaluation is rather limited.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n### S1 - Method\nThe proposed approach is based on the same idea as in UMR[16] and ACSM[15]: separating the correspondence prediction and reconstruction, which is the key to classical structure-from-motion techniques. Unlike [15] and [16], the proposed method adopts an iterative optimization procedure in the reconstruction branch, whereas [15] and [16] use networks to predict shape and pose directly. Specifically, the camera is solved by L-BFGS optimization and the shape is solved in closed form with a optimized shape basis.\n\nThis has the advantage of being more robust against local minima given the shape pose ambiguity, which is a key challenge in unsupervised 3D reconstruction.\nIn contrast to existing methods that rely on additional keypoint supervision or multi-camera hypotheses, the proposed method is more data- and (perhaps) computation-efficient, although it still involves an iterative optimization process.\n\n### S2 - Writing\nThe paper is well-written. The motivation is clearly introduced, and methodology clearly explained.\n\n\n## Weaknesses\n### W1 - Method\n- W1.1: One major drawback of this optimization-based solution is that the same optimization procedure is also required at inference time, even though the authors show that even one iteration can already produce accurate results. How fast is each optimization iteration?\n- W1.2: Does this optimization always yield a good solution? It seems to me the camera viewpoint can fall into local minima with BFGS optimization as well. Moreover, it relies heavily on the predicted correspondences, and cannot be robust against erroneous correspondences as a learning approach could with learned priors.\n- W1.3: Another drawback is the way the correspondences are predicted using points. It does not guarantee continuity, as is also shown in the visualizations where the points can get messy in some parts. CSM [15] and part segmentation [16] have the advantage of producing smoother correspondences with an image. This also makes more concerned with the robustness of the optimization against errors in correspondence prediction.\n- W1.4: One more major limitation is the need for a category template shape similar to [15] and [16]. This has largely simplified the problem of category-specific shape reconstruction and at the same time restricted the reconstructed shape from deviating too much from the template. The reconstructed airplanes for example are almost always the same despite the variation of the inputs (see Fig 9 in the sup. mat.).\n\n### W2 - Evaluation\nThe evaluation has a number of limitations.\n- W2.1: Why not compare with UMR[16]? It obtains part correspondences through a self-supervised part segmentation model, and does not require a template shape.\n- W2.2: The numerical results are not very convincing. Mask mIoU only measures the shape from one single viewpoint, and does not evaluate the 3D shape. Keypoint reprojection is also very sparse and noisy, and may not be indicative of the 3D shape accuracy. It is unclear whether the comparison on these noisy metrics is actually meaningful.\n- W2.3: The evaluation on Pascal3D+ with 3D mIoU is more meaningful (although the 3D shape is also \"pseudo GT\"), but the gap is relatively small. But more importantly, I wonder how well the template shape alone achieves in terms of metric?\n- W2.4: Most of the visual results are visualized in the input viewpoint, which makes it hard to tell the how good the reconstructed 3D shapes are. From the only one novel view provided in Fig. 2 and 3 and in the sup. mat., the reconstructed 3D shapes do not look very plausible. The birds for example appear quite flat. The authors should provide videos or visualizations with multiple viewpoints to showcase the reconstructed 3D shapes. How do they compare to UMR and ACMR visually?\n- It probably also makes sense to plot distribution of the predicted viewpoints. Is the model confused with some viewpoint, eg front vs back, left vs right?\n- It could also help with understanding the pose and shape prediction to color the predicted mesh with the same color pattern in the template mesh.\n\n\n## Clarification needed\n- How exactly is texture symmetry enforced? The method samples texture from the input image using the predicted 2D points. In line 182, it briefly describes the texture symmetry enforced by averaging textures from two symmetric viewpoints. Does it mean taking the average of the sampled colors of two symmetric points (defined by the template)? If so, does this also rely on the visibility prediction? More details should be provided.\n- This is interesting because texture symmetry alone can provide pretty strong cue for shape and pose. Does the model utilize this cue for reconstruction?\n- There are 9 loss terms in total (if not more). How are they balanced?\n\n\n## Additional comments\n- Eq (3): $\\mathbf{V}$ should be $\\mathbf{T}$, input $\\mathbf{C}$ should be lower case $\\mathbf{c}$.\n- Line 207: \"vertical flipping\"? ie, about the horizontal axis and the objects would be upside down?\n\n\n---\n## Post-Rebuttal\nI appreciate the authors' efforts in the rebuttal, which has addressed many of my questions, including the computational cost, comparison with UMR, and template shape baseline performance on Pascal3D+. Regarding the robustness, it is still not clear how robust the correspondence prediction and subsequently the pose estimation are. This is hard to quantify, but more discussion on failure cases in the final version would be very helpful.\n\nOverall, this paper has sufficient interesting material for acceptance. I will keep my original review as borderline accept.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "A few failure examples are provided in the supplementary, but they all seem very challenging cases, eg flying birds, heavy occlusion and small crops.\nAre there failure cases on more regular images? Does the viewpoint prediction get stuck in local minima? Does the correspondence prediction fail dramatically, and what happens in the shape and pose optimization then?\n", "review_text": "", "extra_scores": {"ethical_concerns": "I do not see immediate concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "ExmjlIlsFrU", "reviewer": "Reviewer_xnWE", "summary": "The authors propose to separate out camera view point and template deformation using a differentiable optimization layer, and only delegate correspondence estimation to a CNN. The authors demonstrate improved performance on standard benchmarks with such a modular approach.   ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points\n- Formulating the camera viewpoint and template optimization as a differentiable layer and enabling end-to-end training\n- Geometry driven losses requiring no annotation\n\nWeak Points\n- The experiments are limited. Especially, there is no experiment discussing the performance with/without the silhouette as is done in the related work, \n- The improvement due to separating the optimization tasks should be better motivated and substantiated with additional experiments. For e.g., what happens if we were to use a cnn to regress the view point/deformation parameters with correspondence as input. How does it compare with the differentiable optimizer? What is the inflection point in performance wrt to number of iterations? As neural networks are general function approximaters, the burden of multi task optimization can be alleviated with more network capacity or additional data. All of these are design choices that are not discussed. \n- The main idea of separating out using a CNN for correspondence and classical optimization for pose and shape regression is not new. Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera by Mueller et. al. were the first to do so to the best of my knowledge and there is related work in body/hand shape estimation which is not discussed in this paper. Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop by Kolotouros et. al. is also relevant.  \n- No mention of run time or compute analysis. \n\nPose Rebuttal Review:\nI apologize for missing the fact that mask is necessary for the approach. It was a detail overlooked by me while skimming related work as this is not my primary domain of research. Feeding correspondence to a network can be trivially accomplished by passing tuple of positions or their embeddings thereof. As demonstrated in transformers, this information can help the network learn better. I am still curios of such an analysis. I would also be more enthusiastic about an acceptance if the code is made available, as replication of the approach is non-trivial and marginal improvement over SOTA might be difficult to achieve in practice without the authors divulging the 'secret sauce' as common among the best of contributions in this domain. I am slightly upgrading the rating based on other reviews and their assessment along with the additional experiments on latency. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Augmented reality is here and this work directly impacts it. The authors should include a broader discussion of negative impacts from an augmented reality perspective. Research in nascent stage is the best place to identify potential negative impact on society. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "AiyGzTX7TLw", "reviewer": "Reviewer_YB2E", "summary": "The paper proposes a method for category-level 3D mesh reconstruction from single image. The key innovation is a correspondence-driven optimization layer that produces camera pose and object deformation given a single image, as opposed to using conventional pose CNNs. Overall, the idea is sensible, and the results are better than prior methods under the same setup, either given keypoint annotation or without keypoint annotation. My concerns is mostly on significance and writing.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality and significance:\n- The problem is well-motivated and the discussion of bad local minima in NR-SFM is helpful for understanding the significance of the problem. \n- The proposed solution of casting the pose estimation problem as correspondence regression + optimization is also sensible. I can see how the extra optimization step could produce more accurate shape at test time. However, it's not obvious why it will address the bad local minima issue at training time. It would help to provide further insights or experiment evidence.  \n\nClarity:\n- The presentation is clear and easy to follow in general.\n\nMethod:\n- The method section is well written in general. However, some details are missing from Sec 3.2 and makes it difficult to evaluate the technical significance. For instance, l136 the authors mentioned the optimization is lightweight, but without mentioning the complexity of each iteration and number of iterations needed for convergence.  In table2, the number of iterations is reported but it's not clear whether this is at test time or training time. It will help to compare to prior CNN-based methods in terms of accuracy and speed.\n- From an input image, TTP first predicts 2D-to-3D-template correspondence, and then solves a nonrigid perspective-n-points problem that produces camera pose and shape deformation. However, I'm able to find a concrete description of how is correspondence trained. Is it fair to say the correspondence is trained by cycle consistency loss L_cycle?\n- It is not clear how much deformation can the proposed approach handle. In Eq (7) the authors models shape deformation with a linear shape basis, that has a closed-form solution. However, a linear basis is likely to be insufficient for modeling large motion, such as articulation of human bodies.\n\n\nMinor\n- l106: I'm not following why \"3D to 2D\". From the description, it's the other way around.\n- l142: weigh->weight", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "7QGFmV4bSFS", "reviewer": "Reviewer_P1yY", "summary": "The paper tackles the problem of 3D shape texture and pose estimation from a single image without 3D, multi-view nor camera calibrations. This is an extremely challenging problem. Concurrent works are CMR, U-CMR, CSM and A-CSM. **The key novelty** is to avoid learning pose, texture and shape with several networks heads. Instead, the author propose to learn the 2D positions corresponding to the 3D template vertices, then solve a per-sample optimisation to estimate the pose and template deformation, enabling texture recovery through differentiable rendering. They use implicit differentiation to differentiate through the L-BFGS solver of their inner PnP problem. They achieve successful reconstruction of cars, birds and motorcycles. They outperform their competitors on the Cub dataset and Pascal3D+ dataset, showing exciting successes even without keypoint supervision.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n# Writing\n\nThe method is complicated, and the high quality of the writing helps communicate hard concepts.\nHow the method relates to the state-of-the-art is not really discussed  in the related work section. I am not sure \"Non-rigid structure from motion (NR-SfM)\" deserves a section.\n  \n\n\n\n# Strengths\n\nThe method is simpler than U-CMR and sets the new state of the art in shape, texture, and pose estimation from a single image. Avoiding to learn things that can be optimised per-sample seems like a more robust and simple approach.\nI found the approach very original and am excited by the adoption of implicit differentiation in 3D deep learning. I greatly enjoyed the paper.\n\n # Post-rebuttal comment\n\nI have read the other reviews and thank the author for their response. I found this approach very original and maintain my review.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "# Weaknesses\n\n-   A few things are still unclear to me. How are the unseen parts of the object textured? Using the predicted **u** ? Does this make sense considering they are not not supervised by the texture loss due the the visibility term? You may have discussed this in this sentence that I did not understand: “We also apply the loss on the symmetric texture predictions by using a bilateral symmetric viewpoint and average the two viewpoints. The soft symmetry constraint ensures that the texture of the non-visible side is still inline with the visible side.”\n-   The current limitations of the method are not very clear. Is it easy/hard/impossible to train on a new category like humans? What is the current bottleneck limiting the quality of the reconstructions? Data, deformation model, convergence… ?\n\n  ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presented a new, iterative optimization procedure in the reconstruction branch of the conventional 3D reconstruction framework, instead of training network s to predict shape and pose directly as in UMR and ACSM. In general, the reviewers were a bit mixed in the initial review. After the rebuttal and the discussion phase, R1 increased the rating. Please include many of the rebuttal points in the final version, especially run time analysis and comparison with UMR.
Decision: Accept (Poster)