To The Point: Correspondence-driven monocular 3D category reconstruction
Year: 2022

Abstract:
We present To The Point (TTP), a method for reconstructing 3D objects from a single image using 2D to 3D correspondences learned from weak supervision. We recover a 3D shape from a 2D image by first regressing the 2D positions corresponding to the 3D template vertices and then jointly estimating a rigid camera transform and non-rigid template deformation that optimally explain the 2D positions through the 3D shape projection. By relying on 3D-2D correspondences we use a simple per-sample optimization problem to replace CNN-based regression of camera pose and non-rigid deformation and thereby obtain substantially more accurate 3D reconstructions. We treat this optimization as a differentiable layer and train the whole system in an end-to-end manner. We report systematic quantitative improvements on multiple categories and provide qualitative results comprising diverse shape, pose and texture prediction examples. Project website: https: //fkokkinos.github.io/to_the_point/.

text: We present To The Point (TTP), a method for reconstructing 3D objects from a single image using 2D to 3D correspondences learned from weak supervision. We recover a 3D shape from a 2D image by first regressing the 2D positions corresponding to the 3D template vertices and then jointly estimating a rigid camera transform and non-rigid template deformation that optimally explain the 2D positions through the 3D shape projection. By relying on 3D-2D correspondences we use a simple per-sample optimization problem to replace CNN-based regression of camera pose and non-rigid deformation and thereby obtain substantially more accurate 3D reconstructions. We treat this optimization as a differentiable layer and train the whole system in an end-to-end manner. We report systematic quantitative improvements on multiple categories and provide qualitative results comprising diverse shape, pose and texture prediction examples. Project website: https: //fkokkinos.github.io/to_the_point/.

heading: 1 Introduction
text: Monocular 3D reconstruction of general categories is a task that humans perform with ease, yet remains challenging for computer vision due to its inherently ill-posed nature: the observed 2D image is the result of a confluence of multiple sources of variation, including non-rigid intra-category shape variation, rigid transforms due to camera pose, as well as appearance variation. CNNs can easily learn to discard appearance variation, yet the treatment of the geometric sources of variability remains elusive. Even though strongly-supervised approaches have delivered compelling results e.g. for human reconstruction [1], for general categories we need to rely on weaker forms of supervision as well as self-supervision stemming from the know-how of computer vision.
3D vision has traditionally relied on correspondences to recover both rigid scenes from 2D images for the Structure-from-Motion (SFM) problem [2, 3, 4] as well as the more challenging problem of recovering Non-Rigid structure from 2D point tracks (NR-SFM) [5, 6, 7, 8]. In all those problems 3D reconstruction is accomplished by minimizing the reprojection error between the 3D positions of the inferred 3D scene and their 2D image correspondences. While these solutions have been developed for the (potentially deformable) single-instance case, the idea of relying on correspondences to supervise monocular 3D reconstruction has transpired in recent deep learning works.
CNN-driven monocular 3D category reconstruction [9, 10, 11, 12] has largely relied on selfsupervision for 3D recovery expressed in terms of correspondence-based loss terms. For instance the geometric cycle loss terms of [13, 14, 15] are explicitly phrased in terms of correspondence established from UV maps while the texture-driven loss terms of [9, 13, 12, 16, 11] are implicitly relying on pixel correspondence. The common ground of such loss terms is that if the 3D shape is predicted correctly, it should project to the image in a way that is consistent with the 2D observations, as measured in a pixel-by-pixel sense. These correspondence terms are typically used in
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
tandem with explicit geometric priors such as 3D symmetry [10, 9], predefined camera viewpoint ranges [14, 15, 10], or predefined object scales [14, 15, 10, 12] in order to tackle the ill-posed nature of the problem and the presence of multiple local minima in the associated learning problem.
Local minima however emerge even in the simpler single-instance case of NR-SFM, while highly sophisticated optimization schemes have been introduced to address them, e.g. [17]. Current CNNbased approaches seem to ignore this problem and further exacerbate it by delegating the solution of 3D reconstruction to back-propagation with SGD: separate network heads are tasked with regressing the camera pose and non-rigid deformation given an image and are trained in an end-to-end manner, aiming to minimize the correspondence-driven losses. We argue that this is making optimization harder: network training aims at simultaneously establishing the association between images and rigid and non-rigid pose parameters as well as solving the 3D reconstruction problem in terms of these parameters. Each of these problems is hard enough in isolation and putting them together makes the optimization even harder.
This challenge is reflected in the complicated numerical schemes currently used to mitigate local minima; for instance [10, 12] use multiple camera hypotheses during both training and testing. The number of hypotheses can range from 8 to up to 40 for a single reconstruction and the hypotheses have to be accompanied with a probabilistic method to select the most accurate pose either predicted by an MLP [14] or using heuristic loss-based weighting schemes [10]. Another example of brittle optimization, even when keypoint supervision is available, are the works of [9, 18] where in a first stage SFM/NR-SFM is used to get the camera pose right based on keypoint supervision, which is then followed by optimization with image-based losses to recover a mesh. This challenge has been observed also in the strongly-supervised case of human pose estimation, and the use of per-sample numerical optimization [19] was shown to improve performance in [20, 21, 22, 23, 24].
In this work we deviate from the current practice of using a CNN to regress camera and mesh deformation estimates. Instead, during both training and testing we solve a per-sample optimization problem that explicitly aims at providing a 3D reconstruction that projects “To The Point" (TTP). We take as input the 2D coordinates corresponding to the 3D vertices of a mesh and recover the 3D vertex positions by optimizing with respect to the rigid and non-rigid pose parameters through differentiable optimization [25, 26]. We obtain the 2D points required by our layer by only relying on mask annotations and optionally a small number of 2D semantic keypoint annotations, as well as self-supervision coming from the 3D reprojection loss. We jointly learn the 2D point regression and the 3D modes of shape variability through end-to-end optimization, while treating the per-instance rigid and non-rigid pose parameters as latent variables that are optimized on-the-fly, per sample.
We claim that predicting the correspondences is not only sufficient, but also more appropriate for driving monocular 3D reconstruction: it spares us from the use of any additional geometric priors and also yields state-of-the-art results while only relying on a single camera hypothesis. We evaluate our approach on 3D shape, pose and texture reconstruction on four objects categories using realworld datasets CUB [27] and PASCAL3D+ [28]. We demonstrate competitive 3D reconstruction quality to previous state-of-the-art methods and our ablation study confirms the importance of the self-supervised losses we employ.

heading: 2 Related Work
text: Monocular 3D reconstruction Recent works on this problem [29, 9, 14, 15, 16, 12, 30] have relied on varying forms of supervision. Earlier approaches [9, 29] treat the problem of 3D reconstruction from single images using known masks and manually labelled keypoints from single viewpoint image collections. Recent works [10, 16, 14] have removed the need for keypoints but introduced multiple viewpoint and deformation hypotheses accompanied with a probabilistic method to select the most accurate pose. [13, 31, 32] jointly recover cameras and non-rigid 3D meshes with single hypothesis-based networks, but limit themselves to simpler, almost planar categories like faces, or exploit symmetry priors, limiting their broader applicability.
Closer to our work is Canonical Surface Mapping (CSM) and Articulated Canonical Surface Mapping (ACSM) [14, 15] where the 3D representation is produced in the form of a rigid or articulated template using a 2D-to-3D cycle-consistency loss.
Non-rigid structure from motion (NR-SfM) The aim of NR-SfM is the recovery of the 3D shape and accompanying camera pose given only 2D landmarks without any explicit 3D supervision [7, 8, 33, 34]. Lately, several deep learning [13, 35, 36] methods have been proposed that surpassed the performance of traditional methods while being considerably faster. All of the aforementioned methods employ different priors to tackle the under-constrained problem of NR-SfM. The priors are embedded into the methods using low-rank subspaces [37, 7, 17], spatio-temporal domains [34, 38], equivariance constraints [35] or sparse basis coefficients using L1 constraints [36, 39, 40].
Learnable Optimization: Common methods for incorporating optimization as layers in deep neural networks include implicit function differentiation [25, 26, 41, 42, 43] and optimization unrolling [44, 45, 36]; we refer to [26, 25] for a survey. In 3D reconstruction recent works address the challenge of incorporating RANSAC in an end-to-end trainable pipeline for camera pose estimation based on the Perspective-n-Point (PnP) problem, such as differentiable blind PnP [42, 43] or DSAC [46]. Unlike these works, we do not have to address the combinatorial nature of correspondence, but rather focus on regressing the 2D image positions of a 3D template with a fixed number of vertices.

heading: 3 To-The-Point Monocular 3D Mesh Reconstruction
text: We start in Sec. 3.1 by introducing the 2D quantities predicted by our network, we then present our differentiable camera and mesh optimization layer in Sec. 3.2, and finally present the losses driving our end-to-end training in Sec. 3.4.

heading: 3.1 Predicting 3D to 2D Correspondences
text: Our method assumes that the template of our object category can be described in 3D in terms of N points. As shown in Fig. 1, given an image, we use a CNN, φθ, to regress the 2D coordinates u ∈ RN×2 corresponding to these N 3D points. We also predict a visibility vector v where every vi ∈ [0, 1] indicates whether the 2D to 3D correspondence is occluded in the image. Recognition of occluded points allows for accurate camera pose estimation by eliminating the influence of noisy predicted points belonging to the non-visible areas of the object.

heading: 3.2 Estimation of Pose and Deformation
text: Our aim is to estimate the camera pose and object deformation using only the predicted 2D points u, the visibility vector v and the template mesh T. We first introduce our assumptions about the rigid and non-rigid part of the shape and then turn to the resulting optimization problem.
Firstly, as in [9, 10, 14, 15, 16, 11], we model the 3D-to-2D projection through weak perspective [2]. This involves a 2× 3 3D-to-2D “scaled orthographic” projection matrix of the following form:
C = [ s 0 0 0 s 0 ] , (1)
where the scaling factor s accounts for global scaling due to depth variation; given a set of 2D image points this is set to their standard deviation, yielding invariance of the pose parameters to similarity transforms of the 2D and 3D coordinate frames. The rigid pose parameters comprise a rotation R and translation t that account for viewing a 3D object V from a given camera position. The parametric estimates for the 2D projections of a 3D object can thus be obtained as follows:
û(R, t) = C(RV + t) (2)
where V is 3×N , u is 2×N , and we overload notation for + assuming that t is replicated N times.
Having covered the rigid object case, we now turn to the modeling of non-rigid categories. For this we rely on the deformable template paradigm [47] commonly used also in the NR-SFM literature [7, 8], and obtain a shape estimate V by adding offsets ∆V = Bc to a template shape T, yielding V = ∆V +T. Combined with Eq. 2, we have the following parametric estimate of the 2D positions:
û(C,R, t) = C(R(T+Bc) + t) (3)
which is bilinear in R, t and c.
So far we do not deviate substantially from recent works [9, 10, 16, 11] in terms of modelling: these also rely on the morphable model paradigm [47] and regress a shape update ∆V and a rotation matrix through network heads. A minor modelling difference is that we have a low-rank model for Bc, while their shape update is driven by a high-dimensional latent vector. Our main difference is that rather than delegating to the network the task of predicting the ‘right’ values of R, t,V, we directly optimize for them through a lightweight and differentiable optimization scheme by exploiting our regressed 2D correspondences. This ‘optimizes out’ these parameters and ensures our 3D inference will project as accurately as possible to the 2D points, rather than delegating the optimization to backprop and the network heads.
Our “To-The-Point” approach aims at minimizing the following re-projection error between the predicted 2D points u and the 2D point estimates deliver by the parametric, 3D-based prediction:
l(R, t, c) =
N∑
i=1
vi ‖ui − ûi(C,R, t)‖ 2 2 + γ ‖c‖ 2 2 , (4)
where we weigh the discrepancy between the two quantities by the regressed visibility, ensuring that the reconstruction process is robust to occluded points. We also add a regularization weight γ on the expansion coefficients to avoid instabilities in the first stages of training, when the basis B is still unknown and can lead to prematurely committing to large arbitrary deformations.
To minimize the loss term in Eq. (4), we use an alternating optimization scheme and perform separate updates for camera pose estimation and mesh deformation:
R̂t, t̂t = argmin R,t l(R, t, ĉt), subject toR ∈ SO(3) (5)
ĉt+1 = argmin c l(R̂t, t̂t, c) (6)
where at each step we use some of the previously estimated quantities (denoted by hat) as fixed and optimize with respect with the remaining ones to update their estimates.
Starting from the optimization in Eq. 5, we satisfy the constraint that R ∈ SO(3) by using the angleaxis representation r of the rotation Rr such that Rr = exp[r]× where [·]× is the skew symmetric operator . The underlying non-linear problem is solved using the L-BFGS optimizer [48] and when t > 1 is initialized with the estimate of the previous iteration. To backpropagate through L-BFGS we use the implicit function theorem as described in [42, 43].
Given the rotation matrix R, we solve Eq 6 in closed form as follows:
ct+1 = (ΩTXΩ+ γI)−1ΩTXΥ. (7)
Each row of Ω ∈ R2N×K is defined as Ωi = CR t+1Bi and X ∈ R 2N×2N is a diagonal matrix containing the visibility vector v. Finally, the vector Υ ∈ R2N is defined as Υ = ui −CR
t+1Ti − Ctt+1. For clarity we provide the analytic derivation of the closed form solution in the supplementary material. We also backpropagate through matrix inversion, meaning that the chaining of the update steps can be treated as a differentiable layer and be used in tandem with end-to-end training.
Even though the only quantities regressed by our CNN are the 2D positions and associated visibilities, the basis B that represents the shape variability of our category in Eq.3 is a parameter of this layer, is randomly initialized, and is estimated through back-propagation. As shown in the supplemental material, the basis elements learned this way can be intuitively understood, while our experimental results indicate that they suffice for the accurate recovery of intricate mesh deformations.

heading: 3.3 Texture
text: The final part of monocular 3D reconstruction is the estimation of the texture of the reconstructed 3D shape. The texture of the object is sampled from the input image utilizing the predicted 2D points u closely resembling the sampling-based texturing method of CMR [9]. Unlike CMR, we do not predict uv locations to sample pixel values for the image with a dedicated learnable regressor, but use the predicted 2D points u that drive our whole 3D reconstruction process. For this we use a sampling-based texture approach where the face color is computed by interpolating the u coordinates and then sampling from the texture map, i.e the input image in our case. Any losses applied on the estimated texture back-propagate information to the predicted correspondences allowing us to use appearance image cues in addition to foreground masks to enable accurate correspondence predictions.

heading: 3.4 Loss terms
text: To train our approach we incorporate different losses focusing on pose estimation, texture prediction as well as mesh regularization.
Texture Loss compares the rendered textured image Ĩ and the image appearance in terms of the perceptual similarity metric of [49] after masking by the silhouette S:
Lpixel = dist ( Ĩ ⊙ S, I ⊙ S ) .
We also apply the loss on the symmetric texture predictions by using a bilateral symmetric viewpoint and average the two viewpoints. The soft symmetry constraint ensures that the texture of the non-visible side is still inline with the visible side. This constraint has been employed in prior works [9, 16], however, unlike the proposed method it is commonly applied in a hard-coded manner by symmetrizing the texture across an axis.
Points Chamfer Distance enforces points to lie inside and cover the silhouette of the depicted object [29, 15]. In order to formulate our loss term we define Cmask as the Chamfer distance field of the binary mask of silhouette S. Silhouette consistency simply enforces the predicted 2D correspondences of an instance to lie inside its silhouette. This can be achieved by penalizing the points projected outside the instance mask by their distance from the silhouette. Silhouette coverage enforces the predicted points ui to fully cover the mask of the depicted object and allows us to predict better camera poses and mesh deformations.
LChamfer = ∑
i
Cmask(ui)
︸ ︷︷ ︸ silhouette consistency
+ ∑
p∈S
min ui ‖ui − p‖2
︸ ︷︷ ︸
silhouette coverage
.
Region similarity loss compares the object support computed from the mesh by a differentiable renderer [50] to instance segmentations S provided either by manual annotations or pretrained CNNs using their absolute distance:
Lmask = ∑
i
|Si − frender(Vi, πi)|.
Cycle and Visibility loss Similarly to CSM [14] we use a cycle loss between the regressed 2D correspondences u and the projected 3D points to ensure that regressed points, that form neighborhoods in the template shape, remain close in the image space. The cycle loss is defined as
Lcycle = ∑ i ‖ui − π (V)‖ 2 2. Furthemore, visibility of correspondences aids the camera pose estimation in weakly supervised cases. The visibility loss encourages the predictor φθ to encode the visible area of the mesh in an image by enforcing the predicted visibilities to be similar to those of the rendered z-buffer vgt
Lvis = ∑
i
∥ ∥vi − v gt i ∥ ∥ 1 .
Equivariance Loss The point regressor should be robust to the pose variations. For each training image, we draw a random spatial transform Ts(·) from a predefined parameter range. We use random affine transformations (scale, rotation, and shifting) for spatial transforms as well as vertical flipping Tv(·). The detailed transform parameters are present in the supplementary material. We pass both the input image I and transformed image I ′ = Ts (I) through the φθ network and obtain the corresponding predictions u and u′. For vertical flipping we retrieve two pose estimations R and R′ for I and the flipped image Tv(I). We compute the equivariance loss as follows:
Leqv = ∑
i
‖u′i − Ts(ui)‖1 + arccos 1
2 (Tr(Tv(R)R
′)− 1) .
(Optional) Keypoint reprojection loss While we are primarily interested in training without any manual annotations, our approach can be extended to leverage an arbitrary number of high-level semantic keypoints. This is achieved by setting manually the 3D keypoints on the template mesh and encoding them as a matrix K acting on the mesh. The structure of K entails that each Ki is a fixed vector that regresses the i−th 3D semantic keypoint from the mesh. Given the 2D annotations for an image I and a camera π, a keypoint reprojection loss is formed between the groundtruth annotation and the projected 3D points:
Lkp = ∑
i
‖ki − π (KiV)‖1 .
As-rigid-as-possible (ARAP) constraint Without any mesh deformation regularization, the predicted mesh deformation will lead to arbitrary deformations exhibiting spikes and other anomalies. As such, we use the as-rigid-as-possible (ARAP) [51] constraint as a loss function similar to [11]. The predicted shape V is a locally rigid transformation from the predicted base shape T by:
Larap (T,V) = 1
N
N∑
i=1
∑
j∈N (i)
wij ∥ ∥ ( Vi −Vj ) −Ri ( Ti −Tj )∥ ∥ 2
where N (i) represents the neighboring vertices of a vertex i, wij the cotangent weights and Ri the best approximating rotation matrix, as described in [51]. Beyond mesh regularization, the same loss is applied on each basis component that leads to smooth and locally rigid components.
Even with ARAP there are cases where the network will squeeze the non-visible side of the reconstructed object. This erroneous deformation is not penalized by the ARAP loss, as long as it is locally rigid, and causes the method to predict flattened meshes. We further apply an l2 constraint to the deformations to penalize the method to retain the original volume of the template.

heading: 4 Experiments
text: Datasets We present extensive ablation results and comparisons on bird reconstruction, as well as quantitative results on three more object categories (planes, cars, motorbikes). For birds we use the CUB [27] dataset for training and testing on birds which contains 6000 images. The train/val/test split we use for training and report is that of [9]. For the rest of the objects we use the Pascal3D+ dataset [28] and the associated pre-defined training and validation sets. Similarly to [9], we use both PASCAL VOC and Imagenet images to train our models and use Mask-RCNN [52] to obtain foreground masks for the ImageNet subset. For templates, we use identical to those of CSM [14] for CUB dataset and for PASCAL3D+ we select one of the available CAD models for each object.
Evaluation Metrics We evaluate our model on the CUB dataset [27] and report both the mean Intersection over Union (mIoU) and keypoint reprojection accuracy (PCK) following CMR [9].
For Pascal3D+ we report a canonical 3D mean Intersection over Union metric which measures the 3D overlap between the groundtruth and predicted deformed mesh; in order to compute the overlap, both meshes are voxelized using a 32 grid size before computing the 3D mIoU as in [10, 9, 53, 29].
Network Architecture Following prior work [9], we use a ResNet18 encoder to map an image I to a latent feature map z ∈ R4×4×256. The position regressor is a fully connected layer having as input the flattened feature map z and outputs the regressed 2D positions u ∈ R|V |×2 and their respective visibility v ∈ R|V |×1. The number of basis components is set to K = 16 and the number of iterations for the camera and deformation estimation is four.
Network Training To train the 3D reconstruction model we first warm up the model without applying any deformation for 100 epochs. This warm-up process allows the model to find the best pose possible given the rigid template using available cues like masks, texture and optional keypoints. We then train the full 3D reconstruction network with deformation enabled and all available cues for another 100 epochs. All training details can be found in the supplementary material. All experiments were run on a single RTX 2080 Ti GPU.

heading: 4.1 Quantitative Results
text: Evaluation on CUB In Table 1 we evaluate TTP on the CUB dataset and report the average and standard deviation of 5 experiments with different seeds; the rigid part of the results table indicates the performance of models that do not use a deformable component, while the non-rigid part amounts to the more challenging problem of estimating both camera and mesh deformations.
We observe that our method outperforms the baseline models on both reported metrics, i.e. mean IoU and keypoint re-projection accuracy, while requiring no camera priors. When using 2D keypoint supervision (upper part of the table) our method achieves the best results, outperforming the closest baseline by almost 8 accuracy points. For the case where no 2D keypoints are used the only published result in the literature is the ACSM approach of [15] which relies on 8 camera hypotheses during both training and testing, while also using manual annotation of part-based rigs to bootstrap the deformation model. Our work outperforms this baseline while relying on a single camera hypothesis and without requiring any manual mesh annotation.
We posit that using multiple cameras in [15] aims at mitigating the local minima in network training and optimization. We have therefore rerun the system of [15] with a single camera and five different optimization seeds and observed a further gap in performance compared to our work, as well as a larger variance in the reconstruction accuracy compared to that of our work for the non-rigid case, suggesting a potentially higher chance of getting stuck in local minima.
Ablation study We ablate various terms in our learning objective and report the mIoU and the semantic keypoint reprojection (PCK) metrics. In particular we examine the impact of removing any of the utilized losses in Table 3a. When using keypoint supervision the differences in performance are small. However in the absence of keypoints, the method struggles to align the template with the depicted object when we remove the visibility loss. While mIoU remains high, PCK score decreases substantially meaning that the pose and deformation of the template cover the foreground mask when rendered but not from a proper viewpoint of the object. Similar performance drop occurs without the equivariance loss since the method produces pose estimates biased towards one vertical direction. Finally, removing the texture loss causes mIoU performance to drop significantly.
In Table 3b we study how performance changes as a function of the number of basis elements. Increasing the number of components tends to increase performance but up to 16 elements for both set of experiments. When training with semantic keypoints increasing the number of basis components further improves performance, however since the same does not apply to the mask-only case we have set K = 16 for all of our experiments.
Finally, a key aspect of TTP is the iterative pose and deformation estimation process. In Table 2, we provide the mIoU and PCK scores for every iteration for two experiments trained with and without keypoint supervision. Multiple iterations have to be executed to get the best performance, however TTP’s performance with a single iteration still outperforms prior work for both metrics.
We are complementing these quantitative results with qualitative results in Figure 2 where we show that we can correctly deform the template mesh to produce highly accurate 3D reconstructions.
Evaluation on Pascal3D+ While our primary evaluation is on the CUB dataset, we run supplementary experiments on the cars, airplanes and motorcycle categories of PASCAL3D+ dataset. For cars and aeroplanes we provide comparisons against CMR [9], UCMR [10], a volumetric prediction network [53] and a fitting based method [29]. Three of the methods use segmentation masks, cameras and keypoints for supervision except UCMR that does not require keypoints.
We train our method with and without keypoint supervision and provide our 3D mIoU results in Table 4. We observe that our method performs considerably better than competing methods even when

heading: 4.3 Run-time Analysis
text: In Table 5 we provide a run time analysis of our and prior methods on the task of self-supervised 3D reconstruction. The analysis was performed on a machine with a NVidia RTX 2080Ti, an Intel Xeon W-2255 and 32 GBs of RAM. We report the average of 20 runs for the reconstruction of a 256x256 image. To ensure that the benchmark is fair for all methods, we compute the execution time between the moment an image is given as input to a network up to the moment where the network predicts the mesh, the camera pose and the texture; the implementations of the methods used for comparison are the publicly available ones from the original authors. As reported in Table 5, TTP is faster than ACSM by a considerable margin even on the CPU. CMR is the fastest method of all due to its simplicity, however, it requires keypoint supervision and has substantially lower results as indicate in Table 1.
We note that our implementation relies on PyTorch and we have used PyTorch’s python-based LBFGS optimizer for convenience and autograd for Jacobian computation; understandably for AR applications the LBFGS timing can become substantially faster in C or custom CUDA kernels and with explicitly coded Jacobian matrix computation.

heading: 5 Discussion
text: We have proposed a method to reconstruct 3D meshes, poses and textures of generic objects in the wild without any direct supervision. We learn unsupervised correspondences between 2D image locations and 3D template vertices and use them to compute the camera pose and deformation of the object. Even though our CNN architecture predicts substantially fewer outputs - compared e.g. to [9] where all of the 3D vertices and the camera are directly regressed by the network, we deliver substantially better results. We attribute this to the use of a direct optimization scheme to optimize the 3D reconstruction problem both during training and testing. The resulting optimization problem is particularly lightweight, meaning that it can be used for interactive applications, e.g. in Augmented Reality, while our results indicate that even a single step of the optimization suffices for accurate mesh recover. In future work we aim to extend our approach to cover categories with diverse topologies (e.g. chairs) as well as exploit video-based supervision [12] to further improve accuracy.

Review summaries:
This paper tackles the problem of learning single-image 3D object reconstruction from single-view collections. The proposed method couples a correspondence prediction network with an iterative optimization procedure that solves shape and pose based on the predicted correspondences, starting with a category template shape. The claim is that, with this iterative optimization procedure in the loop, it alleviates additional supervision for camera viewpoint such as keypoints, or multiple manual hypotheses.  The results show that this method is able to achieve comparable reconstruction accuracy without keypoints compared to previous methods that require keypoints. Overall, this paper presents an interesting approach of incorporating iterative optimization in a shape learning framework, but the method also has its limitations and the evaluation is rather limited. 

The paper tackles the problem of 3D shape texture and pose estimation from a single image without 3D, multi-view nor camera calibrations. This is an extremely challenging problem. Concurrent works are CMR, U-CMR, CSM and A-CSM. **The key novelty** is to avoid learning pose, texture and shape with several networks heads. Instead, the author propose to learn the 2D positions corresponding to the 3D template vertices, then solve a per-sample optimisation to estimate the pose and template deformation, enabling texture recovery through differentiable rendering. They use implicit differentiation to differentiate through the L-BFGS solver of their inner PnP problem. They achieve successful reconstruction of cars, birds and motorcycles. They outperform their competitors on the Cub dataset and Pascal3D+ dataset, showing exciting successes even without keypoint supervision. 

The paper proposes a method for category-level 3D mesh reconstruction from single image. The key innovation is a correspondence-driven optimization layer that produces camera pose and object deformation given a single image, as opposed to using conventional pose CNNs. Overall, the idea is sensible, and the results are better than prior methods under the same setup, either given keypoint annotation or without keypoint annotation. My concerns is mostly on significance and writing.