Question: What is the relationship between x, y, z and i, j?
Evidence: A distinct difference compared to existing works is that our spatial gradient ∇ x,y,z fΘ (ϕ; x, y, z) is also conditioned on the pixels in ϕ. This is in stark contrast to existing works. devised the model f Θ to fit to a single scene, and the spatial gradient ∇ x,y,z fΘ (x, y, z) can be conveniently computed because it does not involve the sampling procedure.; used a non-spatial global feature for inference and hence bypassed sampling. In our learning framework, the spatial gradient computation must undergo the sampling procedure.
We name our gradient computation involving the sampling operation as the Pixel Conditioned Gradients and derive a closed-form solution, Differentiable Gradient Sampling (DGS), for handling forward and backward propagation. Figure(a) provides an illustration of our training pipeline. Each layer in our network tracks both the response of the layer as well as its spatial gradient w.r.t. (x, y, z). While it is well established to track the layer-wise spatial gradient for fully-connected (FC) layers or convolutions in existing works, tracking the spatial gradients and back-propagating the loss to the feature map pixels ϕ through the sampling module has not been studied. To this end, we derive the closed-form sampling scheme for tracking and propagating the spatial gradients ∂ϕ ∂x,y,z through the sampling layer. Background -2D Differentiable Sampling. Differentiably sampling pixel values from a grid of 2D feature map with the given pixel locations (i, j) is a common operation. Throughout our paper, we define the pixel coordinates in the Normalized Device Coordinate (NDC) system that ranges from -1 to 1. As illustrated in Fig., given the feature map ϕ and the sampling locations (i, j), the resulting sampled value ϕ(i, j) is
Please refer to Fig.(c) for the definitions of α, β and ϕ A , ϕ B , ϕ C , ϕ D . Without loss of generality we use bilinear interpolation. During training, the gradient from the loss can be back-propagated via
2D Differentiable Gradient Sampling. Our learning framework (Sec. 3.1, Fig.(a)) requires the extension of the sampling capability from just the feature value response ϕ i,j to its spatial gradient ∇ i,j ϕ(i, j). During the forward and the backward propagation, both the sampled feature response ϕ i,j and its spatial gradient ∇ i,j ϕ(i, j) are recorded for further propagation (Fig.(e)):
Hence, during the forward pass, we compute the spatial gradient via Eq. 5 in addition to the existing value sampling (Eq. 3). During the backward pass, we compute the loss gradient over the spatial gradient via
where w and h are the width and height of a pixel in NDC, s.t. w = 2/W and h = 2/H for the feature map with the size W × H.
3D Differentiable Gradient Sampling. We now extend the sampling to 3D. We model the camera as the pin-hole camera. For any point (x, y, z) in the camera space, we seek for its projected 2D locations (i, j) based on the focal length
During the backward propagation procedure, the DGS accumulates the gradient via Comparison with state-of-the-art approaches. We choose to compare to recent representative state-of-the-art approaches, including OccNet, DISN, CoReNet and D 2 IM-Net. Quantitative results are reported in Tab. 3 for the low-realism evaluation setting, and in Tab. 1 for the high-realism setting, where our results outperform all the state-of-the-art results in all the settings. Please refer to Sec. A for qualitative comparison, implementation details and the evaluation metrics.
For our low-realism evaluation setting, all the baseline approaches reported their results in the papers. For OccNet and DISN, the reported results are based on knowing the category canonical view prior, which demonstrates considerable privilege with respect to accuracy. Hence, we mark the results from the literature as OccNet-Privilege and DISN-Privilege (OccNet-Priv. and DISN-Priv. for short). Compared to their privileged setting, our results demonstrate superior results even without the category canonical view prior privilege. To further provide the the baseline results where these two approaches are without the category canonical view prior, we retrain their models with the released codes, and report the results in the "OccNet" and "DISN" row respectively. The results further provide evidence that the category canonical view prior demonstrates privilege on accuracy, as observed in.
For our high-realism evaluating setting, since the results for Occnet and DISN are not directly available from the paper, we retrain their model from the released codes (without the category canonical view prior) on the high-realism data provided by. We also retrain CoReNet using their released codes and obtain results better than reported in their original paper (61.5 as mIoU instead of 59.5). Our model still consistently outperforms all the baselines on almost all the categories. We emphasize that our comparison to CoReNet is a direct ablation comparison where we directly follow their experimental setup and model size. Furthermore, our approach only utilizes the closeto-surface occupancy labels, which is a subset of the whole occupancy labels for all the query point.
Our model still outperforms the baseline and provides convincing evidence that our supervision loss function demonstrate better results than the original CoReNet approach. We attribute the superiority of our approach to our particular focus toward the difficult close-to-surface region similar to hard negative mining. Ablation Study. We further conduct ablation studies to further validate the importance of derived DGS module, via attempting work-around training methods without DGS. i) NoGrad -To test the performance of the baseline when only the surface data are provided, we train this ablation model in exactly the same way compared to its original model, with the only exceptions that only the near-surface points are equipped with training labels, and we mask out all the training labels from the non-surface query points (where it is not necessary to backpropagate the loss gradient of the spatial gradient using DGS). To further evaluate how the rate of known voxels affects the learning performance, we enlarge the near-surface region and evaluate when the rate is 10%, 30% and 50%. An increase in the performances among these baselines would indicate the importance of knowing more voxel labels if our proposed gradient loss (Eq. 1) is not imposed.
ii) FixedE -We train with both the near-surface as well as the spatial gradient supervision, without DGS -meaning the loss gradient of the spatial gradient would not back-propagate to any module before the sampling module -in our case, the feature encoder network. Note all the other losses without gradient sampling can still back-propagated to the feature maps. We report the ablation results in Tab. 1. Both experiments are conducted with the high-realism ShapeNet data. Once again, we comfortably outperform all the baselines, validating the essential roles of DGS in our learning framework.
Figure: Qualitative comparisons on one challenging test case on ScannetV2. For each predicted surface with red and sky-blue colors, sky-blue indicates "positive precision" for that surface region, while red indicates "negative precision". For each ground truth surface on the top-right corner of each prediction with gold and navy-blue colors, navy-blue indicates "positive recall", while gold indicates "negative recall". The larger the blue region is, the higher F1 score would be. 4.2 LEARNING FROM REAL SCANNED DATASETS (SCANNETV2) Dataset. We use ScannetV2  for training and evaluating the performance of the models on the real images. We follow the standard training / testing split as used in and, where 1513 scenes are for trainval (with 1201 for training and 312 for validation), and 100 for testing. Each scene is provided with multiple image capture as well as the associated camera pose. We train the models with all the views given in the training / validation set (2423872 frames in total, after filtering out frames with invalid extrinsic poses), while for testing, we select 10 frames with different extrinsic poses for each test scene. Practically, since all the frames of the scenes are in the form of video clips, with adjacent frames associated with similar extrinsic cameras poses, we select the 10 frames for each frame via extracting every 100 frames from each scene video (e.g. Frame 1, 101, 201, ..., 901), resulting in 1000 frames in total in our test set (10 frames per scene, with 100 test scenes).
Metrics. We use the same evaluation metrics following and.
Since we are the first, to our knowledge, to evaluate single view 3D implicit reconstruction on the ScannetV2 benchmark, and here we only evaluate the geometries within the camera view frustum rather than the whole scene geometries as in;. In addition, we also only evaluate the geometries in front of the amodal depth for each pixel ray. We define the amodal depth for a pixel ray to be delineated by the minimum between the closest structure-category surface (e.g. walls and doors, etc) and the farthest surface. In practice, in order to accommodate the evaluation of the surfaces right on the amodal depth, we slack the evaluation scope with a factor λ (1.05 in our case) multiplied with the amodal depth. This evaluation protocol would be equivalent to the "single-layered" protocol used in, within our single-view scenario. Due to the inherent ambiguity of the scaling and shifting of the predicted 3D single view geometry, we follow them by regressing the best scale and shift comparing the predicted depth with the ground truth depth. For approaches that predict 3D surfaces, the rendered depth map from the predicted mesh would be used for calculating the scale and shift.
Baselines. Since we demonstrate the first attempt to conduct the implicit reconstruction of singleview scenes, very few exiting works provide a direct baseline performance to our task. For Occ-Net, since its image feature extraction is not local, and its gradient propa- Figure: Two representative failure cases of our approach. gation does not require sampling, we use its direct application with Gropp et al. () "OccNet + GeoReg" as one baseline. We train DISN with the TSDF voxelization labels. For CoReNet, we stick to its own voxelization and internal filling toolbox for obtaining the training labels. Since CoReNet can only predict geometries within the fixed range of space, we tried our best to pick the best cube location based on the dataset statistics. We also incorporate depth approaches for comparison by finetuning their weights on ScannetV2. Lastly, we compare with the two ablation models NoGrad and FixedE as introduced in Sec. 4.1.
Results. We provide quantitative comparison in Tab. 2 and qualitative comparison in Fig. respectively. Our model outperforms all state-of-the-art approaches as well as the ablation models. Compared to the synthetic data scenario in Tab. 3 and 1, our model demonstrates even larger advantages when compared to existing state-of-the-art approaches, as our motivation stems from addressing learning from imperfect 3D labels directly from the real scan data. Compared to single-view depth prediction approaches trained on massive data, our approach does not prevail on "Acc" and "Prec". This is due to the fact that these two metrics only project the predicted surface to the ground truth surface, giving advantages to approaches that only predict the visible surface. Our approach still prevails for other metrics (Chamfer Distance and F1), which are considered as the most important metrics. Generalization to Unseen Scenes (Pix3d and Open-Domain Images). We further test our model without further finetuning directly on unseen scenes for evaluating the generalizability of the learned model. We tried our model on Pix3d as well as test images downloaded directly from the internet. We provide a detailed qualitative comparison in Fig. and more results in Fig.. The results further indicate our learning framework exhibits promise for unseen scene generalization. Failure Cases. We provide two representative failure cases of our approach in Fig.. The first case (the first row) demonstrates difficulties in predicting the floor occupancy as a result of the noisy and non-watertight mesh during training. The second case (the second row) shows that our model cannot identify objects (e.g. chairs) and well predicts the invisible part of them.
A distinct difference compared to existing works is that our spatial gradient ∇ x,y,z fΘ (ϕ; x, y, z) is also conditioned on the pixels in ϕ. This is in stark contrast to existing works. devised the model f Θ to fit to a single scene, and the spatial gradient ∇ x,y,z fΘ (x, y, z) can be conveniently computed because it does not involve the sampling procedure.; used a non-spatial global feature for inference and hence bypassed sampling. In our learning framework, the spatial gradient computation must undergo the sampling procedure.
We name our gradient computation involving the sampling operation as the Pixel Conditioned Gradients and derive a closed-form solution, Differentiable Gradient Sampling (DGS), for handling forward and backward propagation. Figure(a) provides an illustration of our training pipeline. Each layer in our network tracks both the response of the layer as well as its spatial gradient w.r.t. (x, y, z). While it is well established to track the layer-wise spatial gradient for fully-connected (FC) layers or convolutions in existing works, tracking the spatial gradients and back-propagating the loss to the feature map pixels ϕ through the sampling module has not been studied. To this end, we derive the closed-form sampling scheme for tracking and propagating the spatial gradients ∂ϕ ∂x,y,z through the sampling layer. Background -2D Differentiable Sampling. Differentiably sampling pixel values from a grid of 2D feature map with the given pixel locations (i, j) is a common operation. Throughout our paper, we define the pixel coordinates in the Normalized Device Coordinate (NDC) system that ranges from -1 to 1. As illustrated in Fig., given the feature map ϕ and the sampling locations (i, j), the resulting sampled value ϕ(i, j) is
Please refer to Fig.(c) for the definitions of α, β and ϕ A , ϕ B , ϕ C , ϕ D . Without loss of generality we use bilinear interpolation. During training, the gradient from the loss can be back-propagated via
2D Differentiable Gradient Sampling. Our learning framework (Sec. 3.1, Fig.(a)) requires the extension of the sampling capability from just the feature value response ϕ i,j to its spatial gradient ∇ i,j ϕ(i, j). During the forward and the backward propagation, both the sampled feature response ϕ i,j and its spatial gradient ∇ i,j ϕ(i, j) are recorded for further propagation (Fig.(e)):
Hence, during the forward pass, we compute the spatial gradient via Eq. 5 in addition to the existing value sampling (Eq. 3). During the backward pass, we compute the loss gradient over the spatial gradient via
where w and h are the width and height of a pixel in NDC, s.t. w = 2/W and h = 2/H for the feature map with the size W × H.
3D Differentiable Gradient Sampling. We now extend the sampling to 3D. We model the camera as the pin-hole camera. For any point (x, y, z) in the camera space, we seek for its projected 2D locations (i, j) based on the focal length
During the backward propagation procedure, the DGS accumulates the gradient via Comparison with state-of-the-art approaches. We choose to compare to recent representative state-of-the-art approaches, including OccNet, DISN, CoReNet and D 2 IM-Net. Quantitative results are reported in Tab. 3 for the low-realism evaluation setting, and in Tab. 1 for the high-realism setting, where our results outperform all the state-of-the-art results in all the settings. Please refer to Sec. A for qualitative comparison, implementation details and the evaluation metrics.
For our low-realism evaluation setting, all the baseline approaches reported their results in the papers. For OccNet and DISN, the reported results are based on knowing the category canonical view prior, which demonstrates considerable privilege with respect to accuracy. Hence, we mark the results from the literature as OccNet-Privilege and DISN-Privilege (OccNet-Priv. and DISN-Priv. for short). Compared to their privileged setting, our results demonstrate superior results even without the category canonical view prior privilege. To further provide the the baseline results where these two approaches are without the category canonical view prior, we retrain their models with the released codes, and report the results in the "OccNet" and "DISN" row respectively. The results further provide evidence that the category canonical view prior demonstrates privilege on accuracy, as observed in.
For our high-realism evaluating setting, since the results for Occnet and DISN are not directly available from the paper, we retrain their model from the released codes (without the category canonical view prior) on the high-realism data provided by. We also retrain CoReNet using their released codes and obtain results better than reported in their original paper (61.5 as mIoU instead of 59.5). Our model still consistently outperforms all the baselines on almost all the categories. We emphasize that our comparison to CoReNet is a direct ablation comparison where we directly follow their experimental setup and model size. Furthermore, our approach only utilizes the closeto-surface occupancy labels, which is a subset of the whole occupancy labels for all the query point.
Our model still outperforms the baseline and provides convincing evidence that our supervision loss function demonstrate better results than the original CoReNet approach. We attribute the superiority of our approach to our particular focus toward the difficult close-to-surface region similar to hard negative mining. Ablation Study. We further conduct ablation studies to further validate the importance of derived DGS module, via attempting work-around training methods without DGS. i) NoGrad -To test the performance of the baseline when only the surface data are provided, we train this ablation model in exactly the same way compared to its original model, with the only exceptions that only the near-surface points are equipped with training labels, and we mask out all the training labels from the non-surface query points (where it is not necessary to backpropagate the loss gradient of the spatial gradient using DGS). To further evaluate how the rate of known voxels affects the learning performance, we enlarge the near-surface region and evaluate when the rate is 10%, 30% and 50%. An increase in the performances among these baselines would indicate the importance of knowing more voxel labels if our proposed gradient loss (Eq. 1) is not imposed.
ii) FixedE -We train with both the near-surface as well as the spatial gradient supervision, without DGS -meaning the loss gradient of the spatial gradient would not back-propagate to any module before the sampling module -in our case, the feature encoder network. Note all the other losses without gradient sampling can still back-propagated to the feature maps. We report the ablation results in Tab. 1. Both experiments are conducted with the high-realism ShapeNet data. Once again, we comfortably outperform all the baselines, validating the essential roles of DGS in our learning framework.
Figure: Qualitative comparisons on one challenging test case on ScannetV2. For each predicted surface with red and sky-blue colors, sky-blue indicates "positive precision" for that surface region, while red indicates "negative precision". For each ground truth surface on the top-right corner of each prediction with gold and navy-blue colors, navy-blue indicates "positive recall", while gold indicates "negative recall". The larger the blue region is, the higher F1 score would be. 4.2 LEARNING FROM REAL SCANNED DATASETS (SCANNETV2) Dataset. We use ScannetV2  for training and evaluating the performance of the models on the real images. We follow the standard training / testing split as used in and, where 1513 scenes are for trainval (with 1201 for training and 312 for validation), and 100 for testing. Each scene is provided with multiple image capture as well as the associated camera pose. We train the models with all the views given in the training / validation set (2423872 frames in total, after filtering out frames with invalid extrinsic poses), while for testing, we select 10 frames with different extrinsic poses for each test scene. Practically, since all the frames of the scenes are in the form of video clips, with adjacent frames associated with similar extrinsic cameras poses, we select the 10 frames for each frame via extracting every 100 frames from each scene video (e.g. Frame 1, 101, 201, ..., 901), resulting in 1000 frames in total in our test set (10 frames per scene, with 100 test scenes).
Metrics. We use the same evaluation metrics following and.
Since we are the first, to our knowledge, to evaluate single view 3D implicit reconstruction on the ScannetV2 benchmark, and here we only evaluate the geometries within the camera view frustum rather than the whole scene geometries as in;. In addition, we also only evaluate the geometries in front of the amodal depth for each pixel ray. We define the amodal depth for a pixel ray to be delineated by the minimum between the closest structure-category surface (e.g. walls and doors, etc) and the farthest surface. In practice, in order to accommodate the evaluation of the surfaces right on the amodal depth, we slack the evaluation scope with a factor λ (1.05 in our case) multiplied with the amodal depth. This evaluation protocol would be equivalent to the "single-layered" protocol used in, within our single-view scenario. Due to the inherent ambiguity of the scaling and shifting of the predicted 3D single view geometry, we follow them by regressing the best scale and shift comparing the predicted depth with the ground truth depth. For approaches that predict 3D surfaces, the rendered depth map from the predicted mesh would be used for calculating the scale and shift.
Baselines. Since we demonstrate the first attempt to conduct the implicit reconstruction of singleview scenes, very few exiting works provide a direct baseline performance to our task. For Occ-Net, since its image feature extraction is not local, and its gradient propa- Figure: Two representative failure cases of our approach. gation does not require sampling, we use its direct application with Gropp et al. () "OccNet + GeoReg" as one baseline. We train DISN with the TSDF voxelization labels. For CoReNet, we stick to its own voxelization and internal filling toolbox for obtaining the training labels. Since CoReNet can only predict geometries within the fixed range of space, we tried our best to pick the best cube location based on the dataset statistics. We also incorporate depth approaches for comparison by finetuning their weights on ScannetV2. Lastly, we compare with the two ablation models NoGrad and FixedE as introduced in Sec. 4.1.
Results. We provide quantitative comparison in Tab. 2 and qualitative comparison in Fig. respectively. Our model outperforms all state-of-the-art approaches as well as the ablation models. Compared to the synthetic data scenario in Tab. 3 and 1, our model demonstrates even larger advantages when compared to existing state-of-the-art approaches, as our motivation stems from addressing learning from imperfect 3D labels directly from the real scan data. Compared to single-view depth prediction approaches trained on massive data, our approach does not prevail on "Acc" and "Prec". This is due to the fact that these two metrics only project the predicted surface to the ground truth surface, giving advantages to approaches that only predict the visible surface. Our approach still prevails for other metrics (Chamfer Distance and F1), which are considered as the most important metrics. Generalization to Unseen Scenes (Pix3d and Open-Domain Images). We further test our model without further finetuning directly on unseen scenes for evaluating the generalizability of the learned model. We tried our model on Pix3d as well as test images downloaded directly from the internet. We provide a detailed qualitative comparison in Fig. and more results in Fig.. The results further indicate our learning framework exhibits promise for unseen scene generalization. Failure Cases. We provide two representative failure cases of our approach in Fig.. The first case (the first row) demonstrates difficulties in predicting the floor occupancy as a result of the noisy and non-watertight mesh during training. The second case (the second row) shows that our model cannot identify objects (e.g. chairs) and well predicts the invisible part of them.
As illustrated in Fig. 4(d), given the feature map ϕ and the sampling locations (i, j),****For any point (x, y, z) in the camera space, we seek for its projected 2D locations (i, j) based on the focal length f0 via i = yf0/z, j = xf0/z.
Answer: i, j are the 2D projection from the 3D camera coordinate system to the 2D pixel coordinate system (the normalized device coordinate NDC used by sampling). We added their definition and relation before Eq. 3 and Eq. 7 in the revision.