Question: What is the engineering constraint?
Evidence: As obtaining the full labels f (ϕ; x, y, z) for query points (x, y, z) from all the locations P I is nontrivial for complex geometry from real-world (Fig.) and due to additional engineering constraints on runtime loading and memory bottleneck for the sufficiently dense pre-computed query point occupancy labels, we propose to incorporate the loss w.r.t. the spatial gradients. For occupancy,
(1) where P 0+ I and P 0− I denotes inward and outward near-surface query points, and λ or represents the loss weight of the occupancy geometric regularization. The facing direction of the mesh surface (determining inward or outward) can be determined by normals (if available), or via rendering the surface in all views as in the RGBD captures  (with the surface facing toward camera as the "outward" side). The three terms in Eq. 1 correspond to the three types of losses in Fig.(c) ("∇", "+", "−") respectively. For SDF, we set the loss similar to:
where P 0 I denotes the query points on the ground truth surface only, and λ sr and λ sn represents the loss weight for the signed distance geometric regularization term and normal term respectively. The three terms represents the Eikonal regularization, surface zero-SDF loss, and the surface normal loss respectively, with the last term optional. Note now the loss functions in Eq. 1, 2 no longer require the implicit ground truth label for query points far away from the ground truth surface, enabling training with the real-world imperfect scanned data.
In both cases, the availability of surface normals is optional.
Practically, we found only our proposed loss for the occupancy version (Fig.(c), Eq. 1) demonstrates capability to be applicable for the real-world single-view feed-forward scenario, and use Eq. 1 rather than Eq. 2 in all of our experiments. First, learning with the Eikonal term (Eq. 2) suffers severely from its sensitivity of model initialization. We found either the SDF or the Truncated SDF (TSDF) representation model cannot easily converge during training. Specifically, we found it poses numerical difficulties when the model is learned to predict a large distance value or a constant truncation value for the majority of the query points in the air that is far from any surfaces. This becomes a major problem when we extend from the single objects scenario or single scene fitting. Second, our loss function (Eq. 1) can significantly save memory footprint during training and enable large batch size training, which is crucial in our learning framework. Unlike in Eq. 2 where all the query points require spatial gradient computation, which leads to 3× higher of memory footprint, in our loss function (Eq. 1), only the non-surface query points do. We set the query point batch size of the non-surface points (512 in practice) to be much smaller than the critical near-surface points (4096 in practice), enabling learning with the image batch size of 32 in our real-scene training.
As obtaining the full labels ˜f(ϕ; x, y, z) for query points (x, y, z) from all the locations PI is nontrivial for complex geometry from real-world (Fig. 2) and due to additional engineering constraints
on runtime loading and memory bottleneck for the sufficiently dense pre-computed query point
occupancy labels, we propose to incorporate the loss w.r.t. the spatial gradients.
Answer: We revised the caption of Fig 2 and the sentence before Eq. 1. The engineering constraint is referring to the difficulties of storing and run-time loading the sufficiently dense pre-computed occupancy labels for all the samples in the batch (batch size is 32) when training with the pre-computed full occupancy labels (our baseline of DISN + TSDF Voxelization), as we introduced in the introduction section. Hence for this baseline, we train with the relatively sparse labels (the voxels), whose resolution is lower than the pre-computed dense occupancy labels.