SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\tau$  is a temperature hyperparameter and  $\Psi_{k}$  is the superquadric inside-outside function [1] associated to the block  $k$ , such that  $\Psi_{k}(\mathbf{x}) \leq 1$  if  $\mathbf{x}$  lies inside the superquadric and  $\Psi_{k}(\mathbf{x}) > 1$  otherwise. Given a set of  $M$  3D points  $\Omega$ , our final regularization term can be written as:
$$
\mathcal {L} _ {\text {o v e r}} = \frac {1}{M} \sum_ {\mathbf {x} \in \Omega} \max  \left(\sum_ {k = 1} ^ {K} \mathcal {O} _ {k} ^ {\mathrm {3 D}} (\mathbf {x}), \lambda\right). \tag {7}
$$
Note that in practice, for better efficiency and accuracy, we only sample points in the region where blocks are located, which can be identified using the block poses  $\mathbf{p}_{1:K}$ .
**Optimization details.** We found that two elements were key to avoid bad local minima during optimization. First, while transparent meshes enable differentiability w.r.t. the number of primitives, we observed a failure mode where two semi opaque meshes model the same 3D region. To prevent this behavior, we propose to inject gaussian noise before the sigmoid in the transparencies  $\alpha_{1:K}$  to create stochasticity when values are not close to the sigmoid saturation, and thus encourage values that are close binary. Second, another failure mode we observed is one where the planar ground is modeling the entire scene. We avoid this by leveraging a two-stage curriculum learning scheme, where texture maps are downscaled by 8 during the first stage. We empirically validate these two contributions in Section 4.3. We provide other implementation details in the supplementary material.
4.1 DTU Benchmark
Benchmark details. DTU [25] is an MVS dataset containing 80 forward-facing scenes captured in a controlled indoor setting, where the 3D ground-truth points are obtained through a structured
Table 1: Quantitative results on DTU [25]. We use the official DTU evaluation to report Chamfer Distance (CD) between 3D reconstruction and ground-truth, best results are highlighted. We also highlight the average number of primitives found (#P) in green (smaller than 10) or red (larger than 10). Our performances correspond to a single random run (random) and a run automatically selected among 5 runs using the minimal rendering loss (auto). We augment the best concurrent methods with a filtering step removing the ground from the 3D input.
Method	Input	Chamfer Distance (CD) per scene	Mean CD	Mean #P
S24	S31	S40	S45	S55	S59	S63	S75	S83	S105
EMS [42]	NeuS-mesh	8.42	8.53	7.84	6.98	7.2	8.57	7.77	8.69	4.74	9.11	7.78	9.6
EMS [42]	3D GT	6.77	5.93	3.36	6.91	6.52	3.50	4.72	7.08	7.25	6.10	5.82	7.4
MBF [57]	NeuS-mesh	3.97	4.28	3.56	4.76	3.33	3.92	3.63	5.58	5.3	6.07	4.44	53.5
MBF [57]	3D GT	3.73	4.79	4.31	3.95	3.26	4.00	3.66	3.92	3.97	4.25	3.98	16.4
Ours (random)	Image	5.41	3.13	1.57	4.93	3.08	3.66	3.40	2.78	3.94	4.85	3.67	4.6
Ours (auto)	Image	3.25	3.13	1.16	3.02	2.98	2.32	3.40	2.78	3.43	5.21	3.07	5.0
EMS [42] + filter	3D GT	6.32	4.11	2.98	4.94	4.26	3.03	3.60	5.44	3.24	4.43	4.23	8.3
MBF [57] + filter	3D GT	3.35	2.95	2.61	2.19	2.53	2.47	1.97	2.60	2.60	3.27	2.65	29.9
light scanner. We evaluate on 10 scenes (S24, S31, S40, S45, S55, S59, S63, S75, S83, S105) that have different geometries and a 3D decomposition that is relatively intuitive. We use standard processing practices [77, 76, 8], resize the images to  $400 \times 300$  and run our model with  $K = 10$  on all available views for each scene (49 or 64 depending on the scenes). We use the official evaluation presented in [25], which computes the Chamfer distance between the ground-truth points and points sampled from the 3D reconstruction, filtered out if not in the neighborhood of the ground-truth points. We evaluate two state-of-the-art methods for 3D decomposition, EMS [42] and MonteboxFinder (MBF) [57], by applying them to the 3D ground-truth point clouds. We also evaluate them in a setup comparable to ours, where the state-of-the-art MVS method NeuS [70] is first applied to the multi-view images to extract a mesh, which is then used as input to the 3D decomposition methods. We refer to this input as "NeuS-mesh".
Results. We compare our Chamfer distance performances to these state-of-the-art 3D decomposition methods in Table 1. For each method, we report the input used and highlight the average number of discovered primitives #P in green (smaller than 10) or red (larger than 10). Intuitively, overly large numbers of primitives lead to less intuitive and manipulative scene representations. Our performances correspond to a single random run (random) and a run automatically selected among 5 runs using the minimal rendering loss (auto). We augment the best concurrent methods with a filtering step using RANSAC to remove the planar ground from the 3D input. Overall, we obtain results that are much more satisfactory than prior works. On the one hand, EMS outputs a reasonable number of primitives but has a high Chamfer distance reflecting bad 3D reconstructions. On the other hand, MBF yields a lower Chamfer distance (even better than ours with the filtering step) but it outputs a significantly higher number of primitives, thus reflecting over-decompositions.
Our approach is qualitatively compared in Figure 3 to the best EMS and MBF models, which correspond to the ones applied on the 3D ground truth and augmented with the filtering step. Because the point clouds are noisy and incomplete (see  $360^{\circ}$  renderings in our supplementary material), EMS and MBF struggle to find reasonable 3D decompositions: EMS misses some important parts, while MBF over-decomposes the 3D into piecewise planar surfaces. On the contrary, our model is able to output meaningful 3D decompositions with varying numbers of primitives and very different shapes. Besides, ours is the only approach that recovers the scene appearance (last column). Also note that it produces a complete 3D scene, despite being only optimized on forward-facing views.
4.2 Real-Life Data and Applications
We present qualitative results on real-life captures in Figure 4. The first row corresponds to the Campanile scene from Nerfstudio repository [66] and the last four rows correspond to BlendedMVS scenes [75] that were selected in [76]. We adapt their camera conventions to ours and resize the images to roughly  $400 \times 300$ . From left to right, we show a subset of the input views, a rendering overlaid with the primitive edges, the primitives, as well as two novel view synthesis results. For each scene, we run our model 5 times and automatically select the results with the minimal rendering loss. We set the maximum number of primitives to  $K = 10$ , except the last row where it is increased to
Figure 3: Qualitative comparisons on DTU [25]. We compare our model to state-of-the-art methods (augmented with a preprocessing step to remove the 3D ground) which, unlike ours, find primitives in the ground-truth point cloud that is noisy and incomplete. Additionally, our approach is the only one able to capture the scene appearance (last column).
Figure 4: Qualitative results on real-life data. We run our default model ( $K = 10$ ) on scenes from Nerfs-tudio [66] (first row) and BlendedMVS [75] (all other rows). The last row corresponds to results where the maximum number of primitives is increased to  $K = 50$ , yielding 17 effective primitives found.
Amodal view synthesis - Nerfacto [66]
Amodal view synthesis - Ours
Scene editing - Removing ears
Scene editing - Moving arm
Figure 5: Applications. (top) Given a set of views constrained to limited viewpoint variations, we compare amodal view synthesis results using Nerfacto [66] and our approach. (middle) After optimization, we can easily modify the rendered scene by editing the different parts. (bottom) Our primitive-based representation enables straightforward physics-based simulations, such as throwing a ball at the objects or pouring water on the scene.
Table 2: Ablation study on DTU [25]. We report metrics averaged over five runs: number of primitives (#P), Chamfer Distance (CD) and rendering metrics (PSNR in dB and SSIM, LPIPS in%). Best and second best are highlighted, #P variability is emphasized in green (smaller than 5) and red (larger than 5).
Method	#P ↓	CD ↓	PSNR ↑	SSIM ↑	LPIPS ↓
Complete model	4.60 ± 0.23	3.63 ± 0.23	20.5 ± 0.2	73.5 ± 0.6	23.9 ± 0.5
w/o Lparsi	8.86 ± 0.27	3.65 ± 0.22	20.6 ± 0.1	73.7 ± 0.4	23.2 ± 0.4
w/o Lover	4.38 ± 0.19	3.80 ± 0.30	20.4 ± 0.3	73.2 ± 0.7	24.1 ± 0.7
w/o curriculum	4.66 ± 0.30	3.99 ± 0.17	20.4 ± 0.2	72.7 ± 0.5	24.5 ± 0.4
w/o noise in α1:K	3.60 ± 0.21	4.13 ± 0.28	20.0 ± 0.2	72.0 ± 0.6	25.6 ± 0.6
w/o LTV	4.04 ± 0.18	4.58 ± 0.42	19.7 ± 0.3	70.8 ± 1.3	26.5 ± 1.2
w/o Lperc	3.22 ± 0.17	4.80 ± 0.20	19.7 ± 0.1	72.7 ± 0.3	40.0 ± 0.4
$K = 50$  due to the scene complexity. These results show that despite its simplicity, our approach is surprisingly robust. Our method is still able to compute 3D decompositions that capture both appearances and meaningful geometry on a variety of scene types. In addition, increasing the maximum number of primitives  $K$  allows us to easily adapt the decomposition granularity (last row).
In Figure 5, we demonstrate other advantages of our approach. First, compared to NeRF-based approaches like Nerfacto [66] which only reconstruct visible regions, our method performs amodal scene completion (first row). Second, our textured primitive decomposition allows us to easily edit the 3D scene (second row). Finally, our optimized primitive meshes can be directly imported into standard computer graphics softwares like Blender to perform physics-based simulations (bottom).
Ablation study. In Table 2, we assess the key components of our model by removing one component at a time and computing the performance averaged over the 10 DTU scenes. We report the final number of primitives, Chamfer distance and rendering metrics. We highlight the varying number of primitives in green (smaller than 5) and red (larger than 5). Results are averaged over five runs,
Table 3: Effect of hyperparameters on DTU [25]. We evaluate the influence of two key hyperparameters of our model: the maximum number of primitives  $K$  (left) and the parsimony regularization  ${\lambda }_{\text{parsi }}$  (right).
Method	#P↓	CD↓	PSNR↑	SSIM↑	LPIPS↓
K = 10 (default)	4.60	3.63	20.5	73.5	23.9
K = 25	7.00	3.58	21.0	74.6	22.5
K = 50	9.26	3.52	20.9	74.7	22.8
Method	#P↓	CD↓
λparsi = 0.001	7.44	3.61
λparsi = 0.01 (default)	4.60	3.63
λparsi = 0.1	1.30	6.88
(a) Missing parts
(b) Unnatural decomposition
Figure 6: Failure cases. We show typical failure cases of our approach. All models are optimized with  $K = 10$  except the rightmost model which is optimized with  $K = 50$ . See text for details.
(c) Parsimony/fidelity trade-off
we report the means and standard deviations. Overall, each component except  $\mathcal{L}_{\mathrm{parsi}}$  consistently improves the quality of the 3D reconstruction and the renderings.  $\mathcal{L}_{\mathrm{parsi}}$  successfully limits the number of primitives (and thus, primitive duplication and over-decomposition) at a very small quality cost.
Influence of  $K$  and  $\lambda_{\mathrm{parsi}}$ . In Table 3, we evaluate the impact of two key hyperparameters of our approach, namely the maximum number of primitives  $K$  and the weight of the parsimony regularization  $\lambda_{\mathrm{parsi}}$ . Results are averaged over the 10 DTU scenes for 5 random seeds. First, we can observe that increasing  $K$  slightly improves the reconstruction and rendering performances at the cost of a higher effective number of primitives. Second, these results show that  $\lambda_{\mathrm{parsi}}$  directly influences the effective number of primitives found. When  $\lambda_{\mathrm{parsi}} = 0.1$ , this strong regularization limits the reconstruction to roughly one primitive, which dramatically decreases the performances. When  $\lambda_{\mathrm{parsi}}$  is smaller, the effective number of primitives increases without significant improvements in the reconstruction quality.
Limitations and failure cases. In Figure 6, we show typical failure cases of our approach. First, for a random run, we may observe bad solutions where parts of the geometry are not reconstructed (Figure 6a). This is mainly caused by the absence of primitives in this region at initialization and our automatic selection among multiple runs alleviates the issue, yet this solution is computationally costly. Note that we also tried to apply a Gaussian kernel to blur the image and propagate gradients farther, but it had little effect. Second, our reconstructions can yield unnatural decompositions as illustrated in Figure 6b, where tea boxes are wrongly split or a single primitive is modeling the bear nose and the rock behind. Finally, in Figure 6c, we show that increasing  $K$  from 10 (left) to 50 (right) allows us to trade-off parsimony for reconstruction fidelity. However, while this provides a form of control over the decomposition granularity, the ideal decomposition in this particular case does not seem to be found: the former seems to slightly under-decompose the scene while the latter seems to over-decompose it.
We present an end-to-end approach that successfully computes a primitive-based 3D reconstruction given a set of calibrated images. We show its applicability and robustness through various benchmarks, where our approach obtains better performances than methods leveraging 3D data. We believe our work could be an important step towards more interpretable multi-view modeling.
We thank Cyrus Vachha for help on the physics-based simulations; Antoine Guédon, Romain Loiseau for visualization insights; François Darmon, Romain Loiseau, Elliot Vincent for manuscript feedback. This work was supported in part by ANR project EnHerit ANR-17-CE23-0008, gifts from Adobe and HPC resources from GENCI-IDRIS (2022-AD011011697R2, 2022-AD011013538). MA was supported by ERC project DISCOVER funded by the European Union's HorizonEurope Research and Innovation programme under grant agreement No. 101076028. Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union. Neither the European Union nor the granting authority can be held responsible for them.
[1] A. H. Barr. Superquadrics and Angle-Preserving Transformations. IEEE Computer Graphics and Applications, 1981.
[2] I. Biederman. Recognition-by-components: a theory of human image understanding. Psychological review, 1987.
[3] T. Binford. Visual Perception by Computer. In IEEE Conference on Systems and Control, 1971.
[4] R. C. Bolles and M. A. Fischler. A RANSAC-Based Approach to Model Fitting and Its Application to Finding Cylinders in Range Data. In *IJCAI*, 1981.
[5] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu. ShapeNet: An Information-Rich 3D Model Repository. arXiv:1512.03012 [cs.CV], 2015.
[6] T. Chaperon and F. Goulette. Extracting cylinders in full 3D data using a random sampling method and the gaussian image. In VMV, 2001.
[7] W. Chen, J. Gao, H. Ling, E. J. Smith, J. Lehtinen, A. Jacobson, and S. Fidler. Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer. In NeurIPS, 2019.
[8] F. Darmon, B. Bascle, J.-C. Devaux, P. Monasse, and M. Aubry. Improving neural implicit surfaces geometry with patch warping. In CVPR, 2022.
[9] P. E. Debevec, C. J. Taylor, and J. Malik. Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach. In SIGGRAPH, 1996.
[10] T. Deprelle, T. Groueix, M. Fisher, V. Kim, B. Russell, and M. Aubry. Learning elementary structures for 3d shape generation and matching. In NeurIPS, 2019.
[11] M. A. Fischler and R. C. Bolles. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography. Communications of the ACM, 1981.
[12] D. F. Fouhey, A. Gupta, and M. Hebert. Data-Driven 3D Primitives for Single Image Understanding. In ICCV, 2013.
[13] Q. Fu, Q. Xu, Y.-S. Ong, and W. Tao. Geo-Neus: Geometry-Consistent Neural Implicit Surfaces Learning for Multi-view Reconstruction. In NeurIPS, 2022.
[14] Y. Furukawa and C. Hernández. Multi-view stereo: A tutorial. Foundations and Trends® in Computer Graphics and Vision, 2015.
[15] Y. Furukawa and J. Ponce. Accurate, Dense, and Robust Multi-View Stereopsis. In CVPR, 2007.
[16] S. Galliani, K. Lasinger, and K. Schindler. Massively Parallel Multiview Stereopsis by Surface Normal Diffusion. In ICCV, 2015.
[17] J. Gao, W. Chen, T. Xiang, C. F. Tsang, A. Jacobson, M. McGuire, and S. Fidler. Learning Deformable Tetrahedral Meshes for 3D Reconstruction. In NeurIPS, 2020.
[18] A. Geiger and C. Wang. Joint 3D Object and Layout Inference from a Single RGB-D Image. In GCPR, 2015.
[19] S. Goel, G. Gkioxari, and J. Malik. Differentiable Stereopsis: Meshes from multiple views using differentiable rendering. In CVPR, 2022.
[71] Y. Wu, W. Liu, S. Ruan, and G. S. Chirikjian. Primitive-based Shape Abstraction via Nonparametric Bayesian Inference. In ECCV, 2022.
[72] J. Xiao, B. Russell, and A. Torralba. Localizing 3D cuboids in single-view images. In NIPS, 2012.
[73] Y. Yao, Z. Luo, S. Li, T. Fang, and L. Quan. MVSNet: Depth Inference for Unstructured Multi-view Stereo. In ECCV, 2018.
[74] Y. Yao, Z. Luo, S. Li, T. Shen, T. Fang, and L. Quan. Recurrent MVSNet for High-Resolution Multi-View Stereo Depth Inference. In CVPR, 2019.
[75] Y. Yao, Z. Luo, S. Li, J. Zhang, Y. Ren, L. Zhou, T. Fang, and L. Quan. BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks. In CVPR, 2020.
[76] L. Yariv, J. Gu, Y. Kasten, and Y. Lipman. Volume Rendering of Neural Implicit Surfaces. In NeurIPS, 2021.
[77] L. Yariv, Y. Kasten, D. Moran, M. Galun, M. Atzmon, R. Basri, and Y. Lipman. Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance. In NeurIPS, 2020.
[78] A. Yu, V. Ye, M. Tancik, and A. Kanazawa. pixelNeRF: Neural Radiance Fields from One or Few Images. In CVPR, 2021.
[79] J. Zhang. Visibility-aware Multi-view Stereo Network. In BMVC, 2020.
[80] J. Y. Zhang, G. Yang, S. Tulsiani, and D. Ramanan. NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild. In NeurIPS, 2021.
[81] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In CVPR, 2018.
[82] E. Zheng, E. Dunn, V. Joic, and J.-M. Frahm. PatchMatch Based Joint View Selection and Depthmap Estimation. In CVPR, 2014.
[83] Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li. On the Continuity of Rotation Representations in Neural Networks. In CVPR, 2019.
Supplementary Material for Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitives
In this supplementary document, we provide additional results (Appendix A), details on the DTU benchmark (Appendix B) as well as implementation details (Appendix C), including design and optimization choices.
Videos for view synthesis, physical simulations and amodal completion. We present additional results in the form of videos at our project webpage: www.tmonnier.com/DBW. Videos are separated in different sections depending on the experiment type. First, we provide view synthesis videos (rendered using a circular camera path), further outlining the quality of both our renderings and our primitive-based 3D reconstruction. Second, we include videos for physics-based simulations. Such simulations were produced through Blender by simply uploading our output primitive meshes. Note that for modeling primitive-specific motions in Blender (e.g., in our teaser figure), primitives should not overlap at all, thus requiring a small preprocessing step to slightly move the primitives for a clear separation. Because each primitive is its own mesh, this operation is easily performed within Blender. Finally, we provide video results where we perform scene editing and compare our amodal view synthesis results to the ones of Nerfacto introduced in Nerfstudio [66]. Models for amodal synthesis are optimized on a homemade indoor scene built from a forward-facing capture only. We use Nerfstudio for data processing and data convention.
Rendering comparison with SOTA MVS.
For completeness, we provide the rendering performances of NeRF [45], a SOTA MVS method that does not predict multiple parts. In Table 4, we compare PSNR for our approach and NeRF using the results reported in [76] on the intersected set of 6 DTU scenes.
Table 4: PSNR comparison on DTU.
Method	S24	S40	S55	S63	S83	S105	Mean
NeRF	26.2	26.8	27.6	32.0	32.8	32.1	29.6
Ours	19.1	21.8	22.6	23.4	22.3	20.8	21.7
B DTU Benchmark
In Figure 7, we show for each scene a subset of the input images as well as  $360^{\circ}$  renderings of the GT point clouds obtained through a structured light scanner. To compute performances, we use a Python version of the official evaluation: https://github.com/jzhangbs/DTUeval-python.
Icosphere and superquadric UV mapping. We use spherical coordinates that we correct to build our texture mapping for the unit icosphere. Figure 8 shows our process with an example. Specifically, we retrieve for each vertex its spherical coordinates  $\eta \in [-\frac{\pi}{2}, \frac{\pi}{2}]$  and  $\omega \in [-\pi, \pi]$  which are linearly mapped to the UV space  $[0,1]^2$ . Because such parametrization presents discontinuities and strong triangle deformations at the poles, we perform two corrections. First, we fix discontinuities by copying the border pixels involved (using a circular padding on the texture image) and introducing new 2D vertices such that triangles do not overlap anymore. Second, we avoid distorted triangles at the poles by creating for each triangle, a new 2D vertex positioned in the middle of the other two vertices. As detailed in the main paper, we derive a superquadric mesh from a unit icosphere in such a way that each vertex of the icosphere is continuously mapped to the superquadric vertex. As a result, the texture mapping defined for the icosphere is directly transferred to our superquadric meshes without any modification.
Design choices. Except constants related to the world scale, orientation and position in the 3D space w.r.t. to the known cameras, all our experiments share the same design choices. Specifically, all the following design choices are defined for a canonical 3D scene assumed to be centered and
Figure 7: DTU [25] scenes with ground-truth. We show a subset of the input images as well as renderings of the GT point clouds. From top to bottom, scenes are: S24, S31, S40, S45, S55, S59, S63, S75, S83, S105.
(a) Raw spherical coordinates
(b) Fixing border discontinuities
Figure 8: Our icosphere UV-mapping. We illustrate different UV parametrizations using raw spherical coordinates (a) as well as our modified coordinates to fix discontinuities (b) and to prevent distortions at the poles (c). For each parametrization, we show the texture image with the face edges representing the UV-mapping as well as a rendering example of the associated icosphere.
(c) Fixing distortions at the poles
mostly contained in the unit cube, with a y-axis orthogonal to the ground and pointing towards the sky. We roughly estimate the scene-specific constants related to the world scale and pose (through coarse visual comparisons or using the camera locations), and apply them to our final scene model to account for the camera conventions.