Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitives
Authors: Tom Monnier
Jake Austin
Angjoo Kanazawa
Alexei A Efros
Mathieu Aubry
Venue: NeurIPS 2023 poster (2023)
Keywords: 3D decomposition
3D reconstruction
MVS
primitives
qualitative 3D

Abstract: Given a set of calibrated images of a scene, we present an approach that produces a simple, compact, and actionable 3D world representation by means of 3D primitives. While many approaches focus on recovering high-fidelity 3D scenes, we focus on parsing a scene into mid-level 3D representations made of a small set of textured primitives. Such representations are interpretable, easy to manipulate and suited for physics-based simulations. Moreover, unlike existing primitive decomposition methods that rely on 3D input data, our approach operates directly on images through differentiable rendering. Specifically, we model primitives as textured superquadric meshes and optimize their parameters from scratch with an image rendering loss. We highlight the importance of modeling transparency for each primitive, which is critical for optimization and also enables handling varying numbers of primitives. We show that the resulting textured primitives faithfully reconstruct the input images and accurately model the visible 3D points, while providing amodal shape completions of unseen object regions. We compare our approach to the state of the art on diverse scenes from DTU, and demonstrate its robustness on real-life captures from BlendedMVS and Nerfstudio. We also showcase how our results can be used to effortlessly edit a scene or perform physical simulations. Code and video results are available at https://www.tmonnier.com/DBW.

Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitives
Tom Monnier
Jake Austin²
Angjoo Kanazawa²
Alexei Efros
Mathieu Aubry'
LIGM,, Ecole des Ponts, Univ Gustave Eiffel
PCC Berkeley
Abstract
Given set of calibrated images scene, we present an approach that produces a simple, compact, and actionable 3D world representation by means of 3D primitives. While many approaches focus on recovering high-fidelity 3D scenes, we focus on parsing a scene into mid-level 3D representations made of a small set of textured primitives. Such representations are interpretable, easy to manipulate and suited for physics-based simulations. Moreover, unlike existing primitive decomposition methods that rely on 3D input data, our approach operates directly images through differentiable rendering. Specifically, we model primitives as textured superquadric meshes and optimize their parameters from scratch with an image rendering loss. We highlight the importance of modeling transparency for each primitive, which is critical for optimization and also enables handling varying numbers of primitives. We show that the resulting textured primitives faithfully reconstruct the input images and accurately model the visible 3D points, while providing amodal shape completions of unseen object regions. We compare approach to the state of the art on diverse scenes from DTU, and demonstrate its robustness real-life captures from BlendedMVS and Nerfstudio. We also show- case how our results can be used to effortlessly edit a scene or perform physical simulations. Code and video results available at tmonni /DBW.
Introduction
Recent multi-view modeling approaches, building Neural Radiance Fields [45], capture scenes with astonishing accuracy by optimizing a dense occupancy and color model. However, they do not incorporate any notion of objects, they 3D modeling software, and they are not useful for physical understanding of the scene. In fact, even though these approaches can achieve high-quality 3D reconstruction, the recovered content nothing but a soup of colorful particles! In contrast, we propose an approach that recovers textured primitives, which are compact, actionable, and interpretable.
easily interpretable for human
standard
More concretely, our method takes as input a collection of calibrated images of a scene, and optimizes set of primitive meshes parametrized by superquadrics [1] and their UV textures minimize rendering loss. The approach we present robust enough to work directly from a random initialization. One of its key components the optimization of transparency parameter for each primitive, which helps in dealing with occlusions as well as handling varying number of primitives. This notably requires adapting standard differentiable renderers to deal with transparency. We also show the benefits of using perceptual loss, a total variation regularization on the textures and a parsimony loss favoring the use of a minimal number of primitives.
Our scene representation harks back to the classical Blocks World ideas [59]. An important difference is that the Blocks World-inspired approaches typically bottom-up, leveraging low-level image features, such as edges [59], super- super-pixels [21], or more recently learned features [72, 32], to infer
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Input
Output
Input
Optimized textured 3D primitives
(a) Prior works, e.g., [42]
(b) Our work
(c) Application: physics-basee based simulations
Figure I: Differentiable Blocks World. (a) Prior works fit primitives to point clouds and typically fail for real data where ground-trune truth point clouds are extremely noisy and incomplete. (b) We propose using calibrated multi-view images instead and simultaneously tackle 3D decomposition and 3D reconstruction by rendering learnalle textured primitives differentiable manner. (c) Such textured decomposition highly compact and user-friendly; it enables us to physics-based simulations, e.g., throwing a ball at the discovered primitives.
3D blocks. In contrast, we perform a direct top-down optimization of 3D primitives and texture using rendering loss, starting from a random initialization in the spirit of analysis-b--symte
Unlike related works that fit primitives 3D point clouds [3, 1, 68, 38, 71, 42, 43] (Figure Ia), our approach, dubbed Differentiable Blocks World (or DBW), does not require any 3D reconstruction priori but instead operates directly on a set of calibrated input images, leveraging photometric consistency across different views (Figure lb). This makes our approach more robust since methods based on 3D) are very sensitive to noise in the reconstructions and have difficulties dealing with incomplete objects. Our setting is similar to existing NeRF-like approaches, but model is able to recover significantly more interpretable and parsimonious representation. In particular, such an interpretable decomposition allows to easily play with the discovered scene, e.g., by performing physics baasicc-based simulations (Figure 1c). Code and video results are available on our project webpage: www. tmonnier. com/ DBBW. com/DBW.
2 Related Work
Scene decomposition into 3D primitives. The goal of understanding a scene by decomposing into a set of geometric primitives can be traced back the very fist computer vision thesis by Larry Roberts on Blocks World [59] in 1963. In it, Roberts shows complete scene understanding system for simple closed world of textureless polyhedral shapes by using generic library of polyhedral block components. In the 1970s, Binford proposes the use of Generalized Cylinders as general primitives [3], later refined by Biederman into the recognition-by ceriioon----oooenetttt theory [2]. But applying these ideas to real-world image data has proved rather difficult.
large family of methods does not consider images all, instead focusing finding primitives in 3D data. Building upon the classical idea of RANSAC [1]], works like [4, 6, 62, 61, 39, 50, 57] accurately extract various primitive shapes (e.g., planes, spheres and cylinders for [62, 61, 391) from a point cloud. In particular, MonteBox [57] is a recent RANSAC-based system that robustly extracts cuboids from noisy point clouds by selecting the best proposals through Monte Carlo Tree Search. To avoid the need for RANSAC hyperparameter tuning while retaining robustness, Liu et al. [42] introduce probabilistic framework dubbed EMS that recovers superquadrics [1]. Other methods instead leverage neural learning advances to robustly predict primitive decomposition from a collection of shapes (e.g., ShapeNet [5]), in the form of cuboids [68], superquadrics [55, 53, 71], shapes from small dictionary [38, 36] or learnable prototypical shapes [10, 54, 43]. However, they are typically limited to shapes of known categories and require perfect 3D data. More generally, the decomposition results of all 3D-based methods highly depend on the quality of the 3D input, which always noisy and incomplete for real scenes. For a complete survey of 3D decomposition methods, we refer the reader to [28].

---

More recently. there has been a renewed effort to fit 3D primitives to various image representa- tions, such as depth maps, segmentation predictions or low-level image features. Depth-based approaches [27, 12, 40, 18, 32] naturally associate 3D point cloud to each image which is then used for primitive fitting. However, the resulting point cloud highly incomplete, ambiguous and some- times inaccurately predicted, thus limiting the decomposition quality. Building upon the single-image scene layout estimation [23, 24J, works like [21, 37] compute cuboids that best match the predicted surface orientations. Finally, Façade [9]. the classic image-based rendering work, leverages user annotations across multiple images with known camera viewpoints to render a scene with textured 3D primitives. In this work, do not rely on 3D, depth, segmentation, low-level features, or user annotations to compute the 3D decomposition. Instead, taking inspiration from Façade [9] and recent multi-view modeling advances [69, 51, 45]. our approach only requires calibrated views of the scene and directly optimizes textured primitives through photometric consistency in an end-to-end fashion. That is, we solve the 3D decomposition and multi-view stereo problems simultaneously.
Multi-view stereo. Our work can be seen as end-to-end primitive- hased approach to multi-view stereo (MVS), whose goal is to output a 31D reconstruction from multiple images taken from known camera viewpoints. We refer the reader to [22, 14] for an exhaustive review of classical methods. Recent MVS works can be broadly split into two groups.
Modular multi-step approaches typically rely on several processing steps to extract the final geometry from the images. Most methods [82, 16, 73, 74, 79, 20, 65], including the widely used COLMAP [63], first estimate depth maps for each image (through keypoint matching [63] or neural network predic- tions [73, 74, 79, 20, 65]) then apply a depth fusion step to generate a textured point cloud. Finally, a mesh can be obtained with meshing algorithm [30, 34]. Other multi-step approaches directly rely on point clouds [15, 34] or voxel grids (64, 33, 26, 49). Note that, although works like [26, 49] leverage end-to-end trainable networks to regress the geometry, we consider them as multi-step methods as they still rely on a training phase requiring 3D supervision before being applied to unknown sets of multi-view images. Extracting geometry through multiple steps involves careful tuning of each stage, thus increasing the pipeline complexity.
End-to-end approaches directly optimize a 3D scene representation using photometric consistency across different views along with other constraints in an optimization framework. Recent methods use neural networks to implicitly represent the 3D scene, in the form of occupancy fields [51], signed
distance functions [77] or radiance fields, as introduced in NeRF [45]. Several works incorporate surface constraints in neural volumetric rendering to further improve the scene geometry [52, 76, 70, 8, 13], with a quality approaching that of traditional MVS methods. Other methods [17, 80, 19, 48] instead propose to leverage recent advances in mesh-based differentiable rendering [44, 29, 41, 7, 58, 35] to explicitly optimize textured meshes. Compared to implicit 3D representations, meshes are highly interpretable and are straightforward to use in computer graphic pipelines, thus enabling effortless scene editing and simulation [48]. However, all the above approaches represent the scene as single mesh, making it ill-suited for manipulation and editing. We instead propose to discover the primitives that make up the scene, resulting in an interpretable and actionable representation. A concurrent work PartNeRF [67] introduces parts in NeRFs. However, only synthetic scenes with single object are studied and the discovered parts mostly correspond to regions in the 3D space rather than interpretable geometric primitives.
3 Differentiable Blocks World
Given a set of N views InNN of scene associated with camera poses C1:N, our goal is to decompose the 3D scene into geometric primitives that best explain the images. We explicitly model the scene as a set of transparent superquadric meshes, whose parameters, texture and number are optimized to maximize photoconsistency through differentiable rendering. Note that compared to recent advances in neural volumetric representations [51, 45, 78], we do not use any neural network and directly optimize meshes, which are straightforward to use in computer graphic pipelines.
Notations. We use bold lowercase for vectors (e.g., a), bold uppercase for images (e.g., A), double- struck uppercase for meshes (e.g., A) and write a1NN the ordered set {ai...., J.
3

---

Background BB
Ground
Superquadric
Primitive block Pe
loosphere
Tag pexture
Planc
T texture
Pa pose
Ta texture
8j scale
e shape
Pir pose
ag transpar.
Input (subset)
Init
Iter 200
Iter Ik
Iter 10k
Final
Output
Figure 2: Overview. (top) We model the world as explicit set of learnable textured meshes that assembled together in the 3D space. (bottom) Starting from random initialization, we optimize such representation through differentiable rendering by photometric consistency across different views.
3.1 Parametrizing a World of Blocks
We propose represent the world scene explicit set of textured meshes positioned in the 3D space. Figure 2 summarizes our modeling and the parameters updated (top) during the optimization (bottom). Specifically, model each scene union of primitive meshes: (i) an icosphere IB modeling background dome and centered on the scene, (ii) plane G modeling the ground, and (iii) primitive blocks 1I:K in the form of superquadric meshes, where K fixed and refers maximum number of blocks. Unless mentioned otherwise, we arbitrarly use K = 10. We write the resulting scene mesh B UGUP U..UPPUU.UURE
The goal of the background dome is to model things far from the cameras that can be well approxi- mated with a planar surface infinity. In practice, consider icosphere with fixed location and a fixed scale that much greater than the scene scale. On the contrary, the goal of the planar ground and the blocks is model the scene close to the cameras. We thus introduce rigid transformations modeling locations that will be updated during the optimization. Specifically, we use the 6D rotation parametrization of [83] and associate to each block a pose P = (rk. } E IR" such that every point of the block x e IR3 transformed into world space by Xworld = rot(r&)x + tk. where to E IR3, F6 E IR and rot maps a 6D vector to a rotation matrix [83]. Similarly. we associate a rigid transformation pgr (rgr tgr the ground plane. We next describe how model variable number blocks via transparency values and the parametrization of blocks' shape and texture.
Block existence through transparency. Modeling variable number of primitives is a difficult task as it involves optimizing over a discrete random variable. Recent works tackle the problem using reinforcement learning [68], probabilistic approximations [55] or greedy algorithms [47], which often yield complex optimization strategies. In this work, we instead propose to handle variable number of primitive blocks by modeling meshes that transparent. Specifically, a learnable transparency value d',, parametrized with a sigmoid, that can be pushed towards zero to change the effective number of blocks. Such transparencies are not only used in rendering process to softly model the blocks existence and occlusions (Section 3.2), but also in regularization terms during optimization, e.g., to encourage parsimony in the number of blocks used (Section 3.3).
associate to each block k
Superquadric block shape. We model blocks with superquadric meshes. Introduced by Barr in 1981 [1] and revived recently by [55], superquadrics define family of parametric surfaces that exhibits a strong expressiveness with small number continuous parameters, thus making good candidate for primitive fitting by gradient descent. More concretely, derive superquadric mesh

---

from unit icosphere. For each vertex of the icosphere, its spherical coordinates E- [_T, π| are mapped to the superquadric surface through the parametric equation [1]: 2:22 and
11 cos"1 17 cos
((n,w) w) =
$2 sin $1 17
S3 cos sin $2 w
where s = {81, 82, 83) E IR² represents an anisoptropic scaling and e e - (2) E IR2 defines the shape of the superquadric. Both and are updated during the optimization process. Note that by design, each vertex of the icosphere is mapped continuously to a vertex of the superquadric mesh, so the icosphere connectivity and thus the icosphere faces- is transferred to the superquadric mesh.
Texturing model. We use texture mapping to model scene uppearance. Concretely, we optimize K + 2 texture images (TTbg. T Tı:k} } which are UV-mapped onto each mesh triangle using pre- defined UV mappings. Textures for the background and the ground are trivially obtained using respectively spherical coordinates of the icosphere and simple plane projection. For a given block k, each vertex of the superquadrie mesh is associated to a vertex of the icosphere. Therefore, we can map the texture image Ta onto the superquadric by first mapping it to the icosphere using fixed UV map computed with spherical coordinates, then mapping the icosphere triangles to the superquadric ones (see supplementary material for details).
3.2 Differentiable Rendering
In order to optimize our scene parameters to best explain the images, we propose to leverage recent mesh-based differentiable renderers [41, 7, 58]. Similar to them, our differentiable rendering corresponds to the soft rasterization of the mesh faces followed by a blending function. In contrast to existing weshh differentiable renderers, we introduce the ability to account for transparency. Intuitively, our differentiable rendering can be interpreted as an alpha compositing of the transparent colored faces of the mesh. In the following, we write pixel-wise multiplication with @ and the division of image-sized tensors corresponds to pixel-wise division.
Soft rasterization. Given a 2D pixel location u, we model the influence of the face j projected onto the image plane with the 2D occupancy function of [7] that we modify to incorporate the transparency value ou, associated to this face. Specifically, we write the occupancy function as:
min ( o))
og (u) ak, exp
(2)
where a is scalar hyperparameter modeling the extent of the soft mask of the face and (,) is the signed Euclidean distance between pixel and projected face j. such that (u( < if pixel is
outside face j and A, (u) 2 0 otherwise. We consider the faces belonging to the background and the ground to be opaque, i.e., use a transparency of for all their faces in the occupancy function.
Blending through alpha compositing, For each pixel, we find all projected faces with an occupancy greater than a small threshold at this pixel location, and sort them by increasing depth. Denoting by L the maximum number of faces per pixel, we build image-sized tensors for occupancy O₂ and color C2 by associating to each pixel the f-th intersecting face attributes. The color is obtained through barycentric coordinates, using clipped barycentric coordinates for locations outside the face. Different to most differentiable renderers and as advocated by [46], we directly interpret these tensors as an ordered set of RGBA image layers and blend them through traditional alpha compositing [56]:
C(OI. C1.L) = Σ (IIa-o,) t=1
o O
(3)
P<t
We found this simple alpha composition to behave better during optimization than the original blending function used in [41, 7, 58). This is notably in line with recent advances in differentiable rendering like NeRF [45] which can be interpreted as alpha compositing points along the rays.
3.3 Optimizing a Differentiable Blocks World
We optimize our scene parameters by minimizing a rendering loss across batches of images using gradient descent. Specifically, for each image I, we build the scene mesh as described in Section 3.1
5

---

and use the associated camera pose to render an image using the rendering process detailed in Section 3.2. We optimize an objective function defined as:
C Frnnder + Aparsi + λTVLTV Aoverfover
where Crenter is a rendering loss between I and i, > ÀTV, Auver are scalar hyperparameters and Eepani; LTV, Lover are regularization terms respectively encouraging parsimony in the use of primitives, favoring smoothness in the texture maps and penalizing the overlap between primitives. Our rendering loss is composed of a pixel-wise MSE loss LMSE and perceptual LPIPS loss [81] pere such that
Grender MSE + Apere In all experiments, we use Aprri 0.01, Apere = λTv = 0. and Aover 1. Figure 2 (bottom) shows the evolution of our renderings throughout the optimization.
Encouraging parsimony and texture smoothness. We found that regularization terms were critical to obtain meaningful results. In particular, the raw model typically uses the maximum number of blocks available to reconstruct the scene, thus over-decomposing the scene. To adapt the number of blocks per scene and encourage parsimony, we use the transparency values as a proxy for the number of blocks used and penalize the loss by fiunsi = Ex vot/K. We also use total variation (TV) penalization [60] on the texture maps to encourage uniform textures. Given a texture map T of size x V and denoting by Tlu, v] E IR3 the RGB values of the pixel at location (u, v), we define:
L. UV Σ
| + 1, T(u 0I1 + ||Tlu, +] -(u, x) wlll))
and write LTV = Liv (Tgp) Cr + E. (Ts)) the final penalization.
Penalizing overlapping blocks. We introduce a regularization term encouraging primitives to not overlap. Because penalizing volumetric intersections of superquadries is difficult and computationally expensive, we instead propose to use Monte Carlo alternative, by sampling 3D points in the scene and penalizing points belonging to more than À blocks, in fashion similar [54]. Following [54], is set to 1.95 so that blocks could slightly overlap around their surface thus avoiding unrealistic floating blocks. More specifically, considering block h and a 3D point x, we define soft 3D occupancy function 0 as:
(1- V <x((x)
Ox((x (x) ag sigmoid
(6)
where T is a temperature hyperparameter and V is the superquadric inside-outside function [1] associated to the block k, such that (x) 1 if x lies inside the superquadric and 4 (x) > 1
otherwise. Given a set of M 3D points Ω our final regularization term can be written as:
K Σ
Lover
1 max XEQ
Σ
Ex (x), 2).
M
Note that practice, for better efficiency and accuracy, we only sample points the region where blocks are located, which can be identified using the block poses P1:K-
Optimization details. We found that two elements were key to avoid bad local minima during optimization. First, while transparent meshes enable differentiability w.r.t. the number of primitives, we observed failure mode where two semi opaque meshes model the same 3D region. To prevent this behavior, we propose to inject gaussian noise before the sigmoid in the transparencies Q1:K to create stochasticity when values are not close the sigmoid saturation, and thus encourage values that are close binary. Second, another failure mode we observed is one where the planar ground modeling the entire scene. We avoid this by leveraging two-stage curriculum learning scheme, where texture maps are downscaled by 8 during the first stage. We empirically validate these two contributions in Section 4.3. We provide other implementation details in the supplementary material.
Experiments
4.1 DTU Benchmark
Benchmark details. DTU [25] is an MVS dataset containing forward-facing scenes captured in controlled indoor setting, where the 3D ground-truth points are obtained through structured

---

Table I: Quantitative results on DTU [25]. We the official DTU evaluation report Chamfer Distance (CD) between 3D reconstruction and ground-truth best results are highlighted. We also highlight the average number of primitives found (#P) in green (smaller than 10) or red (larger than 10). Our performances correspond to a single random run (random) and run automatically selected among 5 runs using the minimal rendering loss (auto). We augment the best concurrent methods with a filtering step removing the ground from the 3D input.
Chamfer Distance (CD) per scene
Mcan
Mcan
Method
Input
S24
S31
S40
S45
$55
S59
S63
S75
S83
S105
CD
#P
EMS [42]
NeuS-mesh
8.42
8.53
7.84
6.98
7.2
8.57
7.77
8.69
4.74
9.11
7.78
9.6
EMS [42]
3D GT
6.77
5.93
3.36
6.91
6.52
3.50
4.72
7.08
7.25
6.10
5.82
7.4
MBF [57]
NeuS-mesh
3.97
4.28
3.56
4.76
3.33
3.92
3.63
5.58
5.3
6.07
4.44
53.5
MBF [57]
3D GT
3.73
4.79
4.31
3.95
3.26
4.00
3.66
3.92
3.97
4.25
3.98
16.4
Ours (random)
Image
5.41
3.13
1.57
4.93
3.08
3.66
3.40
2.78
3.94
4.85
3.67
4.6
Ours (auto)
Image
3.25
3.13
1.16
3.02
2.98
2.32
3.40
2.78
3.43
5.21
3.07
5.0
EMS [42] filter
3D GT
6.32
4.11
2.98
4.94
4.26
3.03
3.60
5.44
3.24
4.43
4.23
8.3
MBF [57] + filter
3D GT
3.35
2.95
2.61
2.19
2.53
2.47
1.97
2.60
2.60
3.27
2.65
29.9
light scanner. We evaluate on 10 scenes (S24, S31, $40, $45, S55, $59, S63, S75, S83, S105) that have different geometries and 3D decomposition that relatively intuitive. We use standard processing practices [77, 76, 8], resize the images to 400 x 300 and run our model with K = 10 on all available views for each scene (49 or 64 depending on the scenes). We use the official evaluation presented in [25], which computes the Chamfer distance between the ground-trutt -ruth points and points sampled from the 3D reconstruction, filtered out if not in the neighhorhood of the ground-truth points. We evaluate two state-of-the-arr methods for 3D decomposition, EMS [42] and MonteboxFinde (MBF) [57]. by applying them to the 3D ground-truth point clouds. We also evaluate them in a setup comparable to ours, where the state-of-the-art MVS method NeuS [70] is first applied to the multi-view images to extract a mesh, which is then used as input to the 3D decomposition methods. We refer to this input as "NeuS-mesh".
Results. We compare our Chamfer distance performances to these state-of of-tte--tttttttt 3D decomposition methods in Table 1. For each method, we report the input used and highlight the average number of discovered primitives #P in green (smaller than 10) or red (larger than 10). Intuitively, overly large numbers of primitives lead to less intuitive and manipulative scene representations. Our performances correspond single random run (random) and run automatically selected among 5 runs using the minimal rendering loss (auto). We augment the best concurrent methods with filtering step using RANSAC to remove the planar ground from the 3D input. Overall, we obtain results that are much more satisfactory than prior works. On the one hand, EMS outputs a reasonable number of primitives but has a high Chamfer distance reflecting bad 3D reconstructions. On the other hand, MBF yields a lower Chamfer distance (even better than ours with the filtering step) but it outputs significantly higher number of primitives, thus reflecting cecoppccomooisionns.
Our approach is qualitatively compared in Figure 3 to the best EMS and MBF models, which correspond the ones applied on the 3D ground truth and augmented with the filtering step. Because the point clouds are noisy and incomplete (see 360 renderings in our supplementary material), EMS and MBF struggle to find reasonable 3D decompositions: EMS misses some important parts, while MBF over-decomposes the 3D into piecewise planar surfaces. On the contrary, our model is able to output meaningful 3D decompositions with varying numbers of primitives and very different shapes. Besides, ours is the only approach that recovers the scene appearance (last column). Also note that produces a complete 3D scene, despite being only optimized on wawarddffccigggg views.
4.2 Real-Life Data and Applications
We present qualitative results on real-life captures in Figure 4. The first row corresponds to the Campanile scene from Nerfstudio repository [66] and the last four rows correspond to BlendedMVS scenes [75] that were selected in [76]. We adapt their camera conventions to ours and resize the images to roughly 400 x 300. From left to right, we show a subset of the input views, a rendering overlaid with the primitive edges, the primitives, as well as two novel view synthesis results. For each scene, we run our model 5 times and automatically select the results with the minimal rendering loss. We set the maximum number of primitives to K = 10, except the last row where it is increased to
7

---

Input view
GT point cloud
EMS [42]
MBF [57]
Ours
Ours rendering
EHARA
Figure Qualitative comparisons DTU We compare model slale-of-the-art methods (augmented with a preprocessing step remove the 3D ground) which, unlike ours, find primitives in the ground-trum point cloud that noisy and incomplete. Additionally, our approach the only one able to capture the seene appearance (last column).
Input (subset)
Rendering
Output
Novel views rendering
Novel views output
Figure 4: Qualitative results on real-life data. We run our default model (K 10) on scenes from Nerfs- tudio [66] (first row) and BlendedMVS [75] (all other rows). The last row corresponds to results where the maximum number of primitives is inereased K 50, yiclding 17 effective primitives found.

---

Input (subset)
Amodal view synthesis Nerfacto [66]
Amodal view synthesis
Ours
Input (subset)
Scene editing
Removing cars
Scene editing
Moving arm
Figure 5: Applications. (top) Given a set of views constrained to limited vicwpoint variations, we compare amodal view synthesis results using Nerfacto [66] and our approach. (middle) After optimization, we can easily modify the rendered scene by editing the different parts. (bottom) Our primitive-based representation enables straightforward physics-based simulations, such throwing ball the objects pouring water the
Table Ablation study on DTU [25]. We report metrics averaged over five runs: number of primitives (#P), Chamfer Distance (CD) and rendering metries (PSNR dB and SSIM, LPIPS in %). Best and second best are highlighted, #P variability is emphasized in green (smaller than 5) and red (larger than 5).
Method
#P +
CD 4
PSNR f
SSIM 1
LPIPS
Complete model
4.60 + 0.23
3.63 = 0.23
20.5 0.2
73.5
23.9 0.5
Earii
8.86 0.27
3.65 = 0.22
20.6
73.7 0.4
23.2 1
w/o Loser
4.38 + 0.19
3.80 = 0.30
20.4 0.3
73.2
24.1 0.7
w/o curriculum
1.66 + 0.30
3.99 = 0.17
20.1
72.7
21.5 0.4
w/o noise in On.K
3.60 + 0.21
4.13 = 0.28
20.0
0.6
2.6
Lrv
4.04 0.18
= 0.3
1.3
26.5
w/o Epere
3.22 + 0.17
4.80 = 0.20
0.1
7.3
40.0 0.4
K = 50 due to the scene complexity. These results show that despite its simplicity, our approach surprisingly robust. Our method is still able compute 3D decompositions that capture both appearances and meaningful geometry on variety of scene types. In addition, increasing the maximum number of primitives K allows us to easily adapt the decomposition granularity (last row).
In Figure 5, we demonstrate other advantages of our approach. First, compared to NeRF-hased approaches like Nerfacto [66] which only reconstruct visible regions, our method performs amodal scene completion (first row). Second, our textured primitive decomposition allows to easily edit the 3D scene (second row). Finally, our optimized primitive meshes can be directly imported into standard computer graphics softwares like Blender to perform physics-based simulations (bottom).
4.3 Analysis
Ablation study. In Table 2, we assess key components our model by removing one component at time and computing the performance averaged over the 10 DTU scenes. We report the final number of primitives, Chamfer distance and rendering metrics. We highlight the varying number of primitives in green (smaller than 5) and red (larger than 5). Results are averaged over five runs,

---

Table 3: Effeet of hyperparameters on DTU [255]. We evaluate the influence of two key hyperparameters of our model: the maximum number primitives K (left) and the parsimony regularization Apani (right).
Method
#PJ
CD4
PSNRT
SSIM
LPIPS
Method
#PJ
CDL
K 10 (default)
4.60
3.63
20.5
73.5
23.9
Apansi 0.001
7.44
3.61
K 25
7.00
3.58
21.0
74.6
22.5
Apans 0.01 (default)
4.60
3.63
9.26
3.52
20.9
74.7
22.8
pansa 0.1
1.30
6.88
(a) Missing parts
(b) Unnatural decomposition
Parsimony/fidelity trade-off
Figure 6: Failure cases. We show typical failure cases of our approach. All models are optimized with K = 10 except the rightmost model which is optimized with K 50. See text for details.
we report the means and standard deviations. Overall, each component except Lparsi consistently improves the quality of the 3D reconstruction and the renderings. Gpursi successfully limits the number of primitives (and thus, primitive duplication and over-decomposition) at very small quality cost.
Influence of K and Aparsi- In Table 3, we evaluate the impact of two key hyperparameters of our approach, namely the maximum number of primitives K and the weight of the parsimony regularization Apari. Results are averaged over the 10 DTU scenes for 5 random seeds. First, we can observe that increasing K slightly improves the reconstruction and rendering performances at the cost of a higher effective number of primitives. Second, these results show that Apursi directly influences the effective number of primitives found. When A 0.1, this strong regularization limits the reconstruction to roughly one primitive, which dramatically decreases the performances. When Aprrsi is smaller, the effective number of primitives increases without significant improvements in the reconstruction quality.
Limitations and failure cases. In Figure 6, we show typical failure cases of our approach. First, for random run, we may observe bad solutions where parts of the geometry are not reconstructed (Figure 6a). This is mainly caused by the absence primitives in this region at initialization and automatic selection among multiple runs alleviates the issue, yet this solution is computationally costly. Note that we also tried to apply a Gaussian kernel to blur the image and propagate gradients farther, but it had little effect. Second, our reconstructions illustrated in Figure 6b, where tea boxes are wrongly split or a single primitive is modeling the bear nose and the rock behind. Finally, in Figure 6c, show that increasing K from 10 (left) 50 (right) allows us to trade-off parsimony for reconstruction fidelity. However, while this provides a form of control over the decomposition granularity, the ideal decomposition in this particular case does seem to be found: the former seems to slightly under-decompose the scene while the latter seems to over-decompose it.
yield unnatural decompositions as
5 Conclusion
present an end-to-end approach that successfully computes a primitive-based 3D reconstruction given a set of calibrated images. We show its applicability and robustness through various benchmarks, where our approach obtains better performances than methods leveraging 3D data. We believe work could be an important step towards more interpretable multi-view modeling.
10

---

Acknowledgments and Disclosure of Funding
We thank Cyrus Vachha for help on the physics- based simulations; Antoine Guédon, Romain Loiseau for visualization insights; François Darmon, Romain Loiseau, Elliot Vincent for manuscript feedback. This work was supported part by ANR project EnHerit ANR-17-CE23-006
gifts from Adobe and HPC resources from GENCI-IDRIS (2022-AD01 101 1697R2, 2022-AD01 101358).. MA was supported by ERC project DISCOVER funded by the European Union's 's HorizonEurope Research and Innovation programme under grant agreement No. 101076028. Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union. Neither the European Union nor the granting authority can be held responsible for them.
References
[1] A. H. Barr. Superquadrics and Angle-Preserving Transformations. IEEE Computer Graphics and Applications, 1981.
[2] I. Biederman. Recognitionbby-comppenntt a theory of human image understanding, Psychological review, 1987.
[3] T. Binford. Visual Perception by Computer. In IEEE Conference on Systems ond Control, 1971.
[4] R. C. Bolles and M. A. Fischler. BRANSAC-Basee Approach to Model Fitting and Its Application to Finding Cylinders in Range Data. In IJCAI, 1981.
A. X. Chang, Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. S. Savarese, M. Savva, S. Song,
H. Su, Xiao, L. Yi, and F. Yu. ShapeNet: An Information- Rich 3D Model Repository. arXiv:1512.03012
(cs. CVJ, 2015.
[6] T. Chaperon
Goulette. Extracting cylinders full using a random sampling method and the gaussian image. In VMV, 2001.
[7] Chen, Gao, H. Ling, E. Smith, J. Lehtinen, Jacobson, and Fidler. Learning to Predict 3D Objects with an Interpolation based based Differentiable Renderer. In NewrIPS, 2019.
F. Darmon, B. Bascle, J.-C. Devaux, P. Monasse, and M. Aubry. Improving neural implicit surfaces geometry with patch warping. In CVPR, 2022.
[9] P. E. Debevec, C. J. Taylor, and J. Malik. Modeling and Rendering Architecture from Photographs: hybrid geometry- and maage-based approach. SIGGRAPH, 1996.
[10] T. Deprelle, T. Groueix, M. Fisher, V. Kim, B. Russell, and M. Aubry. Learning elementary structures for 3d shape generation and matching. In NeurlPS, 2019.
[11] M. A. Fischler and R. C. Bolles. Random Sample Consensus: A Paradigm for Model Fitting with Applications Image Analysis and Automated Cartography. Communications the ACM, 1981.
[12] D. F. Fouhey, A. Gupta, and M. Hcbert. Data-Driven 3D Primitives for Single Image Understanding. In ICCV, 2013.
[13] Q. Fu, Q. Xu, Y.S. Ong, and W. Tao. Geo-Neus: Geometry-Consisten Neural Implicit Surfaces Learning for Multi-view Reconstruction. In NeurlPS, 2022.
[14] Y. Furukawa and C. Hernández. Multi-view stereo: A tutorial. Foundations and Trends@ in Computer Graphics and Vision, 2015.
[15] Y. Furukawa and J. Ponce. Accurate, Dense, and Robust Multi- Vieti-Vieww Stercopsis. In CVPR, 2007.
[16] S. Galliani, K. Lasinger, and K. Schindler. Massively Parallel Multiview Stereopsis by Surface Normal Diffusion. In ICCV, 2015.
[17] J. Gao, W. Chen, T. Xiang, C. F. Tsang, A. Jacobson, M. McGuire, and S. Fidler. Learning Deformable Tetrahedral Meshes for 3D Reconstruction. In NeurIPS, 2020.
[18] A. Geiger and C. Wang. Joint 3D Object and Layout Inference from Single RGB-D Image. In GCPR. 2015.
[19] S. Goel, G. Gkioxari, and J. Malik. Differentiable Stereopsis: Meshes from multiple views using differen- tiable rendering. In CVPR, 2022.

---

[20] X. Gu. Z. Fan, S. Zhu, Z. Dai, F. Tan, and P. Tan. Cascade Cost Volume for High-Resolution Multi- Mulei-View Stereo and Stereo Matching. In CVPR, 2020.
[21] A. Gupta, A. A. Efros, and M. Hehert. Blocks World Revisited: Image Understanding Using Qualitative Geometry and Mechanics. In ECCV, 2010.
[22] R. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press,
2003.
[23] D. Hoiem, A. Efros, and M. Hebert. Geometric context from single image. In ICCV, 2005.
[24] D. Hoiem, A. A. Efros, and M. Hebert. Recovering Surface Layout from an Image. IJC,, 2007.
[25] R. Jensen, A. Dahl, G. Vogiatzis, E. Tola, and H. Aanaes. Large Scale Multi-view Stereopsis Evaluation. In CVPR, 2014.
[26] Ji, J. Gall, H. Zheng. Y. Liu, and L. Fang. SurfaceNet: An End-to-end 3D Neural Network for Multiview Stereopsis. In ICCV, 2017.
[27] H. Jiang and J. Xiao. A Linear Approach to Matching Cuboids in RGBD Images. In CVPR, 2013.
[28] A. Kaiser, A. Ybanez Zepeda, and Boubekeur. Survey Simple Geometric Primitives Detection Methods for Captured 3D Data. Computer Graphics Forum, 2019.
[29] Kato, Y. Ushiku, Harada. Neural 3D Mesh Renderer. In CVPR, 2018.
[30] M. Kazhdan and Hoppe. Screened Poisson Surface Reconstraction. ACM Transactions on Graphics,
2013.
[31] D. Kingma and J. Ba. Adam: Method for Stochastic Optimization. In ICLR, 2015.
[32] F. Kluger, H. Ackermann, E. Brachmann, M. Y. Yang, and B. Rosenhahn. Cuboids Revisited: Learning Robust 3D Shape Fitting Single RGB Images. In CVPR, 2021.
[33] K. Kutulakos and S. Seitz. Theory of Shape by Space Carving. In ICCV, 1999.
[34] P. Labatut, J.P. Pons, and R. Keriven. Efficient Multi-View Reconstruction of Large-Scale Scenes using Interest Points, Delaunay Triangulation and Graph Cuts. In ICCV, 2007.
[35] S. Laine, J. Hellsten, T. Karras, Y. Seol, J. Lehtinen, and T. Aila. Modular Primitives for Bigg-PPeffrmance Differentiable Rendering. ACM Transactions Graphics, 2020.
[36] E.-T. Le. M. Sung, D. Ceylan, R. Mech, T. Boubekeur, and N. J. Mitra. CPFN: Cascaded Primitive Fitting Networks for figh-Ressoution Point Clouds. In ICCV, 2021.
[37] D. C. Lee. A. Gupta, M. Hebert, and T. Kanade. Estimating Spatial Layout of Rooms using Volumetric Reasoning about Objects and Surfaces. In NIPS, 2010.
[38] L. Li, M. Sung, A. Dubrovina, L. Yi, and L. J. Guibas. Supervised Fitting of Geometric Primitives to 3D Point Clouds. In CVPR, 2019.
[39] Y. Li, X. Wu, Y. Chrysanthou, A. Sharf, D. Cohen-Or, and N. J. Mitra. GlobFit: Consistently fitting primitives by discovering global relations. ACM Transactions Graphics, 2011.
[40] D. Lin, S. Fidler, and R. Urtasun. Holistic Scene Understanding for 3D Object Detection with RGBD Cameras. In ICCV, 2013.
[41] S. Liu, T. Li. W. Chen, and H. Li. Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning. In ICCV, 2019.
[42] W. Liu, Y. Wu, S. Ruan, and G. S. Chirikjian. Robust and Accurate Superquadric Recovery: Probabilistic Approach. In CVPR, 2022.
[43] R. Loiseau, E. Vincent, M. Aubry, and L. Landrieu. Leamable Earth Parser: Discovering 3D Prototypes in Aerial Scans. arXiv:2304. 204:22304009744 (cs. CV]] 2023.
[44] M. M. Loper and M. J. Black. OpenDR: An Approximate Differentiable Renderer. In ECCV, 2014.
[45] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. NeRF: Representing Scenes Neural Radiance Fields for View Synthesis. In ECCV, 2020.
12

---

[46] T. Monnier, M. Fisher, A. A. Efros, and M. Aubry. Share With Thy Neighbors: Single-Viet View Reconstruction by Cross-Instance Consistency. In ECCV, 2022.
[47] T. Monnier, E. Vincent, J. Ponce, and M. Aubry. Unsupervised I Layered Image Decomposition into Object Prototypes. In ICCV, 2021.
[48] J. Munkberg, W. Chen, J. Hasselgren, A. Evans, T. Shen, T. Muller, J. Gao, and S. Fidler. Extracting Triangular 3D Models, Materials, and Lighting From Images. In CVPR, 2022.
[49] Z. Murez, T. van As, J. Bartolozzi, Sinha, V. Badrinarayanan, and A. Rabinovich. Atlas: End-to-End 3D Scene Reconstruction from Posed Images. In ECCV, 2020.
[50] L. Nan and Wonka. PolyFit: Polygonal Surface Reconstruction from Point Clouds. In JCCV, 2017.
[51] M. Niemeyer, L. Mescheder, M. Oechsle, and A. Geiger. Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision. In CVPR, 2020.
[52] M. Occhsle, S. Peng, and A. Geiger. UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-Viers Reconstruction. In ICCV, 2021.
[53] D. Paschalidou, I. V. Gool, and A. Geiger. Learning Unsupervised Hicrarchical Part Decomposition of 3D Objects from Single RGB Image. In CVPR. 2020.
[54] D. Paschalidou, A. Katharopoulos, A. Geiger, and S. Fidler. Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networks. In CVPR, 2021.
[55] D. Paschalidou, A. O. Ulusoy, and A. Geiger. Superquadrics Revisited: Learning Shape Parsing Beyond Cuboids. In CVPR, 2019.
[56] T. Porter and T. Duff. Compositing Digital Images. In SIGGRAPH, 1984,
[57] M. Ramamonjisoa, S. Stekovic, and V. Lepetit. MonteBoxFinder: Detecting and Filtering Primitives to Fit Noisy Point Cloud. In ECCV, 2022.
[58] N. Ravi, J. Reizenstein, D. Novotny, T. Gordon, W.-Y. Lo, J. Johnson, and G. Gkioxari. Accelerating 3D Deep Learning with PyTorc3D.. arXiv:2007.08501 [cx. CVJ, 2020.
[59] L. G. Roberts. Machine perception of three-dimensional solids. PhD thesis, Massachusetts Institute of Technology, 1963.
[60] I., I. Rudin and S. Oshe. Total variation hased image restoration with free local constraints. ICIP, 1994.
[61] R. Schnabel, P. Degener, and R. Klein. Completion and reconstruction with primitive shapes. Computer Graphics Forum, 2009.
[62] R. Schnabel, R. Wahl, and R. Klein. Efficient RANSAC for Point-Cloud Shape Detection. Computer Graphicy Forum, 2007.
[63] J. L. Schönberger. E. Zheng, J.M Frahm, and M. Pollefeys. Pixelwise View Selection for Unstructured Multi-View Stereo. In RCCV, 2016.
[64] S. Seitz and C. Dyer. Photorealistic Scene Reconstruction by Voxel Coloring. In CVPR, 1997.
[65] A. Sinha, Murez, Bartolozzi, V. Badrinarayanan, and Rabinovich. DELTAS: Depth Estimation by Learning Triangulation and Densification of Sparse Points. In ECCV, 2020.
[66] M. Tancik, E. Weher, F. Ng, R. Li, B. Yi, J. Kerr, T. Wang, A. Kristoffersen, J. Austin, K. Salahi, A. Ahuja, D. McAllister, and A. Kanazawa. Nerfstudio: Modular Framework for Neural Radiance Field Development. In SIGGRAPH, 2022.
[67] K. Tertikas, D. Paschalidou, B. Pan, J. Park, M. A. Uy, I. Emiris, Y. Avrithis, and L. Guibas, PartNeRF: Generating Part-Aware Editable 3D Shapes without 3D Supervision. CVPR, 2023.
[68] S. Tulsiani, H. Su, L. Guibas, A. A. Efros, and Malik. Learning Shape Abstractions by Assembling Volumetric Primitives. In CVPR, 2017.
[69] S. Tulsiani, T. Zhou, A. A. Efros, and J. Malik. Multi-view Supervision for Singlc-view Reconstruction via Differentiable Ray Consistency. In CVPR, 2017.
[70] P. Wang, L. Liu, Y. Liu, C. Theobalt. T. Komura, and W. Wang. NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi- view Reconstruction. In NeurlPS, 2021.
13

---

[71] Y. Wu, W. Liu, S. Ruan, and G. S. Chirikjian. Primitive-based Shape Abstraction via Nonparametric Bayesian Inference. In ECCV, 2022.
[72] J. Xiao, B. Russell, and A. Toralla. Localizing 3D cuboids in single-view images. In NIPS, 2012.
[73] Y. Yao, Z. Luo, S. Li. T. Fang, and L. Quan. MVSNet: Depth Inference for Unstructured Multi-view Stereo, In ,CCV, 2018.
[74] Y. Yao, Z. Luo, S. Li, T. Shen, T. Fang, and L. Quan. Recurrent MVSNet for High-Resolution Multi-View Sterco Depth Inference. In CVPR, 2019.
[75] Y. Yao, Z. Luo, S. Li, J. Zhang. Y. Ren, L. Zhou, T. Fang. and L. Quan. BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks. In CVPR, 2020.
[76] L. Yariv, J. Gu, Y. Kasten, and Y. Lipman. Volume Rendering of Neural Implicit Surfaces. In NeurlPS, 2021.
[77] L.. Yariv, Y. Kasten, D. Moran, M. Galun, M. Atzmon, R. Basri, and Y. Lipman. Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance, In NewrIPS, 2020.
[78] A. Yu, V. Ye, M. Tancik, and A. Kanazawa, pixeINeRF: Neural Radiance Fields from One or Few Images. In CVPR, 2021.
[79] J. Zhang. Visibility-aware Multi-view Stereo Network. In BMVC, 2020.
[80] Y. Zhang, G. Yang. S. Tulsiani, and D. Ramanan. NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild. In NeurIPS, 2021.
[81] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In CVPR, 2018.
[82] E. Zheng, Dunn, V. Jojic, and J.-M. Frahm. PatchMatch Based Joint View Selection and Depthmap Estimation. In CVPR, 2014.
[83] Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li, On the Continuity of Rotation Representations in Neural Networks. In CVPR, 2019.
14

---

Supplementary Material for Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitives
In this supplementary document, we provide additional results (Appendix A), details on the DTU benchmark (Appendix B) as well as implementation details (Appendix C), including design and optimization choices.
A Additional Results
Videos for view synthesis, physical simulations and amodal completion. We present additional results in the form of videos our project webpage: www. tmonnier: com/DBW. Videos separated in different sections depending on the experiment type. First, we provide view synthesis videos (rendered using a circular camera path), further outlining the quality of both our renderings and our primitive-based 31) reconstruction. Second, we include videos for physics-based simulations. Such simulations were produced through Blender by simply uploading our output primitive meshes. Note that for modeling primitive-specific motions in Blender (e.g., in our teaser figure), primitives should not overlap all, thus requiring a small preprocessing step slightly move the primitives for clear separation. Because each primitive is its own mesh, this operation is easily performed within Blender. Finally, we provide video results where we perform scene editing and compare our amodal view synthesis results to the ones of Nerfacto introduced in Nerfstudio [66]. Models for amodal synthesis are optimized on homemade indoor scene built from newward-facin capture only. We use Nerfstudio for data processing and data convention.
Rendering comparison with SOTA MVS.
Table 4: PSNR comparison DTU.
For completeness, we provide the rendering per- formances of NeRF [45], a SOTA MVS method that does not predict multiple parts. In Table 4, we compare PSNR for our approach and NeRF using the results reported in [76] on the inter-
Method
S24
S40
S55
S63
S83
S105
Mean
NeRF
26,2 26.8
27.6
32.0
32.8
32.1
29.6
Ours
19.1
21.8
22.6
23.4
22.3
20.8
21.7
sected set of 6 DTU scenes.
B DTU Benchmark
In Figure 7, we show for each scene a subset of the input images as well as 360 renderings of the GT point clouds obtained through a structured light scanner. To compute performances, we use Python version of the official evaluation: stttps//ggitien com/ jwt.aagss/DTUuaalll-22appa
C Implementation Details
Icosphere and superquadric UV mapping. We use spherical coordinates that we correct to build our texture mapping for the unit icosphere. Figure shows our process with an example. Specifically, we retrieve for each vertex its spherical coordinates [- 2 - and w E [- π| which are linearly mapped to the UV space [0, 1]2 Because such parametrization presents discontinuities and strong triangle deformations at the poles, we perform two corrections. First, we fix discontinuities by copying the border pixels involved (using circular padding on the texture image) and introducing new 2D vertices such that triangles do not overlap anymore. Second, we avoid distorted triangles 2D vertex positioned in the middle of the other
at the poles by creating for each triangle, two vertices. As detailed in the main paper, we derive a superquadric mesh from a unit icosphere in such way that each vertex of the icosphere is continuously mapped the superquadric vertex. As result, the texture mapping defined for the icosphere is directly transferred to our superquadric meshes without any modification.
Design choices. Except constants related to the world scale, orientation and position in the 3D space w.r.t. to the known cameras, our experiments share the same design choices. Specifically, all the following design choices are defined for a canonical 3D scene assumed to be centered and
15

---

Input (subset)
View
View 2
View 3
View 4
View 5
Figure 7: DTU [25] scenes with ground-truth. We show subset of the input images well as renderings of
the GT point clouds. From top to bottom, scenes are: $24, S31, $40, $45, S55, S59, $63, S75, S83, $105.
16

---

(a) Raw spherical coordinates
(b) Fixing border discontinuities
(c) Fixing distortions at the poles
Figure 8: Our icosphere UV-mapping. We illustrate different UV parametrizations using raw spherical coordinates (a) as well modified coordinates to fix discontinuities (b) and to prevent distortions at the poles (c). For each parametrization, we show the texture image with the face edges representing the UV-mapping as well as a rendering example of the associated icosphere.
mostly contained in the unit cube, with a y-axis orthogonal to the ground and pointing towards the sky. We roughly estimate the scene-specific constants related to the world scale and pose (through coarse visual comparisons using the camera locations), and apply them to our final scene model to account for the camera conventions.
The background corresponds to level-2 icosphere (320 faces), the ground plane is subdivided into 128 uniform faces (for visual purposes) and superquadric meshes are derived from level-1 icospheres (80 faces). The scale for the background and the ground is set to 10. The ground is initialized perpendicular to the y-axis and positioned at [0, 00.9, 0j. The poses of our primitive blocks are initialized using a Gaussian distribution for the 3D translation and random 6D vector for the rotation such that rotations are uniformly distributed on the unit sphere. We parametrize their scale with exponential added to minimum scale value of 0.2 to prevent primitives from becoming too small. These scales are initialized with a uniform distribution in [0.5, 1.5) and multiplied by a constant block scale ratio of 0.25 to yield primitives smaller than the scene scale. The superquadric shape parameters are implemented with a sigmoid linearly mapped in [0.1, 1.9] and are initialized at 1 (thus corresponding to raw icosphere). Transparency values are parametrized with sigmoid and initialized at 0.5. All texture images have a size of 256 x 256, are parametrized using a sigmoid and initialized with small Gaussian noises added to gray images.
Optimization details. All our experiments share the same optimization details. We use Pytorch3I) framework [58] build our custom differentiable rendering process and the default hyperparam- eter a = 10 Our model is optimized using Adam [31] with a batch size of 4 for roughly a total of 25k iterations. We learning rates of 0.05 for the texture images and 0.005 for all other parameters, and divide them by 10 for the last 2k iterations. Following our curriculum learning process, we optimize the model for the first 10k iterations by downsampling all texture images by 8. Then, optimize using the full texture resolution during the next 10k iterations. Finally, to further increase the rendering quality, we threshold the transparency values at 0.5 to make them binary, remove regularization terms related to transparencies (i.e., Cparsi and Lover), divide the weights for the other terms Lper and LTv by 10, decrease the smoothness rendering parameter a to 5 x 10 and finetune our model for the final 5k iterations. In particular, this allows the model output textures that are not darken by non-binary transparencies. During the optimization, we systematically kill blocks reaching a transparency lower than 0.01 and inference, we only show blocks with a transparency greater than 0.5. Similar to [54], we use À = 1.95 and T = 0.005 in our overlap penalization.
Computational cost. Optimizing our model on a scene roughly takes 4 hours on single NVIDIA RTX 2080 Ti GPU. Since MBF [57] and EMS [42] directly operate on the 3D point cloud without computing textures, they are much faster and compute primitives in a couple of minutes. To get comparable timings however, we have to account for a method that computes 3D point clouds from the calibrated images, which is typically longer depending on the method. For example, we report MBF and EMS results using the mesh extracted from NeUS [70], which typically takes 14 hours to converge on single DTU scene.
17


Reviews:
[{"review_id": "4j2OCNUyRa", "reviewer": "Reviewer_ZNqf", "summary": "This paper focuses on parsing a scene into mid-level 3D representations made of a small set of textured primitives. The main contributions are as follows:1) Unlike existing primitive decomposition methods which requires 3D data, this approach relies directly on images as input.\n2) The proposed method reconstructs and decomposes 3D models through differentiable rendering based optimization, by modeling primitives as superquadric meshes and optimizing their parameters from scratch with an image rendering loss. 3) A transparency is modeled for each primitive, which is critical for better optimization and also enables handling varying numbers of primitives.", "questions": "1. Can more explanations be added in introduction for the motivation why we should use a differentiable rendering based optimization?\n2. Can comparisons with some SOTA MVS reconstruction and texturing works be added?\n3. Since this method is similar to a vectorized modeling work, can comparisons with some vectorization works like \"PolyFit\" be added?\n4. Since this method is more similar to an evolution-based 3D reconstruction method, can comparisons with some evolution-based 3D reconstruction works like \"Hiep, V.H.; Keriven, R.; Labatut, P.; Pons, J.P. Towards high-resolution large-scale multi-view stereo. In CVPR 2009, pp. 1430–1437\" be added?\n5. The time statistics of the experimental cases should be reported.", "limitations": "As claimed in the paper, the proposed method works better for water-tight objects and is not so suitable for scenes, which will limit its applications. Can you explain more clearly the reason?", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper propose a novel 3D reconstruction, segmentation and texture mapping approach by a differentiable rendering based optimization, which is heuristic for 3D content generation. The technical part sounds correct and reasonable, and the experimental results show better quality to other SOTA methods of primitive fitting.", "weaknesses": "1. The motivation is this paper should be made clear. Why using a differentiable rendering based optimization? What is its advantages to other SOTA MVS reconstruction and texture mapping works.\n2. Although lacking comparisons to SOTA MVS reconstruction and texturing works, the reconstruction quality of this approach seems not so good as these SOTA methods.\n3. The proposed method is more similar to some evolution-based 3D reconstruction methods, like Hiep, V.H.; Keriven, R.; Labatut, P.; Pons, J.P. Towards high-resolution large-scale multi-view stereo. In CVPR 2009, pp. 1430–1437. Usually these kinds of methods consume much time for iterative evolution process. The paper lacks time statistics.\n4. As claimed in the paper, the proposed method works better for water-tight objects and is not so suitable for scenes, which will limit its applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. In the following, we address the main concerns raised in this review. We will incorporate relevant suggestions in the revised version.\n\n***W1, Q1: \"The motivation in this paper should be made clear. Why using a differentiable rendering based optimization? What is its advantages to other SOTA MVS reconstruction and texture mapping works?\"***\\\nA: Optimizing geometry and texture through differentiable rendering is a single stage process that circumvents multi-stage procedures from traditional MVS methods (e.g., COLMAP), in a fashion similar to NeRF. Such an optimization stage is simple and intuitive and we believe it is a key advantage for further developments. Compared to SOTA MVS works, the main advantage of our approach is that it decomposes a scene into different primitives that are independently actionable. In particular, this enables new applications such as part-based physics simulation (e.g., Fig. 1c and videos in section B of the supplementary index.html), simple scene editing (e.g., Fig. 5 second row) and amodal completion (e.g., Fig 5 first row and section C of the supplementary index.html). We will add clarifications in the introduction.\n\n***W2, Q2: \"Can comparisons with some SOTA MVS reconstruction and texturing works be added?\", \"the reconstruction quality of this approach seems not so good as these SOTA methods\"\\\nW3, Q4:  \"Can comparisons with some evolution-based 3D reconstructions works like Hiep et al. be added?\"***\\\nA: These works are indeed related to our work in the sense they tackle the problem of multi-view stereo. However, these works do not fulfill our primary objective to decompose a scene into distinct geometric parts. Nevertheless, we agree that having metrics for some reference MVS methods would be good for completeness; we will add reconstruction results for COLMAP MVS and a NeRF-based method in the supplementary material. \n\n***Q3:  \"Can comparisons with some vectorizations works like \"PolyFit\" be added?\"***\\\nTwo potentially relevant papers are actually named PolyFit:\n\n1) [Nan and Wonka, Polyfit: Polygonal Surface Reconstruction from Point Clouds, ICCV 2017] relies on fitting planes to a point cloud with RANSAC, and is thus similar in spirit to the MBF [Ramamonjisoa et al., ECCV 2022] baseline we report in the paper and one can expect the same kind of limitations. We tried running this comparison during the rebuttal period but did not manage to make it work. Specifically, the optimization froze for a couple of hours without responding, we will investigate further the reason for this issue.\n\n2) [Dominici et al., Polyfit: Perception-aligned Vectorization of Raster Clip-art via Intermediate Polygonal Fitting, SIGGRAPH 2020] is an image vectorization paper, and we believe extending it to 3D data is not obvious. \n\n***W3: \"The paper lacks time statistics.\"\\\nQ5: \"The time statistics of the experimental cases should be reported.\"***\\\nA: We report our approach timings in Section 3.3. 'Optimization details' paragraph. As stated in L230, optimizing our model roughly takes 4 hours on a single NVIDIA 2080 Ti GPU. We will add a small discussion about the computational cost of the baselines. In particular, since they operate directly on the 3D point clouds without computing textures, they are much faster and compute primitives in a couple of seconds. To get comparable timings, one would also have to account for the 3D point clouds computation from calibrated images, which is typically longer; for example, running COLMAP MVS on a DTU scene roughly takes 10 minutes.\n\n***W4, Limitations: \"As claimed in the paper, the proposed method works better for water-tight objects and is not so suitable for scenes, which will limit its applications. Can you explain more clearly the reason?\"***\\\nA: We are not sure where this is claimed in this paper, our approach works for multiple objects (many examples) and we report a few examples of BlendedMVS and NerfStudio scenes (Fig. 4 and supplementary material). Overall, because our approach involves meshes with clearly defined surfaces, it will indeed struggle to model transparent objects like glass or concepts like fog or clouds. We will make this clear. \n"}, {"review_id": "ALQYkLtSAd", "reviewer": "Reviewer_85Rz", "summary": "This paper (DBW) presents a way to fit superquadric blocks to a scene, starting from multiple images with calibrated cameras. Unlike recent other methods, they don't start with 3D data in the form of point clouds or meshes. The compositional nature of the recovered scene is shown by manipulating the recovered shape on standard tools, simulating other physics-based effects, etc. Another feature is the recovery of textures along with the geometry which adds appearance to the composed scene. Results are shown on selected DTU scenes as well as on a few real-world images.", "questions": "Ablation results in Table 2 and in the supplementary are interesting. However, I see that all rows (except possibly the last row that removes L_perc) have results that are very close to one another. I realise only one loss term is removed in each of the table rows and I don't see a dominant factor in any. What happens if multiple losses are removed? I realise the combinatorics is high, but I feel we could have more insights from them\n\nSee my question on the DRC paper elsewhere.\n\nIt is true that given posed camera views (as necessary for this work), a 3D representation of the scene (as volumetric grids, point clouds, or meshes) is available effectively for free (using well-established methods). Why not use them also in the DWC framework? What happens if some 3D information is added?", "limitations": "The primary complaint I have about this work concerns its utility to tasks.\n\n- Compositionality and interpretability are the major advantages of compositional segmentation of 3D scenes into parts. This holds when the geometry was fit into cuboids in the prior work.\n- Replacing cuboids with superquadrics does not increase its compositionality or interpretability, in my view. The recovery shape may appear to fit the scene better, but which downstream tasks are enabled by it?\n- It is good to see the scene being edited, collided/collapsed, water poured over it, etc. However, these are enabled at this level even by a cuboid decomposition.\n- I am glad to see texture being recovered simultaneously; However, the quality of texture combined with the \"boxiness\" of superquadrics make the new views generated incorrect. This can be seen in the supplementary videos. For example, th Scan24 of the \"house\" appears boxy even with texture applied to it.\n- NeRF and its descendants will provide much higher-quality view generation with appearance than this method can ever give as the geometry is approximated here. So, new view synthesis with the recovered texture is not really useful.\n- Simple geometric reasoning of the scene is possible with compositional segmentation done by DBW, but I don't see a whole lot added by this work over doing the same with simple blocks or a simple extension of the DRC work.\n\nI would like to know which use-cases are much better served by the DBW method presented here.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Compositionally recovering a scene as textured super quadrics is what the paper primarily achieves. This is being done directly using differentiable rendering without going through a 3D representation. The optimisation is achieved by different losses to ensure good fit to the observed shape. Recovering texture along with shape is also a plus.", "weaknesses": "I have some concerns about the presentation of the prior work leading upto this one. My understanding is thus:\n\n- Tulsiani et al have 2 papers in CVPR 2017. One is on Differentiable Ray Consistency (DRC) and the other on Learning Shape Abstractions (LSA) ([66] in the present manuscript)\n- The DRC's is the approach in my view that should motivate DBW presented in this paper. It presents an unsupervised (or self-supervised) way to fit geometry from images with camera matrices. The LSA paper fits cuboids to point clouds and is similar in that aspect of the work.\n- Later, Superquadrics were reintroduced in CVPR2019 [53] and fitted probabilistically in CVPR2022 [41], both from point cloud data.\n- I see the DBW paper as introducing Superquadrics to the DRC paper, taking ideas also from papers [41], [53], [66], and others.\n- I am surprised that the DRC paper is neither acknowledged nor referred to. Is there a reason for that?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. In the following, we address the main concerns raised in this review. We will incorporate relevant suggestions in the revised version.\n\n***Weaknesses: \"The DRC's is the approach in my view that should motivate DBW presented in this paper. [...] I see the DBW paper as introducing Superquadrics to the DRC paper [...] I am surprised that the DRC paper is neither acknowledged nor referred to. Is there a reason for that?\"***\\\nA: We agree that DRC [Tulsiani et al., 2017] is a pioneer analysis-by-synthesis method leveraging multiple views of a scene, thus related to our work, and we will add a discussion in the paper. However, it differs from our approach in three important ways. First, the problem setting is different: DRC is a learning approach for single-view reconstruction while ours can be seen as an optimization method for multi-view stereo. This is extremely important since DRC needs access to a training set made of multiple observations of scenes similar to the ones expected at inference, while our method is optimized on a single scene, without a training stage. Second, the 3D representation is different: DRC leverages a dense voxel grid while our approach optimizes textured parametric meshes. Third, the output is different: DRC predicts the 3D of the scene as a whole (a soup of voxels), whereas our method outputs multiple 3D elements that relate to different parts of the scene. This is typically not trivial and motivated fundamental works like [66] or [53], as well as applications (see Fig. 5). \n\n***Q: Ablation results. \"I see that all rows [...] have results that are very close to one another. I realise only one loss term is removed in each of the table rows and I don't see a dominant factor in any. What happens if multiple losses are removed?\"***\\\nA: Our ablation study removes one component at a time, which indeed leads to smaller differences with the complete method than if we were to combine the different rows. Note however that we ran each experiment 5 times, and reported the standard deviation for each experiment in the supplementary material, making it possible to identify which difference is significant. For example, the parsimony loss is essential to obtain a low number of primitives. The improvement in terms of Chamfer distance provided by TV, noise and curriculum are also significant. The overlap penalization indeed has a smaller quantitative impact, but improves the results qualitatively. As pointed out by the reviewer the combinatorics is high (5 runs for 6 factors ablated sequentially in any order would mean 5 * 6! =3600 experiments), but if the reviewer points to some combination that they feel would be relevant we will be happy to add them.\n\n***Q: \"Given posed camera views, a 3D representation of the scene is available effectively for free. Why not use them also in the DBW framework? What happens if some 3D information is added?\"***\\\nA: Our goal was precisely to develop a method which, opposite to prior works, did not rely on a 3D reconstruction on the scene. Indeed, as outlined in the paper, fitting primitives to a 3D representation comes with the burden of dealing with noisy and incomplete data. In particular, the 3D output for unseen regions would be empty or hallucinated (if not random as illustrated by our amodal synthesis experiment Fig. 5) and computing 3D-based losses in these regions does not make much sense. Our proposed approach to simultaneously solve for MVS and 3D decomposition helps dealing with this ambiguity. While we agree that adding to our framework the possibility to use 3D information in addition to images could be appealing (for example, increasing convergence speed), we believe it would not be trivial to do so in a way that is not penalized by noisy and incomplete 3D data.\n\n***Q: \"Replacing cuboids with superquadrics does not increase its compositionality or interpretability, in my view. The recovery shape may appear to fit the scene better, but which downstream tasks are enabled by it?\", \"I don't see a whole lot added by this work over doing the same with simple blocks\"***\\\nA: The novelty of our method is not to replace cuboids with superquadrics: the key novelty of our approach is to compute a 3D decomposition of a scene using image observations, instead of computing 3D primitives from point clouds like previous works. Our work could easily be applied with other types of primitives, and we actually started by reconstructing simple scenes such as the one of Fig. 1 with cuboids. On such scenes, the quality of the reconstruction was slightly worse with cuboids, but the same applications were indeed enabled. However, many other scenes (e.g., scene with fruits or the last 5 rows of Fig. 3) cannot be handled at all with simple cuboids. Considering superquadrics instead enabled the same applications for these scenes and not being restricted to \"boxy\" scenes.\n\n***Q: \"the \"boxiness\" of superquadrics make the new views generated incorrect\", \"NeRF and its descendants will provide much higher-quality view generation with appearance than this method can ever give as the geometry is approximated here\"***\\\nIndeed, our approach cannot be expected to compete with NeRF in terms of geometry or view generation accuracy for parts of the scene seen from multiple viewpoints and well reconstructed by NeRF (see answer to xDAr W2 for a comparison with NeRF). Our quantitative results for rendering aimed at analyzing the contribution of the different parts of our methods, not claiming state-of-the-art results. However, our view generation can be expected to be and actually is much better than the one provided by NeRF when looking at part of the scene seen in a single image or completely unseen, as showcased by our amodal view synthesis results (Fig. 5 first row)"}, {"review_id": "VLUOTx5EnA", "reviewer": "Reviewer_xDAr", "summary": "This paper presents an optimization-based method to recover 3D textured primitives from image observations. The core of the method is a fully differentiable and parsimonious 3D scene representation consisting of a background dome, ground plane, and a collection of multiple superquadrics, where the transformations as well as the UV-mapped textures are jointly optimized. By introducing transparency into the primitives, the authors devise an effective rendering mechanism that is optimizer-friendly and adaptive to different numbers of primitives. The proposed method is compared to 3D-based counterparts that fit primitives to the ground-truth point cloud and shows better representational compactness and fitting tightness. The primitive-based representation allows for various applications such as physics simulation and amodal shape completion.", "questions": "1. If the superquadrics are optimized to a cuboid, the textures along the edge will be squeezed and thus wasted (as it will not be reflected in the final rendering). Is there a possibility that the UV mapping could be also optimized to best utilize the pixels in the texture?\n2. In Eq(2), how is $\\Delta_j(\\mathbf{u})$ computed in a differentiable manner? It would be helpful to visualize $\\mathbf{O}$ (in supplementary maybe) and describe [7] in more detail for better understanding.\n3. How could the method be extended to large scenes? Will a coarse-to-fine optimization strategy help solve the local minima issue?", "limitations": "The limitations are clearly stated in Sec4.3, and the author should include some exemplar failure cases (figures) in the supplementary material.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The task of 3D decomposition from image collections is novel. Previous works either fit 3D primitives directly from point clouds, or build a low-level representation of the scene from images. This paper is the first effort to compute 3D primitives using image observations, and such a task would be useful in many other aspects including data transfer, scene editing, etc.\n2. The idea of optimization through rendering is novel. Inspired by the effective differentiable rendering scheme from NeRF, the paper introduces the transparency of the primitives and uses alpha blending to render, which is fast and effective. Such an idea is clever and inspiring.\n3. The presentation is good and the writing is clear. The paper is very clear in explaining the core ideas and the scene representation. Also, I could not find any grammatical errors or obvious typos in the writing.\n4. Results are compelling. The decomposition of 3D primitives is promising even for very complicated shapes such as robots. The decision boundaries of the primitives are clear and could be even useful for unsupervised semantic segmentation.", "weaknesses": "The main weakness of the paper is in its evaluation. In particular, the following experimental results should be reported:\n1. Running EMS [41] and MBF [55] on the geometries recovered by, e.g., NeuS [13, 67]. From Fig.3 it could be clearly seen that even for 'GT point cloud' provided by DTU dataset, there are missing regions. Hence it would be fairer if the input of EMS and MBF is point clouds sampled from the NeuS-mesh. This would make the experimental results & comparisons more convincing as this aligns the inputs of the proposed method and the baselines.\n2. The rendering quality (e.g., PSNR) should be compared against NeRF-based baselines, or other mesh-based baselines, such as NVDiffRast, PartNeRF, or Nerfacto. Although I could understand that the rendering quality is not the main pursuit of the paper, it would be good to have such a comparison for completeness. Actually, the author could add metrics that compare the representation compactness (such as the file size for a scene) just to show off the advantage of using this mid-level representation. Additionally, it would be very interesting to show how the rendering quality, optimization time, and scene size vary when the number of primitives grows.\n3. Running time of the method and the baselines should be compared (or at least discussed).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. In the following, we address the main concerns raised in this review. We will incorporate relevant suggestions in the revised version.\n\n***W1: Additional baselines using geometry from NeuS***\\\nA: In the following table, we report DTU results for MBF and EMS using the geometry extracted from NeuS [67], instead of the GT point clouds. More specifically, we use the official repository and the official config files to optimize the model and extract a mesh, then use uniformly sampled 3D points as input for the baselines. Our approach still performs better than these baselines. We hypothesize that these baselines are worse than the ones using GT 3D point clouds because NeuS-mesh is extremely accurate in the object region but hallucinates geometry for unseen or ambiguous regions like the background; fitting primitives in these regions penalizes a lot the final scores. Note that these regions are in general filtered using GT object masks before reporting quantitative and qualitative results on DTU (e.g., see [74, 50, 73, 67]). We will add the results for all scenes in the paper.\n\n| Model   \t| Input |  S24 |  S40 |  S63 | S105 | Mean CD | Mean #P |\n|-------------|:-----:|:----:|:----:|:----:|:----:|:-------:|:-------:|\n| EMS w/ NeuS |  Img  | 8.73 | 7.22 | 7.45 | 9.27 |   8.17  |   8.7   |\n| MBF w/ NeuS |  Img  | 4.22 | 3.77 | 3.14 | 6.31 |   4.36  |   53.7  |\n| Ours    \t|  Img  | 3.25 | 1.16 | 3.40 | 5.21 |   3.25  |   4.5   | \n\n***W2: Rendering comparison with NeRF***\\\nA: We agree that having rendering metrics for a reference method would be good for completeness; we will add the results for NeRF in the supplementary material. In the following table, we compare PSNR for our approach and NeRF using the results reported in [73] on 4 DTU scenes. As expected, our results are clearly worse when evaluating rendering on viewpoints close to the ones used for optimization. However, our results (Fig. 5 first row) are much better for very different viewpoints.\n\n| Model |  S24 |  S40 |  S63 | S105 | Mean |\n|-|:-:|:-:|:-:|:-:|:-:|\n| NeRF  | 26.2 | 26.8 | 32.0 | 32.1 | 29.3 |\n| Ours  | 19.1 | 21.8 | 23.4 | 20.8 | 21.3 |  \n\n***W2: Results when the number of primitives grows***\\\nA: In the following table, we report DTU results (effective number of primitives #P, Chamfer Distance and image rendering metrics) averaged over the 10 scenes and 5 random seeds for different values of K (the maximum number of primitives). We can observe that, as expected, increasing K improves the performances. Regarding the computational cost, increasing K does not change the overall runtime (roughly 4 hours for all experiments) but it impacts the memory consumption. We will add a discussion as well as visualizations in the supplementary material. \n\n| Model          \t| #P$\\downarrow$ | CD$\\downarrow$ | PSNR$\\uparrow$ | SSIM$\\uparrow$ | LPIPS$\\downarrow$ |\n|-|:-:|:-:|:-:|:-:|:-:|\n| $K = 10$ (default) | 4.60 | 3.63 | 20.5 | 73.5 | 23.9|\n| $K = 25$ | 7.00 | 3.58 | 21.0 | 74.6 | 22.5 |\n| $K = 50$ | 9.26 | 3.52 | 20.9 | 74.7 | 22.8 |\n\n***W3: \"Running time of the method and the baselines should be compared (or at least discussed)\"***\\\nA: We report our approach timings in Section 3.3. 'Optimization details' paragraph. As stated in L230, optimizing our model roughly takes 4 hours on a single GPU. We will add a discussion about the computational cost of the baselines. Since they operate directly on the 3D point clouds without computing textures, they are typically much faster and compute primitives in a couple of seconds. To get comparable timings, one would also have to account for the 3D point clouds computation from calibrated images, which is typically longer; for example, running COLMAP MVS on a DTU scene roughly takes 10 minutes.\n\n***Q1: \"Is there a possibility that the UV mapping could be also optimized to best utilize the pixels in the texture?\"***\\\nA: Indeed, the UV mapping could theoretically be optimized to best utilize pixels depending on the shape of the superquadric. However, this would imply additional regularization terms or another finetuning stage to prevent the mapping from diverging when the textures are randomly initialized. In addition, as stated L297 in our limitations section, our texturing model could also be improved to account for the primitive geometry. We leave these improvements for future works.\n\n***Q2: \"In Eq(2), how is $\\Delta_j(\\mathbf{u})$ computed in a differentiable manner? It would be helpful to visualize $\\mathbf{O}$ (in supplementary maybe) and describe [7] in more detail for better understanding.\"***\\\nA:  For a given pixel $\\mathbf{u}$ and a face $j$, $\\Delta_j(\\mathbf{u})$ corresponds to the signed Euclidean distance. More specifically, as described in [40] and [7], the edges of the face are projected onto the image plane, and the Euclidean distance between pixel $\\mathbf{u}$ and face $j$ corresponds to the Euclidean distance to the closest edge, which can be computed (almost everywhere) in a differentiable way. We will add clarifications in our supplementary material.\n\n***Q3: \"How could the method be extended to large scenes? Will a coarse-to-fine optimization strategy help solve the local minima issue?\"***\\\nA: Similar to NeRF, extending this method to large-scale scenes is not trivial. In particular, it would involve splitting the scene into multiple regions (e.g., in a spirit similar to Block-NeRF) and dealing with objects of different scales. We believe that a coarse-to-fine optimization could help with the latter, by gradually increasing the number of primitives and decreasing their initial scales.\n\n***Limitations: Exemplar for failure cases***\\\nA: We will add failure case examples in the supplementary material.\n"}, {"review_id": "cEn7IWFlo5", "reviewer": "Reviewer_3Dgw", "summary": "The paper presents a method for producing simple \"blocks world\" 3D geometry for a scene from posed RGB images. Compared with multi-view 3D reconstruction approaches that produce point clouds or nerfs, the emphasis is on producing a simple, robust and (somewhat) object-aligned 3D representation. The representation used here is a collection of superquadric blobs. The main technical contribution is in the adaptation of differentiable rendering techniques to collections of superquadrics with support for transparency. Experiments show fitting various numbers of superquadrics to standard multi-view datasets such as DTU. The results show that the method can fit scenes with multiple sub-objects with reasonable quality, but is limited in the details it can reconstruct due to a limited number of superquadrics (usually ~10, one example shows 17).", "questions": "How was the number K=10 chosen? What happens if K=100, or K=1000? More generally, what is limiting the detail level of the fits?\n\nWhy does the model get caught in bad local minima (in more detail)? Would just increasing the number of restarts avoid the minima?\n\nAre the parameters (sigma) of the soft rasterization adjusted on a schedule during optimization, similar to the original soft rasterizer paper? What is the effect, if any, of the soft rasterizer approximation on the optimization (i.e., what is the effect of varying sigma)? \n\nHow does the model behave for different values of the parsimony parameter? Is there an ablation showing the effect of that parameter?\n\n", "limitations": "As mentioned above, it would be nice to have a more detailed description of the limits of the optimization. Just saying it gets caught in bad local minima isn't all that helpful. What is the cause of these minima? Why doesn't the soft rasterization and the transparency handling smooth out the loss landscape sufficiently?", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper is very well written and the main concepts are easy to understand. Relevant related work is discussed in detail. \n\nThe approach is simple and appealing: given posed images, fit a set of textured primitives such that they reconstruct the images. It is a purely optimization-based approach without a learned prior, so in theory it should apply equally well across any image collection. The main technical novelty beyond the high-level concept is an adaptation of the soft rasterizer idea to handle transparent layers of superquadrics, which may be useful in other contexts.\n\nThe work could encourage others to investigate these sorts of \"blocks world\" representations.", "weaknesses": "My main concern with the paper is the results. The quality of the fits is pretty good for the number of superquadrics used, but still coarse, and the texture map can only do so much if the geometric complexity of the shape is higher than the superquadrics can handle (e.g., pitched roof of the building, top row fig. 3). The most impressive result is the gundam figure with 17 primitives (fig. 4), and the system apparently produces tight fits only for actual \"blocks world\" scenes (fig. 1).  \n\nI understand that the method is not targeted at photorealistic reconstruction. However, the usual criticism of \"blocks world\"-style approaches is that they are unrealistic, because the world isn't actually made out of blocks. The fidelity of the results of this method seem to confirm this criticism, which is a bit disappointing.\n\nThe limitations section suggests that the method is prone to bad local minima. This is a reasonable limitation but it is not surprising; it seems like an obvious limitation of this sort of optimization. Standard techniques such as coarse-to-fine optimization, or primitive splitting, were apparently not explored (unlike, for example, 3D Gaussian Splatting, Kerbl, et al., SIGGRAPH 2023). The paper states that K=10 was chosen for all experiments (l. 239) but the reader is left to wonder why this (small) number of primitives was chosen. \n\nMore minor: the adaptation of soft rasterizer to multiple layers seems like it could get slow for many layers, but this limitation is also not discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. In the following, we address the main concerns raised in this review. We will incorporate relevant suggestions in the revised version.\n\n***Weaknesses: \"The system apparently produces tight fits only for actual \"blocks world\" scenes\",  \"The usual criticism of \"blocks world\"-style approaches is that they are unrealistic, because the world isn't actually made out of blocks. The fidelity of the results of this method seem to confirm this criticism, which is a bit disappointing.\"***\\\nA: While we believe our results show that we can represent objects and scenes of significant complexity with a reasonable accuracy (using multiple superquadrics, e.g., Fig. 4), we agree that a \"blocks world\" approach such as ours will never be able to represent fine geometric details or extremely complex objects. However, while there are plenty of works that aim at photorealism (e.g., NeRF), they typically do not output an actionable representation, for which some sort of \"mid-level representations\" like blocks are necessary, and which is useful for some applications (e.g., we demonstrate physical simulation, scene manipulation and amodal view synthesis). We tackle this less explored problem of decomposing a scene into actionable primitives and we believe we are the first to do so from images alone.\n\n***Weaknesses: \"The method is prone to bad local minima. [...] Standard techniques such as coarse-to-fine optimization, or primitive splitting, were apparently not explored\"***\\\nA: Our curriculum learning can actually be seen as a coarse-to-fine optimization, where we gradually increase the resolution of the texture maps. We found this procedure to be particularly effective in solving the bad local minima where the entire scene is approximated with the textured ground plane. Refining the 3D reconstruction by splitting primitives or adding smaller primitives during the course of the optimization is an interesting path of research but is quite different from our main objective to recover a qualitative 3D representation of the world. We leave this direction for future works.\n\n***Q: \"How was the number K=10 chosen? What happens if K=100, or K=1000? More generally, what is limiting the detail level of the fits?\"***\\\nA: In a flavor similar to [66] and [53] which use $K\\leq20$, our main objective is to discover the most predominant geometric parts of a scene. Looking at DTU scenes, our intuition is that the minimal number of parts that coarsely explain the entire scene geometry is actually relatively small. Therefore, we arbitrarily cap the number of primitives to K=10. The current limiting factor for increasing the detail level of the fits is the memory consumption. In particular, having K=1000 implies optimizing 1000 texture images of relatively high resolution which does not hold in memory. In the following table, we report DTU results for different values of K. We can observe that increasing K improves the performances; we will add a discussion in the supplementary material.\n\n| Model | #P$\\downarrow$ | CD$\\downarrow$ | PSNR$\\uparrow$ | SSIM$\\uparrow$ | LPIPS$\\downarrow$ |\n|-|:-:|:-:|:-:|:-:|:-:|\n| $K = 10$ (default) | 4.60 | 3.63 | 20.5 | 73.5 | 23.9 |\n| $K = 25$ | 7.00 | 3.58 | 21.0 | 74.6 | 22.5 |\n| $K = 50$ | 9.26 | 3.52 | 20.9 | 74.7 | 22.8 |\n\n***Q: \"Why does the model get caught in bad local minima (in more detail)? Would just increasing the number of restarts avoid the minima? [...] Why doesn't the soft rasterization and the transparency handling smooth out the loss landscape sufficiently?\"***\\\nA: More precisely, a typical bad local minima is one where some parts of the geometry are not reconstructed. This is mainly caused by the absence of primitives in this region at the random initialization. Increasing the number of restarts will solve this issue, however this is not ideal. To avoid these restarts, we tried adapting the smoothness parameter sigma of the soft rasterization or equivalently applying a Gaussian kernel to blur the image, but the experiment was not conclusive. Blurring the image allows pixels to be influenced by larger regions in the image, however it introduces approximations and ambiguities that cannot be easily resolved. We will add a discussion in the paper. \n\n***Q: \"Are the parameters (sigma) of the soft rasterization adjusted on a schedule during optimization? What is the effect, if any, of the soft rasterizer approximation on the optimization (i.e., what is the effect of varying sigma)?\"***\\\nA: As stated in our supplementary material L61 and L66, the smoothness hyperparameter $\\sigma$ is set to the default value $10^{-4}$ during most of the optimization (20k iteration), then it is decreased to $5 \\times 10^{-6}$ during the last 5k iteration. We found this second training stage to help compute pixel-accurate shape boundaries and improve the quality of the texture images. Using higher values of $\\sigma$ makes the results blurrier.\n\n***Q: \"How does the model behave for different values of the parsimony parameters? Is there an ablation showing the effect of that parameter?\"***\\\nA: In our ablation study in Table 2, we evaluate the impact of removing the parsimony loss. In the following table, we report additional DTU results for other values of $\\lambda_{\\text{parsi}}$. We can observe that it directly influences the effective number of primitives found. When $\\lambda_{\\text{parsi}} = 0.1$, the regularization is too strong (roughly 1 primitive found) and it dramatically decreases the reconstruction performances. When $\\lambda_{\\text{parsi}}$ is much smaller, the number of primitives increases without significant improvements.\n\n| Model | #P$\\downarrow$ | CD$\\downarrow$ |\n|-|:-:|:-:|\n| $\\lambda_{\\text{parsi}} = 0$ | 8.86 | 3.65 |\n| $\\lambda_{\\text{parsi}} = 0.001$ | 7.44 | 3.61 |\n| $\\lambda_{\\text{parsi}} = 0.01$ (default) | 4.60 | 3.63 |\n| $\\lambda_{\\text{parsi}} = 0.1$ | 1.30 | 6.88 |"}]

Author rebuttal:

Meta-review: The paper had mixed ratings: 1 accept, 2 borderline accept, and 1 borderline reject. Author rebuttals were effective in addressing the concerns of the reviewers. The negative reviewer (borderline reject) did not answer the rebuttal and participated in the discussion. The area chair agreed with the ratings of the positive reviewers and recommend acceptance.
Decision: Accept (poster)