SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tom Monnier<sup>1</sup> Jake Austin<sup>2</sup> Angjoo Kanazawa<sup>2</sup> Alexei A. Efros<sup>2</sup> Mathieu Aubry<sup>1</sup>
$^{1}$ LIGM, Ecole des Ponts, Univ Gustave Eiffel
Given a set of calibrated images of a scene, we present an approach that produces a simple, compact, and actionable 3D world representation by means of 3D primitives. While many approaches focus on recovering high-fidelity 3D scenes, we focus on parsing a scene into mid-level 3D representations made of a small set of textured primitives. Such representations are interpretable, easy to manipulate and suited for physics-based simulations. Moreover, unlike existing primitive decomposition methods that rely on 3D input data, our approach operates directly on images through differentiable rendering. Specifically, we model primitives as textured superquadric meshes and optimize their parameters from scratch with an image rendering loss. We highlight the importance of modeling transparency for each primitive, which is critical for optimization and also enables handling varying numbers of primitives. We show that the resulting textured primitives faithfully reconstruct the input images and accurately model the visible 3D points, while providing amodal shape completions of unseen object regions. We compare our approach to the state of the art on diverse scenes from DTU, and demonstrate its robustness on real-life captures from BlendedMVS and Nerfstudio. We also showcase how our results can be used to effortlessly edit a scene or perform physical simulations. Code and video results are available at www.tmonnier.com/DBW.
Recent multi-view modeling approaches, building on Neural Radiance Fields [45], capture scenes with astonishing accuracy by optimizing a dense occupancy and color model. However, they do not incorporate any notion of objects, they are not easily interpretable for a human user or a standard 3D modeling software, and they are not useful for physical understanding of the scene. In fact, even though these approaches can achieve a high-quality 3D reconstruction, the recovered content is nothing but a soup of colorful particles! In contrast, we propose an approach that recovers textured primitives, which are compact, actionable, and interpretable.
More concretely, our method takes as input a collection of calibrated images of a scene, and optimizes a set of primitive meshes parametrized by superquadrics [1] and their UV textures to minimize a rendering loss. The approach we present is robust enough to work directly from a random initialization. One of its key components is the optimization of a transparency parameter for each primitive, which helps in dealing with occlusions as well as handling varying number of primitives. This notably requires adapting standard differentiable renderers to deal with transparency. We also show the benefits of using a perceptual loss, a total variation regularization on the textures and a parsimony loss favoring the use of a minimal number of primitives.
Our scene representation harks back to the classical Blocks World ideas [59]. An important difference is that the Blocks World-inspired approaches are typically bottom-up, leveraging low-level image features, such as edges [59], super-pixels [21], or more recently learned features [72, 32], to infer
Figure 1: Differentiable Blocks World. (a) Prior works fit primitives to point clouds and typically fail for real data where ground-truth point clouds are extremely noisy and incomplete. (b) We propose using calibrated multi-view images instead and simultaneously tackle 3D decomposition and 3D reconstruction by rendering learnable textured primitives in a differentiable manner. (c) Such a textured decomposition is highly compact and user-friendly: it enables us to do physics-based simulations, e.g., throwing a ball at the discovered primitives.
(c) Application: physics-based simulations
3D blocks. In contrast, we perform a direct top-down optimization of 3D primitives and texture using a rendering loss, starting from a random initialization in the spirit of analysis-by-synthesis. Unlike related works that fit primitives to 3D point clouds [3, 1, 68, 38, 71, 42, 43] (Figure 1a), our approach, dubbed Differentiable Blocks World (or DBW), does not require any 3D reconstruction a priori but instead operates directly on a set of calibrated input images, leveraging photometric consistency across different views (Figure 1b). This makes our approach more robust since methods based on 3D are very sensitive to noise in the reconstructions and have difficulties dealing with incomplete objects. Our setting is similar to existing NeRF-like approaches, but our model is able to recover a significantly more interpretable and parsimonious representation. In particular, such an interpretable decomposition allows us to easily play with the discovered scene, e.g., by performing physics-based simulations (Figure 1c). Code and video results are available on our project webpage: www.tmonnier.com/DBW.
Scene decomposition into 3D primitives. The goal of understanding a scene by decomposing it into a set of geometric primitives can be traced back to the very fist computer vision thesis by Larry Roberts on Blocks World [59] in 1963. In it, Roberts shows a complete scene understanding system for a simple closed world of textureless polyhedral shapes by using a generic library of polyhedral block components. In the 1970s, Binford proposes the use of Generalized Cylinders as general primitives [3], later refined by Biederman into the recognition-by-components theory [2]. But applying these ideas to real-world image data has proved rather difficult.
A large family of methods does not consider images at all, instead focusing on finding primitives in 3D data. Building upon the classical idea of RANSAC [11], works like [4, 6, 62, 61, 39, 50, 57] accurately extract various primitive shapes (e.g., planes, spheres and cylinders for [62, 61, 39]) from a point cloud. In particular, MonteBoxFinder [57] is a recent RANSAC-based system that robustly extracts cuboids from noisy point clouds by selecting the best proposals through Monte Carlo Tree Search. To avoid the need for RANSAC hyperparameter tuning while retaining robustness, Liu et al. [42] introduce a probabilistic framework dubbed EMS that recovers superquadrics [1]. Other methods instead leverage neural learning advances to robustly predict primitive decomposition from a collection of shapes (e.g., ShapeNet [5]), in the form of cuboids [68], superquadrics [55, 53, 71], shapes from a small dictionary [38, 36] or learnable prototypical shapes [10, 54, 43]. However, they are typically limited to shapes of known categories and require perfect 3D data. More generally, the decomposition results of all 3D-based methods highly depend on the quality of the 3D input, which is always noisy and incomplete for real scenes. For a complete survey of 3D decomposition methods, we refer the reader to [28].
More recently, there has been a renewed effort to fit 3D primitives to various image representations, such as depth maps, segmentation predictions or low-level image features. Depth-based approaches [27, 12, 40, 18, 32] naturally associate a 3D point cloud to each image which is then used for primitive fitting. However, the resulting point cloud is highly incomplete, ambiguous and sometimes inaccurately predicted, thus limiting the decomposition quality. Building upon the single-image scene layout estimation [23, 24], works like [21, 37] compute cuboids that best match the predicted surface orientations. Finally, Façade [9], the classic image-based rendering work, leverages user annotations across multiple images with known camera viewpoints to render a scene with textured 3D primitives. In this work, we do not rely on 3D, depth, segmentation, low-level features, or user annotations to compute the 3D decomposition. Instead, taking inspiration from Façade [9] and recent multi-view modeling advances [69, 51, 45], our approach only requires calibrated views of the scene and directly optimizes textured primitives through photometric consistency in an end-to-end fashion. That is, we solve the 3D decomposition and multi-view stereo problems simultaneously.
Multi-view stereo. Our work can be seen as an end-to-end primitive-based approach to multi-view stereo (MVS), whose goal is to output a 3D reconstruction from multiple images taken from known camera viewpoints. We refer the reader to [22, 14] for an exhaustive review of classical methods. Recent MVS works can be broadly split into two groups.
Modular multi-step approaches typically rely on several processing steps to extract the final geometry from the images. Most methods [82, 16, 73, 74, 79, 20, 65], including the widely used COLMAP [63], first estimate depth maps for each image (through keypoint matching [63] or neural network predictions [73, 74, 79, 20, 65]), then apply a depth fusion step to generate a textured point cloud. Finally, a mesh can be obtained with a meshing algorithm [30, 34]. Other multi-step approaches directly rely on point clouds [15, 34] or voxel grids [64, 33, 26, 49]. Note that, although works like [26, 49] leverage end-to-end trainable networks to regress the geometry, we consider them as multi-step methods as they still rely on a training phase requiring 3D supervision before being applied to unknown sets of multi-view images. Extracting geometry through multiple steps involves careful tuning of each stage, thus increasing the pipeline complexity.
End-to-end approaches directly optimize a 3D scene representation using photometric consistency across different views along with other constraints in an optimization framework. Recent methods use neural networks to implicitly represent the 3D scene, in the form of occupancy fields [51], signed distance functions [77] or radiance fields, as introduced in NeRF [45]. Several works incorporate surface constraints in neural volumetric rendering to further improve the scene geometry [52, 76, 70, 8, 13], with a quality approaching that of traditional MVS methods. Other methods [17, 80, 19, 48] instead propose to leverage recent advances in mesh-based differentiable rendering [44, 29, 41, 7, 58, 35] to explicitly optimize textured meshes. Compared to implicit 3D representations, meshes are highly interpretable and are straightforward to use in computer graphic pipelines, thus enabling effortless scene editing and simulation [48]. However, all the above approaches represent the scene as a single mesh, making it ill-suited for manipulation and editing. We instead propose to discover the primitives that make up the scene, resulting in an interpretable and actionable representation. A concurrent work PartNeRF [67] introduces parts in NeRFs. However, only synthetic scenes with a single object are studied and the discovered parts mostly correspond to regions in the 3D space rather than interpretable geometric primitives.
3 Differentiable Blocks World
Given a set of  $N$  views  $\mathbf{I}_{1:N}$  of a scene associated with camera poses  $\mathbf{c}_{1:N}$ , our goal is to decompose the 3D scene into geometric primitives that best explain the images. We explicitly model the scene as a set of transparent superquadric meshes, whose parameters, texture and number are optimized to maximize photoconsistency through differentiable rendering. Note that compared to recent advances in neural volumetric representations [51, 45, 78], we do not use any neural network and directly optimize meshes, which are straightforward to use in computer graphic pipelines.
Notations. We use bold lowercase for vectors (e.g., a), bold uppercase for images (e.g., A), double-struck uppercase for meshes (e.g., A) and write  $a_{1:N}$  the ordered set  $\{a_1, \ldots, a_n\}$ .
$\mathbf{T}_{\mathrm{bg}}$  texture
Icosphere  $\mathbf{T}_{\mathrm{bg}}$  texture
Background B
Ground  $\mathbb{G}$
Plane
Tgr texture
Superquadric
erquadric
$\mathbf{p}_k$  pose
$\mathbf{s}_k$  scale
$\mathbf{e}_k$  shape
Primitive block  $\mathbb{P}_k$
$\alpha_{k}$  transpar.
$\mathbf{T}_k$  texture
Input (subset)
Figure 2: Overview. (top) We model the world as an explicit set of learnable textured meshes that are assembled together in the 3D space. (bottom) Starting from a random initialization, we optimize such a representation through differentiable rendering by photometric consistency across the different views.
Iter 200
Iter 1k
Iter 10k
3.1 Parametrizing a World of Blocks
We propose to represent the world scene as an explicit set of textured meshes positioned in the 3D space. Figure 2 summarizes our modeling and the parameters updated (top) during the optimization (bottom). Specifically, we model each scene as a union of primitive meshes: (i) an icosphere  $\mathbb{B}$  modeling a background dome and centered on the scene, (ii) a plane  $\mathbb{G}$  modeling the ground, and (iii)  $K$  primitive blocks  $\mathbb{P}_{1:K}$  in the form of superquadric meshes, where  $K$  is fixed and refers to a maximum number of blocks. Unless mentioned otherwise, we arbitrarily use  $K = 10$ . We write the resulting scene mesh  $\mathbb{B} \cup \mathbb{G} \cup \mathbb{P}_1 \cup \ldots \cup \mathbb{P}_K$ .
The goal of the background dome is to model things far from the cameras that can be well approximated with a planar surface at infinity. In practice, we consider an icosphere with a fixed location and a fixed scale that is much greater than the scene scale. On the contrary, the goal of the planar ground and the blocks is to model the scene close to the cameras. We thus introduce rigid transformations modeling locations that will be updated during the optimization. Specifically, we use the 6D rotation parametrization of [83] and associate to each block  $k$  a pose  $\mathbf{p}_k = \{\mathbf{r}_k,\mathbf{t}_k\} \in \mathbb{R}^9$  such that every point of the block  $\mathbf{x}\in \mathbb{R}^3$  is transformed into world space by  $\mathbf{x}_{\mathrm{world}} = \mathrm{rot}(\mathbf{r}_k)\mathbf{x} + \mathbf{t}_k$ , where  $\mathbf{t}_k\in \mathbb{R}^3$ ,  $\mathbf{r}_k\in \mathbb{R}^6$  and rot maps a 6D vector to a rotation matrix [83]. Similarly, we associate a rigid transformation  $\mathbf{p}_{\mathrm{gr}} = \{\mathbf{r}_{\mathrm{gr}},\mathbf{t}_{\mathrm{gr}}\}$  to the ground plane. We next describe how we model variable number of blocks via transparency values and the parametrization of blocks' shape and texture.
Block existence through transparency. Modeling a variable number of primitives is a difficult task as it involves optimizing over a discrete random variable. Recent works tackle the problem using reinforcement learning [68], probabilistic approximations [55] or greedy algorithms [47], which often yield complex optimization strategies. In this work, we instead propose to handle variable number of primitive blocks by modeling meshes that are transparent. Specifically, we associate to each block  $k$  a learnable transparency value  $\alpha_{k}$ , parametrized with a sigmoid, that can be pushed towards zero to change the effective number of blocks. Such transparencies are not only used in our rendering process to softly model the blocks existence and occlusions (Section 3.2), but also in regularization terms during our optimization, e.g., to encourage parsimony in the number of blocks used (Section 3.3).
Superquadric block shape. We model blocks with superquadric meshes. Introduced by Barr in 1981 [1] and revived recently by [55], superquadrics define a family of parametric surfaces that exhibits a strong expressiveness with a small number of continuous parameters, thus making a good candidate for primitive fitting by gradient descent. More concretely, we derive a superquadric mesh
from a unit icosphere. For each vertex of the icosphere, its spherical coordinates  $\eta \in \left[-\frac{\pi}{2}, \frac{\pi}{2}\right]$  and  $\omega \in [-\pi, \pi]$  are mapped to the superquadric surface through the parametric equation [1]:
$$
\Phi (\eta , \omega) = \left[ \begin{array}{c} s _ {1} \cos^ {\epsilon_ {1}} \eta \cos^ {\epsilon_ {2}} \omega \\ s _ {2} \sin^ {\epsilon_ {1}} \eta \\ s _ {3} \cos^ {\epsilon_ {1}} \eta \sin^ {\epsilon_ {2}} \omega \end{array} \right], \tag {1}
$$
where  $\mathbf{s} = \{s_1, s_2, s_3\} \in \mathbb{R}^3$  represents an anisotropic scaling and  $\mathbf{e} = \{\epsilon_1, \epsilon_2\} \in \mathbb{R}^2$  defines the shape of the superquadric. Both  $\mathbf{s}$  and  $\mathbf{e}$  are updated during the optimization process. Note that by design, each vertex of the icosphere is mapped continuously to a vertex of the superquadric mesh, so the icosphere connectivity - and thus the icosphere faces - is transferred to the superquadric mesh.
Texturing model. We use texture mapping to model scene appearance. Concretely, we optimize  $K + 2$  texture images  $\{\mathbf{T}_{\mathrm{bg}}, \mathbf{T}_{\mathrm{gr}}, \mathbf{T}_{1:K}\}$  which are UV-mapped onto each mesh triangle using predefined UV mappings. Textures for the background and the ground are trivially obtained using respectively spherical coordinates of the icosphere and a simple plane projection. For a given block  $k$ , each vertex of the superquadric mesh is associated to a vertex of the icosphere. Therefore, we can map the texture image  $\mathbf{T}_k$  onto the superquadric by first mapping it to the icosphere using a fixed UV map computed with spherical coordinates, then mapping the icosphere triangles to the superquadric ones (see supplementary material for details).
3.2 Differentiable Rendering
In order to optimize our scene parameters to best explain the images, we propose to leverage recent mesh-based differentiable renderers [41, 7, 58]. Similar to them, our differentiable rendering corresponds to the soft rasterization of the mesh faces followed by a blending function. In contrast to existing mesh-based differentiable renderers, we introduce the ability to account for transparency. Intuitively, our differentiable rendering can be interpreted as an alpha compositing of the transparent colored faces of the mesh. In the following, we write pixel-wise multiplication with  $\odot$  and the division of image-sized tensors corresponds to pixel-wise division.
Soft rasterization. Given a 2D pixel location  $\mathbf{u}$ , we model the influence of the face  $j$  projected onto the image plane with the 2D occupancy function of [7] that we modify to incorporate the transparency value  $\alpha_{k_j}$  associated to this face. Specifically, we write the occupancy function as:
$$
\mathcal {O} _ {j} ^ {\mathrm {2 D}} (\mathbf {u}) = \alpha_ {k _ {j}} \exp \left(\min  \left(\frac {\Delta_ {j} (\mathbf {u})}{\sigma}, 0\right)\right), \tag {2}
$$
where  $\sigma$  is a scalar hyperparameter modeling the extent of the soft mask of the face and  $\Delta_j(\mathbf{u})$  is the signed Euclidean distance between pixel  $\mathbf{u}$  and projected face  $j$ , such that  $\Delta_j(\mathbf{u}) < 0$  if pixel  $\mathbf{u}$  is outside face  $j$  and  $\Delta_j(\mathbf{u}) \geq 0$  otherwise. We consider the faces belonging to the background and the ground to be opaque, i.e., use a transparency of 1 for all their faces in the occupancy function.
Blending through alpha compositing. For each pixel, we find all projected faces with an occupancy greater than a small threshold at this pixel location, and sort them by increasing depth. Denoting by  $L$  the maximum number of faces per pixel, we build image-sized tensors for occupancy  $\mathbf{O}_{\ell}$  and color  $\mathbf{C}_{\ell}$  by associating to each pixel the  $\ell$ -th intersecting face attributes. The color is obtained through barycentric coordinates, using clipped barycentric coordinates for locations outside the face. Different to most differentiable renderers and as advocated by [46], we directly interpret these tensors as an ordered set of RGBA image layers and blend them through traditional alpha compositing [56]:
$$
\mathcal {C} \left(\mathbf {O} _ {1: L}, \mathbf {C} _ {1: L}\right) = \sum_ {\ell = 1} ^ {L} \left(\prod_ {p <   \ell} ^ {L} \left(1 - \mathbf {O} _ {p}\right)\right) \odot \mathbf {O} _ {\ell} \odot \mathbf {C} _ {\ell}. \tag {3}
$$
We found this simple alpha composition to behave better during optimization than the original blending function used in [41, 7, 58]. This is notably in line with recent advances in differentiable rendering like NeRF [45] which can be interpreted as alpha compositing points along the rays.
3.3 Optimizing a Differentiable Blocks World
We optimize our scene parameters by minimizing a rendering loss across batches of images using gradient descent. Specifically, for each image  $\mathbf{I}$ , we build the scene mesh as described in Section 3.1
and use the associated camera pose to render an image  $\hat{\mathbf{I}}$  using the rendering process detailed in Section 3.2. We optimize an objective function defined as:
$$
\mathcal {L} = \mathcal {L} _ {\text {r e n d e r}} + \lambda_ {\text {p a r s i}} \mathcal {L} _ {\text {p a r s i}} + \lambda_ {\mathrm {T V}} \mathcal {L} _ {\mathrm {T V}} + \lambda_ {\text {o v e r}} \mathcal {L} _ {\text {o v e r}}, \tag {4}
$$
where  $\mathcal{L}_{\mathrm{render}}$  is a rendering loss between I and  $\hat{\mathbf{I}}$ ,  $\lambda_{\mathrm{parsi}}$ ,  $\lambda_{\mathrm{TV}}$ ,  $\lambda_{\mathrm{over}}$  are scalar hyperparameters and  $\mathcal{L}_{\mathrm{parsi}}$ ,  $\mathcal{L}_{\mathrm{TV}}$ ,  $\mathcal{L}_{\mathrm{over}}$  are regularization terms respectively encouraging parsimony in the use of primitives, favoring smoothness in the texture maps and penalizing the overlap between primitives. Our rendering loss is composed of a pixel-wise MSE loss  $\mathcal{L}_{\mathrm{MSE}}$  and a perceptual LPIPS loss [81]  $\mathcal{L}_{\mathrm{perc}}$  such that  $\mathcal{L}_{\mathrm{render}} = \mathcal{L}_{\mathrm{MSE}} + \lambda_{\mathrm{perc}}\mathcal{L}_{\mathrm{perc}}$ . In all experiments, we use  $\lambda_{\mathrm{parsi}} = 0.01$ ,  $\lambda_{\mathrm{perc}} = \lambda_{\mathrm{TV}} = 0.1$  and  $\lambda_{\mathrm{over}} = 1$ . Figure 2 (bottom) shows the evolution of our renderings throughout the optimization.
Encouraging parsimony and texture smoothness. We found that regularization terms were critical to obtain meaningful results. In particular, the raw model typically uses the maximum number of blocks available to reconstruct the scene, thus over-decomposing the scene. To adapt the number of blocks per scene and encourage parsimony, we use the transparency values as a proxy for the number of blocks used and penalize the loss by  $\mathcal{L}_{\mathrm{parsi}} = \sum_{k} \sqrt{\alpha_{k}} / K$ . We also use a total variation (TV) penalization [60] on the texture maps to encourage uniform textures. Given a texture map  $\mathbf{T}$  of size  $U \times V$  and denoting by  $\mathbf{T}[u, v] \in \mathbb{R}^{3}$  the RGB values of the pixel at location  $(u, v)$ , we define:
$$
\mathcal {L} _ {\mathrm {t v}} (\mathbf {T}) = \frac {1}{U V} \sum_ {u, v} \left(\left\| \mathbf {T} [ u + 1, v ] - \mathbf {T} [ u, v ] \right\| _ {2} ^ {2} + \left\| \mathbf {T} [ u, v + 1 ] - \mathbf {T} [ u, v ] \right\| _ {2} ^ {2}\right), \tag {5}
$$
and write  $\mathcal{L}_{\mathrm{TV}} = \mathcal{L}_{\mathrm{tv}}(\mathbf{T}_{\mathrm{bg}}) + \mathcal{L}_{\mathrm{tv}}(\mathbf{T}_{\mathrm{gr}}) + \sum_{k}\mathcal{L}_{\mathrm{tv}}(\mathbf{T}_{k})$  the final penalization.
Penalizing overlapping blocks. We introduce a regularization term encouraging primitives to not overlap. Because penalizing volumetric intersections of superquadrics is difficult and computationally expensive, we instead propose to use a Monte Carlo alternative, by sampling 3D points in the scene and penalizing points belonging to more than  $\lambda$  blocks, in a fashion similar to [54]. Following [54],  $\lambda$  is set to 1.95 so that blocks could slightly overlap around their surface thus avoiding unrealistic floating blocks. More specifically, considering a block  $k$  and a 3D point  $\mathbf{x}$ , we define a soft 3D occupancy function  $\mathcal{O}_k^{3\mathrm{D}}$  as:
$$
\mathcal {O} _ {k} ^ {\mathrm {3 D}} (\mathbf {x}) = \alpha_ {k} \operatorname {s i g m o i d} \left(\frac {1 - \Psi_ {k} (\mathbf {x})}{\tau}\right), \tag {6}
$$