Improved Convex Decomposition with Ensembling and Boolean Primitives

## Abstract

Abstract Describing a scene in terms of primitives – geometrically simple
shapes that offer a parsimonious but accurate abstraction of
structure – is an established vision problem. This is a good model of
a difficult fitting problem: different scenes require different
numbers of primitives and primitives interact strongly, but any
proposed solution can be evaluated at inference time.
The state of the art method involves a learned regression procedure
to predict a start point consisting of a fixed number of primitives,
followed by a descent method to refine the geometry and remove
redundant primitives. Methods are evaluated by accuracy in depth
and normal prediction and in scene segmentation. This paper
shows that very significant improvements in accuracy can
be obtained by (a) incorporating a small number of negative primitives and (b) ensembling over a number of different regression
procedures. Ensembling is by refining each predicted start point,
then choosing the best by fitting loss. Extensive experiments on a
standard dataset confirm that negative primitives
are useful in a large fraction of images, and that our
refine-then-choose strategy outperforms choose-then-refine,
confirming that the fitting problem is very difficult.

## 1 Introduction

Geometric representations of scenes and objects as primitives –
simple geometries that expose structure while suppressing detail –
should allow simpler, more general reasoning. For example, it is easier to plan
moving a cuboid through a stylized free space than it is to plan
moving a particular chair through a particular living room; as
another example, an effective primitive representation should simplify
selecting and manipulating objects in scenes (and so image-based scene
editing []). But obtaining primitive representations that abstract
usefully and accurately has been hard (review
Sec. 2).

There are two main types of method. A descent
method chooses primitives for a given geometry by minimizing a cost
function. Important obstacles include: different geometries require different numbers of primitives; the
choice of primitive appears to be important in ways that are opaque;
the fitting problem appears to have large numbers of local minima;
and finding a good start point is difficult. In particular,
incremental fitting procedures are often defeated by interactions
between primitives. A regression method uses a learned predictor to map
geometry to primitives and their parameters. These methods can pool
examples to avoid local minima, but may not get the best prediction
for a given input.

The SOTA method [] for parsing indoor scenes uses a
regression method to predict a start point consisting of a fixed
set of primitives. An important feature of this class of problem is
that, at run time, one can evaluate a predicted solution
efficiently and accurately. The start point is then polished using
a descent method, applied to a fitting loss that
compares the prediction with depth and segmentation maps predicted by
a suitable pretrained network, with backward selection to remove
redundant primitives. Finally, evaluation is by comparing the primitive
geometry to reference depth, normal and segmentation.

This paper shows two procedures that yield significant (over 30%
relative error) improvements in accuracy. First, we allow a small
number of *negative* primitives in the sense of constructive solid
geometry (CSG). Second, we show that an appropriately constructed
ensembling method produces very strong improvements in accuracy.

For negative primitives, the predicted geometry is the set difference
between the union of positive primitives and the union of negative
primitives. As our ablation experiments show, this significantly
expands the geometries we can encode and significantly complicates the
fitting problem. On their own, negative primitives produce
small improvements in accuracy. With ensembling we obtain significant
improvements in accuracy. We ensemble by using multiple predictors,
each trained to predict a start point with a different number of
primitives; some predictors use only positive primitives, others use
both positive and negative primitives. Each predicted start point is
then polished by minimising a fitting loss, and the best resulting set of primitives
by fitting loss is reported. This polish-then-choose strategy yields
very strong improvements in accuracy. Notably, for some scenes only
positive primitives are used, whereas for others both positive and negative
primitives are used.

Our contributions are:

- 1.
We believe our method is the only one that can fit CSG with a set
differencing operator to indoor scenes.
- 2.
Our ensembling method has a novel structure that results in
large improvements in accuracy.
- 3.
Our primitive decomposition method for indoor scenes is an
effective procedure that substantially outperforms SOTA on established metrics on the benchmark NYUv2 dataset.

Figure: Figure 1: We present a method that advances the SOTA for primitive decomposition of indoor scenes, by using ensembling and boolean primitives. We present qualitative comparison with prior work here. In the fourth column, notice how a negative primitive helps explain free space on the bottom left; in the last column, notice how a negative primitive helps represent the chair in the center.
Refer to caption: x1.png

## 2 Related Work

Primitives date to the origins of computer vision.
Roberts worked with blocks []; Binford with
generalized cylinders []; Biederman with
geons []. Ideally, complex
objects might be handled with simple primitives [],
each primitive is a semantic part [, , ].
Primitives can be recovered from image data [, ], and allow
simplified geometric reasoning [].

For individual objects, neural methods could predict the right set of primitives
by predicting solutions for test data that are “like” those that worked for training data. Tulsiani et
al. parse 3D shapes into cuboids, trained without ground truth segmentations []. Zou et al. parse with a recurrent architecture []. Liu et al. produce detailed reconstructions of objects in
indoor scenes, but do not attempt parsimonious
abstraction []. Worryingly, 3D
reconstruction networks might rely on object
semantics []. Deng et al. (CVXNet)
represent objects as a union of convexes, again training without
ground truth segmentations []. An early variant of CVXNet
can recover 3D representations of poses from single images, with reasonable parses into parts [].
Meshes can be decomposed into near convex primitives, by a form of search [].
Part decompositions have attractive editability []. Regression methods face some difficulty producing
different numbers of primitives per scene (CVXNet uses a fixed number; [] predicts the
probability a primitive is present; one also might use Gumbel softmax []).
Primitives that have been explored include: cuboids [, , , , , , , ];
superquadrics [, , ]; planes [, ]; and generalized cylinders [, , ].
There is a recent review in [].

Neural Parts [] decomposes an object given
by an image into a set of non-convex shapes.
CAPRI-Net [] decomposes 3D objects given as point
clouds or voxel grids into assemblies of quadric surfaces.
DeepCAD [] decomposes an object into a sequence of
commands describing a CAD model, but requires appropriately
annotated data for training. Point2Cyl [] is similar, but predicts the 2D shapes in form of an SDF.
Notably, [, , ] also utilise CSG with negative primitives or parts but, unlike our work, focus on CAD models of single objects instead of complex real-world scenes.

Hoiem et al parse outdoor scenes into vertical and horizontal
surfaces [, ]; Gupta et al
demonstrate a parse into blocks []. Indoor scenes can be
parsed into: a cuboid [, , ]; beds and some furniture as
boxes []; free space []; and plane
layouts [, ]. If RGBD is
available, one can recover layout in detail []. Patch-like primitives can be imputed from
data []. Jiang demonstrates parsing RGBD images into
primitives by solving a 0-1 quadratic
program []. Like that work, we evaluate
segmentation by primitives (see [], p. 12), but
we use original NYUv2 labels instead of the drastically simplified
ones in the prior work. Also, our primitives are truly convex.
Monnier et al and Alaniz et al decompose scenes into sets of superquadrics using differentiable rendering, which requires calibrated multi-view images as input [, ].
Most similar to our work is that of Kluger et al, who identify
cuboids sequentially with a RANSAC-like [, , ] greedy
algorithm [, ].

The success of a descent method depends critically on the start point, typically dealt with using
greedy algorithms (rooted in []; note the prevalence of RANSAC in a recent review []);
randomized search [, ]; or multiple starts.
Regression methods must minimize loss over all training data, so at inference
time do not necessarily produce the best representation for the
particular scene. The prediction is biased by the need to get other
scenes right, too. To manage this difficulty, we use a mixed
reconstruction strategy – first, predict primitives using a network,
then polish using descent.

Figure: Figure 2: Inference Overview: We feed an RGBD image into an ensemble of independently trained convolutional neural networks. Each network predicts the parameters of a set of convexes $\mathcal{C}_{i}$. The number of convexes predicted by each network varies between $12$ and $28$, with up to two of them being negative. We refine each set of convexes by minimizing the training loss w.r.t. the input depth map. Our final decomposition consists of the set of refined convexes $\mathcal{C}_{i}$ which yields the lowest absolute relative depth error.
Refer to caption: x2.png

## 3 Method

Our work is based on the architecture and losses of [] and maintains its basic inference procedure:

- 1.
Predict initial convex parameters from an RGBD image via a convolutional neural network.
- 2.
Refine the fit by directly optimizing convex parameters against the training losses.

Unlike [], we employ an ensemble of multiple networks that predict varying numbers of convexes, and select the prediction which yields the lowest error after refinement (Sec. 3.1).
This allows us to abandon the pruning heuristic used by [] to control the number of convexes for each scene.
We furthermore introduce *negative* boolean primitives for scene decomposition (Sec. 3.2).
As visualised in Fig. 3, boolean primitives allow for a more parsimonious description of complex geometry.
An additional loss, annealing schedule and data augmentation yield further accuracy gains (Sec. 3.3).
Fig. 2 provides an overview of our inference pipeline.

### 3.1 Ensembling

We remark that much of the literature on primitive decomposition fits a fixed number of primitives [].
Other work starts from a fixed number of primitives and removes excess primitives according to a greedy algorithm [].
The problem with these approaches is that it is difficult to know a priori what initial settings are best for a given test image.
A solution we employ in this work is ensembling the prediction from multiple networks, and selecting the best one.
We train networks with different numbers of primitives to construct an ensemble; an even richer ensemble can be constructed by varying weights for some of the training losses, which can substantially effect the behavior of the resulting decomposition even when the number of primitives is the same. To temper the compute requirements, we focus on varying the number of initial primitives and show that this ensemble is sufficient to boost final quality.

For a given test image, we can evaluate by running each test image on each network, evaluating the generated primitive depth map against an inferred or GT depth, and use the best network for subsequent refinement. In practice, we observed refine-then-choose to perform better, whereby we refine each method first then choose the one with the best error metrics. Even though this involves more compute, the quality gains are substantial (see Table 2).

### 3.2 Boolean primitives

A traditional collection of primitives is represented by an indicator function $O:\mathbb{R}\rightarrow[0,1]$, with $O(x)=0$ indicating free space, and $O(x)=1$ indicating a query point $x\in\mathbb{R}^{3}$ is inside the volume. When introducing negative primitives, the final indicator can be composed of a CSG operation between the union of positive primitives and union of negative primitives. Let $O^{+}(x)$ be the indicator of positive primitives only, and $O^{-}(x)$ be the indicator of negative primitives only. The final indicator for our representation is simply

$$ $O(x)=\operatorname*{ReLU}(O^{+}(x)-O^{-}(x))$ (1) $$

Our modified representation allows re-using the existing sample loss, unique parametrization loss, and manhattan world loss [, ] for both $O^{+}(x)$ and $O^{-}(x)$. However, for negative primitives only, we must modify the samples on which the overlap loss, guidance loss, and localization loss are applied. During each training iteration, we select samples for which the ground truth label for a point is outside, $x=0$, but the indicator function is positive, $O(x)=1$. Thus if a negative primitive moves to such a sample, its classification will become $O(x)=0$, matching ground truth.

Our early experimentation showed that we are better off pretraining with positive primitives only, and then introducing negative primitives midway through training. Conceptually, this procedure allows positive primitives to explain the scene at a high level, and then negative primitives to subsequently improve the representation.

Figure: Figure 3: Boolean primitives are parameter-efficient. Representing a simple box with a hole punched in it can be challenging even with several traditional primitives, as shown in (a), where five primitives get stuck in a local minimum. In contrast, two primitives - one positive and one negative - can represent the geometry successfully because of the enriched vocabulary of operations. Two views are shown in (b) and (c).
Refer to caption: x3.png

### 3.3 Performance improvements

Biasing sample loss The primary loss for training a convex decomposition network is

$$ $L_{approx}=\mathbb{E}_{x\sim\mathbb{R}^{3}}||\hat{O}(x)-O(x)||^{2}.$ (2) $$

We postulate that negative primitives would be most useful in regions that the positive primitives over-explain certain geometry, i.e. they explain more inside samples correctly than outside samples. In effect, if the positive primitives are “too big", then negative primitives will help the network carve away unnecessary geometry. In other words, there will be more useful regions that negative primitives can exist. We can achieve this bias by simply introducing an additional sample loss but only apply it to points where the GT label is inside, $O(x)=1$

$$ $L_{inside}=\mathbb{E}_{x\sim\mathbb{R}^{3}}||\hat{O}(x)-1||^{2}.$ (3) $$

We weight $L_{inside}$ by 0.1, and ablate that choice in Table 4.

Annealing loss weights Further, we found more stable training by annealing the weight of the overlap loss and alignment loss, starting from 0 at the beginning of training, up to the target weight midway through training. We preserve the annealing of the surface sample weight, whereby early in training free space samples are prioritized in the losses, and by midway of training, all samples have an equal weight. All these performance improvements are intended to aid the network in predicting high-level geometric structure of the scene early in training, then getting the details right towards the end.

Data augmentation Prior art did not successfully implement data augmentations in the form of vertical and horizontal flips. A correct implementation needs to take into account the effect of camera calibration parameters on the point cloud. We do so here and in practice, we observe substantial improvements across the board – see Table 5.

Augmentations are especially valuable given that the NYUv2 dataset is relatively small - though clearly sufficient for getting good results. Our procedure uses the standard 795/654 train/test NYUv2 split []. We use this dataset primarily to maintain consistency with existing evaluation metrics. We do not consider the volume loss or segmentation loss from [] in our experimentation, as they were shown to have an approx. neutral effect in prior work.

We implement our procedure in tensorflow and train all networks with Adam optimizer, learning rate $1.5\times 10^{-4}$, batch size 48, for 12000 iterations, requiring 2.5 hours on a single A40 GPU. Our method is RGBD input. Refining a single test image requires $\sim 55$ seconds for each network in an ensemble. Computing all results in Sec. 4 required approximately $800$ GPU hours in total, including preliminary experiments.

**Table 1: Evaluation without refinement: Regressors with a fixed number of primitives tend to work better when negative primitives are introduced, but negative primitives are not in themselves a major improvement. For some scenes, the fitting difficulties created by the negative primitives outweighs the improvements, but on average there tends to be some advantage. Each row is a regressor; $K^{total}$ gives the number of primitives; $K^{-}$ the number of negative primitives. Metrics are standard for depth, normals and segmentation.**
| Ensemble | $K^{total}$ | $K^{-}$ | $\text{AbsRel}\downarrow$ | $\text{Normals}_{\text{mean}}\downarrow$ | $\text{Normals}_{\text{median}}\downarrow$ | $\text{Seg}_{\text{acc}}\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| No | 12 | 0 | 0.130 | 38.219 | 34.285 | 0.588 |
| No | 12 | 1 | 0.132 | 38.719 | 34.771 | 0.585 |
| No | 12 | 2 | 0.131 | 39.289 | 35.018 | 0.590 |
| No | 16 | 0 | 0.126 | 38.404 | 35.177 | 0.608 |
| No | 16 | 1 | 0.128 | 38.332 | 34.460 | 0.605 |
| No | 16 | 2 | 0.131 | 38.776 | 34.933 | 0.598 |
| No | 20 | 0 | 0.125 | 41.606 | 36.726 | 0.635 |
| No | 20 | 1 | 0.130 | 39.423 | 35.549 | 0.619 |
| No | 20 | 2 | 0.128 | 38.857 | 35.405 | 0.621 |
| No | 24 | 0 | 0.120 | 38.930 | 35.744 | 0.647 |
| No | 24 | 1 | 0.123 | 39.250 | 35.255 | 0.640 |
| No | 24 | 2 | 0.124 | 38.670 | 34.513 | 0.639 |
| No | 28 | 0 | 0.127 | 38.717 | 35.071 | 0.637 |
| No | 28 | 1 | 0.124 | 39.769 | 35.417 | 0.647 |
| No | 28 | 2 | 0.126 | 38.864 | 34.856 | 0.643 |
| pos | 21.1 | 0 | 0.111 | 38.861 | 34.839 | 0.638 |
| pos + neg | 21.9 | 0.89 | 0.106 | 38.535 | 34.265 | 0.639 |

## 4 Experiments

We perform extensive quantitative and qualitative evaluation of our method. To do so, we use established evaluation procedures on the depth, normals, and segmentation inferred from the generated primitives. Ensembling regression alone is beneficial. In Table 1, we present several networks with different numbers of positive and negative primitives. We do not apply refinement at test time. Individually, each procedure performs quite well across a range of initial primitives. In some cases, introducing negative primitives helps on average (we test $K^{-}\in[0,1,2]$). When we ensemble the five networks without negative primitives, we get substantially better error metrics, particularly as measured by AbsRel of the depth map. To select the best method, we simply compare the depth of generated primitives against GT. Ensembling with negative primitives boosts quality. Further, when we enrich the vocabulary of operations with negative primitives, all error metrics get better across the board (pos+neg).

**Table 2: Evaluation with refinement: Refinement – polishing the convexes predicted by the regressor against the depth – improves fits significantly (cf. top block with Table 1), but the best strategy by far is to ensemble (bottom block). Ensembles with negative primitives available do best. Generally, refine-then-select (R$\rightarrow$S) is significantly better than select-then-refine (S$\rightarrow$R), likely because the fitting problem is extremely hard, so the start point for refining is a poor guide to how well the refinement will proceed. In the bottom block, the $K^{-}$ indicates the average number of negative primitives used per image, suggesting the best fit for a significant fraction of images has one or more negative primitives. First two rows: Even with fewer primitives, any individual network performs better than prior art.**
| Ensemble | Refine | $K^{total}$ | $K^{-}$ | $\text{AbsRel}\downarrow$ | $\text{Normals}_{\text{mean}}\downarrow$ | $\text{Normals}_{\text{median}}\downarrow$ | $\text{Seg}_{\text{acc}}\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| No (V. et al []) | Yes | 13.9 | 0 | 0.098 | 37.355 | 32.395 | 0.618 |
| No (V. et al []) | Yes | 15.7 | 0 | 0.096 | 37.355 | 32.700 | 0.630 |
| No | Yes | 12 | 0 | 0.086 | 35.646 | 30.851 | 0.611 |
| No | Yes | 12 | 1 | 0.090 | 36.211 | 31.401 | 0.600 |
| No | Yes | 12 | 2 | 0.088 | 36.383 | 31.386 | 0.607 |
| No | Yes | 16 | 0 | 0.079 | 35.412 | 31.099 | 0.632 |
| No | Yes | 16 | 1 | 0.083 | 35.762 | 31.031 | 0.626 |
| No | Yes | 16 | 2 | 0.087 | 36.138 | 31.792 | 0.613 |
| No | Yes | 20 | 0 | 0.078 | 37.520 | 31.771 | 0.657 |
| No | Yes | 20 | 1 | 0.083 | 36.416 | 31.680 | 0.637 |
| No | Yes | 20 | 2 | 0.081 | 35.831 | 31.535 | 0.639 |
| No | Yes | 24 | 0 | 0.074 | 35.472 | 31.511 | 0.667 |
| No | Yes | 24 | 1 | 0.074 | 35.334 | 30.441 | 0.662 |
| No | Yes | 24 | 2 | 0.075 | 35.378 | 30.539 | 0.661 |
| No | Yes | 28 | 0 | 0.073 | 35.050 | 30.403 | 0.672 |
| No | Yes | 28 | 1 | 0.073 | 35.942 | 30.825 | 0.670 |
| No | Yes | 28 | 2 | 0.077 | 35.321 | 30.501 | 0.667 |
| pos | S$\rightarrow$R | 21.1 | 0 | 0.076 | 35.721 | 30.932 | 0.657 |
| pos + neg | S$\rightarrow$R | 21.9 | 0.89 | 0.076 | 35.677 | 30.836 | 0.655 |
| pos | R$\rightarrow$S | 24.41 | 0 | 0.066 | 35.138 | 30.149 | 0.662 |
| pos + neg | R$\rightarrow$S | 24.83 | 0.80 | 0.064 | 35.085 | 29.950 | 0.676 |

Refinement improves all methods. In Table 2, we apply our refinement procedure on all test images using the GT depth map. Consistent with previous work, refining is essential to getting the best results. Observe how all error metrics, particularly AbsRel, get better with refinement, comparing Table 2 against Table 1. In particular, the negative primitives we introduced get better with refinement. While we get strong results across all numbers of primitives, the introduction of negative primitives only occasionally helps on average, in some cases slightly hurting metrics, which indicates that our test scenes are quite diverse and different settings are optimal for each scene. Refine-then-select performs better than select-then-refine. When we ensemble the five positive-only networks, rows pos, all error metrics get better than any method alone. However, the fact that we get better numbers when we select after refining indicates that this is an extremely difficult fitting problem whereby what appears to be the best start point may not necessarily yield the best endpoint. When comparing ensembles with negative primitives (pos+neg), we again observe that we are better off refining then selecting. Further, on average the network picks $0.80$ negative primitives in our best ensemble - which means they are genuinely helpful on some scenes. In Fig 4 we present histograms showing how many total and negative primitives were chosen on our test set. In practice, (left) our procedure is able to handle larger numbers of primitives better than prior work, observing that more primitives is generally better, and right, negative primitives can be quite helpful, noting that they are selected from the ensemble about half the time.

A biased sampling loss should be part of the ensemble. We ablate our decision to bias the sampling loss to favor classifying “inside” points correctly via $L_{inside}$. In Table 4, we test $w_{inside}\in[0.0,0.1,0.2,0.4]$ with $K^{-}=1$. This loss is clearly helpful across all numbers we tried. Yet, when more positive primitives are present, $K^{total}=20$, the loss hurts quality. We thus let $w_{inside}=0.1$ in our experimentation as a reasonable mid-ground.

Data augmentation yields more accurate decompositions. As Table 5 shows, augmenting RGBD input data with horizontal and vertical flips during training reduces the AbsRel depth error and increases the segmentation accuracy measurably, albeit with a marginal penalty w.r.t. normal accuracy.

Any individual network we train beats baselines. Without ensembling, with or without negative primitives, our method beats all baselines on nearly every metric - see Tables 2, 3.

Our combination of improvements strongly outperforms existing baselines. We can evaluate our approach using the occlusion-aware distance metric from []. Our ensembles with positive and positive+negative primitives strongly outcompete existing baselines.

**Table 3: Baseline comparisons: Ensembling strongly outperforms two recent SOTA methods, using the metrics reported by [], and using negative primitives in the ensemble produces further improvements. We ensemble with refine-then-select, and show results with only positive primitives present (Ours (pos), five networks, $K^{total}\in[12,16,20,24,28]$)), as well as with positive and negative primitives (Ours (pos+neg), 15 networks, $K^{-}\in[0,1,2]$). Our ensembles significantly outperform existing work. Further, we present results on the fifteen methods we trained, where $K^{total}/K^{-}$ is shown. Even without ensembling, any individual method we trained performs better than the baselines across every metric except $\text{mean}_{cm}$.**
|  | ${\text{AUC}_{@50}}{\uparrow}$ | ${\text{AUC}_{@20}}{\uparrow}$ | ${\text{AUC}_{@10}}{\uparrow}$ | ${\text{AUC}_{@5}}{\uparrow}$ | ${\text{mean}_{cm}}{\downarrow}$ | ${\text{median}_{cm}}{\downarrow}$ |
| --- | --- | --- | --- | --- | --- | --- |
| Vavilala et al.  [] | $86.9$ | $72.5$ | $56.5$ | $38.2$ | $26.6$ | $10.1$ |
| Kluger et al.  [] | $77.2$ | $62.7$ | $49.1$ | $34.3$ | $20.8$ | - |
| 12/0 | 88.4 | 75.7 | 59.6 | 39.8 | 24.2 | 8.9 |
| 16/0 | 89.3 | 77.7 | 62.2 | 42.3 | 22.8 | 8.2 |
| 20/0 | 89.2 | 78.2 | 63.4 | 44.2 | 22.9 | 7.9 |
| 24/0 | 90.1 | 79.6 | 64.2 | 43.3 | 21.4 | 7.6 |
| 28/0 | 90.2 | 80.7 | 66.9 | 47.1 | 21.3 | 7.0 |
| 12/1 | 87.5 | 74.2 | 58.3 | 39.3 | 25.5 | 9.3 |
| 16/1 | 88.5 | 76.7 | 61.5 | 42.2 | 24.1 | 8.4 |
| 20/1 | 88.3 | 76.4 | 61.0 | 42.1 | 24.7 | 8.3 |
| 24/1 | 89.9 | 79.8 | 65.8 | 46.7 | 21.6 | 7.1 |
| 28/1 | 89.9 | 79.9 | 66.2 | 47.4 | 21.6 | 7.0 |
| 12/2 | 87.4 | 74.7 | 59.2 | 40.5 | 25.7 | 9.2 |
| 16/2 | 87.8 | 75.3 | 59.1 | 39.6 | 25.6 | 9.2 |
| 20/2 | 88.7 | 77.4 | 62.0 | 42.1 | 24.1 | 8.4 |
| 24/2 | 89.8 | 79.9 | 65.8 | 46.3 | 22.1 | 7.2 |
| 28/2 | 89.3 | 79.7 | 66.1 | 47.0 | 22.7 | 7.2 |
| Ours (pos) | 91.2 | 81.5 | 67.1 | 47.3 | 19.3 | 6.79 |
| Ours (pos + neg) | 91.5 | 82.0 | 68.0 | 48.4 | 18.8 | 6.58 |

**Table 4: Biased sampling loss: Weighting errors on inside samples higher than those on outside samples helps, particularly in the regime where there is a negative primitive and there are few positive primitives. This is likely a form of crude boosting effect, that forces the negative to “cut out” incorrectly classified inside samples. $w_{inside}=0$ is no bias; increasing values are more biased.**
| $w_{inside}$ | $K^{total}$ | $K^{-}$ | $\text{AbsRel}\downarrow$ | $\text{Normals}_{\text{mean}}\downarrow$ | $\text{Normals}_{\text{median}}\downarrow$ | $\text{Seg}_{\text{acc}}\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| 0.0 | 12 | 1 | 0.122 | 37.693 | 33.136 | 0.556 |
| 0.1 | 12 | 1 | 0.090 | 36.211 | 31.401 | 0.600 |
| 0.2 | 12 | 1 | 0.093 | 36.292 | 31.736 | 0.594 |
| 0.4 | 12 | 1 | 0.097 | 36.424 | 31.482 | 0.607 |
| 0.0 | 20 | 1 | 0.080 | 35.707 | 31.301 | 0.651 |
| 0.1 | 20 | 1 | 0.083 | 36.416 | 31.680 | 0.637 |
| 0.2 | 20 | 1 | 0.086 | 36.521 | 31.620 | 0.637 |
| 0.4 | 20 | 1 | 0.090 | 36.388 | 32.470 | 0.642 |

**Table 5: Data augmentation: Unlike prior work [], we utilize data augmentation of 3D samples in the form of random vertical and horizontal flips during training. This yields better decompositions as measured by AbsRel depth error and segmentation accuracy. We consider the slightly worse normals an acceptable trade-off and they are worth continued investigation.**
| Augment | $K^{total}$ | $K^{-}$ | $\text{AbsRel}\downarrow$ | $\text{Normals}_{\text{mean}}\downarrow$ | $\text{Normals}_{\text{median}}\downarrow$ | $\text{Seg}_{\text{acc}}\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| Yes | 24 | 0 | 0.074 | 35.472 | 31.511 | 0.667 |
| No | 24 | 0 | 0.080 | 35.414 | 30.773 | 0.652 |

Figure: Figure 4: There are a wide range of active participants in the ensemble. This figure shows the frequency with which ensemble members are chosen by refine-then-select for 654 test scenes, broken out by number of positive primitives and number of negative primitives. Note how often regressors with large numbers of positive primitives are chosen; and that negative primitives are used frequently.
Refer to caption: x4.png

## 5 Discussion

The key goal of primitive representations since the
1960s has been to demonstrate representations that can (a) be computed
from data and (b) genuinely simplify reasoning tasks.
We have demonstrated a method that can produce accurate fits of
multiple convex primitives, some “negative”, to complex indoor
scenes represented in RGBD images. Our method really can be computed
from data, and in accuracy significantly
outperforms SOTA.

#### Limitations

The method requires ensembling a number of regressors, with consequent costs in
training and inference time. While we can evaluate accuracy, it is
difficult to usefully assess the extent to which the method is
parsimonious, apart from looking at the relatively small number of
primitives used. We have shown partial progress on simplifying
reasoning tasks (the segmentation implied by the primitives is fair,
but not competitive with the best semantic segmenters). In general,
demonstrating that primitives are useful remains open, as it has since
the 1960s.

## 6 Appendix / supplemental material

Figure: Figure 5: Ablation on number of finetuning steps. Polishing the prediction from regression is essential to obtaining good results. This is true both when GT depth is available (solid line) or depth must be inferred from RGB (dashed line). A different error metric is shown in each row: AbsRel, Segmentation accuracy, and mean normal error (in degrees). Even when introducing negative primitives (columns 2 and 3), polishing helps significantly. In this work, we perform quantitative evaluation with 300 finetuning steps wherever we do perform polishing, a reasonable tradeoff between quality and speed. In this figure, $K^{total}=16$, and we evaluate on 80 random test images.
Refer to caption: x5.png

**Table 6: Evaluation with inferred depth: Our procedure still works well even when ground truth depth isn’t available. Without ensembling, any of our fifteen networks with $K^{total}\in[12,16,20,24,28]$ and $K^{-}\in[0,1,2]$ perform better than prior work [], even with fewer primitives available. We can ensemble these networks by evaluating the primitive decomposition against an inferred depth map (we use []). $S\rightarrow R$ means that we evaluate AbsRel of the primitives generated by the network against the MIDAS depth map across all members of the ensemble, select the best one, then refine that one (using the MIDAS depth map for supervision in the losses). $R\rightarrow S$ means that we refine predictions from all members of the ensemble on a particular test image, then choose the best one as measured by AbsRel against MIDAS. Ensembling generally helps, particularly when we use $R\rightarrow S$. However, introducing negative primitives into the ensemble yields slightly worse normal and segmentation in exchange for better depth. The implication is that effectively fitting negative primitives remains hard.**
| Ensemble | Refine | $K^{total}$ | $K^{-}$ | $\text{AbsRel}\downarrow$ | $\text{Normals}_{\text{mean}}\downarrow$ | $\text{Normals}_{\text{median}}\downarrow$ | $\text{Seg}_{\text{acc}}\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| No (V. et al []) | Yes | 14.4 | 0 | 0.144 | 38.235 | 33.621 | 0.615 |
| No | Yes | 12 | 0 | 0.1437 | 36.440 | 32.491 | 0.588 |
| No | Yes | 12 | 1 | 0.1375 | 37.288 | 32.765 | 0.597 |
| No | Yes | 12 | 2 | 0.1351 | 37.285 | 32.606 | 0.607 |
| No | Yes | 16 | 0 | 0.1384 | 36.706 | 32.369 | 0.620 |
| No | Yes | 16 | 1 | 0.1340 | 36.858 | 32.411 | 0.627 |
| No | Yes | 16 | 2 | 0.1365 | 37.120 | 32.871 | 0.611 |
| No | Yes | 20 | 0 | 0.1305 | 37.893 | 32.814 | 0.656 |
| No | Yes | 20 | 1 | 0.1341 | 37.562 | 33.089 | 0.637 |
| No | Yes | 20 | 2 | 0.1312 | 37.169 | 33.083 | 0.641 |
| No | Yes | 24 | 0 | 0.1298 | 36.552 | 32.261 | 0.669 |
| No | Yes | 24 | 1 | 0.1291 | 36.738 | 32.181 | 0.662 |
| No | Yes | 24 | 2 | 0.1296 | 36.809 | 32.238 | 0.660 |
| No | Yes | 28 | 0 | 0.1300 | 36.405 | 32.101 | 0.672 |
| No | Yes | 28 | 1 | 0.1288 | 37.298 | 32.596 | 0.667 |
| No | Yes | 28 | 2 | 0.1298 | 36.808 | 32.272 | 0.668 |
| pos | S$\rightarrow$R | 23.284 | 0 | 0.1300 | 36.647 | 32.142 | 0.659 |
| pos + neg | S$\rightarrow$R | 22.514 | 0.786 | 0.1296 | 36.765 | 32.161 | 0.654 |
| pos | R$\rightarrow$S | 27.908 | 0 | 0.1294 | 36.395 | 32.092 | 0.672 |
| pos + neg | R$\rightarrow$S | 27.939 | 1.997 | 0.1287 | 36.929 | 32.396 | 0.668 |

**Table 7: Baseline comparisons with inferred depth: Ensembling strongly outperforms two recent SOTA methods, using the metrics reported by [], and using negative primitives in the ensemble produces further improvements. Here, we evaluate when GT depth is not available and must be inferred. We ensemble with refine-then-select, and show results with only positive primitives present (Ours (pos), five networks, $K^{total}\in[12,16,20,24,28]$)), as well as with positive and negative primitives (Ours (pos+neg), 15 networks, $K^{-}\in[0,1,2]$). Our ensembles significantly outperform existing work. Further, we present results on the fifteen methods we trained, where $K^{total}/K^{-}$ is shown. Even without ensembling, any individual method we trained performs better than the baselines across every metric except $\text{mean}_{cm}$.**
|  | ${\text{AUC}_{@50}}{\uparrow}$ | ${\text{AUC}_{@20}}{\uparrow}$ | ${\text{AUC}_{@10}}{\uparrow}$ | ${\text{AUC}_{@5}}{\uparrow}$ | ${\text{mean}_{cm}}{\downarrow}$ | ${\text{median}_{cm}}{\downarrow}$ |
| --- | --- | --- | --- | --- | --- | --- |
| Vavilala et al.  [] | $77.3$ | $47.6$ | $26.8$ | $13.9$ | $40.2$ | $26.2$ |
| Kluger et al.  [] | $57.0$ | $33.1$ | $18.9$ | $10.0$ | 34.5 | - |
| 12/0 | 77.8 | 48.1 | 27.1 | 13.9 | 39.4 | 26.2 |
| 16/0 | 78.8 | 49.2 | 27.8 | 14.4 | 37.9 | 25.1 |
| 20/0 | 80.1 | 50.7 | 28.7 | 14.9 | 36.7 | 24.1 |
| 24/0 | 80.4 | 50.8 | 28.8 | 15.0 | 36.2 | 24.0 |
| 28/0 | 80.3 | 50.8 | 28.7 | 14.8 | 36.2 | 24.1 |
| 12/1 | 78.1 | 48.7 | 27.5 | 14.3 | 39.2 | 25.7 |
| 16/1 | 79.0 | 49.8 | 28.1 | 14.6 | 38.4 | 25.1 |
| 20/1 | 78.9 | 49.6 | 28.1 | 14.6 | 38.6 | 24.8 |
| 24/1 | 79.9 | 50.8 | 28.9 | 15.0 | 37.1 | 24.2 |
| 28/1 | 79.9 | 50.7 | 28.9 | 15.0 | 37.2 | 24.2 |
| 12/2 | 78.6 | 49.3 | 27.9 | 14.5 | 38.8 | 24.8 |
| 16/2 | 78.4 | 49.1 | 27.9 | 14.4 | 39.1 | 25.5 |
| 20/2 | 79.5 | 50.1 | 28.5 | 14.7 | 38.0 | 24.7 |
| 24/2 | 79.9 | 50.7 | 28.8 | 14.9 | 37.2 | 24.1 |
| 28/2 | 79.8 | 50.7 | 28.9 | 15.0 | 37.3 | 24.1 |
| Ours (pos) | 81.1 | 51.6 | 29.4 | 15.3 | 35.0 | 23.16 |
| Ours (pos + neg) | 81.4 | 52.3 | 30.0 | 15.7 | 34.5 | 22.75 |

Figure: Figure 6: Qualitative evaluation when GT is present, best method chosen by ensemble boxed in green. Row label shows the $K^{total}/K^{-}$.
Refer to caption: x6.png

Figure: Figure 7: Qualitative evaluation when GT is present, best method chosen by ensemble boxed in green. Row label shows the $K^{total}/K^{-}$.
Refer to caption: x7.png

Figure: Figure 8: Qualitative evaluation when GT is present, best method chosen by ensemble boxed in green. Row label shows the $K^{total}/K^{-}$.
Refer to caption: x8.png

Figure: Figure 9: Qualitative evaluation when GT is present, best method chosen by ensemble boxed in green. Row label shows the $K^{total}/K^{-}$.
Refer to caption: x9.png

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 5
presentation: 2
questions: See above.
soundness: 3
strengths: 1.This paper tackles a challenging and unresolved task of scene convex decomposition from a single RGBD image and significantly advances the performance on the NYUv2 dataset.
2.It is reasonable to introduce negative primitives as a subtraction operator in CSG, which adds flexibility to primitive-based scene representations. The authors show examples demonstrating that boolean primitives are parameter-efficient for fitting certain geometric shapes. Additionally, the ensembling strategy is straightforward and easy to follow.
3.The paper includes extensive ablation experiments to validate the contribution and performance improvement of each proposed module.
summary: This paper presents an approach for decomposing indoor scenes into convex primitives from a single RGBD image. It introduces two key strategies for convex decomposition: the ensembling process and the use of boolean primitives. The ensembling strategy involves employing multiple networks, each predicting a different number or type of primitives, and selecting the optimal primitive set based on refinement loss. The boolean primitives strategy utilizes constructive solid geometry (CSG) operations, allowing up to two negative primitives for subtraction, which enhances geometric accuracy by trimming unnecessary parts of positive primitives. Additionally, the paper proposes skills to improve fitting accuracy, such as biasing sample loss, annealing loss weights, and data augmentation. Experiments on the NYUv2 dataset demonstrate that the proposed strategies enhance geometric representation accuracy and outperform previous state-of-the-art methods in depth estimation, normal prediction, and scene segmentation accuracy. While the method proposed in the paper enhances task performance, some design aspects remain somewhat puzzling and could be further improved.
weaknesses: 1.The main contributions of this paper are ensembling predictions from multiple networks and introducing negative primitives. However, these strategies do not appear to yield consistent performance gains. In Table 1, the naive method '24/32/40' outperforms both 'Pos - S→R' and 'Pos + Neg S→R', suggesting that ensembling and negative primitives are not accurate enough to help candidate selection before refinement. Although 'Pos - R→S' and 'Pos+Neg - R→S' achieve better performance, this is likely due to the availability of more final results for selection, indicating that the "refinement then selection"  is still necessary and impactful. In Table 2, it is also somewhat puzzling that, in some cases, introducing negative primitives reduces fitting accuracy.
2.The ensembling strategy is simple but underdeveloped. While it provides more candidate sets of convexes, it also increases computation time significantly. Besides, the number of primitives is preset, meaning the approach is only effective if sufficient networks are available to generate adequate candidates across diverse scenarios. A more effective approach might be to fuse all convexes from multiple networks and then learn to select the best results.
3.Boolean primitives convincingly provide a more accurate and efficient scene representation. However, it is confusing that the maximum number of negative primitives is limited to only 2, and in some cases, the negatives actually reduce performance, as shown in Table 2. Additionally, the paper notes that the primary performance gains from the negative operation occur when a negative primitive occupies empty floor space, whereas it seems more reasonable that negative primitives should have greater potential to remove excess space occupied by positive primitives.
4.In the experiments, the paper mainly compares its results to "Convex decomposition of indoor scenes," but lacks a comparison to "Robust Shape Fitting for 3D Scene Abstraction, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024."  Besides, while shape abstraction is a lighter-weight representation, it would be appreciated to include application examples of convex decomposition for completeness.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: I wonder how each proposed component makes a qualitative difference.
soundness: 3
strengths: - The paper is well-written and easy to follow.
- The ensemble of results to avoid local optima, though straightforward in general, is novel in learning-based shape abstraction. This ensembling and "pick-best" strategy is well-suited to the setup, where multiple inference results can be evaluated against the given depth map. Interestingly, using negative primitives alone shows limited performance gain, and the paper empirically demonstrates that a refine-then-choose strategy is more effective than the reverse.
- The proposed method outperforms previous works across all metrics, with a notable improvement in depth accuracy over the current SOTA.
- The bias loss term is well-motivated for negative primitives.
summary: This paper presents a shape abstraction method using convex primitives for indoor scene understanding. The core contribution lies in the observation that while negative (in the CSG sense) primitives alone provide limited benefits, their combination with model ensembling yields notable performance gains. The method is evaluated on the NYU v2 dataset, using reconstruction accuracy metrics for depth and normal, and segmentation accuracy, following the experimental setups of previous works. Notably, the proposed method significantly outperforms the most relevant SOTA method in depth accuracy.
weaknesses: - According to the descriptions of Figures 10 and 11, the negative primitive is used only for the floor. This raises concerns about whether the negative primitive is functioning as intended, as indicated in Figure 2.
- While following the experimental setup of previous works, the proposed method is only evaluated on the NYU v2 dataset, which is relatively small compared to other commonly used benchmarks for indoor RGB-D data, like SUN-RGBD and ScanNet. Evaluation on these larger datasets would strengthen the paper significantly.
- Although well-motivated, the impact of the bias loss term does not seem significant or consistently effective in improving accuracy, as shown in Figure 8.
- An ablation study on learning rate annealing is missing.
- The paper lacks a theoretical explanation on why using negative primitives alone provides limited benefits and combining them with the ensemble method yields better performance.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: - What exactly is the criteria for evaluating a decomposition result? In L195, the authors mention that they use depth map error to decide which decomposition is the best. However, this criteria might not be perfect, since the user might want a decomposition with fewer primitives and clearer structure, despite having larger reconstruction error.
soundness: 2
strengths: - The problem studied in this paper is interesting and important. Decompositing 3D indoor scene geometry into several primitives could facilitate the better understanding of the structure of 3D scene geometry.
- The proposed method in this paper appears to be sound and effective. Performing decomposition with multiple networks will indeed improve the reconstruction result for sure, and adding negative primitives indeed might reduce the representation complexity under certain scenarios.
- Extensive ablation studies are conducted in the paper to show the effectiveness of the proposed compoenents.
summary: This paper studies two techniques to improve the problem of fitting a set of primitives to indoor scenes. Specifically, they first propose to use an ensemble of networks to predict a variety of solutions, and to choose the best one according to certain metrics. They further propose to predict negative primitives to complement with the positive primitives and perform boolean operations, which improves the performances as well. Experiments are conducted to show the effectiveness of the proposed components.
weaknesses: - The writing of this paper is hard to follow. Specifically the paper is based on a prior work, yet the prior work is not fully explained beforehand. Therefore the general method reads disconnected and fails to explain the method well enough.
- It is not clear on the relationship between the proposed method and works that perform CSG decomposition, such as [Du et al. 2018]. However, the negative boolean primitive discussed in this work is relevant to the traditional CSG decompostion of 3D geometry. Moreover, can the proposed method be applied to more general CSG grammar?
- The experiment setting of the evaluation section is not clearly illustrated. Specifically the major metric is the reconstruction metric, which might not convey enough message since the main goal of doing decomposition is to facilitate understanding. Moreover the experiment setting of the proposed method and prior works can be pretty different, since prior works do not predict negative primitives.
- Bringing negative primitives to the prediciton result, although improving reconstruction accuracy, yet can bring other issues. For example, the segmentation and understanding can become more tricky when there are negative primitives.

[Du et al. 2018] InverseCSG: automatic conversion of 3D models to CSG trees.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Can the model generalize to more varied datasets given its high resource demands?
soundness: 3
strengths: 1. Innovative Use of Boolean Primitives: The introduction of negative primitives is a notable addition, enabling more efficient geometric abstraction by allowing finer control over free space representation.

2. Effective Ensembling Strategy: The refine-then-choose approach in ensembling leads to significant improvements in fitting accuracy by allowing multiple regression methods to predict different start points.
summary: This paper proposes an improved approach to convex decomposition by integrating ensembling techniques and Boolean primitives. The method enhances the accuracy of depth, normal prediction, and segmentation, addressing issues in scene fitting and reducing reliance on a fixed number of primitives by allowing a flexible start point that is iteratively refined.
weaknesses: 1. High Computational Demand: The ensemble-based approach requires significant training and inference resources, which may limit scalability and real-world applications.

Meta-review: This paper introduces ideas to improve a previous method for reconstructing an indoor scene into a set of primitives from an RGBD image. Building upon the method by Vavilala and Forsyth, the authors propose two main ideas: 1) incorporating negative primitives, and 2) ensembling multiple regressors. The experiments demonstrate the improvement of these two ideas compared to the baseline method.

Most reviewers gave negative scores and raised concerns about the lack of explanation and justification for why negative primitives and ensembling improve the results, as well as the lack of qualitative analysis. The post-rebuttal discussion quickly converged to reject the submission, and the AC agrees with the reviewers' comments.

Decision: Reject