Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper describes a method for sequentially assembling Lego bricks into 3D structures resembling objects from training classes. Efficient GPU-based methods are developed to filter out invalid attachment points and to learn to identify high-value placements. Unlike prior work, the method handles a heterogeneous mix of brick types (a mix of 2x4 and 2x2 bricks is demonstrated).
strength_and_weaknesses: The paper is well-written and presents a plausible and efficient solution to the problem. Long assembly sequences with a wide branching factor can be difficult to control, and the presented results look quite good considering this challenge. The method guarantees that the assemblies are valid by construction. The training is also self-supervised and does not appear to use explicit GT construction sequences.

That said, the (not great in absolute terms) quality of the results does leave one wondering if combining a voxel-based generator with a subsequent (non-ML) step that fits an arrangement of bricks to the voxel pattern would be a better way to generate (still structurally valid) assemblies of Lego bricks, rather than to impose distributional constraints on very long Markov sequences. The presented method is an interesting technical demo of controlling long assembly sequences with tight constraints, but its practical utility in the context of Lego construction seems limited. Autoregressive methods have their place, but in this case it seems like a particularly challenging paradigm to employ, and there don't seem to be obvious takeaways transferable to other sequential assembly problems. The "Potential Applications" section mentions building construction and robotic assembly, and in each of these tasks there seem to be easier ways to achieve the end goal.

Finally, even though the paper claims that the assembly sequences are ones humans can actually follow ("can... provide instructions to build the structure"), the sequences don't actually take things like phsyical stability and connection strength of intermediate structures into account, so this may not be true in all cases.

clarity,_quality,_novelty_and_reproducibility: The paper is very clear and appears to be original. The method appears to be novel though the high-level problem has been addressed in other papers. The tech is non-trivial and plausible. For other quality assessments, please see my comments above.

summary_of_the_review: I think this paper does a good job in developing an improved algorithm for sequential Lego brick assembly. That said, there are some weaknesses which I have identified above. Hence, I think the paper, while interesting, does not represent a very significant step forward. If the authors could argue for greater impact of the algorithm, that assessment might change.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an approach for the sequential brick assembly problem (SBA). To the best of my understanding SBA is a special case of sequential assembly problem with a finite number of actions (possible brick placements). The proposed approach is based on a combination of a convolutional NN, which is traned to predict the score of possible placements, and convolutions for quickly filtering out infeasible solutions with respect to given physical constraints.
strength_and_weaknesses: Strengths:
- The proposed approach leverages tensorial computations with a significant performance gain
- The code will be released upon publication

Weaknesses (for details, see Clarity section below):
- The presentation could be significantly improved 
- Some crucial aspects of the problem are not empirically validated
clarity,_quality,_novelty_and_reproducibility: **Presentation**

    "Thompson et al. (2020) propose to use masks along with their
    method, but the utilization of masks degrades assembly
    performance."

What are masks?

---

    "these methods share common limitations: they only use a single
    brick type"

They *can't* be adapted to use multiple shapes or they were simply
tested with a single type?

    "inevitably become slower as the number of bricks increases due to
    an exponentially increasing search space."

I'm quite sure that this is true for the proposed method too. The
difference is that the proposed method can be easily parallelized on GPUs. If my understanding is correct, the computation for different candidate shapes can be carried out in parallel regardless of the assembly technique. In principle, the other techniques could be parallelized, right?  

---

    "To tackle the limitations above, we devise a novel brick assembly method with a U-shaped sparse
    3D convolutional neural network utilizing a convolution filter to validate complex brick constraints
    efficiently."

What is the intuition though? I wouldn't describe in detail on how
the core idea is implemented ("U-shaped 3D CNN") in the first sentence
after the related work.

At the same time, the specific choice of neural architecture is never motivated in the rest of the paper.
Why using a U-Net? Have you considered (and performed ablation studies with) different architectures?

---

    "BrECS performs the best with 117% higher average classification scores than the next best one."

I have no idea what 117% higher average classification score
means. Does it mean "more than twice as accurate"? I wouldn't report
numerical results here. Even if the metric is well-known or clearly
explained, it is not really useful to the reader at this point.


---

    "Unlike the sequential brick problem, a sequential part assembly
    problem also shares similar difficulties to the combinatorial
    optimization problem [...]"

Unlike? Is 'sequential brick problem' different than 'sequential brick
ASSEMBLY problem'? If not, isn't the sequential brick assembly problem
a special (simpler) case of sequential part assembly problem?


---

    "these constraints encourage us to accentuate the nature of
    combinatorial optimization since there exist a huge number of
    possible combinations in the presence of complex physical
    constraints."

What do you mean with 'to accentuate the nature of CO'?

---

    "Moreover, we expect that our neural network produces a likely
    complete or potentially next-step 3D structure, which is
    represented by a probability of voxel occupancy."

It is not clear what Bt' should represent. An example would definitely
help. Is it a probability mass function over a x a x a? How do you
ensure that the U-net outputs a valid distribution?

---

    "In the case of assembling a structure with r brick types, we
    prepare r convolution filters of the same size with r brick types
    and repeat the above process for each convolution filter."

I guess that the actual number should be at most r * 4 (different
rotations for asymmetric shapes). Here you should briefly mention how
convolutions have to be adjusted for asymmetric shapes.

---

    "a score for next brick positions A(t+1) in R^(a×a×a) is computed by
    sliding a convolution filter"

An example here would also help. I don't get the intuition on why the
dimensionality of A(t+1) should be the same as Bt', isn't A(t+1)
scoring the different options in terms of brick positioning? These
should be less than R^(a×a×a).

The same doubts apply to V(t+1).

---

    "The motivation of our method to select a pivot brick is that a
    pivot brick with more attachable positions should be more
    preferable than one with less attachable positions, rather than
    choosing a pivot brick that is connected to a position with the
    highest score of C(t+1)."

Shouldn't the method actively try to maximise the objective?

    "Since a NN tends to memorize training samples and their assembly
    sequences, choosing a position with the highest score fails to
    create novel structures."

The fact that you don't expect your predictor to generalize doesn't
seem a good reason for disregarding the score.

Overall, I think that the sampling procedure is not clearly
presented. I would also expect a discussion on the
exploration-exploitation trade-off, i.e. how to control diversity
vs. quality of the generated structures.

---

    "By replacing Bt' , i.e., the output of the U-Net, with a
    ground-truth voxel occupancy, we generate an assembly sequence
    following the procedure of BrECS."

Isn't BrECS the proposed approach?


---

It is not clear which limitations are
inherent to the proposed approach and which ones can be addressed with
future work. Can the proposed approach accomodate any constraint
besides the ones considered here (e.g. constraints on the frequency of
certain blocks, or budgets)? My intuition is that filtering out infeasible configurations with convolutions only works for a restricted number of constraints, but I might be wrong.

---

**Experiments**

    "Generated sequences are unique and diverse, since the
    stochasticity is injected from the sampling strategy introduced in
    Section 4.1.3."

Isn't uniqueness implying diversity? More importantly, diversity is not measured in any way.

    "We demonstrate that our model generates diverse structures with
    high-fidelity satisfying physical brick constraints in the
    experiments on the completion of brick structures."

The trade-off between quality and diversity is not investigated in the
experiments.

High-fidelity is often mentioned in the paper but never formally
defined. After re-reading the paper I realized that the goal is to
generate structures that are similar to a reference model.
This aspect could be clarified.





summary_of_the_review: While the proposed approach addresses some computational challenges of the problem considered, its presentation should be improved before publication. Little insights are provided on why the technique is effective and what are its limitations. The experimental section disregards important  aspects in this sequential generation, such as the exploration vs. explotation tradeoff.
All these aspects require a major revision of the paper. I cannot recommend the current version for publication.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method that generates high-fidelity brick structures satisfying physical constraints for two tasks: partial structure completion and novel structure generation for a given class of shape. The main contributions are:
- A parallelizable brick position predictor and validator based on convolution filters, making the inference of the proposed brick sequence generation method efficient on GPU for large brick structures.
- The proposed method is agnostic to the brick type, with experimental demonstrations of assembling structures using both 2x4 and 2x2 bricks.
strength_and_weaknesses: Strengths:
- Technically sound methodology. The technical flow in Figure 1 is intuitive.
- Good empirical performance in terms of shape completion accuracy, semantic generation quality, validity, and efficiency compared to baselines.

Weaknesses:
- The paper is poorly written with confusing logic, and the task definition is unclear until the end of the paper. Such poor clarity is unacceptable for more potential readers. See more details below.
- The wider impact or applications of the research problem are not clear. Though the authors believe this method can be extended to building design or industrial robots, the proposed method seems highly specific to LEGO bricks and hard to extend.

Paper writing and clarity issues:
- The abstract does not illustrate which problem/goal specifically you are targeting. Obviously, you are working on brick sequence generation, but is the generation conditioned on certain target shapes or shape classes? From the current abstract, it seems to be that you are generating ANY assembly sequence that satisfies the physical constraints, which confused me because I don't see why you need such a complicated approach to achieve that.
- The detailed definitions of completion and generation tasks are only vaguely described in Sec 5.1-5.2. In other words, most of the time when I was reading the paper, I didn't have a clear clue of which specific problem the authors were addressing. The terms "completion task" and "generation task" are frequently used in the middle of the paper without explanation. So please bring the task definitions forward - let people know what you are addressing even before describing the technical approach.

Questions:
- Sec 4.1.3 details the pivot brick selection procedure, where the authors propose sampling to avoid being deterministic. This is reasonable for the novel structure generation task. But for the shape completion task, does adding stochasticity help?
- In Sec 4.2, I am not sure if I understand this correctly: "By replacing $B_t^{'}$ with a ground-truth voxel occupancy, we generate an assembly sequence ...". Would you mind elaborating more on this procedure?
clarity,_quality,_novelty_and_reproducibility: - Clarity is not enough in my opinion for the reasons mentioned above.
- The problem formulation and technical approaches are novel compared to previous works.
- Reproducibility is not an issue for me if the authors are willing to release the code.
summary_of_the_review: Though the results seem good, I cannot give acceptance based on the current state of the paper, as there is so much confusion. I look forward to the authors' clarification on important aspects.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Authors propose a method for sequential generation of Lego models. This addresses two tasks – (i) generating new class-conditional Lego models a priori; (ii) completing partial models. The proposed approach consists of a neural network to predict how good candidate locations for the 'next' brick is, and various checks to ensure constraints on fabricability are fulfilled. The method is demonstrated on shapes from the ModelNet dataset, and shown to perform favorably versus the chosen baselines.
strength_and_weaknesses: - the first task considered is generation of models of certain object classes, from Lego. However, this is straightforwardly accomplished by using an existing generative model of ModelNet shapes (outputting any representation that can be converted to voxels), then applying Brickr [Testuz, Eurographics 2013] to convert this to a valid Lego model. That approach already supports many more brick types than the proposed method, and supports all the relevant constraints.

- the second task considered is completing partial lego models of ModelNet classes. This can again be addressed easily using existing techniques – convert to a point cloud, run a point-cloud completion model, convert back to voxels, and run Brickr on the 'new' part, giving the uppermost part of the input model as the first Brickr layer.

- very few qualitative generation examples are given (three examples from two classes) – making it impossible to judge whether generated shapes are in fact reasonable. Those which are given, are not particularly impressive, and it's not specified whether they are curated.

- there is no quantification (based on nearest-neighbors) of whether the method is in fact generating novel shapes, or simply memorising its training set

- the constraints in sec. 3 need more explanation. Do they correctly disallow 'squeezing in' blocks? Why must structure be fully-connected from start (submodules assembled separately then combined are common in real Lego)?

- the overall approach is a complex engineered pipeline, very specific to the task, and most stages of this are (in my opinion) not of great interest to the majority of the ICLR community

- stochasticity in the generative process appears to be introduced only by randomly selecting which of a candidate set of points to consider at each step. This is fine a-priori; however, the set of candidates appears to be determined not by the learnt model directly, but by the number of possible locations that bricks could be affixed to them. Why not let the model learn the distribution over 'next' attachment points? Also, why is the selection of 'pivot brick' artificially separated from choosing which particular location on this brick to attach to?

clarity,_quality,_novelty_and_reproducibility: - the flow is unclear in places – e.g. sec. 4.1 says it looks at satisfying constraints, yet 4.1.1 instead addresses generating unconstrained 'possible next step' voxelisations

- aside from this the writing is generally good – clear and fluent, with very few typos

- the precise tasks addressed are not defined very clearly early in the paper – inputs, outputs, etc.

- the task is novel, as is the overall approach proposed. Within the pipeline, all the components are fairly standard

- it appears that sufficient details are given to allow reimplementation of the proposed method (particularly if the authors make the code public as promised)

summary_of_the_review: As noted above, I believe the tasks can be addressed straightforwardly using existing methods (at very least, these should be included as a baseline). Moreover, the pipeline is not particularly interesting technically, nor do there seem to be many generalisable insights that would be beneficial to the community.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper could potentially be interesting, however it is poorly written. Detailed explanations of certain parts are OK, but the overall structure is confusing and shouldn't be published in the current form. To be specific, even the problem formulation takes a while to piece together. The basic problem and proposed solution should be outlined in the beginning and be clear. For example: 1. What is the problem space (e.g. 3d locations of 2x4 and 2x2 blocks or something else)?, 2. What is the objective (e.g. here is a 3d structure specified as such and such, find a sequence of brick locations that would assemble it or something else), 3. What is the objective optimised (e.g. overlap with target structure, sampled from a dataset or something else), 4. What is the dataset (structures, sequences?, made of lego or just generic shapes) and how do you use it. 5. What is the basic idea of your solution (e.g. RL on the objective function (no) or something else...).
As as second point, as reviewers pointed out, there are methods to solve this problem. Depending on who you ask, proposing the method in this paper might or might not be of value. In my opinion, if the method is interesting and general, it can still be of value even if solved by other means, but finding a problem where other methods don't work well would make a stronger case for the method.
justification_for_why_not_higher_score: The paper is poorly written and the case for this method is weakened by other methods that work well on this problem.
Decision: Reject