Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The proposed combinatorial construction is a new formulation for combinatorial problems
which have a vast number of possible combinations. This requires the agent to assemble LEGO
units sequentially to build the desired structure based on the 2D target image. The authors use a
reinforcement learning method with Graph-Neural-Network to solve this formulated problem
which requires a comprehensive understanding of the incomplete input information and long-
term planning. The authors also introduce an action validation network to help eliminate
invalid actions when the action space increases as the assembly processes. The
proposed method is evaluated on three types of tasks with different inputs and is compared
with three types of baselines: Bayesian optimization-based method, fully-MLP architecture, and supervised learning method.
main_review: *Strengths*:   
First of all, the paper is well organized and written, and the reader can easily follow the logic of the
paper. The proposed method has a good generalization ability among different types of 2D
inputs: single view and multiple views. The experiment results show the proposed RL based
method outperforms the fully-MLP based method which indicated the effectiveness of using
GNN architecture for embedding the input data. From the qualitative results on both MINST
and ModelNet, it is shows that the proposed promise a plausible 3D construction with the
partial information.

*Weakness*:
1. The proposed formulation for this combinatorial problem is interesting, but I don’t see it
would be easily transfer to real construction problems, especially when the authors claims that they
"focus on the real-world construction procedure". Because it is hard to use the
action validation network to eliminate the invalid actions when the action space will be
continuous in real world problem.
2. The action validation network is pre-trained when used in the experiments. It is
necessary to show how the performance would be when the action validation network is
trained jointly with the reinforce pipeline.
3. It is also interesting to show some ablation study on this action validation network.
4. Would the performance be improved when using a more complete 3D model as input for
the proposed method? Is it reasonable to assume only images as input, given that 3D models are available?
5. There are also some concerns about the baseline selection and justification (more on this below).
limitations_and_societal_impact: The proposed propose an interesting formulation of one combinatorial problem and
implement this in the OpenAI Gym environment. Although, the proposed method works well
on this combinatorial construction problem and outperforms some baselines, the paper lacks
some necessary experiments to fully support the usage the action validation network.

Also, the motivation of the problem setup does not seem to be well justified, such as the use 
of images instead of 3D models, and why only DRL is picked as baselines? What about other 
type of baseline solutions such as search-based method, or even supervised learning?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a new task of constructing a 3D object by assembly of Lego-like primitive bricks, given only image/s as the target. They propose an RL agent that by incorporating Graph neural networks and action validation module is able to tackle the proposed task. They demonstrate the agent’s ability on three different sets of data.
main_review: The proposed problem is very interesting and hard due to its combinatorial properties, but I think it would be useful if the authors could elaborate more on the justification of why this is an interesting problem, how it could have a broader impact, and potential applications.

Some of the methods introduced in the paper such as the issue of dealing with variable and large action space are a great contribution beyond the use case in the current work.

I do not think there is much evidence that humans learn about objects by decomposing objects into very primitive elements. Rather the segmentation is done semantically. Would be interested to be pointed to literature that demonstrates otherwise.

Section 2. An illustration of brick in this section could help the reader better understand the text. For example in L66: an illustration of some of the combinations of two bricks could be very informative for the reader to visualize the combinatorial space. Same for the offset locations on a brick.

L115: Is x_t in a quantized space due to the nature of the bricks? 

What happens if the action validation network is wrong and the proposed action is invalid?

Most of the experimental comparisons are more like ablation of the B3. Would it be possible to adopt some of the prior work such as those listed in Table 1 for this task and show comparison results with them?
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a combinatorial construction task consisting of assembling unit primitives (i.e. LEGOs) sequentially. The desired target is specified using one or more 2D images. The main challenges in this setting are dealing with incomplete information (of the target) and long-term planning with a variable-sized action space containing invalid actions in a combinatorially large space. To address this, the authors adopt an action validation network that filters invalid actions to an actor-critic network. 
main_review: ## Strengths
Constructing structures from primitives is an important task for AI and RL. The use of a 2x4 brick as the unit primitive greatly simplifies the problem, but still allows for enough complexity to be interesting and relevant. 

## Weaknesses
The main technical novelty of the approach is the introduction of the action validation network, but lacks any ablation studies that demonstrate its effectiveness. 

The experimental results are also a bit underwhelming. In particular, it seems like the authors are training on single classes individually for each of the datasets (e.g. a single digit for MNIST). It is not clear why this is done, and would be very limiting if the training could not be done more generally. Additionally, besides the ablations, only one baseline is tested, a Bayesian optimization-based approach (BO). However, since BO requires exact volumetric information it would be helpful to include others that are more comparable. 

## Correctness
The methodology seems correct to the best of my knowledge.

## Clarity
Writing is clear.

## Relation to Prior Work
Prior work on assembly is discussed, but it is unclear why none were used as baselines in this work (see weaknesses).

## Additional Feedback

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposes a GNN+RL pipeline to use 2x4 LEGO bricks for reconstructing 3D shapes from observation of a single image or a set of images. The authors claim three main contributions: 1) the novel problem formulation of combinatorial construction; 2) the RL agent Brick-by-Brick (B^3) to solve the problem of growing action space and invalid actions in the sequential part assembly process; 3) implement an OpenAI gym environment, plus benchmark and evaluations.
main_review: **Originality:**

(positive) The paper has the technical contribution of proposing the novel problem of combinatorial construction and formulating a GNN+RL pipeline for addressing the task. Also, the authors make contributions at 1) designing the action space to make it not growing with the increasing number of bricks during assembly, by defining the action space as a tuple of <pivot brick, offset>; 2) training a neural network to predict if an action is valid or not, to reduce the growing checking time if not using the network.

(negative) However, I have some major concerns regarding the claimed contributions. Firstly, it is weird to claim "combinatorial construction" as the novel formulation proposed in this paper, as this is a general term and many previous works [19, 38] are also essentially solving the combinatorial part assembly problem. Secondly, I don't understand the necessity of training a network to predict invalid actions. If checking for O(|A_off|t^2) for an accurate answer is too computationally heavy, can you simply maintain a voxel-grid logging the brick occupancy and adding a new brick is simply a O(1)-time checking?

**Quality:**

(positive) The method design is reasonable, easy to read, and well-stated. The paper conducts a lot of experiments over different datasets. 

(negative) I have many concerns regarding the experiments: 1) it seems weird to do the MNIST experiment as MNIST images are 2D shapes. Why using 3D bricks for reconstruction is an interesting task? Also, it seems that you will remove one dimension and only use a 1x2 or 1x4 2D brick to doing a 2D reconstruction, right? 2) The qualitative result for the Table shape in Fig.  6 is pretty bad. Considering that this is the only result shown for Table in the main paper, is this representing your best result? Can you achieve better results than this? Actually, this may slightly exaggerate my concern why using 2x4 bricks as the only primitive is a good choice, given such bad reconstruction quality for real ModelNet shapes? 3) Can you compare to [19, 38] as these two published papers are so related to the task in this work? In my opinion, it should be not impossible to convert them to work with image conditions. Also since your method also uses 3D ground-truth to provide rewards in Eq. 1,  I think the task settings (input/output/supervision/goal) in this paper are quite similar to [19, 38]. I thus think that a fair comparison is necessary to prove the improvement of the proposed method. 4) In table 1, it's weird to put "AVN" as a term for comparison. Do previous works not consider invalid overlapping at all? I don't think this is a big point to claim difference. 

**Clarity:**

the paper is overall well-written and easy-to-read. However, the comparison to [19, 38] is not clearly described, in Sec 1/2/5. 

**Significance:**

overall, I think the claimed contributions are not strong to meet NeurIPS bar, or not well-proven to be necessary. Besides, necessary detailed discussions and experimental comparisons to significantly similar works [19, 38] are missing.


limitations_and_societal_impact: yes, the authors adequately addressed these issues.
review_title: null

Meta-review: The paper addresses a problem a construction a 3d object with lego, in the action space of locations of placement of a new block, purely from visual observation during test time and trained with reinforcement learning to match the volume of the target object.

The main positives of this paper are that this is an interesting setup/problem that has not been addressed before as far as we can tell (the lego construction yes, but from purely visual, trained with RL), they propose an interesting solution of graph neural network with action validation network and succeed at building basic shapes. 

The main drawback is that while they do succeed in making 3d shape objects (like an airplane), the quality of the constructed objects is not very high and the objects are not so complex (e.g. don’t have that many blocks). However, as this has not been done before, it’s hard to know what to expect - how difficult this problem really is. 

Because of this and because the setup is novel and this topic is quite uncommon and yet interesting, important and should be studied more, I think the paper should be accepted. 

For the authors I primarily recommend pushing the performance - getting larger, more diverse 3d shapes constructed of at higher level of quality.

There were also few things unclear in the discussion, I also recommend for the authors to explain them with better writing. 
Decision: Accept (Poster)