Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work proposes a pipeline to train a robot to learn to assemble blocks to match the given target shape. A graph neural network is utilized to encode the state information, and a deep reinforcement learning algorithm is applied to maximize the reward. The authors conduct extensive experiments to demonstrate the effectiveness of the proposed approach.
strengths_and_weaknesses: Strengths
1) This work proposes to leverage a graph neural network to extract the state information used to learn an assembly policy in a reinforcement learning fashion.
2) Extensive evaluation experiments are conducted to demonstrate the effectiveness of the proposed approach.

Weaknesses:
1) The technical contributions are limited. Using graph neural networks to represent the state and deal with the combinatorial problem in the assembly process has been investigated by several works. For example, "Generative 3D Part Assembly via Dynamic Graph Learning. Huang et al. Neurips 2022."
2) Some important descriptions are missing. For example, it's not clear why a set of points is sufficient to represent the target shape. How the topology information is encoded in the set of points?  
3)  The current setting is too simple. It's not clear what's the performance if more complicated objects are used. Whether the proposed pipeline could deal with more complicated objects? How to define the action space since there are more grasping poses? 
 

summary_of_recommendation: This work proposes a pipeline to train a robot to learn to assemble blocks to match the given target shape leveraging the graph neural network to encode the state. However, the technical novelty is limited and the current setting is too simple. It's unclear whether the proposed pipeline could work with complicated objects.
issues: 1) in Line 102, "desired shape that needs to be assembled consists of a set of points parameterizing this target shape". It's unclear how the target shape is parameterized by the set of points. 
2) In Line 111, "we have to decide over Na = 2 × 2 × 5 = 20 actions per block." It's difficult for the reviewer to understand these 20 actions per block. It would strengthen the paper if a visualization image is introduced here.
3) In Line115-115, what is the function of F in equation (1). 
4) In Line 118-119, "a threshold ∆ is exceeded, or whenever there are no more unplaced blocks available, or after an invalid action. " What is an invalid action?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper addresses the problem of autonomous assembly. Goals are defined by a set of 3D points forming a polygon shape that might be 2D or 3D. The robot's actions then consist of picking and placing blocks to construct the structure defined by the shape formed from the 3D points. The task notably has challenging characteristics like combinatorial explosion from a growing number of action, as well as others. A GNN representation with multi-headed attention mechanisms is proposed to represent the state of the constructed blocks and several RL methods are explored to perform model-based and model-free training. The work compares to the prior work structure2vector in one experiment, showing that the proposed architecture has more representational power essential to abstract robot assembly.
strengths_and_weaknesses: Strengths:
-The paper is clear, concise, and well written. 
-The benchmarking environment presents another interesting problems for researchers.
-The approach presents an interesting formulation of techniques to improve robot assembly from scratch

Weakness:
-The paper differentiated itself often in the related work that had a predefined goal in the form of a demonstration, but also has a predefined goal in the form of the 3D points. Perhaps some emphasis should be place on the abstraction level of the goal.

Side comments:
-Interesting future direction I thought would define the goals in natural language.
summary_of_recommendation: I am unfamiliar with existing works on the robot structure assembly problem although I am very familiar with GNNs that include multi-headed attention mechanisms. As a result, I am assuming the main contributions applying the GNN with multi-headed attention mechanism to the robot assembly problem, robot assembly benchmark, and different combinations of learning algorithms for the problem. If other reviewers had strong opinions about missing baselines or prior work I would be open to those discussions. This is the reason for weak accept instead of strong.
issues: Would it be possible to write Section 2.2 to rely less on existing algorithms from prior work? This might help readers that are unfamiliar with those works.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a data-driven method for robotic assembly by leveraging structured state representations in conjunction with learning-based algorithms to effectively solve tasks. Specifically, the state representation is based on an graph neural network that incorporates attention mechanisms, and the learning algorithm is a hybrid model-based and model-free reinforcement learning algorithm. Experiments on a simulated assembly environment demonstrate that the proposed method can construct a variety of structures, outperforming variants with alternative state representations and learning algorithms.
strengths_and_weaknesses: Strengths:
* The method is explained in a thorough yet concise manner
* The experiments perform a systematic study of the state representation and learning algorithm 
* The experimental analysis is generally thorough: when there is a positive or negative result, the paper accompanies that result with a discussion of the underlying reasons behind the result

Weaknesses:
* The environment seems a bit simple and artificial —  the blocks are of the same shape and size and the unplaced blocks are laid out in a regularly spaced manner. Intuitively, it seems that the robot can choose to pick up any of the unplaced blocks, as they all have the same functionality. This makes the problem seem artificially simple. Furthermore, the target configurations for the blocks seem quite arbitrary, how do they translate to real-world structures that are of interest in assembly/construction applications? 
* The learning algorithms adopt elements from MCTS, a model-based algorithm. This raises the question, where does the underlying model come from? If making model predictions is based on having access to a simulator rather than a learned model, then this approach is unlikely to scale up due to the expensive computational load of simulators
* Experiments can still go deeper, ideally answering the following questions: (1) how much of a difference does multi-head attention make compared to single-head attention? (2) what forms of generalization can the proposed method achieves and what forms can it not achieve? (3) are the representations learned by the proposed framework useful for transfer to semantically related tasks?
summary_of_recommendation: This paper is well-written, clear, and the experimental analysis is generally thorough. Ultimately, my primary concern is with this paper’s overall motivation and message. The paper positions itself as an applications-focused work, seeking to advance robotic automation for assembly and construction. Yet the experimental domain does not appear to be a convincing or inspiring medium through which to advance the state-of-the-art on these applications. On the other hand, the paper does not seek to make broader points about how its findings are useful to the general robot learning community. As a result, the impact of this paper (in its current form) appears limited. I recommend either revising the message of the paper to be more universal, or revising the experiments to demonstrate more impressive results in assembly / construction.

**Update after rebuttal** my recommendation stays the same. Please refer to my followup post for a detailed explanation.
issues: See the weaknesses list. In addition:
* Many of the experiments are conducted without the robot — what is the motivation behind this? Perhaps the reason is that introducing the robot may introduce confounding factors in the analysis, in which case the paper should clearly justify why this may be the case     
* Numerous citations to GNN-based works are missing, including but not limited to (1) Relational deep reinforcement learning, by Zambaldi et al, (2) Relational inductive bias for physical construction in humans and machines, by Hamrick et al, (3) Contrastive learning of structured world models, by Kipf et al
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper studies the problem of 3D assembly, more specifically having a 7 DOF robot arm stack blocks into a target shape. The paper aims to maximize combinatorial generalization for this task, by studying different choices of graph representations (through the use of GNNs and multi head attention) and monte carlo tree search algorithms (combined with Q-learning). The experiments show that a partially connected architecture for graph representation learning is superior than the fully connected version, and this is then tested in conjunction with DQN+MCTS algorithm for the best results.
strengths_and_weaknesses: Strengths:
- The flow of the paper is great, starting with a clear goal in mind, proposing very reasonable choices for each component of the pipeline with just the right amount of explanation, and testing the different components (representation and algorithm) separately.
- The experimental section is thorough and clear. I really appreciate the targeted experimentation on each of the components, as it makes the results more reproducible and scientifically valuable for future work. The design of the novel benchmarking environment is also a contribution.

Weaknesses:
- It may be helpful to include some visualizations of the DQN+MCTS search procedure for a small toy example (e.g. with possible future states/actions and their estimated Q-functions). The toy example can also help better distinguish the different search methods described in section 2.2.
summary_of_recommendation: The problem of 3D assembly is obviously important, and the paper provides valuable contributions in terms of empirical evaluation of graph representation learning and Q-learning / MTCS applied to this task. The presentation is clear and the experiments are thorough and provides potentially valuable insights for future work.
issues: See above.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a method to learn structure assembly policies via RL where using Graph Neural Networks. The input to the network is a graph representation of the scene, where each node represents an object (rectangular building block). The output of the network is the Q-value for different actions. The task is specified by stacking blocks to fill up a target volume. The authors evaluate a variety of RL algorithms in simulation and achieve a success rate of 85% in the best case and 49% in the hardest case (harder cases have more blocks and larger structures).
strengths_and_weaknesses: Strengths:
-	The proposed method of incorporating graph-based inductive bias in the construction task is suitable.
-	The experiments and results are thorough.
-	The analyses provide insights on comparing various graph-based representations, like multiheaded-attention and structure-to-vector.
Weaknesses
-	My main criticism of the paper is that it is poorly written, or at least, difficult to parse. The wording is dense both from a language and visual perspective. There is some key information I was unable to easily find in the paper. I list them as the following questions.
-	Is the task (target build volume) encoded in the observations, or do we need to train a new Q function for a new target? 
-	One of the main benefits of GNNs is its ability to generalize across different numbers of nodes and edges. In the case of this paper, can the GNNs trained on small number of objects generalize to bigger ones?
-	Is it the case that the current action encoding limits newly placed blocks can only be placed next to previously placed blocks? If so, it is unclear how the action encoding can be made more flexible to avoid this limitation when needing to build structures that have hollow volumes.
-	The task volume visualization is also difficult to understand, both in the paper and in the videos. I think it may be clearer to represent target structures with semi-transparent volumes rather than outlines. It is also unclear how a set of rectangular blocks is supposed to construct some of the visualized structures that contain slanted sides.
-	The improvement in performance when training with robot-in-the-loop may come at a cost of overfitting to the robot’s geometry and kinematics. How well would this transfer to new robots or even the same robot with a different end-effector?
-	It would be helpful to see an additional baseline comparison against using MCTS with a simple hardcoded heuristic (value) function. This will illustrate the additional information learned by DQN.
-	While many experiments were performed, it is difficult to understand their significance and what solid conclusions should be drawn from them. Clearer discussions on this would have been helpful.

summary_of_recommendation: My recommendation of weak reject is primarily due to the lack of clarity in the paper writing, which made it difficult for me to clearly assess the paper’s technical merits and contributions.

**Update** I have updated my recommendation to weak accept after the authors' revisions.
issues: Please see the Strengths and Weaknesses section.
review_title: null

Meta-review: # Strengths 
- The novelty of leveraging a graph neural network with multi-headed attention mechanism in manipulation assembly setting was noted by multiple reviewers 
- Extensive evaluation experiments to demonstrate the effectiveness of the proposed approach.

# Weaknesses 
- There are mixed reviews about writing clarity
- Multiple reviewers raised concerns about the simplicity of the task settings and connections to real world problems.
- Multiple reviewers noted that the technical contributions are limited
- Multiple reviewers highlight missing experimental and technical details.


# Recommendations
- Please outline limitations of the work 
- Thoroughly go through the recommendations provided by each reviewers to revise the paper
- This is a borderline paper with mixed reviews, please converse and address reviewers concerns

# Decision summary
- Extensive clarifications and additional experiments were received from the authors during the rebuttal period and were considered by the reviewers before final recommendation was made.
- Authors clarifications were acknowledged and factored in. Some concern around the experiments still remains.

We hope that the feedback received will help with the future directions of the work under submission. Thank you for participating in CoRL

Decision: Accept (Poster)