Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Authors use energy based models (EBMs) and object graph minimization to perform instruction-guided spatial rearrangement. Authors use EBMs to represent each spatial predicate (binary or n-ary). Objects are represented using the 3D or 2D overhead box coordinates. Authors train the EBMs using gradient descent on the sum of energies with respect to the coordinate space, providing the final configurations that best satisfies the instructions. 
Using this, essentially, the model reason/plan over a pixel-abstract, but bounding-box aware space to interpret the instruction, predicting a goal state. Using a low-level and pick-and-place policy execution, the objects are finally shifted to the goal position. 

Authors make following empirical claims:
1. The model outperforms the baselines, especially in complicated instructions
2. Generalization over  1) novel compositions,  2) novel instructions, and 2) novel objects and attributes.
strength_and_weaknesses: The idea about moving from spatial reasoning in language space to space of object 2D 3D coordinates and representing predicates though EBMs are quite interesting. I feel there are some challenges to scale the work in real-world settings. Also, a few aspects of the generalization claims seem unsupported through experiments. 

A few concerns:
1. I am wondering is the set of binary and n-ary spatial concepts open? You at least need the knowledge of which is a binary vs n-ary predicate? How do you infer that in care you are not using a list of predicates? I see you are training one EBM per predicate, which means the predicate list should be fixed right?

2. In general, works such as Palm-Saycan [1] have also showed at least one experiment on real environments, such as kitchen environment, moving from table-top ones. As far as I remember, even LLMPlanner [2] uses a real-world table-top environment. I think, showing the work in such setting would help; especially given questions such as the first one.

3. Generalization claim: Also, I do not see experiments performance over novel objects are shown. Among novel attributes, do you only mean shape and color? Is there anything else considered?

4. LLMPlanner such as PaLM-saycan uses success predictors to give some feedback whether goal state has been reached. 
Though such goal state detection through energy minimizer is discussed, I do not see whether such a feedback mechanism exist? Some part has been discussed as physics based issues in Limitations. However, this is important, as the model only outputs final goal configuration; implying many goal configurations may be hard to achieve through pick-and-place policies in reality. What is the effect of this?

[1] Do As I Can, Not As I Say: Grounding Language in Robotic Affordances, Ahn et al. 2022

[2] Inner Monologue: Embodied Reasoning through Planning with Language Models
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper reads well, barring some minor comments, detailed below.

Quality & Novelty: I have discussed the strengths and weaknesses above. I think the basic idea is definitely novel. But, it may not be easy to extend to real-world scenarios. 

Reproducibility: Authors have shared the code in supplementary, and is willing to share the code upon publication.

Minor Comments:
1. Page 5: "For detailed …" It is slightly unclear to me, whether there is any other complexities involved while minimizing the sum of energy functions. It is better to explain at least in Appendix rather than sending the readers directly to the code.

2. Page 5: "Referential …" Parts of the implementation and method description is sometimes intertwined. Might be helpful to summarize the whole process as an algorithm, provided the modular nature of things.

3. Page 6: "scene rearrangement …" Its a bit out of the blue. How do you create this benchmark? You can describe it in short at the least.

4. In Table 1, what is meant by "Task progress"? 

summary_of_the_review: The idea of representing spatial predicates as EBMs is novel; alongwith reasoning over the 2D or 3D object coordinate space for instruction-guided spatial rearrangement. I also agree that language space may not be the best option to reason spatially about images. However, some experiments are lacking to support the generalization claims about generalizing to novel objects and attributes, alongwith experiments in real-world setting. Training each EBM to correspond to a 2-ary or n-ary predicate also raises the question of how many spatial relations that this model can support. Also, given the goal configuration is output and needs to be executed by a low-level policy executor, its easy to think of scenarios where the executor may not be able solve this using a greedy policy (stacks, tower of hanoi, inadvertently creating obstacles etc.). So, I am borderline on this. I feel some more experimentation is required, some of which may be non-trivial.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper formulates the problem of spatial arrangement of objects as minimizing the energy function over the configurations of the object graph. The graph is constructed by parsing the language instruction and utilizing visual language grounding. The graph energy function is composed of predicates parameterized as neural networks. The model is trained using contrastive divergence with Langevin dynamics. Finally real object arrangement is done by a Transporter Network. Empirical results show that this formulation exhibits zero-shot generalization capability and works better than existing methods on the tasks evaluated.
strength_and_weaknesses: Strength
=======
- A nice formulation of the spatial arrangement as object graph energy minimization. Although this treatment has been studied for decades, it is still interesting in the context of language instruction for robotics. The setting also requires a minimal amount of training data, as most heavy lifting parts (linguistic, visual and robotic) are handled by existing systems.
- The empirical results are promising.

Weaknesses
==========
- The limitations discussed in the paper (temporal constraints, location parameterization, simple predicates) are important to address, even though energy minimization can be theoretically suitable for solving these problems.
- There are many moving parts in the overall system, it is important to see how robust the system is if the parts are not too reliable (e.g., semantic parsing, object grounding, robotic failure).
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, and the concept of object graph optimization is useful under the setting of language to action. The work is reproducible. A more focused contribution, in depth analysis of the core idea, and more sophisticated experimental settings would make the paper more interesting to read.
summary_of_the_review: The paper introduces a way to treat of spatial arrangement as object graph energy minimization under the language to action context. The formulation itself isn't new, but the overall solution is interesting and applicable in certain settings. The are important limitations, however.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an optimization scheme to arrange a set of objects so they conform to a set of spatial constraints, that are either binary directional relationships, or specific spatial arrangements e.g. line or circle. An energy based model (EBM) is learned for each of the constraint type, which maps a set of obvious bounding boxes and locations to score that indicates how well the constraint is satisfied. These models are learned by enforcing that the ground truth layouts scores higher than perturbed layouts. At test time, a gradient-descent scheme is used to optimize the position of the objects, with the guide of the learned EBMs. The paper also incorporates off-the-shelf methods that maps language to constraints, and uses the optimized layout to guide a robotic agent. However, I am not sure if either of these relates to the central contribution of the paper. Evaluation show that the proposed method performs better than general purpose embodied agents on a set of tasks that seem to be tailored towards the proposed method. 
strength_and_weaknesses: #Strength
- Good empirical performance on the set of tasks evaluated. Although these tasks are rather biased towards the proposed method, it is still nice to see.
- Optimization guided by learnable priors is a good idea and should be fairly generalizable. The ability to learn these priors with a simple MLP over object attributes is also nice to know.

#Weaknesses
### Novelty
- Constraint-guided layout optimization is a long studied problem, see earlier works such as "Make it home: automatic optimization of furniture arrangement", "Interactive Furniture Layout Using Interior Design Guidelines" for optimization, and works such as "Example-based Synthesis of 3D Object Arrangements" for learnable spatial priors. In some sense, the types of constraints dealt with in these works are much more complex than those in this work.
- While learnable spatial priors are nice, the set of relations dealt with these work is rather simple, so it is unclear whether the techniques introduced here can apply to more complex scenarios (many work suggests that neural networks struggle to perform complex spatial reasoning, especially if based on axis-aligned bounding boxes).
- It is not clear to me what is the significance of the language to constraint module and the robotic planning module. Seems that the work can exist without either of these and neither adds much to the contribution of this work.

### Design Choices
- I am not sure whether the proposed EBMs can generalize to more complex spatial relations, especially if semantics are involved. Relations such as "to the left of" are rathe easy to learn and I am not seeing examples of more complex cases e.g. circle. It seems that the benefit of a learnable prior really starts to emerge for more complex cases, whereas the type of constraints studied in this paper can be easily hardcoded (in a way that allows a gradient-descent based optimization scheme).
- I am concerned about the optimization landscape formed by the union of all EBMs. It seems that there can easily be scenarios where the optimization get stuck in a local minimum (e.g. object A starts above B, needs to move below B, but additional constraints enforce that A cannot appear to either the left or the right of B). There is a reason why many of these works I mentioned above resorts to methods more complex than simple hill climbing e.g. MCMC with large jumps, simulated annealing, etc.
- It seems that the method will struggle in cases where not all constraints can be satisfied. Since the learned functions lacks a consistent magnitude, it can be tricky to balance the weights of different constraint and ensure that the most number of constraints can be satisfied. 

### Evaluation
- Would be really helpful to visualize the learned energy function. Since the constraints studied here is rather simple, the models really don't need to learn much to lead to good results. A visualization showing a smooth function that consistently satisfies the constraints can increase my confidence in the proposed method.
- A baseline with hardcoded rules is needed - the method won't be of much use if it performs much worse than such a baseline.
clarity,_quality,_novelty_and_reproducibility: ### Clarity
More detail should be provided for the core contribution of the paper (the EBM-based optimization scheme).

### Quality
OK. See weaknesses above for my major concerns.

### Novelty
Optimization-based scene rearrangement is not new, there has been decades of research on this topic (see weaknesses above). Learnable EBM seems novel to me, at least with respect to the specific task, but whether they are useful is questionable.

### Reproducibility
Code is provided, looks reasonable, but probably need a bit of cleanup and a README file.
summary_of_the_review: I don't know much about robotics, but it appears to me that the core of the method is the learned-prior-guided layout optimization. This works without visual language grounding or the transporter network. I have many concerns with this optimization scheme, especially if it can generalize to more complex scenarios (see weaknesses above). As a result, while I think this method can be potentially useful, I lean towards rejection.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
This paper introduces an energy based layout model to perform spatial rearrangement actions conditioned on language, which can then be combined with a pick and place low-level policy in robotics tasks. The reviewers generally appreciated the idea of using energy based models in this context and to go from language to reasoning in spatial (bounding box) space. 

The paper has received three expert reviews and was a tough nut to crack. The reviews were initially mixed and slightly unfavorable, and the paper actually stayed in the borderline area throughout the full evaluation period. It has been estimated to lie on the decision boundary by the reviewers and the AC and good arguments have been made for its rejection. In particular, several issues were raised:

- Positioning of the paper with respect to literature on layout models, in particular since existing work is more complex (in terms of constraints) compared to the submitted paper. This was the one remark where the authors' response is considered as hand-wavy.
- Missing baselines in comparisons
- Generalization capabilities 
- Writing of the paper. This paper spends 4 pages on introduction and descriptions of related work and only a single page on the description of the method, which boils down to references of existing papers and a single equation. Details of the model are not given, the paper refers to the appendix, but even in the appendix details are omitted. This was considered as a major problem, as this paper is very far from being reproducible in its current form.

On the upside, the authors responded to (most) of the reviewers' comments and did a large number of new experiments, requested by the reviewers. While not all of them were considered convincing, they did address many of the raised points.

This paper has been extensively discussed in a meeting involving the AC and all three reviewers. No clear commitment has been made by the reviewers, who agreed that the paper had merit and was interesting to the community but also suffered from unfortunate choices made in presentation and evaluation. All reviewers agreed that there were shortcomings in evaluation, which had the potential to skew the perception of the problem and the field by a potential reader, but no consensus was reached on whether this was sufficient grounds for rejection.

The AC then discussed this paper with the SAC, and they considered all available information. Their message to the authors is very encouraging, and they thank them for their hard work during the rebuttal phase, providing additional experiments, which in the long run will be an important contribution to the paper. However, they also decided that in its current state the paper is not yet ready for publication. The points weighting most in this decision were the lack of positioning and evaluation (noted by all reviewers), but also the lack of reproducability. As it is written, the paper simply does not provide enough information to the reader on the model and its workings.
justification_for_why_not_higher_score: This paper was on the decision frontier accept/reject.
It could have been accepted
justification_for_why_not_lower_score: No lower score possible
Decision: Reject