Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work introduces a novel approach to tackling object rearrangement. The main idea is to learn a scoring function in the form of a gradient field. The learned gradient field could be used by a model-based planning algorithm to find collision-free arrangement plans or as a reward function to train RL agents. Experiment results show promising results of the proposed method for rearranging a room and sample efficient policy learning.
strengths_and_weaknesses: Strength:
1. The paper is well-written with a clear introduction of the implementation details.
2. I really enjoy reading the paper following the motivation of how the learned gradient field could be used to tackle three major challenges of object arrangement, which is easy to follow and makes a lot of sense.
3. The velocity-based action space of the designed task showcases that the proposed method could be used in complex continuous control scenarios. The discussion of collision avoidance planning with ORCA further strengthens the practicality of the proposed approach in real-world tasks.
4. It is impressive to see such a gradient-based scoring function could be used as a reward function for training the RL agent. Also, the gradient-based action could further alleviate the burden of the action generation network so that only a small residual policy network needs to be trained, which largely improves the sample efficiency of the policy learning process. 

Concerns:
1. How does the RL residual policy learn to avoid collision? As mentioned in the model-based planning method, the gradient field does not have environment information so the gradient cannot avoid collision. Since the residual policy is based on gradient-based action, does it need an additional collision penalty reward during training? In the paper, you mentioned the centralized and decentralized reward but more details here would be better.
2. The gradient field might struggle to handle multi-model distributions. What if there are multiple arrangement goals for the same environment. Will the gradient become a mean of different distributions which will mislead the planner or the policy learning? I feel a probabilistic model would make more sense for this situation.
3. The authors mentioned that the room-arrangement task is too complex for planning-based methods. I would suggest trying with bi-level planning which first plans high-level arrangements plans with the learned gradient and then use a collision-avoidance trajectory planner to optimize the path to avoid the collision.

questions: 1. When talking about collision-free room arrangement, do we consider the 2d rotation of the furniture during the movements? Basically, what’s the action space, and why it is hard compared to the ball arrangement?
2. Did you test the generalization of the proposed method? e.g. Testing in an unseen room with novel furniture.
limitations: 1. Might fail to handle multi-model goal distributions.
2. Missing some details of the experimental setups in the main paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: ## Post Rebuttal

I thank the authors for their response. I have increased my score.

## Pre Rebuttal

This paper attempts to tackle the problem of learning to rearrange a set of objects to a "sensible" state without a pre-defined reward function. Towards this end, they propose Target Gradient Fields (TarGF). TarGF estimates the gradient of the likelihood that the current environment state is a goal (or "sensible") state for the objects.

TarGF is learned by randomly permuting correct arrangements.

The output of TarGF is used to estimate the best action to take at every timestep and/or used to estimate the reward.

The method evaluated in two scenarios. In the first a set of balls must be rearranged into either a circle, clusters by color, or a combination. In the second the furniture in rooms must be rearranged back to its initial states.
strengths_and_weaknesses: # Strengths
The proposed method is able to learn solely from examples and does not require any additional data other than correct examples (negatives are generated by adding noise to the correct examples).

In the circle scenario, the proposed method outperforms various baselines across a series of metrics -- namely, collision count and success.

# Weaknesses

The evaluation is very simple. There isn't an agent that rearranges these objects, instead the objects all rearrange themselves. This is a good initial evaluation that shows the method provides a reasonable reward signal, but it leaves a lot of questions since this reduces the time-horizon, isn't representative of how the objects would be moved by a single agent (one object moves at a time), and removes the initial exploration to find the objects. An evaluation that is closer to a real scenario would be helpful.

Ideally, the authors would evaluate on something like HAB (Szot et al 2021), or AI2Thor rearrangement.

The residual policy derived from the gradient-based action seems quite important to overall performance of the model free approach. However, this residual policy requires an oracle entity-based representation of the state -- requires all object positions, object categories, and bounding boxes. While it is fine to assume this at training time, this is also required at evaluation time. I am concerned by the assumption that such a representation is available as the task complexity increases.

On this topic, it is unclear if the baselines receive the same information. The supplement says they operate on state, but state is defined just as object position, not position + category + bounding box, in the paper.

Finally, in the most realistic scenario, Room Arrangement, the proposed method likely doesn't outperform GAIL + SAC by a statistically significant margin. Can the authors report confidence intervals instead of variance?
questions: Is (1) a well-defined optimization problem? For any non-trivial starting state, won't p_tar(s_0) always be negative infinity, so the discounted sum will always be negative infinity

Comments:
Rearrangement should be used instead of arrangement to be inline with what the community is calling this task. See Batra et. al. 2020.
limitations: Limitations are addressed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper learns a reward function / score function for objective rearrangement tasks, making use of recent advancements in score-based generative modeling. The authors term this score function a “target gradient field”, and show that it can be used with either a path planner or a reinforcement learning algorithm to solve object rearrangement tasks in a toy setting (i.e. where you can directly control object velocities). The authors compare against a number of baselines and competing methods in these simulated object rearrangement domains, and show that their method performs well across several different metrics. 

strengths_and_weaknesses: Strengths
- Learning reward functions is an interesting application of score-based generative modeling, and has not been explored before, as far as I know. 
- Proposed approach performs well against other reward learning approaches on the object-rearrangement task. 

Weaknesses
- The paper is somewhat limited in scope -- it is only applied to a very specific robotics problem (that of object rearrangement), and even here some major simplifying assumptions had to be made (such as the fact you can directly control the velocity of any object. 
- The paper only shows results in low-dimensional domains (small graphs). Since score-based generative modeling also works in high-dimensional domains (such as images), it would be interesting to see if the method can be used for reward learning from scene images, for example. 
- While the paper is nicely structured, it contains a large number of typos and grammatical errors. These errors must be fixed before the paper is ready for publication. I will provide a non-exhaustive list here, but I would also suggest making useful of professional proof-reading services if possible. 

Typos and grammatical errors:
- “We do also tries to find a example-based control method”
- “Differently, our proposed approach focus on learning the target gradient fields from the examples.”
- “Besides, both RL method and traditional planner can be supported by our target gradient fields in the object arrangement task.”
- “even if we are accessible to the target distribution”
- “In address these problems”
- "reducing the risk of objects being collided”

questions: Some questions

- “learning a score function to qualify a state” → what does “qualify” mean here? Can this sentence be re-written to make it less ambiguous? 
- How do you ensure collision avoidance when using model-free RL? Or is collision avoidance not pursued in this setting? 

limitations: I don't think the authors have done a good job of addressing the limitation of the work. In the limitations section, the only limitation that the authors mentioned is the fact that their work only considered planar settings, and not 3D settings. There is only a very brief mention of the fact that future work could use real robots. I think the authors should provide a more detailed limitations section (by reducing space used elsewhere). Some ideas for limitations can be seen in the weaknesses section above. A somewhat more comprehensive plan on how the method could be applied to more realistic settings would also be useful.
review_title: null

Meta-review: metareview: After a strong rebuttal from the authors and an extensive discussion among the reviewers, I believe this work will be a valuable contribution to NeurIPS. I recommend it for acceptance and encourage the authors to address the reviewers comments for the camera-ready version of the paper, especially the point about the simplistic evaluation of the method - please consider a more realistic evaluation scenario.
confidence: Certain
Decision: Accept