Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The main contributions of this paper could be concluded as following:
(1). Proposed a GNN-based VAE model named NeatNet to deal with rearrangement tasks, which can extract more latent information like the spatial information of input scene images
(2). Innovatively bring out the new form of rearrangement tasks: rearrange the given scene into personalized style with preference extracted from provided train samples
(3).Gathered preference samples from users and validate the effectiveness of proposed model with experiments

strengths_and_weaknesses: Strengths:
1.	Innovatively proposed the method that use Graph Neural Network layers in VAE, a traditional generative model which may allow the model directly learn from arrangement provided by users in format of graphs
2.	Designed a well-considered attention mechanism and update rules to make the model extract the most important preference in samples.
3.	Comprehensively explore the performance of propose model in different situations with conducting experiments, such as generate arrangements for known scenarios, fit a new kind of object in current scenario and generalize a learned preference to a new scenario.
Weakness:
1.	More realistic and complex scenarios need to be used to validate the effectiveness and generalizability of proposed model. For reality, images of some real scenarios with same arrangements should be concluded. For complexity, scenarios with larger views such as a whole scene of dining room instead of just the table.
2.	Baselines should be more convincing instead of only naive methods implemented in experiment.

summary_of_recommendation: I recommend this paper for its contributions on forming a innoative type of robot re-arrangement tasks and proposing the GNN-based  VAE model to deal with such kind of rearrangement tasks. The GNN-based model allow the generative model  directly learn from arrangement provided by users in format of images and extract the preference information from latent space. And the experiments validated its effectiveness with comparison to several naïve baselines. 
issues: 1. More realistic and complex samples should be included
2. A more complicate and effective baseline should be used in comparison to make the results more convincing
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper tackles the scene rearrangement task by proposing a novel architecture that learns low-level representations of user preferences in regards to tidying up a space. These representations can subsequently be used to generate object arrangements according to the specific user's spacial preferences and further generalize to include unseen objects through the use of word embeddings. Towards achieving the above, the paper also presents a simulator used for data collection through user interaction. Results receive positive evaluation by users in terms of matching their spatial preferences.
strengths_and_weaknesses: The paper was very well written. The structure and language significantly helped to follow and comprehend the work done. The combination of VAEs and GNNs was an interesting approach towards learning user preferences and scene representation. Furthermore, generalization to new objects is an important aspect of the problem and the paper did not leave this unaddressed. Combining images with word embeddings for objects in order to address the generalization, was creative.  

In terms of weaknesses, one general philosophical question stood out: Important spatial preferences such as left/right handedness are very few and represent a much more constrained space of a user's rationale behind the way they prefer their space organization. For instance, there are people that left/right handedness doesn't necessarily play a part in how they organize their space and/or they are not consistent in a particular location for objects in their space. Secondly, rating organized scenes by humans is not entirely convincing as a good measure of how well the approach works, i.e. due to the inherent subjectivity of user preferences, the same user could be giving different ratings on the same scene at different points in time. 

summary_of_recommendation: Overall, the paper presented an interesting line of work. The problem of user spatial preferences and their use for scene arrangement is inherently difficult due to its subjectivity. The work conducted employed a creative approach that yielded notable results. Along with the paper being well written and easy to follow, it made for a captivating read. In terms of impact, it's not entirely convincing that it can spawn novel directions in the community. However, it can potentially spawn very interesting conversations about the work itself as well as the philosophy of human preferences and thus, would make a solid addition to CoRL 2021. 
issues: Ln 15. "Given goal states stipulating where each object should go, many approaches exist for planning a sequence of actions to manipulate objects into the desired arrangement" was followed by only one reference. Could you please cite a few more approaches?

In the related work, an overview of overall existing work on scene rearrangement was expected.

Could the dataset described in https://arxiv.org/abs/2103.16544 be potentially used for further evaluation of the method (instead of or complementary to the developed simulator) as it provides a wider set of scenes? It is very recent work, therefore not expected to have been used in this paper but perhaps it could be useful for future work? 

In regards to rearrangement data collection from users, why did you have to build your own simulator? There are many simulators out there, for embodied AI and scene rearrangement. 

Ln 19. "Several factors involved in spatial preferences are universal": Please consider rephrasing. One of the two universality examples you give right after, happens to not apply to me. Also, especially since it is subsequently mentioned that "It is clear that spatial preferences are a complex, multi-faceted, and subjective concept" (ln 30). Subjectivity is quite contrary to universality. 

Ln 78. "secondly because our method applies prior knowledge from other users": why is this desired behavior? User preferences should be unique vectors in the latent space i.e. other users' preferences should not matter when tailoring a scene to a specific user?

Semantic embeddings section was a bit too brief. Unclear how the embeddings loaded from FastText are tied to the visual scenes and thus, allow for generalization to new objects. Are all objects in a scene labelled by the simulator somehow? Could you please add some more details?
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This work provides a method to enable a robot arrange/tidy objects according to a user’s preferences. NeatNet is a variational autoencoder that learns a representation of the user’s preferences and uses it to position objects in a scene. Input scenes are encoded using graph neural network layers.


strengths_and_weaknesses: The paper is clearly written and organized. The authors propose a novel scheme to learn user preferences of spatial object structures by combining graph neural network layers with a variational autoencoder architecture. The series of questions answered in the evaluation section seem comprehensive. 

However, the paper could be strengthened by adding baselines that also model the user’s preferences. Currently, none of the baselines model user preferences which provides for an unfair comparison. For instance, in Tables 1 and 4, it’s not surprising that the rest of the baselines (Mean-Positions, NetNet-No-Prefs, and Pose-Graphs, Mean-Position, Random-User) would do poorly. Adding baselines that take into account user preferences would also shed light on why using GNN layers + VAE the way the paper did is the best way to learn the user’s spatial preferences rather than doing something like inverse reinforcement learning. 

I am also curious why the Nearest Neighbor baseline performs so poorly in Table 3. Do the Nearest Neighbor algorithm and NeatNet use the same word embeddings? Currently, it seems like the Nearest Neighbor algorithm places the computer where the desktop was because “desktop” is the most similar word. If so, wouldn’t expanding k, the # of nearest neighbors, help alleviate this problem? 

Providing some examples of distinct user preferences in the environments (especially dining and office scenes) would be helpful. Since there are established norms in the non-abstract environments (e.g., keyboard and mouse go in front of the monitor, plate goes in between fork and knife), I’m curious to see how varied these preferences can get. For instance, is the main difference in preferences just left vs right-handedness? Adding some statistics on the variance of user preferences in each of these scenes would help the reader get a better sense of the difficulty of the problem.

Including some visual examples for each of the experiments (especially Table 4) would be great :-)

Please report standard deviation for all means in all the tables. 

summary_of_recommendation: I vote for a weak accept conditioned on the fact that the authors are able to provide stronger baselines in their experiments. 

EDIT: Thank you for your reply. I have read it and am raising my score. 
issues: - Addition of stronger baselines (see comments above)
- Providing more examples of both experimental results as well as different user preferences. 
- Including standard deviation for all means in all tables.
review_title: null

Meta-review: In this paper, the authors propose the novel NeatNet considering user preference for robotic rearrangement task. The variational autoencoder is used to learn a low-dimensional preference vector and the scenes are modeled with GNN. All reviewers recognized that the paper is well written and organized. It will make the paper more convincing if some stronger baselines, for example, considering user preference, are provided. Also, more qualitative results are expected to make the experimental results more illustrative.

In the rebuttal session, the authors have responded to the reviewers' concerns and more details are provided. And the reviewers have reached a consensus of accepting this paper.
Decision: Accept (Poster)