My House, My Rules: Learning Tidying Preferences with Graph Neural Networks
Authors: Ivan Kapelyukh
Edward Johns
Venue: CoRL2021 Poster (2021)
Keywords: graph neural networks
preference learning
rearrangement tasks

Abstract: Robots that arrange household objects should do so according to the user's preferences, which are inherently subjective and difficult to model. We present NeatNet: a novel Variational Autoencoder architecture using Graph Neural Network layers, which can extract a low-dimensional latent preference vector from a user by observing how they arrange scenes. Given any set of objects, this vector can then be used to generate an arrangement which is tailored to that user's spatial preferences, with word embeddings used for generalisation to new objects. We develop a tidying simulator to gather rearrangement examples from 75 users, and demonstrate empirically that our method consistently produces neat and personalised arrangements across a variety of rearrangement scenarios.

My House, My Rules: Learning Tidying Preferences with Graph Neural Networks
Ivan Kapelyukh The Robot Learning Lab Imperial College London ikb17@imperial.. ac. uk
Edward Johns The Robot Learning Lab Imperial College London e johns@imperial.ar ac. uk
Robots that arrange household objects should do so according to the user's preferences, which are inherently subjective and difficult to model. We present NeatNet: novel Variational Autoencoder architecture using Graph Neu- ral Network layers, which can extract low-dimensional latent preference vector from a user by observing how they arrange scenes. Given any set of objects, this vector can then be used to generate an arrangement which is tailored to that user's spatial preferences, with word embeddings used for generalisation to new objects. We develop a tidying simulator to gather rearrangement examples from 75 users, and demonstrate empirically that our method consistently produces neat and per- sonalised arrangements across a variety of rearrangement scenarios.
Abstract:
Keywords: graph neural networks, preference learning, rearrangement tasks
Introduction
Rearranging objects in unstructured environments fundamental task in roboties. For example, a domestic robot could be required to set a dinner table, tidy a messy desk, and find a home for a newly-bought object. Given goal states stipulating where each object should go, many approaches exist for planning sequence of actions to manipulate objects into the desired arrangement [1, 2, 3].
But how does robot know what these goal states should be, for a rearrangement to be considered successful, tidy, and resthetically pleasing? The ability of a robot to understand human preferences, especially regarding the handling of personal objects, is a key priority for users [4]. Several factors involved in spatial preferences are shared across many users: symmetry and usefulness are preferable to chaos. Stacking plates neatly inside cupboard rather than scattering them randomly across sofa common sense. However, many of the factors involved are inherently personal, as is their relative prioritisation. For example, is the person left or right- handed? Do they want their favourite book tidied away neatly on shelf, or placed on the table nearby for convenience? How high is their risk tolerance do they deliberately keep fragile glasses further from the edge of the shelf, even if makes them harder to reach? Do they order food in the fridge with the tallest objects at the back and the shortest the front? Or do they keep the most frequently used items the front and easily reachable? In this case, how does the robot know which items are most frequently used and ought be placed near the front? This in itself depends on personal preference.
is clear that spatial preferences are complex, multi-faceted, and subjective concept. In this paper, propose object rearrangement strategy which models these preferences with latent vec- tor representation, inferred by directly observing how a user organises their home environment. The network is trained on arrangements made by set of training users. For test user, the robot will ob- serve how they have arranged their home. This can be achieved by a robot using existing techniques for localisation, object identification and pose estimation [5, 7]. By passing these example scenes through the Variational Autoencoder, the network will extract this test user's 's preference vector. This can then be used by a robot to tidy up these scenes, and also predict personalised arrangements for new objects and new scenes based on how training users with "similar" preferences to this test user arranged them. These predicted arrangements can then be used as target states for planning and manipulation algorithms [1]-
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

scenes arranged user
tidy known scene
encoder
place new item
user preference vector
arrange
Figure 1: NeatNet infers a user preference vector, and can then predict personalised arrangements.
We make the following contributions, adding a layer of personalisation to robotic rearrangement:
Novel Graph VAE Architecture. We present NeatNet: architecture for learning spatial pref- erences. Users are encoded through representation learning with Variational Autoencoders. Graph Neural Network (GNN) layers are used to learn from scenes arranged by users.
Word Embeddings for Object Generalisation. Our network extracts semantic features for an ob- ject from the word embedding of its name, enabling generalisation to objects unseen during training.
User-Driven Evaluation. We develop TidySim: a 3D simulator web app for users to arrange scenes. With rearrangement data from 75 users, we demonstrate empirically that our method can learn spa- tial preferences, and predict personalised arrangements which users rate highly. Visualisations of predicted arrangements are available in the supplementary material and on our project page, to- gether with our code and a summary video: www.robot-leariing uk/mmy-house-my-uunn
2 Related Work
A crucial challenge in the field of robotic rearrangement is predicting position for each object in order to form a tidy arrangement. Prior approaches discretise the problem by semantically grouping objects into containers [8, 91. learn Dirichlet Process models of common object locations [10, 11]. or optimise a cost function based on one user's 's examples [12]. We now discuss these three approaches, and then briefly reference existing neural methods for learning non-spatial user preferences.
Grouping semantically related objects. Collaborative filtering [8] can predict how user would categorise an object based on how similar users categorised it. Data is mined from object taxonomies such as shopping websites to generalise to new objects: preferences about known objects nearby in the hierarchy are used to select a shelf for the new object. Object taxonomies can also be used to place an object into the correct cupboard, viewed as a classification task [9]. While these approaches effective in grouping objects into boxes cupboards,
method addresses the general case of learning continuous spatial relations, like how to set a dining table or tidy an office desk.
Dirichlet Process models. Recent work [10] learns spatial concepts: i.c. the relationship between an object, its tidied location, and name describe that location (useful for voice commands). Model parameters are learned with Gibbs Sampling. Another approach with Dirichlet Process mod- elling uses context from sampled human poses [11] to position objects conveniently, Density func- tion parameters are shared across objects of the same type. This approach is evaluated on a training dataset of arrangements made by 3-5 users, with further 2 users scoring arrangements. While these methods predict generally tidy arrangements, it is not shown whether they can tailor them to indi- vidual preferences. We develop a tidying simulator to gather data on a larger scale, demonstrating that neural encoder makes personalised predictions method uses word embeddings to predict positions for objects unseen during training, and learns transferable representations of tidying preferences which can be applied to arrange new scenes.
a dataset of 75 users. Furthermore,
Example-driven tidiness loss function. Another approach is to ask each user to provide several tidy arrangements for a scene, to serve as positive examples [12]. To tidy a scene, the robot computes target tidy layout using a cost function which encourages altering the object-object distances in the untidy scene to match those in the "closest" tidy positive example. Simulated re-annealing is used to optimise this cost function efficienlly. We add value firstly because our method generalises to unseen objects without requiring users to arrange them manually, and secondly because our method

---

combines knowledge about the current user's preferences with prior knowledge about "similar" training users, correcting errors in individual examples and allowing for stronger generalisation.
Neural recommender systems. Neural networks have been used for making recommendations based user preferences [13]. The YouTube neural recommender system [14] predicts user's expected watch time for a future video based on past vidcos watched and searches made. However, these methods do address the challenges of learning spatial relations by observing scenes and predicting tidy arrangements for any set of objects. We apply GNN components to solve this.
3 Method
We introduce NeatNet: a novel architecture for learning spatial preferences. First, we describe Variational Autoencoder for learning latent representations of a user's preferences (Section 3.1). Then we introduce the GNN layers which allow NeatNet to learn from scenes arranged by users, modelled as graphs of objects (Section 3.2). Next, we adapt the architecture to learn from multiple example scenes per user (Section 3.3). Finally, we illustrate how object semantic embeddings are inferred from word embeddings, for generalisation to unseen objects (Section 3.4).
3.1 Encoding User Preferences with VAEs
Our objective is learn user encoder function Eg. represented by a neural network with learned parameters ø. Its output is user's preference vector u. Its input is representation of scene arrangement made by that user. Each scene is defined by the terms s (the semantic identities of the objects in the scene, further explained in Section 3.4), and (the position encodings for those objects, e.g. coordinates). These are not passed using GNN layers (Section 3.2), since scenes can have variable number of objects. We also train position predictor function Da. Given a set of objects identified by s, and a user preference vector u, this predicts a position for each object which reflects those spatial preferences.
the VAE directly: scenes are first encoded
user preference latent space
left-handed arrangement
right-handed arrangement
Eg
E
Figure 2: Two users being mapped to points in the user latent space based on how they arrange dining scene. Users close together this latent vector space share similar spatial preferences.
These networks trained end-to-end as Variational Autoencoder [15]. A scene represented by and p is passed into the encoder Eg- The output user preference vector u, along with the semantic embeddings of the objects s, is passed into the position predictor Dg. which acts decoder by reconstructing the original scene. Since the user preference vector a low-dimensional bottleneck, this encourages the network to learn important spatial preferences like Seddrightthannddness which are useful for predicting the positions of many objects at once. For variational inference, the encoder Eg predicts Gaussian distribution over the possible scene. At training time, we sample from this distribution to obtain u, and we take the mean at inference time. The VAE objective is thus Ø*, 0 argming,e L(ø,0) 0).
vectors which could explain the input
L( 0) A Epeaca (s.p) [[)
pll 3KL
Here, Pdato represents the training data distribution. p(u) is prior over the user preference vec- tors: a standard Gaussian, with zero mean and unit variance. KI. represents the Kullback-Leibler divergence (standard in VAEs [15]) which encourages the estimated distributions to stay close to
3

---

the p(u) Gaussian prior over the user space, and the hyperparameter / scales this prior term to make the latent space less sparse and encourage disentanglement [16].
3.2 Encoding Scenes with GNN Layers
We will now encode scenes as vectors so that they can be passed through the VAE. To do this, we Graph Neural Network (GNN) layers. GNNs can operate on scenes with a variable number of arbitrary nodes, which is critical since the robot cannot foresee all the objects it will ever encounter. Each node represents object in the scene. Node has feature vector x; formed by concatenating the semantic embedding for that object s; with its position encoding Pt. The scene graph is fully connected, avoid making assumptions about which object-object relations relevant not.
semantic embeddlng
position encoding
graph encoder
preference extractor
preference vector
graph encoding vector
graph of 4 object nodes in scene
Figure 3: The encoder network, all together representing the function Eg defined in Section 3.1. The input is node feature matrix with row for each node's s feature vector x; _ 8; || p;- The graph encoder consists of GAT layers (described later) separated by ELU non-linearities [17]. Then we pass the final feature vector of each node through network of fully-connected layers, which has the same output size as the graph encoding vector. To aggregate the output node vectors into a single graph encoding vector, we apply global add-pooling layer. This graph encoding vector is passed through a user preference extractor: series of fully-connected layers separated by LeakyReLU activations [18]. The output is mean and variance for the user preference vector distribution, from which we sample to obtain u. This is a latent vector which may not always be interpretable, but it might capture features such as whether the user prefers convenience over aesthetics.
semantic embedding
position encoding
position predictor
graph of objects to be placed
predicted positions
user preference vector
Figure 4: The position predictor, which of objects for which we wish to predict positions, according to a user preference vector u. We set the node feature vectors to be x; - pullu. This time, the output is a position vector for each node. Our experiments focus positions, but the position vector be extended to include orientation information. Similarly to the graph encoder, the position predictor consists of a sequence of GAT layers, after which each node is passed through series of fully-connected layers, the final output of which is the predicted position vector for that node.
the decoder Dg. The input consists of a
Both the graph encoder (Figure 3) and position predictor (Figure 4) use GNN layers. As is standard for GNN layers, we need to compute hidden feature vector ht for each node i, based on its current feature vector x; as well the node feature vectors ny of its neighbours. To compute hts we choose the Graph Attention (GAT) layer [19]. h, computed as a weighted sum of the node feature vectors zj in the current node's ' neighbourhood, where the weights ag generated by the soft

---

attention mechanism. Since our graph is fully connected, we calculate attention weights for all the other objects the scene: therefore the attention mechanism allows the network to autonomously decide which object-to-objed relations are the most important for predicting correct arrangements. In order to further increase the power of the graph network, we can add second GAT layer which takes as input the h; features produced by the first layer. A GAT layer computes h; follows, where represents concatenation, and the learned parameters are the weights and a:
exp (LeakyReLU (a" [Wa;]] Wa,l) KKEA exp (LeakyReL.U (a [Wa]] Wox))
h4
Orig Waj
Ozj
jEM.
3.3 Generalising Across Scenes
Many spatial preferences, such as eeftrightthandedness will be exhibited by the user consistently across scenes. We now extend NeatNet to learn from multiple scenes arranged by same user, as shown in Figure Every example scene arranged by the current user is passed through the single- scene encoder network & in Figure 3. Each scene may produce subtly different estimate of the user's preferences, they are aggregated through a global pooling layer produce average preference vector u. On the decoding side, we use the same u vector as the input to the position predictor reconstruct all the scenes created by this user. Since is the architecture's bottleneck, this encourages the network to learn important preference features which generalise across scenes, such left/right bandedness preference for arranging objects compactly rather than spaciously.
office objects semantic embeddings
preference estimate based on office
position predictor
encoder
global pooling layer
copy
overall preference vector
user encader
position predictor
preference estimate based dining
dining objects
semantic embeddings
Figure 5: Learning one generalised user preference vector across multiple example scenes per user.
3.4 Semantic Embeddings
Each object's identity is represented by its semantic embedding vector. One-hot encoding is straightforward approach. limitation this method is that it does not generalise readily to new objects. An alternative is feature vectors: example entries include the height of an object, RGB values describing its colour palette. However, this may not capture all features necessary for tidying. We also develop method using word embeddings. Objects which appear in similar linguistic con- texts are used for similar activities, and so are often arranged in related way: for example "pen" and "pencil", or "salt" and "pepper". We load word embeddings from FastText model [20], pre- trainadidd on the Common Crawl dataset of over 600 billion tokens. Names of objects are provided to NeatNet either by the simulator by an object detection system if deployed physically. These names are converted to word embeddings and passed through the semantic embedding extractor (Figure 6). yielding the semantic embeddings referred in Figure This allows the robot to predict positions for new objects never seen before in any tidy arrangement during training.
5

---

general word
Word Embedding Space (T-SNE)
embedding
semantic embedding
Latent dimension
plate
bowl
cup
phone
"pen"
embedding model
word
semantic embedding extractor
computer laptop
pencil
pen
keyboard
pre-trained literature
trained end-to-end
Latent dimension
Figure 6: Left: Word embedding space. Object names which appear in similar linguistic contexts will be close together in this latent space. Right: the semantic embedding extractor outputs object's semantic embedding vector, using fully-connected layers. This is trained end-to-end with NeatNet, to extract the object features which most relevant for spatial arrangement task.
4 Experiments
4.1 Collecting Rearrangement Data from Users
Gathering data from human users at scale is significant challenge for robotics experiments. We developed TidySim: a 3D object rearrangement simulator deployed as a web app, allowing users arrange series of scenes through an intuitive interface. Features include: moving the camera, clicking and dragging objects to move or rotate them, and an inventory for unplaced objects.
Figure 7: Scenes arranged by users. From left: (1-2) are abstract scenes, where users must split the objects into two lines, either by shape or by colour, (3) is dining room, and (4) is an office desk.
4.2 Experimental Procedure
We designed the scenes in Figure 7 to maximise experimental value while keeping the user workload low (~ 5 minutes total per user). Rearrangement data from 75 users was gathered by distributing the experiment link social media. Each submits tidy arrangement for number of scenes. We set aside a group of 8 test users, male and female, and majority non-roboticists. The training dataset was formed from the 67 remaining users. NeatNet was trained by passing training users through the Graph VAE and updating parameters with a neural network optimiser according to the loss function in Equation 3. For each test user, we passed their example scenes through NeatNet to extract their spatial preferences, and predict personalised arrangement for the test scene (varies by experiment). Arrangements were also produced by the baseline methods in Section 4.3. The user was then asked to score the arrangements produced by each method based on how tidy they were according their preferences, on scale from (completely untidy) 10 (perfectly tidy).
4.3 Baselines
In Section 2, we qualitatively compare with existing methods to show where we add new capabili- ties. For fair quantitative evaluation of our method on this dataset, we test against variety base- line methods depending on the specific rearrangement task. Desitive-Exampla to tidy known

---

scene, this baseline copies the example arrangement which was supplied by the test user (which the NeatNet method uses extract this user's tidying preferences). This baseline ignores priors from other users and is unavailable when generalising to new objects or scenes. Random-Posto Piniiiion sets the missing object's predicted position randomly, while Mean-Position uses the object's mean position across all the example arrangements from training users. NeatMett-NoPreefs is an ab- lation baseline: the test user's preference vector is replaced with the zero vector (representing a neutral preference, which test users might not find tidy). The Pose-Graph method produces an arrangement which some users will find tidy, but is still not personalised to the current user's pref- erences. It learns a probabilistic pose graph of relative object positions from the training example arrangements, and solves it using Monte Carlo sampling and scoring to find the most likely ar- rangement. When placing an unseen object. other objects in the scene, and adds small random offset to minimise collisions with other objects. -Niihhbou finds the object already placed by the user which has the closest word em- bedding vector to the unseen object, and places the new object next to it, i.e. would place the salt next the pepper. Neighted-kMUl -wl---leeee-iiion uses the k = 3 nearest neighbours, weighted by word distance, to compute an average position. For the task of arranging a new scene, the Random-User baseline copies how a random training user arranged that scene, without personali- sation. PWW--SeenePPojectiion projects the user's preferences from the example scene onto the new scene, by placing each object in the new scene based on how the user placed the k most similar objects in the example scene, similar to the Weighted -NNeedd----------NN---------N- Reg--esegessonn algorithm.
Saaa-WWtt-Offfee uses the mean position of the
4.4 Experiment Results
We designed a series of experiments to test our method's capabilities. Visualisations of predicted arrangements are available in the supplementary material and on our project page: wuw.uuu-oooou------rlll..e
www.robot-learinng.o www.roboelarrinnaa///yy-looo-s--
Can NeatNet Tidy a Known Scene? (Table 1) Suppose that the robot has seen the user tidy room. After the room becomes untidy, the robot must tidy up each object. This tests how well the network can reconstruct the scene based on the user preference vector and its prior knowledge. For each scene separately, we trained NeatNet as described in Section 4.2. Then for each test user we pass their scene through the VAE and ask them to rate the reconstructed scene.
Tidying Method
Abstract
Abstract 2
Dining Table
Office Desk
Average
Mean-Position
3.25 +2.04
3.00
11.4
2.34 +1.17
2.22 +1.48
2.70 +1.50
NeatNett-No-Pres
4.50 +1.73
3.75 11.31
2.58 1.06
3.70 +1.1
3.63 +1.38
Pose-Graph
6.98 +1.53
7.82 +0.86
7.40 11.2
Positive-EXample
8.75
1.26
7.50
0.55
8.60 10.58
9.18 10.81
8.51 10.98
NeatNet
8.75
1.46
8.25
1.45
9.12 10.49
8.90 10.40
8.76 11.17
Table 1: Tidying known scene. Rows are tidying methods and columns are scenes. The mean and standard deviation of tidiness scores given by test users are shown, along with cross-scene average.
NeatNet can reconstruct personalised tidy arrangements from a user preference vector, out- performing baselines and approaching the user's handmade Dositive-Example In a direct com- parison, all test users ranked NeatNet over Pose-Graph. Surprisingly, users sometimes even prefer the reconstructed tidy scene over their own example. This because in each individual user-created arrangement, there is noise in how they placed each object (realistically this would also include error from localisation/poss estimation). Our method uses prior knowledge from other users to correct for noise in individual examples (analogous to VAE image denoising [21]).
Can NeatNet Find a Home for a New Object? (Table If a user brings new object (e.g. cup) into the scene, the robot should predict a tidy position for it. NeatNet is trained as before, but with the positions of random nodes masked out from input examples to encourage generalisation. At test time, we mask out the new object's position from the test user's example arrangement, but the position predictor must still place it. This analogous to the use of generative models for "inpainting" missing or obstructed regions of input images [22]. This experiment is repeated for each object. NeatNet outperforms the baselines by successfully combining what it knows about the current user's preferences with prior knowledge of how similar users placed the new object.
7

---

Prediction Method
Cup
Fork
Knife
Plate
Average
Random- Position
26.70 19.07
15.76
15,51
13.96 18.03
7.10 14.49
16.38 18.19
NeatNet-No-Proet
13.20 16.52
14.98
15.01
13.38
15.77
2.88 1.002
11.12 16.23
Mean-Position
11.42 4:5.9
15.06
+5.50
13.48
+6.27
1.64
+0.65
10.40
+6.14 14
NeatNet
5.72 1.46
1.18
1076
1.44
#1.15
1.96
10.67
2.58 11.34
Table 2: Placing new object into the scene. Figures show the mean and standard deviation of the absolute error between the predicted and true, unmasked positions of each object (in centimetres).
Can NeatNet Generalise to Objects Unseen During Training? (Table 3) The network must place a new blue box into an abstract scene, and a laptop into the office scene, though it has never seen laptop during training. The training process is similar the previous experiment. NeatNet's placement of the new objects (box and laptop) is satisfactory to users. To place the laptop, NeatNet considers how the user placed several semantically similar objects: the computer, laptop and key- board. The hearest-Neighhee their desktop computer, thus misplacing it under the desk. Weighted Beiigheed-kNNRReee -NNN--eee-eeeee-- on performs well, but NeatNet was still ranked higher by every user because it can extract the semantic informa- tion from the word vectors which is most useful for tidying, since the network is trained end-to-end. Furthermore, NeatNet is able to learn and extrapolate the order-of-size pattern in the abstract scene.
baseline places the laptop based solely on how the user placed
Tidying Method
Abstract Scene
Office Desk
Average
Mean -ith-witth-fffet
2.58 1..35
4.68 11.31
3.63
Nearost-Neighboun
5.78 11.32
1.70 10.67
3.74
Weiighted--ll--RRgreess-s NeatNet
4.22 +0083
7.80 +1.64
6.01
8.10 1.224
9.16 1.28
8.63
Table 3: Placing an object unseen during training, with user scores for tidiness shown.
Can NeatNet Predict a Personalised Arrangement for New Scene? (Table 4) At training time, the network is shown how the user has arranged both abstract scenes, with random scene masking applied to encourage generalisation. At test time, the network is shown one abstract scene and asked to predict how the test user would arrange the other. kNN-Scene WWIN-Scene-Projecee Projecction produces person alised arrangements, but NeatNet lines up objects more neatly because it combines personalisation with learned prior knowledge about these objects from training users. The Random-Use baseline performs moderately well, especially when by luck the random training user which was chosen has similar preferences to the test user. However, when those preferences differ, the neural network method out-performs it. This shows the importance of accounting for preferences, and demonstrates that the network can learn preferences which enable generalisation across scenes.
Tidying Method
Abstract
Abstract 2
Average
Mean-Position
2.04 1.00
2.30 +0.97
2.17
Random-User
5.52 +1.12
6.24 +1.67
5.88
KNN-Scene-Pron PrcenneProjection
6.36 10.96
7.32 11.48
6.94
NeatNet
9.58 10.53
9.72 10.44
9.65
Table 4: User scores for predicted arrangements of a new scene.
5 Conclusions
Findings. Our NeatNet architecture learned latent representations of user spatial preferences by training as a Variational Autoencoder. Graph Neural Network components allowed it to learn di- rectly from user-arranged scenes, represented as graphs, and word embeddings enabled predic- tions for unseen objects. NeatNet's ability to generalise and make personalised predictions was demonstrated through experiments with real users, adding new capabilities to existing methods. Future work. Rather than using the word embedding of the object's 's name, convolutional layers can be used to generate a semantic embedding for an object based on its visual appearance (e.g. teacups look similar to mugs and are placed similarly). Furthermore, we can combine our approach with model-based priors from existing methods, e.g. using human pose context [11] as part of our loss function to ensure objects are placed conveniently, while also tailoring to individual preferences.
8

---

Acknowledgments
This work was supported by the Royal Academy of Engineering under the Research Fellowship scheme.
References
[1] D. Batra, A. X. Chang. S. Chernova, A. J. Davison, J. Deng, Koltun, S. Levine, J. Malik, I. Mordatch, R. Mottaghi, M. Savva, and H. Su. Rearrangement: A challenge for embodied Al. ArXiv, abs/201 1.01975, 2020.
[2] C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-Pérez. Integrated task and motion planning. Annual Review of Control, Robotics, and Autonomous Systems, 4(1):265-299) 2021.
[3] E. Karpas and Magazzeni. Automated planning for robotics. Annual Review of Control, Robotics, and Autonomous Systems, (()::417-499,000
[4] C. Pantofaru, Takayama, T. Foote, and B. Soto. Exploring the role of robots in home organization. In Proceedings of the Seventh Annual ACM/IEEE International Conference on Human-Rolee Interaction, HRI '2, page 327-334, New York, NY, USA, 2012. Association for Computing Machinery. ISBN 9781450310635.
[5] McCormac, R. Clark, M. Bloesch, J. Davison, and S. Leutenegger. Fusion++: Volumetric object-level slam. 2018 International Conference on 3D Vision (3DV), pages 32-41, 2018.
[6] J. Redmon, S. K. Divvala, R. B. Girshick, and A. Farhadi. You only look once: Unified, real- time object detection. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 779-788, 2016.
[7] C. Wang, D. Xu, Zhu, R. Martin-Martin, C. Lu, L. Fei-Fei, and S. Savarese. DenseFusion: 6D object pose estimation by iterative dense fusion. 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 3338-33333333 1338-3347,2 2019.
[8] N. Abdo, C. Stachniss, L. Spinello, and W. Burgard. Robot, organize my shelves! Tidying up objects by predicting user preferences. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 1557-1564, 2015.
[9] M. Schuster, Jain, M. Tenorth, and M. Beetz. Learning organizational principles in human environments. In 2012 IEEE International Conference on Robotics and Automation, pages 3867-3874, 2012.
[10] A. Taniguchi, S. Isobe, L. E. Hafi, Y. Hagiwara, and T. Taniguchi. Autonomous planning based on spatial concepts to tidy up home environments with service robots. Advanced Robotics, 35 (8):411 1897,1499,11 2021.
[11] Y. Jiang, M. Lim, and A. Saxena. Learning object arrangements in 3D scenes using human context. Proceedings of the 29th International Conference on Machine Learning, ICML 2012, 06 2012.
[12] M. Kang. Y. Kwon, and S.-E. Yoon. Automated task planning using object arrangement opti- mization. In 2018 15th International Conference Ubiquitous Robots (UR), pages 334-341, 2018.
[13] X. He, L. Liao, H. Zhang, L. Nie, X. Hu, and T.S.. Chua. Neural collaborative filter- ing. In Proceedings of the 26th International Conference on World Wide Web, WWW '17, page 173-182. International World Wide Web Conferences Steering Committee, 2017. ISBN 978 9781450349130.
[14] P. Covington, J. Adams, and E. Sargin. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems, New York, NY, USA, 2016.

---

[15] D. P. Kingma and M. Welling, Auto-encoding variational Bayes. In 2nd International Confer- ence on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Confer- ence Track Proceedings, 2014.
[16] I. Higgins, L. Matthey, A. Pal, C. P. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner. beta- VAE: Learning basic visual concepts with a constrained variational frame- work. In ICLR, 2017.
[17] D. Clevert, T. Unterthiner, and S. Hochreiter. Fast and accurate deep network learning by exponential linear units (ELUs). In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
[18] A. L. Maas, A. Y. Hannun, and A. Ng. Rectifier nonlinearities improve neural network acoustic models. In Proc. ICML, volume 30, page 3, 2013.
[19] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio. Graph Attention Networks. International Conference on Learning Representations, 2018.
[20] Bojanowski, E. Grave, A. Joulin, and Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5, 07 2016.
[21] D. Im, S. Ahn, R. Memisevic, and Y. Bengio. Denoising criterion for variational auto- encoding framework. In Proceedings of the Thirty-Firsl AAAI Conference on Artificial Intelli- gence, AAAI' 17, page 1059-2065. AAAI Press, 2017.
[22] R. A. Yeh, C. Chen, T.-Y. Lim, A. G. Schwing, M. A. Hasegawa- and M. N. Do. Semantic image inpainting with deep generative models. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6882 6890, 2017.
10


Reviews:
[{"review_id": "glB0qFwwZvf", "reviewer": "Reviewer_vXaq", "summary": "The main contributions of this paper could be concluded as following:\n(1). Proposed a GNN-based VAE model named NeatNet to deal with rearrangement tasks, which can extract more latent information like the spatial information of input scene images\n(2). Innovatively bring out the new form of rearrangement tasks: rearrange the given scene into personalized style with preference extracted from provided train samples\n(3).Gathered preference samples from users and validate the effectiveness of proposed model with experiments\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.\tInnovatively proposed the method that use Graph Neural Network layers in VAE, a traditional generative model which may allow the model directly learn from arrangement provided by users in format of graphs\n2.\tDesigned a well-considered attention mechanism and update rules to make the model extract the most important preference in samples.\n3.\tComprehensively explore the performance of propose model in different situations with conducting experiments, such as generate arrangements for known scenarios, fit a new kind of object in current scenario and generalize a learned preference to a new scenario.\nWeakness:\n1.\tMore realistic and complex scenarios need to be used to validate the effectiveness and generalizability of proposed model. For reality, images of some real scenarios with same arrangements should be concluded. For complexity, scenarios with larger views such as a whole scene of dining room instead of just the table.\n2.\tBaselines should be more convincing instead of only naive methods implemented in experiment.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend this paper for its contributions on forming a innoative type of robot re-arrangement tasks and proposing the GNN-based  VAE model to deal with such kind of rearrangement tasks. The GNN-based model allow the generative model  directly learn from arrangement provided by users in format of images and extract the preference information from latent space. And the experiments validated its effectiveness with comparison to several naïve baselines. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "1. More realistic and complex samples should be included\n2. A more complicate and effective baseline should be used in comparison to make the results more convincing\n", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "H6yURpHDaB", "reviewer": "Reviewer_Ftgd", "summary": "This work provides a method to enable a robot arrange/tidy objects according to a user’s preferences. NeatNet is a variational autoencoder that learns a representation of the user’s preferences and uses it to position objects in a scene. Input scenes are encoded using graph neural network layers.\n\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is clearly written and organized. The authors propose a novel scheme to learn user preferences of spatial object structures by combining graph neural network layers with a variational autoencoder architecture. The series of questions answered in the evaluation section seem comprehensive. \n\nHowever, the paper could be strengthened by adding baselines that also model the user’s preferences. Currently, none of the baselines model user preferences which provides for an unfair comparison. For instance, in Tables 1 and 4, it’s not surprising that the rest of the baselines (Mean-Positions, NetNet-No-Prefs, and Pose-Graphs, Mean-Position, Random-User) would do poorly. Adding baselines that take into account user preferences would also shed light on why using GNN layers + VAE the way the paper did is the best way to learn the user’s spatial preferences rather than doing something like inverse reinforcement learning. \n\nI am also curious why the Nearest Neighbor baseline performs so poorly in Table 3. Do the Nearest Neighbor algorithm and NeatNet use the same word embeddings? Currently, it seems like the Nearest Neighbor algorithm places the computer where the desktop was because “desktop” is the most similar word. If so, wouldn’t expanding k, the # of nearest neighbors, help alleviate this problem? \n\nProviding some examples of distinct user preferences in the environments (especially dining and office scenes) would be helpful. Since there are established norms in the non-abstract environments (e.g., keyboard and mouse go in front of the monitor, plate goes in between fork and knife), I’m curious to see how varied these preferences can get. For instance, is the main difference in preferences just left vs right-handedness? Adding some statistics on the variance of user preferences in each of these scenes would help the reader get a better sense of the difficulty of the problem.\n\nIncluding some visual examples for each of the experiments (especially Table 4) would be great :-)\n\nPlease report standard deviation for all means in all the tables. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I vote for a weak accept conditioned on the fact that the authors are able to provide stronger baselines in their experiments. \n\nEDIT: Thank you for your reply. I have read it and am raising my score. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "- Addition of stronger baselines (see comments above)\n- Providing more examples of both experimental results as well as different user preferences. \n- Including standard deviation for all means in all tables. "}}, {"review_id": "8F7TQbSQBM", "reviewer": "Reviewer_8rCv", "summary": "The paper tackles the scene rearrangement task by proposing a novel architecture that learns low-level representations of user preferences in regards to tidying up a space. These representations can subsequently be used to generate object arrangements according to the specific user's spacial preferences and further generalize to include unseen objects through the use of word embeddings. Towards achieving the above, the paper also presents a simulator used for data collection through user interaction. Results receive positive evaluation by users in terms of matching their spatial preferences.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper was very well written. The structure and language significantly helped to follow and comprehend the work done. The combination of VAEs and GNNs was an interesting approach towards learning user preferences and scene representation. Furthermore, generalization to new objects is an important aspect of the problem and the paper did not leave this unaddressed. Combining images with word embeddings for objects in order to address the generalization, was creative.  \n\nIn terms of weaknesses, one general philosophical question stood out: Important spatial preferences such as left/right handedness are very few and represent a much more constrained space of a user's rationale behind the way they prefer their space organization. For instance, there are people that left/right handedness doesn't necessarily play a part in how they organize their space and/or they are not consistent in a particular location for objects in their space. Secondly, rating organized scenes by humans is not entirely convincing as a good measure of how well the approach works, i.e. due to the inherent subjectivity of user preferences, the same user could be giving different ratings on the same scene at different points in time. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, the paper presented an interesting line of work. The problem of user spatial preferences and their use for scene arrangement is inherently difficult due to its subjectivity. The work conducted employed a creative approach that yielded notable results. Along with the paper being well written and easy to follow, it made for a captivating read. In terms of impact, it's not entirely convincing that it can spawn novel directions in the community. However, it can potentially spawn very interesting conversations about the work itself as well as the philosophy of human preferences and thus, would make a solid addition to CoRL 2021. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "Ln 15. \"Given goal states stipulating where each object should go, many approaches exist for planning a sequence of actions to manipulate objects into the desired arrangement\" was followed by only one reference. Could you please cite a few more approaches?\n\nIn the related work, an overview of overall existing work on scene rearrangement was expected.\n\nCould the dataset described in https://arxiv.org/abs/2103.16544 be potentially used for further evaluation of the method (instead of or complementary to the developed simulator) as it provides a wider set of scenes? It is very recent work, therefore not expected to have been used in this paper but perhaps it could be useful for future work? \n\nIn regards to rearrangement data collection from users, why did you have to build your own simulator? There are many simulators out there, for embodied AI and scene rearrangement. \n\nLn 19. \"Several factors involved in spatial preferences are universal\": Please consider rephrasing. One of the two universality examples you give right after, happens to not apply to me. Also, especially since it is subsequently mentioned that \"It is clear that spatial preferences are a complex, multi-faceted, and subjective concept\" (ln 30). Subjectivity is quite contrary to universality. \n\nLn 78. \"secondly because our method applies prior knowledge from other users\": why is this desired behavior? User preferences should be unique vectors in the latent space i.e. other users' preferences should not matter when tailoring a scene to a specific user?\n\nSemantic embeddings section was a bit too brief. Unclear how the embeddings loaded from FastText are tied to the visual scenes and thus, allow for generalization to new objects. Are all objects in a scene labelled by the simulator somehow? Could you please add some more details?\n"}}]

Author rebuttal:

Meta-review: In this paper, the authors propose the novel NeatNet considering user preference for robotic rearrangement task. The variational autoencoder is used to learn a low-dimensional preference vector and the scenes are modeled with GNN. All reviewers recognized that the paper is well written and organized. It will make the paper more convincing if some stronger baselines, for example, considering user preference, are provided. Also, more qualitative results are expected to make the experimental results more illustrative.

In the rebuttal session, the authors have responded to the reviewers' concerns and more details are provided. And the reviewers have reached a consensus of accepting this paper.
Decision: Accept (Poster)