SORNet: Spatial Object-Centric Representations for Sequential Manipulation
Authors: Wentao Yuan
Chris Paxton
Karthik Desingh
Dieter Fox
Venue: CoRL2021 Oral (2021)
Keywords: Object-centric Representation
Spatial Reasoning
Manipulation

Abstract: Sequential manipulation tasks require a robot to perceive the state of an environment and plan a sequence of actions leading to a desired goal state, where the ability to reason about spatial relationships among object entities from raw sensor inputs is crucial. Prior works relying on explicit state estimation or end-to-end learning struggle with novel objects or new tasks. In this work, we propose SORNet (Spatial Object-Centric Representation Network), which extracts object-centric representations from RGB images conditioned on canonical views of the objects of interest. We show that the object embeddings learned by SORNet generalize zero-shot to unseen object entities on three spatial reasoning tasks: spatial relationship classification, skill precondition classification and relative direction regression, significantly outperforming baselines. Further, we present real-world robotic experiments demonstrating the usage of the learned object embeddings in task planning for sequential manipulation.

SORNet: Spatial Object-Centric Representations for Sequential Manipulation
Wentao Yuan University of Washington
Chris Paxton NVIDIA
Karthik Desingh University of Washington
Dieter Fox University of Washington, NVIDIA
https: httpss///wetttorn.gtthhhaa tttttoun.../sorrett
Abstract:
Sequential manipulation tasks require robot to perceive the state of an environment and plan a sequence of actions leading to a desired goal state, where the ability reason about spatial relations among object entities from raw sensor inputs is crucial. Prior works relying on explicit state estimation or end-to- end learning struggle with novel objects
tasks. In this work, we propose SORNet (Spatial Object-Centrry Representation Network), which extracts object- centric representations from RGB images conditioned on canonical views of the objects of interest. We show that the object embeddings learned by SORNet gen- eralize zero-shot unseen object entities on three spatial reasoning tasks: spatial relation classification, skill precondition classification and relative direction re- gression, significantly outperforming baselines. Further, we present worldworld robotic experiments demonstrating the usage of the learned object embeddings in task planning for sequential manipulation.
Keywords: Object-centric Representation, Spatial Reasoning, Manipulation
Object embeddings
left (red block, pallow blol)
in front, _alnfrooooliii offiew block, rod block) stackee foee, block, gower, blools
Spatial relations
approachirobet, block) True obj/racc,,
SORNet
clear(red
RGB
Skill Preconditions
Canonical Object
Relative Direction
We propose SORNet (Spatial Object-Centric Representation Network) that learns object embed- dings useful for spatial reasoning tasks such as predicting spatial relations, classifying skill preconditions, and regressing relative direction between objects.
Figure 1:
1 Introduction
A crucial question for complex multi-step robotic tasks is how represent relations between entities in the world, particularly as they pertain to preconditions for various skills the robot might employ. In goal-directed sequential manipulation tasks with long-horizon planning, it is common use state estimator followed by a task and motion planner or other model- based system [1, 2, 3, 4, 5, 6]. A variety of powerful approaches exist for explicitly estimating the state of objects in the world, e.g. [7, 8, 9, 10]. However, is challenging to generalize these approaches to an arbitrary collection of objects. In addition, the objects often in contact in manipulation scenarios, where works explicitly addressing the problem of generalizing to unseen objects [11, 12] still struggle.
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Fortunately, knowing exact poses of objects may not be necessary for manipulation. End-to-ent methods [13, 14, 15] leverage that fact and build networks that generates actions directly from sensor inputs without explicitly representing objects. Nevertheless, these networks are very specific to the tasks they are trained on. For example, it is non-trivial to use network trained on stacking blocks to unstack blocks.
In this work, we take an important step towards a manipulation framework that generalizes zero-shot to novel tasks with unseen objects. Specifically, we propose a neural network that extracts object- centric embeddings from raw RGB images conditioned on object queries, which we call SORNet, or Spatial Object-Centre Centric Representation Network. The design of SORNet allows it to generalize to novel objects without retraining or finetuning. The object-centric embeddings produced by SORNet can be combined with readout networks to inform a task and motion planner with implicit object states relevant to goal-directed sequential manipulation tasks, e-g. logical preconditions for primitive skills or continuous 3D directions from the end effector to objects in the scene.
To summarize, our contribution are: (1) method for extracting object-centric embeddings from RGB images that generalizes zero-shot to different number and type of objects; (2) framework for learning object embeddings that capture continuous spatial relations with only logical supervi- sion; (3) a dataset containing sequences of RGB observations labeled with spatial predicates during various tabletop rearrangement manipulation tasks.
We empirically evaluate the object-centric embeddings produced by SORNet on three different downstream tasks: 1) classification of logical predicates for action preconditions: 2) visual question and answering for spatial relations; 3) prediction of relative 3D direction between entities in ma- nipulation scenes. In all three tasks, the models are tested held-out objects that did not appear in training data. In all tasks, SORNet obtains significant improvements over the baseline methods. Finally, we evaluated SORNet with real-world robot experiments to showcase the transfer of the learned object-centric embeddings to real-world observations.
2 Related Work
Sequential Manipulation In goal-directed sequential manipulation tasks with long-horizon plan- ning, a class of work uses a pipeline approach requiring state estimator followed by a task and motion planner [1, 2, 3, 4, 5, 6]. but the state estimator with an explicit state representation is often restricted to an environment or a set of objects. Another class of work employs neural networks to learn motor controls directly from raw sensor data, such as RGB images and joint encoder read- ings [14, 16, 15, 17, 18, 19], but they are often not transferable to new tasks, especially tasks in- volving long-horizon planning. Our work combines network that estimates implicit states with a symbolic planner to obtain the best of both worlds: generalization to new objects and to new tasks.
Learning Spatial Relations Learning spatial relations between object entities have been studied in the field of 3D vision and robotics. Methods such as [20, 21, 22] predict discrete continuous pair- wise object relations from 3D inputs such as point clouds or voxels, assuming complete observation of the scene and segmented objects with identities. In contrast, our approach does not make any assumptions regarding the observability of the objects and does not require pre-processing of the sensor data. The learning framework by Kase al. [23] is most related to our approach, which takes a sequence of sensor observations and classifies set of pre-defined relational predicates which is then used by a symbolic planner to produce a suitable operator. Compared to our approach, theirs is limited by the number of objects in the scene and fixed set of spatial predicates.
Visual Reasoning Recently, several advancements have been made on visual reasoning benchmarks [24, 25, 26] using transformer networks [27]. Toward solving spatio-temporal reasoning task from CLEVRER [25] and CATER [26], Ding et al. [28] proposed an object-based attention mechanism and Zhou et al. [29] proposed a multi-hop transformer model. Both works assume segmentation model produce object segments and performs language grounding to the segments to perform rea- soning. Our SORNet architecture is simpler and can solve ppatial-reasoning tasks for unseen object instances without requiring a segmentation or object detection module. Furthermore, our work fo- cuses on a relatively complex manipulation task domain involving manipulator in the observations. Although our current work focuses on predicting spatial relations from single RGB frame, the object-centric embeddings could potentially be used for solving temporal-reasoning tasks.
2

---

m
m
m b
m
b
B
Transformer Encoder
Positional Encoding
Linear Projection Flattened Patches
Context Patches
Canonical Object Viows
Figure 2: SORNet architecture, Input the network is RGB image and canonical views of the objects interest. The RGB image broken context patches which have same size canonical views. These patches are flattened and added with positional encoding and passed through multi-layer multi-head transformer [30]. The embeddings corresponding to the canonical views are used for the downstream tasks.
Figure 3: Examples of canonical object views used for training CLEVR (left) and Leonardo(right) dataset. Lighting, texture and object poses vary across different views, and sometimes there occlusion by the robot.
Methods
3.1 SORNet: Spatial Object-Centric Representation Network
Our object embedding network (SORNet) (Fig. 2) takes an RGB image and an arbitrary number of canonical object views and outputs an embedding vector corresponding to each input object patch. The architecture of the network is based the Visual Transformer (ViT) [30]. The input image broken into a list of fixed-sized patches, which we call context patches. The context patches concatenated with the canonical object views to form patch sequence. Each patch is first flattened and then linearly projected into a token vector, then positional embedding is added to the sequence of tokens. Following [30], we token vectors as positional embeddings. The transformer encoder, which includes multiple layers of multi-head self-attention The transformer encoder outputs a sequence of embedding vectors. We discard the embedding for context patches and keep those for the canonical object views.
of learnable vectors with the same dimension the tokens are then passed through
We apply the same positional embedding to the canonical object views to make the output embed- dings permutation equivariant. We also mask out the attention among canonical object views and the attention from context patches to canonical object views to ensure the model uses information from the context patches to make predictions. In this way, we pass in arbitrary number of canonical object views in arbitrary order without changing model parameters during inference.
Intuitively, the canonical object views can be viewed as queries where the context patches serve as keys to extract the spatial relations' values. Note that canonical object views are not crops from the input image, but arbitrary views of the objects that may not match the objects' appearance the scene. Our model learns identify objects even under drastic change in lighting, pose and occlusion. Fig. 3 shows some examples of canonical object views used in our experiments.

---

768
Example:
Unary
top, is sleapiuus) lop ___heargroonn
Predicatoo invalving
M,x MLPs
obyact-rooll
on surfacofolos, leff
emboodiings
odjechiccaion
relations
alend, effector, wfond, offector, grean)
effector
effecctor grean)
1536
Example:
steckedtbiue growe)) smakeddfaiu red) arackedioreen
Binary MLP
(N-1)
M, Z-ayer MLPs
Predicateo invalving otyickatyect refations
wihiptts,, greas) algrest wnlnfotow, algred witffroen, bise)
binary abject embeddings
sjerus, grean)
ayöroe, red)
Afgreon, blus
Figure 4: Architecture of the readout networks, which uses the object embeddings from SORNet to predict spatial relations, such logical statements that The readout network flexible accommodate any number input object embeddings without changing its parameters. My and Mp denote the number unary and binary relation types respectively.
skill preconditions continuous 3D directions.
3.2 Readout Networks
The readout networks (Fig. 4) are responsible for predicting a list of relations using object embed- dings. The relations can be logical statements, e.g., whether the blue block is stacked onto the green block, or continuous quantities, e.g. which direction should the end effector move to reach the red block. The readout networks consist of collection of 2-layer MLPs, Here we focus on unary and binary relations. Unary relations involve a single object or an object and the environment, which could be the robot region the table. Binary relations involve two objects and, optionally, the environment. In principle, our framework is extensible to relations involving more than objects, but we leave that for future work.
for each type of relations.
The readout network for unary relations takes the list of object embeddings and outputs relations pertaining the object that the embedding is conditioned on. Taking the top.is-cledt classifier for an example, if the input embedding is conditioned on the blue block, the network will output whether there is any object top of the blue block. If the input embedding is conditioned on the red block, the network will output whether there any object on top of the red block.
The readout network for binary relations takes list of binary object embeddings created by con- catenating pairs of object embeddings and outputs relations corresponding to a pair of objects, c.g., whether the blue block is on top of the red block. Thus, with V object embeddings, there will be (N - 1) binary object embeddings and N (N - 1) output relations.
Parameters of the readout network are independent of the number of objects. The number of output relations dynamically changes with the number input object embeddings. For example, when are 7 unary relations and binary relations, with 4 objects, the network generates 7 4x +x (4-1) 52 outputs; with 5 objects, the network generates 5 x x (5 - = 75 outputs. In this way, our overall model generalizes zero-shot to scenes with an arbitrary number of objects.
The Leonardo Dataset
To test the generalization of our model to new objects and tasks, we created simulated tabletop environment named Leonardo, where Franka Panda robot manipulates set of randomly colored blocks. The robot is given a goal formulated as a list of predicates simple task planner [5] to find sequence of actions to achieve that goal and sample-base motion planner [31] to generate trajectories and choose grasps. we know the ground truth poses of blocks in the simulator, we can compute ground-truth logical predicates at every step of the planning process. We used NVISII [32] to render the RGB observations. Domain randomization including random lighting, background and perturbations to the camera position is applied while rendering.
satisfied, and then use

---

Initial
Findl
Training scenario (1)
Training scenario (2)
Training scenario (3)
Testing scenario (a)
Testing scenario (b)
Testing scenario (c)
Figure 5: Sample seenes from training and testing scenarios in Leonardo dataset. Top row shows the initial configuration of task scenario and the bottom row shows the goal condition. The training scenarios contains 4 blocks with random colors with single tower stack goal condition. The testing scenarios contain 4-6 blocks with specific colors with various goal conditions involving multi-tower stacking scenarios.
The training data contains 133796 sequences of single task stacking 4 blocks in a tower. The block colors are randomly chosen from 405 xked colors'. The testing data contains 9526 sequences with 4-6 blocks, chosen from colors that are not included in the training data: red. green, blue, yellow, aqua, pink, purple. The testing tasks consists of 7 tasks different from training. Fig. 5 shows some examples. Please see the supplement for full description of the testing tasks.
5 Results
5.1
CLEVR-CoGenT
We first evaluate our approach on a variant of the CLEVR dataset [24], a well established benchmark for visual reasoning. CLEVR contains rendered RGB images with most 10 objects per image. There are 96 different objects total (2 sizes, 8 colors, 2 materials, 3 shapes). Each image is labeled with 4 types of spatial relations (right, front, left, behind) for each pair of objects.
Specifically, we use the CoGenT version of the dataset, which stands for Compositional General- ization Test, where the data is generated in two different conditions. In condition A, cubes are gray, the opposite:
blue, brown, or yellow and cylinders are red, green, purple, or cyan. Condition cubes are red, green, purple, cyan and cylinders are gray, blue, brown, yellow. Spheres can be any color in both conditions. The models are trained on condition A and evaluated on condition B. The training set (trainA) contains 70K images and the evaluation set (valB) contains 15K images. Several prior works [33, 34] show significant generalization gap on CLEVR-CoGenT caused by the visual model learning strong spurious biases between shape and color.
We generate a question for each spatial relation in the image, e.g. "Is the large red rubber cube in front of the small blue metal sphere?" We filter out any query that is ambiguous, e.g. if there were two large red cubes, one in front and one behind the small blue sphere. This results around million questions for both valA and valB sets. We compare against MDETR [34], which reports state-of-the-an zero-shot result on CLEVR-CoGeNT. i.e. there no fine-tuning on any example from condition B. The results summarized in Table 1. Our model performs drastically better on classifying spatial relations unseen objects and shows a much smaller generalization gap between valA and valB sets.
Unlike MDETR which takes text queries, our model takes visual queries in the form of canonical object views (i.e. 2 canonical views for the objects mentioned in the question). To eliminate the in- fluence of those factors, we report the performance of the MDETR model trained on the full CLEVR dataset, denoted MDETR-oracle, We can see that although SORNet is only trained condition A. it is able to achieve similar performance to MDETR-oracke The zero-shot generalization ability of our model can potentially be combined with other reasoning pipelines improve generalization performance on other types of queries as well.
"pttps:/ //wwt////////ee/ wttp:////ccoo///////
5

---

MDETR [34]
MDETR-orach [34] SORNet(ours)
ValA Accuracy 84,950
97.944
99,006
ValB Accuracy
59.627
98.052
98.222
Zero-shot relation classification accuracy on CLEVR-CoGenT [24]. The MDETR-oracle model has seen all objects during training, where as MDETR and SORNet have only see objects condition A. SORNet takes canonical views as queries whereas MDETR takes text queries.
Table I:
5.2 Predicate Classification
Next, we evaluate the task of predicate classification on the Leonardo dataset. We compare against baselines that do not use object conditioning. The first two baselines use a ResNet18 and a ViT-B/32 respectively to directly predict 52 predicates. The last baseline uses the same architecture as ours, but the embedding tokens come from 4 fixed class embedding vectors. We report 3 metrics: accuracy, F-1 score and all-match accuracy across all predicates and among predicates within a category. The accuracy and F-1 score are computed per predicate and averaged. For all-match accuracy, the predicates are considered as a single vector, which most faithfully represents how the predicates are treated by a planner. If any predicate is classified incorrectly, the prediction for the entire category is considered as wrong. The models are tested on images where the objects are completely unseen during training.
The results are summarizzd in Table Models labeled with M-View are trained on different views of the scene as data augmentation. During testing, we aggregate the predictions from 3 views by adding the logits. The M- View models are better at handling occlusions by leveraging multi- nalli-view formation. The gripper state is concatenated to the object embedding in the SORNet variant denoted by (G) in Table 2, which is better at classifying predicates correlated to the opening/closing of the gripper, e.g. has_obj. We can see that the non-object-conditione applied to unseen objects zero-shot. Even after fine-tuning on 100 examples, they still significantly underperform zero-shot SORNet. This demonstrates the generalizability of our model obtained via conditioning on canonical object views.
baselines fail drastically when
Further, we tested our model on scenes with 5 to 6 objects, while it has only been trained on 4-object scenes. We first treat the additional objects as distractors, which shows that it is not necessary to acquire canonical views of every object in the scene if they are not of interest. We then treat all objects as objects of interest. In this case, the number of binary predicates increases quadratically with the number of objects in the scene, so the all-match accuracy naturally drops even if the average accuracy and F-1 score remain the same. Note that none of the baselines can even be applied to these scenes with more objects without introducing additional model parameters and retraining the model.
Finally, we run our best performing model on 30 real-world images of robot performing various manipulation tasks. The quantitative results are in Table 2 and Fig. 6 shows two qualitative examples of the predicates predicted by our model. Our model transfers to real-world without losing much accuracy. It does make some mistakes when encountered with novel scenarios never seen during training. such as one block stacked in between two blocks (right plot in Fig. 6).
5.3 Skill Executability
We further evaluate the predicate prediction in the context of task planning. Each frame in the Lconardo datasct is labeled with a primitive skill that will be executed, c.g. grasp the red block. Each skill has a list of preconditions that needs to be satisfied before it can be executed, which can be formulated as a vector of predicate values. In other words, if all of the predicates in the preconditions are classified correctly, we can determine the executability of that skill correctly. In Table 3, we report the accuracy for classifying the executability of skills in the Leonardo test set using the predicate predictions.
This evaluation puts more emphasis on the predicates relevant to the manipulation of objects, e.g- approach, al gned, which are rarely true in the training data. Our model is able to identify these predicates correctly whereas the baselines fail completely on skills relevant to object manipulation, i.e. grasp, al i gn, pl ce and i ft.

---

Accuracy
Method
pred
all
on.surface
has.obj
topucilear
stacked
aligned
approsach
Majority
52
88.2
79.3
88.4
75.1
91.7
99.2
92.9
8 M-Head O-shot
52
88.5
80.0
88.4
76.2
91.7
99.2
92.9
VIT-B/32 M- V--iew 0-shot
88.5
76.2
91.7
92.9
B/T-B/52 M. M-Ilead M. View D-shot
52
88.5
80.2
88.4
76.0
91.7
99.2
92.9
ResNet18 M-Head 100-shet
52
88.5
80.0
88.4
76.4
91.7
99.1
92.9
VIT-B/32 M-View 100-shot
52
88.6
79.9
88.3
78.7
91.7
99.
92.8
VIE-B/32 M-Head M-View 100-shot
87.8
92.4
SORNet O-shor
97.6
96.9
96.6
98.7
99.2
96.1
SORNet M- View 0-sbot
52
98.9
99.0
95.9
99.2
99.6
99.4
97.3
SORNet View (G) 0-shot
52
98.9
98.9
98.8
98..5
99.5
99.3
96.6
SORNet M-View distractor O-shot
52
98.3
98.6
96.6
95.2
98.6
99.5
97.5
SORNet
Shst
97.7
SORNet M-View (G) 5 obj O-shot
98.5
98.5
99.4
95.8
98.2
99.6
97.4
SORNet M View (C) 6 Orshot
102
98.0
98.3
99.6
93.9
96.8
99.7
97.7
SORNet M-View (G) real-world O-shot
96.3
93.3
97.1
95.6
All-match Accuracy
Method
* pred
all
surface
has
top clear
stacked
aligned
approach
ResNet18
0.0
30.4
71.6
VIT-B/32 M-View 0-shot
52
0.0
0.4
53.5
30.3
30.1
89.8
71.6
VIT-B/32 M-Head M-View 0-shot
0.0
53.5
30.3
30.1
89.8
71.6
R M.Head 100-shot
52
0.0
0.3
$3.5
30.9
30.1
89.8
71.6
VT--B/32 M-Vicw 100-shot
0.2
55.0
40.5
31.0
72.5
VIT-B/12 M.Heaal M.View 100
52
4.1
21.7
62.4
63.8
40.0
89.8
75.0
SORNet O-sbot
52
43.8
66.4
81.4
87.8
87.9
90.5
86.1
SORNet M. View O-shot
52
60.2
86.3
83.5
96.8
95.8
92.4
89.5
SORNet M- Vd-View (G) 0-shot
63.6
84.5
95.8
94.0
94.3
92.0
86.6
SORNet M.View
50.1
91.0
SORNet M- View 2 distractors 0-shot
52
39.2
76.3
87.2
68.2
78.3
95.0
90.4
SORNet M. View (G) 5 obj 0-shot
70
45.5
74.8
97.6
81.1
72.6
92.0
87.5
SORNet M-View (G) 6 obj 0-shot
102
29.7
68.9
97.7
70.2
52.4
92.0
87.5
SORNet M. View (G) real world O-shot
52
26.7
63.3
93.3
80.0
76.7
90.0
80.0
F.1 Score
Method
# pred
all
on.surface
has.obj
top-clear
stacked
aligned
approach
R M-Head O-shot
52
9.7
23.6
0.0
31,5
0.0
0.0
0.0
VIT-B/32 M- Liew 0-shot
52
11.9
37.4
28.9
0.0
0.0
0.1
VIT-B/32 M-Ileud Viewew Dishot
52
12.2
32.5
27.7
0.0
0.0
R M-Head 100-shet
0.0
32.6
0.0
0.0
VIT-B/32 M. Niew 100-shol
52
0.0
37.7
6,3
46.5
0.0
0.0
7.3
M-Head M-View shot
0.0
73.2
27.2
SORNet O-shot
52
83.2
92.2
79.7
93.0
91.2
63.8
74.9
SORNet View 0-sbot
88.9
97.5
82.0
98.4
97.3
70.5
81.7
SORNeL M View (G) 0-shot
52
89.5
97.1
94.7
96.8
96.4
69.9
76.7
SORNet M-View distractor (-shot
52
85.1
96.3
81.8
90.1
88.8
67.5
80.1
SORNet View 2 distractors O-shot
52
80.2
95.2
80.2
79.4
81.4
60.7
75.3
SORNet M-View (G) 5 obj O-shot
70
85.3
96.0
96.7
91.3
83.6
69.8
78.1
View obj
87.5
69.2
77.9
SORNet View (G) seal-world O-shot
76.5
90.7
85.7
80.3
69.1
68.9
Table 2: Zero-shot predicate classification results on the Leonardo test data where the objects are held-out from training. SORNet significantly outperforms baselines that are not abject-conditioned, and is able generalize to scenes with a different number of objects without retraining or finetuning.
surteceggren, Modi lett) goo an suttacejjee act, fwrt www.iatttyytoooo....... bock. top
on earfecered Mook nahd)
nigh)
wo in cleatted plock)
green block)
toy Ruskousoorin Booo (deak) dissttytiion beekt
stackeolgronn Mock bux Llod) eaceeddytiooe black, block)
Jucrrush pwwper/obaai..ooooooo rod, slock) stcceedggronn blod biue block)
Figure 6: Visualization of predicted predicates in real-world scenes. Black means true positive, blue means false positive and red means false negative. True negatives not shown due limited space.
7

---

Accuracy
Method
all
approach
grasp
align
place
lift
go.home
ResNet18 M- M-View
27.0
88.7
0.0
0.0
0.0
0.0
100.0
ViT-B/32 M-View
27.4
90.6
0.0
0.0
0.0
0.0
100.0
ViT-B/32 M-Head M-View
27.3
90.4
0.0
0.0
0.0
0.0
100.0
SORNet SORNet M-View
63.7
64.3
72.4
67.4
99.4
53.0
97.6
63.6
65.1
69.1
99.8
52.9
69.7
99.5
SORNet M-View (G)
76.3
98.7
68.1
99.9
99.9
69.7
100.0
Skill exccutability accuracy on the Leonardo test data. SORNet is the only model that able to correctly determine the exccutability of skills relevant to the handling of objects.
Table 3:
Method
ResNet18
8 (MV)
ResNet18 (P)
CLIP-VIT
CITP- (P)
SORNer (P)
SORNet (P MV)
Obj Obj
0,4308
0.6068
0,9876
0.9875
0.6145 0.4960
0.1679
0.1458
FF-Ohj 0.3251
0.3464
0.5929
0.6544
0.1962
0.1777
Table Euclidean error on regression of continuous 3D unit vector between entities in scene. The regres- sors are trained on 1000 examples with unseen objects. Methods labeled with are pretrained on the Leonardo dataset. Methods labeled with MV use 3 views.
5.4 Open Loop Planning
In this demo, we incorporate SORNet as part of an open-loop planning pipeline a real-world ma- nipulation scenario. Specifically, given an initial frame, we use the predicates predicted by SORNet M-Viev (G) to populate a state vector. A task and motion planner takes the state vector and desired goal (formulated
list of predicate values be satisfied), and outputs sequence of primitive skills. The robot then exccutes this sequence skills in an open loop fashion. This demonstrate that how SORNet can be applied to sequential manipulation of unseen object in a zero-shot fashion, i.c. without any fine-tuning on the test objects. Please refer to our supplementary video for the demo.
5.5 Relative Direction Prediction
Although the training objective of SORNet is purely logical, with large-scale pre-training, the object embeddings learned by SORNet contains continuous spatial information. We demonstrate this fact by using SORNet embeddings to predict the relative 3D direction between entities. Specifically, we trained a regressor (same architecture as the classifier in Sec. 3.2) on top of frozen SORNet embeddings to predict the continuous direction between two objects (Obj-Obj) or the direction the end effector should move to reach a certain object (EE-Obj). The regressor is trained using L2 loss on 1000 examples with unseen objects, and tested on 3000 examples with the same objects.
The results are summarized in Table 4. SORNet is able to outperform models trained from scratch with pose supervision (ResNet18) as well as models initialized with weights from large-scale Lunguage-image pretraining [35] (iT).-ViT). This demonstrates that our representation learning technique is more suited to manipulation scenarios where more precise spatial information is cru- cial. In our supplementary video, we showed that by predicting the direction for movement in an online fashion, model can be used for visual servoing to guide the robot reach target object.
Conclusion
We proposed SORNet (Spatial Object- Ceneric Representation Network) that learns object-centric representations from RGB images. We show that the object embeddings produced by SORNet cap- ture spatial relations which can be used in a downstream tasks such as spatial relation classification, skill precondition classification and relative direction regression. Our method works on scenes with an arbitrary number of unseen objects zero-shot fashion. With real-world robot experiments, we demonstrate how SORNet can be used in manipulation of novel objects.

---

Acknowledgments
This work was supported in part by the National Science Foundation under Contract NSF-NRI- 2024057 and in part by Honda Research Institute as part of the Curious Minded Machine initiative.
References
[1] R. Fikes and N. J. Nilsson. Strips: new approach to the application theorem proving to problem solving. Artificial intelligence, 2(3-4):189-008, 1971.
[2] M. Fox and D. Long. PddI2. 1: An extension to pddl for expressing temporal planning domains. Journal of artificial intelligence research, 20:61 1-24, 2003.
[3] S. Balakirsky, Z. Kootbally, C. Schlenoff, T. Kramer, and S. Gupta. An industrial robotic knowledge representation for kit building applications. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1365-1370. IEEE, 2012.
[4] F. Rovida. B. Grossmann, and V. Krüger. Extended behavior trees for quick definition of flexible robotic tasks. In 2017 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 6793-6800. IEEE, 2017.
[5] C. Paxton, N. Ratliff, C. Eppner, and D. Fox. Representing robot task plans robust -yniicll--yamicall systems. 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5588 5595. IEEE, 2019.
[6] Z. Sui, L. Xiang, C. Jenkins, and Desingh. Goal-directed robot manipulation through axiomatic scene estimation. The International Journal of Robotics Research, 36(1):86-100 10486-14,,55 2017.
[7] Y. Xiang, Schmidt, V. Narayanan, and D. Fox. Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes. arXiv preprint arXiv:1711.000 2017.
[8] Sundermeyer, Z.-C. Marton, M. Durner, M. Brucker, and Triebel. Implicit 3d orientation learning for 6d object detection from rgb images. In Proceedings of the European Conference on Computer Vision (ECCV), pages 699-715- 2018.
[9] Y. Li, G. Wang. X. Ji, Y. Xiang, and D. Fox. Deepim: Deep iterative matching for 6d pose estimation. In Proceedings of the European Conference Computer Vision (ECCV), pages 683-698, 2018.
[10] X. Deng, A. Mousavian, Y. Xiang, F. Xia, T. Bretl, and D. Fox. Poserbpf: A rao-blackwellized particle filter for 6-d object pose tracking. IEEE Transactions on Robotics, 2021.
[11] C. Xie, Y. Xiang, A. Mousavian, and D. Fox. The best of both modes: Separately leveraging rgb and depth for unseen object instance segmentation. In Conference on robot learning, pages 1369-1378. PMLR, 2020.
[12] Y. Xiang, C. Xie, A. Mousavian, and D. Fox. Learning rgh-d feature embeddings for unseen object instance segmentation. arXiv preprint arXiv:2007.15157. 2020.
[13] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 000900299900... 2015.
[14] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-137, 2016.
[15] Y. Duan, M. Andrychowicz, B. C. Stadie, Ho, J. Schneider, I. Sutskever, Abbeel, and W. Zaremba. One-shot imitation learning. In NIPS, 2017.
[16] A. Ghadirzadch, A. Maki, D. Kragic, and M. Björkman. Deep predietive policy training using reinforce- ment learning. In 2017 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 3351-2356, IEEE, 2017.
[17] D.-. Huang, S. Nair. D. Xu, Y. Zhu, A. Garg, L. Fei-Fei, S. Savarese, and J. Niebles. Neural task graphs: Generalizing to unseen tasks from a single video demonstration. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8565-8574, 2019.
[18] C. Paxton. Y. Bisk, J. Thomason, A. Byravan, and D. Foxl. Prospection: Interpretable plans from language by predicting the future. In 2019 International Conference on Robotics and Automation (ICRA), pages 6942-6948. IEEE, 2019.

---

[19] T. Yu, P. Abbeel. S. Levine, and C. Finn. One-shot hierarchical imitation learning of compound visuomo- tasks. arXiv preprini 1 2018.
[20] B. Rosman and S. Ramamoorthy. Learning spatial relationships between objects. The International Journal of Robotics Research, (31(11)::328-3330 2011.
[21] S. Fichtl, A. McManus, W. Mustafa, D. Kraft, N. Krüger, and E. Guerin. Learning spatial relationships from 3d vision using histograms. In 2014 IEEE International Conference Robotics and Automation (ICRA), pages 501-508. IEEE, 2014.
[22] M. Sharma and O. Kroemer. Relational learning for skill preconditions. arXiv preprint arx1v:0012.01693, 2020.
[23] K. Kase, C. Paxton, Mazhar, T. Ogata, and D. Fox. Transferable task execution from pixels through deep planning domain learning. 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 10459-10465. IEEE, 2020.
[24] Johnson, B. Hariharan, L. van Maaten, Fei-Fei, L. Zitnick, and R. Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017.
[25] Yi, C. Gan, Li, Kohli, J. Wu, A. Torralba, and J. B. Tenenbaum. CLEVRER: collision events for video representation and reasoning. In ICLR, 2020.
[26] R. Girdhar and D. Ramanan. CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning. In ICLR, 2020.
[27] Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, N. Gomez, Kaiser, and I. Polosukhin. Atteation all you need. arXiv preprint arXiv:1706.03766 2017.
[28] D. Ding, Hill, A. Santoro, and M. Botvinick. Ohject-based attention for spatio-temporal rea- soning: Outperforming neuro-symbolic models with flexible distrihuted architectures. arXiv preprint arXiv:2012.08500. 2020.
[29] H. Zhou, A. Kadav, Lai, A. Niculescu-Mizil, M. R. Min, M. Kapadia, and H. P. Graf. Hopper: Multi- hop transformer for spatiotemporal reasoning. arXiv preprint 10Xiv:2003.10544 2021.
[30] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghami, M. Min- derer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 19999, 2020.
[31] C. R. Garrett, T. Lozano-Pérez, and L. P. Kaelbling. Pádistream: Integrating symbolic planners and blackbox samplers via optimistic adaptive planning. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 30. pages 440-448. 2020.
[32] N. Morrical, J. Tremblay, S. Birchfield, and Wald. VISII: Virtual imaging interface, 2020. https: / com/ www//wwooowooo//o.../..ooo..o..o www--wooeeetteeeeeoooeo jeeeejettiiiiie /VSSII/.
[33] K. Yi, J. Wu, C. Gan, A. Torralba, P. Kohli, and J. B. Tenenbaum. Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. arXiv preprint urXiv:1810.02338,, 2018.
[34] A. Kamath, M. Singh, Y. LeCun, I. Misra, G. Synnaeve, and N. Carion. Mdetr-modulated detection for end-to-end modal understanding. arXiv preprint arXiv:2104. 127633 2021.
[35] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, Askell, P. Mishkin, Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint 1eXiv:2103.00020 2021.
10


Reviews:
[{"review_id": "l2H-akOJTtI", "reviewer": "Reviewer_nXEV", "summary": "This paper introduces a network architecture based on vision-transformers to produce object-centric latent embeddings, conditioned on canonical views of the objects in question. The learned embeddings are shown to be useful to estimate logical predicates that can be used for question answering and symbolic planning. Particularly interesting is their generalization to novel objects, settings, and from simulation to a real robot setting.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper presents a very interesting idea that is at the heart of where current approaches to robot learning struggle: generalization to unseen objects and settings. The proposed method is novel but not overly complex (which is great). The method is tested creatively in a variety of scenarios including on a real robot. The paper is also very well organized and written. I was not able to find some details of the experiments (see below) but I am sure that this can be easily fixed.\n\nThe part that I am most worried about are the different baselines. As the paper uses novel tasks or task variations that others have not published on, it is important to make sure that the baselines are the right ones and that they have been properly tweaked and optimized. Based on the text, I am not fully convinced that this is the case here (see below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This is an overall very interesting paper with novel ideas that could have a significant impact on the field. Right now, there is some information missing from the experimental section and the baselines have to be improved. If these issues are resolved, I would be happy to update my rating accordingly.\n\nEDIT: The issues were addressed by the rebuttal.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "## CLEVR experiments\n\nL159: For the 2 million questions, are only the validation parts of condition A and B counted? How many questions are used for training?\n\nHow are the canonical object views generated here? As mentioned in the paper, providing these, simplifies the task compared to the baseline as the network does not need to infer visual appearance from text. This discrepancy and the fact that the evaluation does not use the questions from the dataset should be clarified in Table 1. Are there baselines that also take in canonical views and therefore would allow a better direct comparison?\n\nAlso, are the questions turned into predicates for the proposed method but asked as text to MDETR? Could this cause issues in the comparison?\n\nTable 1: Why does MDETR-oracle achieve a higher ValA accuracy that MDETR? Shouldn’t training on both conditions only help in condition B? Condition A is already used in the non-oracle version, right? Or is this version also trained on validation data in addition to training data? Either way, it would be good to clarify this in Table 1. Also, would it make sense to add a SORNet-oracle variant to this comparison? Why or why not?\n\nL167-169: Based on the results, I agree that the bottleneck is generalizability, but I do not think that the data proves that “generalizability of the visual model” is the bottleneck. It could also be generalizability between the concrete scenarios and questions, e.g. between training and validation splits.\n\n## Predicate Classification\n\nHow do the baseline methods classify predicates for unseen objects (L184) if they are not conditioned on these objects?\n\nTable 2: The fact that the baselines do not outperform simple majority prediction indicates that there might be an issue with training the baselines on the imbalance dataset. Did the authors attempt to address label imbalance, e.g. by adapting the sampling between positives and negatives or by weighing the loss terms appropriately?\n\n## Relative Direction Prediction\n\nI am not sure how the relative direction prediction task is formulated here, especially for the baselines. Are the objects fixed and the models always predict the direction between all pairs of objects?\n\nTable 5b shows that the baselines overfit here, which is not surprising given the small dataset size used in this experiment. However, there are a number of ways to reduce overfitting, such as early stopping, regularization losses, dropout etc. Did the authors apply any of these techniques to reduce overfitting in the baselines?\n"}}, {"review_id": "Qx9fIFY0EOY", "reviewer": "Reviewer_eyTp", "summary": "The paper presents Spatial Object-Centric Representation Network (SORNet), an approach that takes an RBG image and several canonical object views, and outputs object-centric embeddings which are then used in 3 spatial reasoning tasks, including: spatial relationship classification, skill precondition classification and relative direction regression. In addition to this, real-world robot manipulation experiments are shown to demonstrate the ability to use these representations for task planning.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The system offers a flexible way to accommodate a number of objects in the scene.\n- The paper is clear and presents thorough results.\n- Convincing real world results.\n\n**Weaknesses**\n\n- This list of contributions seem superfluous. To recap, the states contributions are (1) architecture, (2) dataset, (3) experiments, (4) real robot demo. It's not clear to me how the architecture is a contribution, as it is essentially just a Visual Transformer (ViT). Moreover, experiments to back up your method should not be counted as a contribution; these are required in any paper to support a presented method. I would argue that there are only 2 real contributions here: the method and the dataset.\n- Related work could use a bit more attention; currently only has a small paragraph each on spatial and visual reasoning. What about considering other sequential manipulation papers that don't fall within these areas, and explain why perhaps Spatial Object-Centric Representations are better suited for the style pf problem?\n- Relative Direction Prediction experiment need to be motivated more; it's not clear why this is  useful thing to learn and doesn’t seem to fit with the rest of the paper. Perhaps this can be expanded upon. However, the video does briefly motivate with a visual servoing section.\n\n**Minor**\n- Missing domain rand citations (particularly in the manipulation domain); more details on the domain randomisation? Amount of noise, etc\n- Typo \"… direction between in the 3D space …\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper has limited technical novelty however, the proposed system instead gives strong experimental results.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "**Questions** \n- After you get the plan, how do you locate the pose of the blocks in the real world? This detail seems missing in the real-world section. ", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "KL48HMfJT0U", "reviewer": "Reviewer_zJy5", "summary": "This paper addresses the problem of classifying objects' spatial relationships from vision. To do this, it proposes a novel architecture which takes in the scene image along with close ups of each object and embeds each object view and each image patch into an embedding space. A transformer jointly processes all of these embeddings to output one embedding per object, which can then be passed to downstream tasks, such as classifying spatial relations. The results show a high level of performance on a block rearrangement task and on the clevr dataset.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed method enables learning object agnostic relationships by conditioning on the specific objects at inference time. \n- The real world robot results show that this model can transfer well form sim to real and is robust enough for planning through stacking tasks.\n- The clevr results show the model generalizes to novel object-attribute compositions. \n- While this model will not work for all manipulation tasks, it looks like this could be used to quickly put together controllers for new tasks which could be useful for robotics practitioners.\n\nLimitations:\n- As proposed, the method does not address the case of objects appearing different from different angles, or the case of needing to take the object's pose into account when manipulating it (e.g. a trapezoid or a long prism).\n- The results as shown require a high supervision burden of labeling object relationships explicitly. Many prior approaches in this area (e.g. relation networks, [25]) instead aim to learn these relationships implicitly from sparser supervision. \n\nQuestions:\n- If the objects look different at different orientations, a single canonical view could be insufficient for finding the object in the scene. I wonder if simply averaging the embeddings from multiple views would be enough to address this limitation. This would be an interesting experiment to improve the paper. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I like the method and overall results. The model is novel but not groundbreakingly so, and has only been shown to work in highly supervised settings. However, the object generalization and real world results are promising for future work.\n\n+++\nThe update from the authors has addressed my comments and I can strongly recommend acceptance.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "The paper does not compare to any existing object-attention based methods, nor does it show any results with realistic objects (vs solid colored, convex, geometrical shapes). Although the paper writes that [24] \"assume a segmentation model\", my understanding is that [24] uses an unsupervised learning method to learn the segmentations. Other than this, the model looks very similar to SORNet, so some further explanation of sets SORNet apart would help explain the novelty. \n"}}]

Author rebuttal:

Meta-review: Post-rebuttal, there is clear and consistent agreement from all three reviewers that this new method is useful and novel, and potentially important. This is a well-executed paper built around a cool idea that follows through to real-robot experiments.
Decision: Accept (Oral)