Toward Learning Geometric Eigen-Lengths Crucial for Robotic Fitting Tasks
Authors: Yijia Weng
Kaichun Mo
Ruoxi Shi
Yanchao Yang
Leonidas Guibas
Venue: Submitted to ICLR 2023 (2023)
Keywords: Visual Representation Learning
Shape Understanding

Abstract: Some extremely low-dimensional yet crucial geometric eigen-lengths often determine whether an object can be fitted in the environment or not. For example, the {\em height} of an object is important to measure to check if it can fit between the shelves of a cabinet, while the {\em width} of a couch is crucial when trying to move it through a doorway. Humans have materialized such crucial geometric eigen-lengths in common sense since they are very useful in serving as succinct yet effective, highly interpretable, and universal object representations. However, it remains obscure and underexplored if learning systems can be equipped with similar capabilities of automatically discovering such key geometric quantities in doing robotic fitting tasks. In this work, we therefore for the first time formulate and propose a novel learning problem on this question and set up a benchmark suite including the tasks, the data, and the evaluation metrics for studying the problem. We explore potential solutions and demonstrate the feasibility of learning such eigen-lengths from simply observing successful and failed fitting trials. We also attempt geometric grounding for more accurate eigen-length measurement and study the reusability of the learned geometric eigen-lengths across multiple tasks. Our work marks the first exploratory step toward learning crucial geometric eigen-lengths and we hope it can inspire future research in tackling this important yet underexplored problem. 



Under review as a conference paper at ICLR 2023
TOWARD LEARNING GEOMETRIC EIGEN-LENGTHS CRUCIAL FOR ROBOTIC FITTING TASKS
Anonymous authors Paper under double-bling review
ABSTRACT
Some extremely low-dimensional yet crucial geometric eigen-lengths often deter- mine whether an object can be fitted in the environment or not. For example, the height of object is important to mcasure to check if can fit between the shelves a cabinet, while the width of a couch is crucial when trying to move it through doorway. Humans have materialized such erucial geometric eigen-lengths in common sense since they are very useful in serving as succinct yet effective, highly interpretable, and universal object representations. However, remains obscure and underexplored if learning systems can be equipped with similar capabilities of automatically discovering such key geometric quantities in doing robotic fitting tasks. In this work, we therefore for the first time formulate and propose a novel learning problem on this question and set up benchmark suite including the tasks, the data, and the evaluation metrics for studying the problem. We explore poten- tial solutions and demonstrate the feasibility of learning such eigen-lengths from simply observing successful and failed fitting trials. We also attempt geometric grounding for more accurate eigen-length measurement and study the reusability of the learned geometric eigen-lengths across multiple tasks. Our work marks the first exploratory step toward learning crucial geometric eigen-lengths and we hope it inspire future research in tackling this important yet underexplored problem.
INTRODUCTION
Fit under the table:
Place into the box: width and length
Put into and close the drawer:
Stack plates: diameter
height and width
height, width, and length
(a)
(b)
(c)
(d)
Figure 1: Example tasks and the hypothesized crucial geometric measurements by humans.
Consider a robot tasked with placing many small objects on warehouse shelves, where both the objects and the shelves have diverse geometric configurations. While the robot can simply try to accomplish the task by trial-and-errol be attempted because they will obviously fail. For example, we should not attempt to place tall object on shelf whose height is too low. We base this judgement on the estimation of critical geometric eigen-length or measurement, the height of the object and the shelf, whose comparison allows a quick estimate of task feasibility.
us as humans. it clear that certain placements should not
While object height is example of important eigen-lengths of an object that is crucial for the above shelf placement task, it is not hard to think of many other types of object eigen-lengths for other fitting tasks. Figure presents some other example tasks together with the presumable geometric eigen-lengths based on human common sense. For example, the geometric eigen-length diameter is important for the task of stacking plates in different sizes (Figure 1] (a)), while the width and length

---

Under review as a conference paper at ICLR 2023
of object are crucial geometric eigen-lengths for deciding if one can put arbitrary shape object into an open box (Figure [ (c)).
Having such extremely low-dimensional yet crucial geometric eigen-lengths extracted as the repre- sentations for objects is certainly beneficial for designing learning systems for robotic fitting tasks. One telling evidence is that we humans have naturally built up the vocabulary of geometric key quantities, such as height, width, and diameter, when perceiving and modeling everyday objects, and used them perform various object fitting tasks. Besides being succinet yet effective abstractions of objects for quickly estimating the feasibility for the downstream fitting tasks, such crucial geometric eigen-lengtin are also highly interpretable, which exposes the principled reasoning process behind the feasibility checking, and universal, as they are generally applicable to objects with arbitrary shape and useful across different downstream tasks.
Current research in representation learning for computer vision and robotics has mostly been focusing on learning nigh-dimensional latent codes or injecting human knowledge as inductive bias for learning structured representations. While learning high-dimensional latent codes provides total flexibility learning any useful feature for mastering the downstream tasks, these latent codes are high- dimensional, hard to interpret, and may be prone overfitting to the training domain. For structured representations, though researchers have explored using different kinds of object representations, such as bounding boxes (Tulsiani et al. 2017) and key points (Manuelli al. 2019), to accomplish various downstream tasks in computer vision and robotics, these structure priors are manually specified based on human knowledge about the tasks. In contrast, we aim to explore the automatic discovery of low-dimensional yet crucial geometric quantities for robotic fitting tasks while injecting the minimal human prior knowledge- only assuming that we are measuring eigen-lengths of the input objects.
In this paper, we first propose to study a novel learning problem on discovering low-dimensional geometric eigen-lengths crucial for fitting tasks and set up the benchmark suite for studying the problem. As illustrated in Figure 2 given a fitting task (putting the bowl inside the drawer of the table) that involves a scene geometry (the table) and an object shape (the bowl). we are interested in predicting whether the object can fit in the scene accomplishing the task or not, via discovering a few crucial geometric eigen-lengths and composing them into task program which outputs the final task feasibility estimation. To study the problem, we also define set of commonly seen robotic fitting tasks, generate large-scale data for the training and evaluating on each task, and set up a set of quantitative metrics for evaluating and analyzing the method performance and if the emergent geometric eigen-lengths match the desired ones humans usually use.
We also explore potential solutions to the proposed learning problem and present several of our key findings. First of all, we will show that learning such low-dimensional key geometric eigen-lengths are achievable from only using weak supervision signals such as the success failure of training fitting trials. Secondly, the learned crucial geometric eigen-lengths can be more accurately measured if geometric grounding allowed and attainable for certain fitting tasks. Finally, we make an initial stab at exploring how to share and re-use the learned geometric eigen-lengths across different tasks and even for novel tasks. Marking the first step defining and exploring this important yet underexplored problem, we hope our work can draw people's 's attention to this task and inspire future research in designing solutions tackling it.
To summarize, this work makes the following contributions:
We propose a novel learning problem on discovering low-dimensional geometric eigen-lengths crucial for fitting tasks;
We set up a benchmark suite for studying the problem, including a set of fitting tasks, the dataset for each task, and a range of quantitative and qualitative metrics for thorough performance evaluation and analysis;
We explore potential solutions to the proposed learning problem and present some key take-away messages summarizing both the successes and unresolved challenges.
2 RELATED WORK
Learning Geometry Abstraction. A long line of research has focused on learning fow-dimensiona. and compact abstraction for input geometry. Given as input a 2D 3D shape, past works have studied learning various geometric abstraction as the shape representation, such as bounding boxes (Tulsiani
et al. 2017 Sun al. 2019), convex shapes (Deng et al. 2020), Gaussian mixtures (Genova et al.
2

---

Under review as a conference paper at ICLR 2023
Eigen-Lengths
Program
AND
Output
Task
Geometry
OBJW
Proposed Learning Paradigm where we first predict a set of geometric eigen- from the input geometries, then compose them using a task program to get the final task output.
Figure 2:
2019 2020), superquadratics (Paschalidou al. 2019 2020),
parametric curves (Reddy al. 2021
and surfaces (Sharma et al. 2020; Smirnov et al. (020)) elc.. Most of these works use geometry fitting as the primary objective. Our work, however, focuses on discovering geometric abstraction that can help solve the downstream manipulation tasks instead of reconstruction.
There are also previous works exploring ways to learn task-specific geometry representation for manipulation tasks. For example, researchers have tried to learn key points (Manuelli et al.) 2019 Qin
al. 2020 Wang al. 2020] Chen et al.] [020 Jakab et al. [202]] Chen et al. 2021] and affordance information (Kim & Sukhatme 2014] Mo al., 202 la by Turpin et al. 2021 Deng al. 202I 1) for robotic manipulation tasks. These works mostly pre-define the types geometry abstraction and the
downstream policies to use the extracted shape summaries, and the abstraction is mostly dense or high dimensional. In this paper, we aim for useful geometric lengtts and ways to automatically discover and compose them for solving manipulation tasks.
Disentangled Visual Representation Learning. Another line of work focuses on unsupervised
representation learning techniques that pursue disentangled and compositional latent representations for visual concepts. For example, InfoGAN (Chen et al. (0166). beta-VAE (Higgins et al. (017), and many more works (Higgins et al., 2016 Siddharth et al. 2017 Yang et al. (020)) discover disentangled features, each of which controls certain aspect of visual attributes, usually with reconstruction as the objective. In contrast to their primary objectives of controllable reconstruction or generation, we explore the problem of learning geometric eigen-lengths driven by the goal of accomplishing downstream fitting tasks. Also, our task involves reasoning over two geometric inputs and comparing the extracted eigen-lengths on both inputs, while these previous works on disentangled visual representation learning factor out visual attributes for a single input datum.
3 LEARNING PROBLEM FORMULATION
Given robotic fitting task T E T, we aim to learn very few but the crucial geometric eigen-lengths LT (e.g., width, length. height) of the object shape O E O and the environment geometry E E that are useful for checking the feasibility of fitting O into E under the task T'. Figure 2 presents an example of the proposed learning problem where the task is to put the bowl (O) inside the drawer of the cabinet (E). In this example, the width, length, height of the drawer and the bowl are the crucial desired geometric eigen-lengths (Lr) and we can compose them in a task program to output the final task feasibility prediction. We consider each eigen-length L LT function mapping from the input object shape O or the environment geometry E to scalar value for the eigen-length measurement, i.e. : O U + R. After obtaining the eigen-length measurements for both the object and environment inputs, i.e. {L(O) E L and {L(E) E Lr}, we perform pairwise comparisons between the corresponding eigen-lengths checking if L(O) < L(F)) holds for every E Lr. The task of fitting O in E under the task 7 is predicted as successful if all the conditions hold and as failed if any condition does not hold. This format of task program is based on the intuition that in fitting tasks, we require the object to be "smaller" than the parts of the environment affording the action. Durining training, the learning systems see many fitting trials over different objects and environment geometric configurations together with their ground-truth fitting feasibility, i.e. {(O,, E,, Successful /Faieed)) = 0, 1, 2, ... }. The goal is to learn eigen-lengte functions based on which correct prediction of task feasibility given test input (Ores:; Frest) can be made.
3

---

Under review as a conference paper at ICLR 2023
4 CAN GEOMETRIC EIGEN-LENGTHS BE LEARNED FROM BINARY TASK SUPERVISION?
In this work, we are interested in learning geometric eigen-lengths that are crucial for downstream tasks. We hope to achieve automatic discovery of these eigen-lengths from doing tasks as it requires the least human prior and allows maximum flexibility. Therefore, we start with the minimum form of supervision and explore the following question: given only binary task success/failure supervision, is it possible to learn geometric eigen-lengths of input geometries that are sufficient for the task?
4.1
TESTBED FOR EIGEN-LENGTH LEARNING
width
longtth
height
length
holder radius
height
Fedius
(4) Tube Passing
(6) Cylinder Fitting
(6) Sphere Fitting
66) Container Fitting
(e) Countentop Placing
(1) Mug Hanging
Figure 3: Summary of tasks and their huuthnnhypotheszeed key mecssuemmntt/cigee-eenngg
We start by curating set of tasks the testbed for the learning problem, summarized in Fig. For each task, we build a large-scale dataset comprising diverse shapes and configurations.
Task Design Principles We design the tasks to (1) cover a wide range of geometries, including synthetic, simple primitive shapes and more complex ones like ShapeNet objects; (2) facilitate the analysis and interpretation of learned eigen-lengths. Specifically, here we base the analysis on comparisons to numan-hyyotthsizzed eigen-lengths: given a task, humans can identify related key eigen-lengths (referred to as "ground truth" in the following), e.g., object height when putting them on shelves. Comparing the learned eigen-lengths to these "ground truth" may provide important insights. To achieve this, we need accessible ground truth eigen-lengths to begin with. Primitive shapes like cylinders are ideal as they are parameterized by eigen-lengths like radius and height.
Task Specifications
In all tasks, we aim determine whether a placement/mottin of the object exists in a certain environment, specifically:
(a) Tube passing, (Tube) Pass an object through rectangular tube. A tube is a cuboid without the front and back faces. Width and height of the tube/object are the key eigen-lengths to compare.
(b) Cylinder fitting. (Cylinder) Place an object into a cylindrical container. Bounding sphere radius of the object in XY plane and its height, as well as the radius and height of the cylinder container are the key ligen-lengths.
(c) Sphere fitting. (Sphere) Place an object into a spherical container. Radius of the bounding sphere of the object and the container is the key eigen-length.
(d) Container fitting. (Fit) Place an object into cavities in another ShapeNet container object. Example cavities include drawers shelves (See Fig. 3 of furniture. Most cavities have cuboid-like shapes. Thus, key eigen-lengths are width, length and height of cavities and objects.
(e) Countertop placing. (Top) Place an object on top of another Shape environment object, such that its projection along the gravity axis is fully enclosed by the environment countertop. Width and length of the countertop surface and the object are key eigen-lengths.
(I) Mug hanging. (Mug) Hang mug on a mug holder by its handle. The holder is cylinder- shaped rod. Key eigen-lengths are the distance between sides of the mug handle and the diameter (or equivalently, the radius) of the mug holder rod.
For objects to be fitted in tasks (a)-(c), we 1200 common household object models from training and 4 testing categorics in ShapeNet (Chang et al. 2015). following Mo et al. (2021b). During data generation, we apply random scaling to the object model, then sample N 1024 points from the object surface. Note that we also apply random rotation to the object. In (e), we use furniture and appliances from ShapeNet as the environment geometry, including 550 shapes from 7 object categories. In (f), we use ~00 ShapeNet mugs. We randomly sample the parameters of primitive shapes and the scaling factors of ShapeNet shapes, then sample M 1024 points from their surfaces. For all tasks, we generated 75k training and 20k testing environment-object pairs. Please refer to Appendix A.3 for more data generation details.
Data Generation Details
4

---

Under review as a conference paper at ICLR 2023
Faine
Onjhit
1 "
OM Cloud
VectorNiet
Vectory
Projection Langth
Ooj Eipen-Jagth
Prod
(AND))
Foire
Envilet
L
Poloc Clood
WeightNee
Distms
Sum
P.g
L+ v'le-- P)
Highnilenghh
Figure 4: Network architectures. (a) A minimal eigen-length learning pipeline where we separately encode environment and object into eigen-length values, perform pair-wise comparison, and take the logical AND of results. (b) A eeomeery-groundee framework where we first predict vectors and points as the geometry grounding, then compute eigen-length from them.
2rodcton2
R2 m 0.9970
Prodzton
R2 0.9970
Proditio22
R2 = 0,9786
Preduction
R2 0.9933
Predition
R2 0.9972
Prodction
R2 0.9830
6. to
-2.5
1.0
0.5
1.5
0,5 1.5
Tube
Tube
(b) Cyfindder R
(1) Cylindtr
Sphere
(1) Mug Horder
R2 - 0.80 14
R2 0.8890
0.8938
R2 0,9116
0.9360
R2- 0.6485
Prediaton
Prnd
a
Prnd
X
Ooject
FIL Object
(e) Top, Oiject
Top, OsjectY
Top. Object
Figure 5: Correlation Analysis. Each plot shows the relationship between one learned eigen-length (Y coord.) and its matching "ground truth" measurement (X coord.). Higher It2 values imply stronger correlation.
4.2 A MINIMAL NETWORK ARCHITECTURE
Intuitively, we can measure the object and the environment separately and see if the object is "smaller" than the environment. Thus come up with the minimal network architecture shown in Fig. 4 (a). We separately map the object and environment geometries into two sets of eigen-lengths, perform pairwise comparisons between them, and compose comparison results using logical AND.
we encode object point cloud O and environment point cloud E using two Point- Net (Qi et al. 2017) networks, ObjNet and EnvNet. Both networks output S-dim vectors L Lob),, Leiv (L LTTe ,..., ,...,, Lgd). We then compute task success 2 [T (F) > T.ghi (O)). During training, we use differentiable approximation T(E, 0) = II= 1== ( (L eene (E) L (O))/T), where is a learnable parameter. We set S = 1 for (c) Sphere, (f) Mug, S 2 for (a) Tube, (b) Cylinder, S = 3 for (d) Fit, (c) Top.
Concretely,
Lobj = bb Lob
as (E, 0) N==
4.3 ANALYSIS OF LEARNED EIGEN-LENGTHS
We analyze the eigen-lengthe learned by the network by comparing them to "ground truth" eigen- lengths shown in Fig. 3 For each task, we randomly sample N 512 test data points and obtain the corresponding N eigen-length predictions ( Ji= for each of the S learned eigen-lengths, well as N values (T.2 ".o.))......N. for each of the S "ground truth" eigen-lengths.
For each pair of predicted and "ground truth" eigen-lengths (s,8'), we draw scatter plot of points (T.gt I.pred) and perform least squares linear regression over them to get corresponding R We match the predictions and groundtruths by maximizing the sum of R² scores and show the scatter plots of matched pairs in Fig. 5 Note that in (e) Top, since predict S = 3 sigen-lengths while there are only 5 = groundtruth eigen-lengths, we show the unmatched prediction with its most correlated groundtruth. For complete S x S' plots, please refer to Appendix B.2
Learned eigen-lengths are strongly correlated with tuman-typothesized measurements. As Fig. 5 shows, R values between predictions and "ground truths" are close to greater than 0.9 except for the redundant prediction slot 3 in (e) Top. They also have clear one-to-one correspondences with ground truth in tasks with multiple eigen-lengths, suggesting good disentanglement is learned.
Knowing the number of eigen-lengths beforehand is not a requirement for successful learning. The number S of eigen-lengths learn is hyperparameter set before learning. However, it does
5

---

Under review as a conference paper at ICLR 2023
not have to be the exact number of relevant eigen-lengths. As shown in (e) Top, when have more slots for eigen-lengten than needed, "ground truth" eigen-lengths are still captured by the first two predictions. The third prediction does not strongly correlate with any "ground truth". A further probe reveals that comparisons of this eigen-length almost never (only in 0.4% of the cases) contribute to the final result, outputting True most of the time. The network learns pair of degenerate eigen-lengths as there is no more necessary information to capture.
5 CAN GEOMETRY GROUNDINGS BE DISCOVERED FOR EIGEN-LENGTHS!
While Fig. [ shows strong correlation between learned eigen-lengths and "ground truth", their relationship is not always perfectly linear, can be observed in (d) Fit and (e) Top with complex geometries. Even in more linear cases, the scaling and offset make the raw eigen-length value hard to understand, egg., negative "length" values are less intuitive. As eigen-lengths can be seen as measurements of the object, many of them have sparse supports geometry groundings on the objects, e.g., height is the distance between the base plane supporting the object and its highest point. These geometry groundings anchor the corresponding eigen-length values, provide an intuitive explanation of these values, and usually earry geometric/semanti importance themselves. We are therefore interested in the following question: a high level, instead of directly predicting eigen-length values, if we first predict some geometric entities like points, vectors, and planes, then derive eigen-lengths from them, is it possible to learn meaningful eigen-lengths and geometry groundings?
ground the eigen-lengths
geometry? From
5.1 GROUNDING EIGEN-LENGTH PREDICTIONS ON GEOMETRIC PRIMITIVES
voctor
vector U.
11 (v.
langth L
n, (V. 4)
heightsH
voctor V.
vector
Figure 6: Ligen-Length Geometry Groundings. We ground each eigen-length L with unit vector V and two parallel planes 1lp HI, with normal v. L is computed as the distance between 1I. Ilg-
Consider fitting tasks like (d) Container Fitting and (e) Countertop Placing where the spaces affording the task can be roughly described by a set of parallel planes. |
To compute the success label of the task, say fitting an object into nightstand, we can measure the size of the spaces of interest in the environment (the drawer part) along important directions (its main axes) and comparing it to the measurement of the object. Inspired by this, we ground pair of eigen-length, on a tuble of unit vector and two planes (i,, HI. Ig)) as illustrated in Fig. 6 we measure both the object and the environment along U.. We take the object measurement as the diameter of the projection of the object point cloud O on the vector ö. i.e. Lo (O) maxpeo Jf p- millpeo of For the environment, we use a pair of parallel planes Hp. II, with normal ū to separate out a certain region relevant to the task (the drawer), then measure the distance between the planes. In practice, we adopt the (point, normal) plane representation and predict point pair (p,g) q) that determines the plane pair. The environment eigen-length is then computed [eav(E)) (E) 8. (q p)-
Figure 4 (b) illustrates our network architecture. In VectorNet, we employ a PointNet classification backbone to extract global feature of the environment point cloud E E R Mx3 then an MLP to predict S 3D vectors (m.)).m1.... 1. In WeightNet, we employ a PointNet segmentation backbone extract per-point features, then S 2 MLPs with to predict S pairs of probability distributions WP. W over the point cloud. The point coordinates of (p.: 9,) are then computed as the weighted average of original point cloud coordinates, namely p. W E, 9, = W E.
Note that other tasks may require other inductive hias. We focus on this type of tasks to study the feasibility of geometry- grounded ligen-length learning. We leave more versatile system as future work.

---

Under review as a conference paper at ICLR 2023
5.2 ANALYSIS OF LEARNED GEOMETRIC PRIMITIVES AND EIGEN- VALUES
Front View, Pred
Top View Pred
Environment
Top View Pred 1,2,3
Environment
(A) Passing
Container Finting
(4) Countertop Placing
Figure 7: Geometry Grounding Visualizations. We plot the learned vectors (as arrows) and planes (as disks) on top input environment point clouds. We also show the object model next point clouds for clearer view of object structure. For (d) Fit, we visualize predictions in two views for clarity. Please refer to Appendix B. for more visualizations.
0.9998
- 9998
-0.90 0.9983
R2 0.9994
R2 0.9998
R2 0.9993
1.5
1.5
Prodeaton
Protetion
Proddtion
Predation
prea Meseaue
10
1.0
0.5
1.0
1.5
0.5 1.0 1.5
0.5
1.0 1,5
(40 FiL Object
(4) Fl. OsjectY
FR. Object
(4) Tog, Object
X
(4) Top. Object Y
(4) Top, Object
Figure 8: Improved correlation after using geometry groundings. We show scatter plots of predicted eigen-length (Y coord.) and their matching "ground truth" (X coord.) in (d) Fit and (e) Top.
We perform the same correlation analysis and visualize the results in Fig. g Compared to Fig. 5 learned eigen-lengths are now almost equal "groundtruth" thanks the anchoring effect of the geometry grounding. The extra predicted eigen-length in (e) Top also behaves differently, capturing the same "ground truth" as another learned eigen-length. This suggests the regularization from geometry grounding makes learned eigen-lengths more likely to be meaningful measurements. It also reaffirms the fact that the number of eigen-lengths we set in advance can be different from the actual number of necessary eigen-lengths, Please see Appendix E.4 for detailed discussion.
We also visualize the learned geometry groundings in Fig. The Iearned vectors align with the main axes of object geometry. The learned planes overlap with tube surfaces in (a) Tube, surround the edge of countertops in (e) Top, and separate out the region of interest in (d) Fit, e.g. the higher one out of two storage spaces. These meaningful geometric entities provide clear interpretation of learned eigen-lengths, e.g. in (e) Top's case, red and green predictions coincide with each other and both capture the back-to-front length of the countertop.
5.3 A STUDY ON THE ROBUSTNESS AND DATA EFFICIENCY GROMETRY-GROUNDE EIGEN-LENGTIIS
Geometry grounding of eigen-lengths can be seen as form of regularization. We are therefore curious how the introduction of geometry groundings may influence the robustness of models in extreme test setups, as well as their data efficiency. We compared the performance of (1) Direct, a no-eigen-length approach, where MLP directly predicts the final label from the concatenation of object and environment latent features. (2) Implicit, the minimal eigen-length- lint-llenth-beeee pipeline introduced previously; and (3) Grounded, the coometry-grounder version.
Table shows test performances in extreme test conditions with low resolution point clouds or with extraordinary object scalings. bigen-length-basee approaches exhibit much higher robustness.
Fig. shows the trend test performances as we change the size training data. We also plot the difference between "ground truth" eigen-length measurement directions (local up and right) and predicted vectors as a way to quantify eigen-length quality. Results suggest that geometry-gen -rrouuddd version is more data efficient if meaningful geometry groundings emerge. When the training data
7

---

Under review as a conference paper at ICLR 2023
Table 1: Performance on extreme test cases. All methods are trained on size = 1024 point clouds with width ur and height sampled from U[00.4, 1.0)).
1.0
Grunded
Auzuresy
0.8
Arge to
Direct
Implicit
Grounded
Grounded Dieett
Default Points 64
99.00
99.15
99.65
0.8
76.80
93.50
96.17
16
10"
10'
10
10'
10'
wr, UT 0.2, 0.4)) wr U( 3.0.330)
72.10
98.78
99.72
Tmiiinng Samples
Training Samples
82.44
98.82
99.63
Figure 9: Trend of Left: test accuracy and Right: av- erage angle between learned vector groundings and "groud truth" directions w.r.t training samples.
is limited (< 3000 samples), however, the predicted directions of groundings far from ground truth measurement directions, suggesting that the model fails to learn meaningful groundings for eigen-lengths, and thus the final accuracy lower than Direct.
6 CAN EIGEN-LENGTHS BE LEARNED IN MULTI-TASK SETTINGS AND APPLIED TO NEW TASKS?
As humans, we are able to develop library of useful measuremenntseigen--eeue -engghs like height from past experience. Given a new task, instead of trying cluelessly, we would start with known measurements and investigate their role in the task. In this section, we ask if learned eigen-lengths can work in similar way, ie., given a set of training tasks, is it possible to learn set of eigen-lengths from them? Further, given a novel task, can we learn to select subset of learned eigen-lengths that sufficient for it? In other words, can agents accumulate and transfer knowledge in the form of cigen-lengths?
6.1 MULTI-TASK TESTRED
We design a set of tasks that share key eigen-lengths as the testhed for multi-task learning- As shown in T0)a), we consider box-fitting tasks where the box only has a subset of six faces. Each mode of face existence corresponds a different task with different geometric constraints. For example, to be able to fit, an object has to be narrower than the box in task 2 and shorter than the box in task 3. We set aside the box with all six faces present as the length from the training task set, and learn to use all of them during testing. By boxes with partial faces, we aim to mimic different types of cavities in the furniture, e.g., closed drawer box with all faces, an open space on the shelf as box without the front face, etc.
task. We expect to learn width, height, and
Test Task
Train 1
2
Train
OI(L
/
Obj Measure: 1%
O(LL
/
Prod Label
(AND)
Env Measure: L L....L L, "".... L
(a) Setting
(b) Multi-Task Learning Framework
Figure 10: (a) Multi-Task Setting where each train task uses boxes with certain faces missing as the environment geometry, and test task uses complete box; and (b) Learning Framework, where we use trainable masks to select eigen-length comparison results.
6.2 MUL.TI-TASK LEARNING FRAMEWORK
Fig. [0(b) shows the multi-task learning framework we experiment with. From a high level, we learn set of S eigen-length and allow each task to select relevant ones from them. This selection step is implemented as a learnable binary mask {mī ) =21,.....5 over eigen-lengths for each task Tk. We simply insert the mask in the AND-composition and compute the outcome for Tj as
I m. ((I.Tn (E) r.g (O))/T).
8

---

Under review as a conference paper at ICLR 2023
Table 2: Multi-Task learning, novel task adaptation results. We finetune eigen-length-basee methods on novel task for epoch and compare them to the direct method trained from scratch for and 100 epochs.
(Single Task) Direct
(Eigen-Length) Implicit (Eigen-Length) Grounded
Epoch
100
Test Accuracy
73.14
88.47
97.71
99.48
During training, optimize both the eigen-length prediction networks and continuous version of per-task masks m [0, 1]. At test time, we freeze network weights and only learn a mask to choose from eigen-lengths learned during training. Notably, we limit the size of test task data to 10 batches (320 samples) to examine if learned eigen-lengths help in few-shot adaptation scenarios.
6.3 MULTI-TAS LEARNING AND FEW-SHOT TEST TASK ADAPTATION
We experiment with both implicit and geometry-grounder eigen-length prediction networks. To analyze the learned eigen-lengths and per-task masks, we visualize learned geometry groundings that are selected (m, > 0.5) in each task in Fig. [1] Meaningful groundings are learned and correctly selected for each task, including the test task.
To explore whether sigen-lengths learned during training help quicker adaptation to new tasks, we compare the test task performance of Implicit, Grounded to Direct trained from scratch on the test task. All methods are limited 10 batches of test task samples. As shown in Table within one epoch of finetuning, methods based on the reuse of learned eigen-lengths already achieve high performance, surpassing Direct trained from scratch by a large margin, even when the latter has been trained for 100 epochs.
Training Tasks
Test Task
Figure 11: Learned Geometry Grounding in Multi-Task Setting, We only show learned geometry grounding (vectors as arrows, planes as disks) selected by the mask in each task.
CONCLUSION
In this work, formulate novel learning problem of automatically discovering low-dimensional geometric eigen-lengths crucial for fitting tasks. We set up a benchmark suite comprising a curated set of fitting tasks and corresponding datasets, as well as metric and tools for analysis and evaluation. We demonstrate the feasibility of learning meaningful eigen-lengths as sufficient geometry summary only from binary task supervision. We show that proper geometry grounding of the eigen-lengths contributes to their accuracy, interpretability, and robustness. We also make an initial attempt at learning shared eigen-lengths in multi-task settings and applying them to novel tasks.
Our exploration suggests broad opportunities in this new research direction and reveals many chal- lenges. For example, grounding eigen-lengte predictions on geometries requires reasonable choice of geometric primitives, which relies on inductive bias of the specific tasks considered. It would be challenging future direction to build a universal framework that accommodates a wide range of tasks by leveraging all kinds of geometric primitives and inductive biases. In many task instances, have access to signals beyond binary success or failure, e.g., a possible placement position of the object. How leverage these task signals in eigen-length learning remains open problem. As first-step attempt at defining and exploring the challenging problem eigen eegen-length learning, we do hope our work can inspire more researchers to work this important yet underexplored direction.
may

---

Under review as a conference paper at ICLR 2023
REFERENCES
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, al. Shapenet: An information-rich 3d model repository. arXiv preprint arxiv:
2015.
Boyuan Chen, Pieter Abbeel, and Deepak Pathak. Unsupervised learning of visual 3d keypoints for control. In International Conference on Machine Learning, pp. 1539-1549. PMLR, 2021.
Nenglun Chen, Lingjie Liu, Zhiming Cui, Runnan Chen, Duygu Ceylan, Changhe Tu, and Wenping Wang. Unsupervised learning of intrinsic structural representation points. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9121-9130, 2020.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pp.
2180-2188, 2016.
Boyang Deng, Kyle Genova, Soroosh Yazdani, Sofien Bouaziz, Geoffrey Hinton, and Andrea Tagliasacchi. Cvxnet: Learnable convex decomposition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 31-44, 2020.
Shengheng Deng, Xun Xu, Chaozheng Wu, Ke Chen, and Kui Jia. 3d affordancenet: A benchmark for visual object affordance understanding. In Proceedings of the IEEE/CVV Conference on Computer Vision and Puttern Recognition (CVPR), pp. 1778-1787, June 2021.
Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sama,, William Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 7154-7164, 2019.
Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. Local deep implicit functions for 3d shape. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4857-4866, 2020.
Irina Higgins, Loic Matthey, Xavier Glorot, Arka Pal, Benigno Uria, Charles Blundell, Shakir Mohamed, and Alexander Lerchner. Early visual concept learning with unsupervised deep learning. arXiv preprint arXiv:1606.05577 2016.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, 2017.
Tomas Jakab, Richard Tucker, Ameesh Makadia, Jiajun Wu, Noah Snavely, and Angjoo Kanazawa. Keypointefforme: Unsupervised 3d keypoint discovery for shape control. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12783-12792, 2021.
David Inkyu Kim and Gaurav S Sukhatme. Semantic labeling 3d point clouds with object affordance for robot manipulation. In 2014 IEEE International Conference on Roboties and Automation (ICRA), pp. 5578-5584. IEBE, 2014.
Lucas Manuelli, Wei Gao, Peter Florence, and Russ Tedrake. kpam: Keypoint affordances for category- robotic manipulation. arXiv preprint arXiv:1903.0666 090.096933.664 2019.
Kaichun Mo, Leonidas J Guibas, Mustafa Mukadam, Abhinav Gupta, and Shubham Tulsiani. Where2act: From pixels to actions for articulated 3d objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6813-6823, 2021a.
Kaichun Mo, Yuzhe Qin, Fanbo Xiang, Hao Su, and Leonidas Guibas. O20-Afford: Annotation- free large-scale object-object affordance learning. In Conference Robot Learning (CoRL), 202
Despoina Paschalidou, Ali Osman Ulusoy, and Andreas Geiger. Superquadrics revisited: Learning 3d shape parsing beyond cuboids. In Proceedings IEEE Conf. on Computer Vision and Pattrrn Recognition (CVPR), June 2019.
10

---

Under review as a conference paper at ICLR 2023
Despoina Paschalidou, Luc van Gool, and Andreas Geiger. Learning unsupervised hierarchical part decomposition of 3d objects from a single rgb image. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), June 2020.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652-660, 2017.
Zengyi Qin, Kuan Fang, Yuke Zhu, Li Fei-Fei, and Silvio Savarese. Keto: Learning keypoint representations for tool manipulation. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 7278-7285, IEEE, 2020.
Pradyumna Reddy, Michael Gharbi, Michal Lukac, and Niloy J Mitra. Im2vec: Synthesizing vector graphics without vector supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 7342-7351, 2021.
Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7263-7271, 2017.
Gopal Sharma, Difan Liu, Subhransu Maji, Evangelos Kalogerakis, Siddhartha Chaudhuri, and Radomir Mëch. Parsenet: A parametric surface fitting network for 3d point clouds. In European Conference on Computer Vision, pp. 261-276. Springer, 2020.
Narayanaswamy Siddharth, Brooks Paige, Willee Van de Meent, Alban Desmaison, Noah D Goodman, Pushmeet Kohli, Frank Wood, and Philip HS Torr. Learinng disentangled representations with semi-supervised deep generative models. arXiv preprint arXiv: 00400, 2017.
Dmitriy Smirnov, Matthew Fisher, Vladimir G Kim, Richard Zhang, and Justin Solomon. Deep parametric shape predictions using distance fields. Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 561-570, 2020.
Chun-Yo Yu Sun, Qian-Fang Zou, Xin Tong, and Yang Liu. Learning adaptive hierarchical cuboid abstractions of 3d shape collections. ACM Transactions on Graphics (TOG), 38(6):1-13, 2019.
Shubham Tulsiani, Hao Su, Leonidas J. Guibas, Alexei A. Efros, and Jitendra Malik. Learning shape abstractions by assembling volumetric primitives. In Computer Vision and Pattern Recognition (CVPR), 2017.
Dylan Turpin, Liquan Wang, Stavros Tsogkas, Sven Dickinson, and Animesh Garg, GIFT: Generaliz- able Interaction-aware Functional Tool Affordances without Labels. In Proceedings of Robotics: Science and Systems, Virtual, July 2021. doi: 10. (0.15507/RSS.2000 2021 10VIIX...006
Jiay Wang, Shize Lin, Chuxiong Hu, Yu Zhu, and Limin Zhu. Learning semantic keypoint representations for door opening manipulation. IEEE Robotics and Automation Letters, 5(4): 6980-6987, 2020.
Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangehen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, Li Yi, Angel X. Chang, Leonidas J. Guibas, and Hao Su. SAPIEN: A simulated bast- interactive environment. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Yanchao Yang. Yutong Chen, and Stefano Soatto. Learning to manipulate individual objects in an image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2020.

---

Under review as a conference paper at ICLR 2023
A IMPLEMENTATION DETAILS
A.1 NETWORK ARCHITECTURE
The framework in Section consists of PointNet and MLP output head that maps the PointNet global feature to S scalar values. The architecture outlined below, where the numbers in the parenthesis refer to the number of channels in each layer. We use batch normalization and and LeakyReL.U after all FC layers, except for the output layer.
Per-Point MLP(3 + + 128 + 1024)
PointNet
Max Pooling
MLP(1024 + 256 - S)
Output: scalars.
The framework in Section consists of VectorNet and WeightNet. VectorNet consists of PointNet classification backbone and an MLP output head, as outlined below.
Per-Point MLP(3
64 128 1024)
PointNet
Max Pooling
MLP(1024 256 - +5)
Output: S' vectors.
WeightNet consists of PointNet segmentation backbone and 2S parallel MLP output heads, each outputs weight distribution over all points, as outlined below.
Per-Point MLP(3 - 64[per--oni point feature] + 128 - 1024)
PointNet
Max Pooling global feature]
Concat(per-point) feature, global feature)
+
MLP((1024 64) 512 256 - 128)
4
Output Weight MLP,(125 - 256 + 1), 1,2, ...
SoftMax
Output: 2.5 sets of per-point weights.
We use LeakyReLU and batch normalization after each FC layer except for the output layers.
A.2 TRAINING DETAILS
All networks are implemented using PyTorch and optimized by the Adam optimizer, with learning rate starting at 10-3 and decay by half every 10 epochs. Each batch contains 32 data points; each epoch contains around 1600 batches. We train models for ~ 100 epochs on all tasks. The learnable parameter T is initialized with r = 1. All experiments are run on a single NVIDIA TITAN X GPU.
A.3 DATASET DETAILS
Table and summarizes the statistics of environment/object shapes used in dataset. Each shape drawn with probability in inverse proportion to the number of shapes in its category, such that each object category appears with similar frequency in the final dataset.
12

---

Under review as a conference paper at ICLR 2023
Table 3: Environment Shape Statisties.
Box
Microwave
Refrigerator
Safe
Storage Furniture
Table
Washing Machine
Total
Train Test
21
34
21
272
70
13
440
73
25
127
Table 4: Object Shape Statisties.
Train
Basket 77
Bottle 16
Bowl 128
Box
Can 65
Pot 16
Mug 134
TrashCan 25
Total 478
17
Test
Bucket
Dispenser
Jar 528
Kettle 26
Total 554
33
9
During data generation for the tasks where both the environment and the object are ShapeNet objects, we apply random scaling s ~ ([0.9, 1.1]) to the environment objects, set all joints to closed state and sample M = 1024 points from the object model. Given an ohject-environment pair, we randomly sample 7 = 1000 candidate positions in the environment point cloud, and check whether placement of the object at each candidate satisfy the task specification using SAPIEN (Xiang et al. 2020) simulation. If all candidates fail, we label the pair negative, otherwise positive. Specifically. the candidate positions are sampled from "applicable and possible regions" following Mo et al. (2021b)'s 's definition. For example, only consider points with upward facing normals, and for task (e) only consider points with close to highest z coordinates. We generated around 75K training data and 20K testing data for each task.
B ADDITIONAL RESULTS
B. GEOMETRIC GROUNDING VISUALIZATION AND FAILURE CASE DISCUSSION
Fig. 12 and 13 show more visualizations of the learned eigen-lengths the three tasks from the main paper. Our framework is able to learn reasonable eigen-lengths that measure along crucial directions. These eigen-lengths are also grounded by planes that suggest the relevant part of object which supports the task. In experiments with primitive shapes as environments, the learned planes almost overlap with the box/tube faces. In experiments with ShapeNet container objects as environments, especially task (d) (Fir, or container fitting) as shown in Fig. [3] locating the relevant part becomes more challenging. As this usually involves finding cavities in shape and selecting the largest one. Fig. 13 shows examples of our learned eigen-lengths, most which make sense, as shown (a)-(o). We are able to ignore irrelevant parts, e.g. the legs of tables, and find the part of object that affords the "containment" task, c.g. the drawer in (b), the closet in (c). When there are many cavities that afford the same task, the network picks the largest one, c.g. in (d) and (k).
Failure Cases. We also observe some failure cases where the learned eigen-lengths are inaccurate. Fig. (t) shows the most representative ones. (p) shows a relatively complex shape, where the network struggles to find the correct width of the drawer. (q) and (r) show cases where the network finds the wrong cavity. According to our task definition, the object can only be placed in the drawer part in (q). Instead, the network finds the part top of the drawer. In (r), the network finds the second largest cavity instead the largest one at the bottom. (s) shows an extreme case where the height of the pizza box is much smaller than the other two extents. As objects usually have correlated extents, comparing height suffices most of the time. The network probably lacks the motivation to precisely capture the width and the length of the pizza box, resulting in the underestimation of width and length in (s). Finally, our formulation, i.e. the AND clause three eigen-length comparisons, can not fully and precisely describe the nature of this task. The washing machine in (t) has cylinder-shape shaped cavity, which our network tries to approximate by a cuboid, which is reasonable within the range of its expressive power but not accurate. Also, there could be shapes that do not have "largest"
13

---

Under review as a conference paper at ICLR 2023
front view
(a) Tube Passing
top view
instance
(e) Countertop Placing
Figure 12: Additional qualitative results. We visualize the learned vectors and planes for (a) Tube Passing and (c) Countertop Placing. We show all eigen-lengths in the front(a)/top(e) view. We also show the underlying instances in task (e) countertop placing for clearer understanding of the object structure. Note that though some joints are "open" for visualization purpose, all instances in the dataset are at their rest state.
cavity, c.g. some drawers in a closet may be designed for tall and narrow things, while others are designed for flat things. To deal with arbitrary objects, the extents of both types of drawers useful. Introducing more complex and flexible formulations, e.g. in Section D would help better capture the complexity of the task.
B.2 CORRELATION ANALYSIS RESULTS
We show here the scatter plots and correlation Rit values between all prediction eigen-lengths and all presumable geometric measurements. Ri² value, or coefficient of determination, a metric in [0, 1] reflecting linear correlation between two variables. The closer R² is to 1, the more linearly correlated the variables are. Given two of samples F4, Vr. where 1, 2, ..... n, R² defined between and the least squares linear regression of Vi on 24, ((x)):
R² - Σ, (y: ((x.))
E, (y: - 7)
where y +. yi the mean value of yi- Results from Eigen-Lengtth-mmple
are shown in Fig. 14 Results from Higen-Length-Grroune are shown in Fig. [5] We can clearly see the one-to-one correspondence between predictions and presumable measurements. R² is close to greater than 0.9 where the prediction is the match for the measurements, otherwise the value much smaller. Grounded variant, where R²2 values are close to the theoretical bound when it matches. The models can learn a compact and appropriate set of eigen-lengths from binary task supervision. Also note that the extraneous prediction slot in task (e) (Top, countertop placing) become degenerate with another prediction slot, as has mentioned before in main text.
more apparent in the Eigen-Length
C APPLYING RANDOM ROTATIONS TO INPUT ENVIRONMENT GEOMETRIES
C.1 FORMULATION AND IMPLEMENTATION
While the object shape is randomly rotated in all experiments in the main paper, take environment geometry directly from ShapeNerChang et al. (2015) where shapes are axis-at alignel In this section, consider more challenging setting where the environment geometry is also randomly rotated.
14

---

Under review as a conference paper at ICLR 2023
Specifically, we consider "rotated" version of the Container Fitting task and Tube Passing task in the main paper. For each original data point, i.c. a settainner/uue-obbjee pair with a boolean label ((object point cloud P.. environment point cloud P.). success label L), sample random rotation R and apply it to both the container and the object. We feed ( (RP,, RPP.) to the network described in Section 5 and supervise the network with the same label T.
C.2 CORRELATION ANALYSIS AND RESULT VISUALIZATION
We show the correlation analysis of learned eigen-lengths in Fig. 16 A strong, disentangled correlation between learned eigen-lengths and hyothsiipottesizeed ones can still be observed.
We also visualize the learned geometry groundings in Fig. 17 The predicted planes roughly align with the main cavities of the objects. From the results, we can see that the proposed problem setting is still valid and the studied methods can still be applicable and produce reasonable results.
D EXTENDING AND CLAUSES TO DISJUNCTIVE NORMAL FORM (DNF)
D.1 FORMULATION
We employ the AND clause formulation for all tasks shown in the main paper. Namely, after learning a library of paired object/environmert eigen-lengths {[L,"" L we compose them by
f(E,
A 12.,
S (8) 1 (O)]
(selection mask is omitted for clarity), approximated by
T(E,0) = II
1=122
Here show extend this formulation to the more general Disjunctive Normal Form (DNF), where an OR connects multiple AND clauses. Each AND clause composes eigen-length comparison results of subset of eigen-lengths. The result of each AND clause is then aggregated by OR operator. More precisely.
TEE V A (E) > 1.000 (O)).
U.El sEU.
4 = {Ua}a specifies the subset U, of eigen-lengths in each AND clause. We similarly use differentiable approximation during training:
T(E,0)= = 1- II
(1
II SEU,
(((L - (0))/T))
U.EU
The introduction of two-level logic and the OR operator helps express more complex reasoning and deal with wider range of tasks. For example, many realistic tasks have multiple solutions. OR captures the relationship that the task can be executed if any, not necessarily all, of the solutions work.
D.2 TASK AND IMPLEMENTATION DETAILS
To demonstrate our framework's compatibility with this new formulation, experiment with the Mubei-Tubbe Passing task. This is a variant of task (a) (Tube, or tube passing) in the main paper, where we have two tubes of random sizes placed next each other. As long the object can be translated and passed through any of these tubes, the task considered as successful.
Similar to tube passing, randomly sample the extents of the tubes, the shape, scale, and rotation of the object. The center of the two tubes are always at two fixed positions on the y-axis.
We set the number of eigen-lengths namely 2/ = 2). (3. 4) Ideally, the learned eigen-lengths should correspond to the height and width of the tubes. Also, the height and width of the same tube should be in the same AND group.
learn as S and split them into two disjoint AND groups.
15

---

Under review as a conference paper at ICLR 2023
D.3 RESULT VISUALIZATION
Fig. 18 visualizes the learned eigen-lengths, where green and yellow belong to one group. purple and red belong to another group. We successfully learn eigen-lengths that measure along the height/width directions of the tubes. We also learn them in correct groups, where width and height of the same tube are paired together.
E DISCUSSION AND FUTURE WORK
E.I DEFINITION OF EIGEN-LENGTIIS AND APPLICATION SCOPE OF THE EXPLORED FRAMEWORK
In our setting, an eigen-length is whatever scalar measurement (i.e., just a ID scalar) the network invents to best perform its stated downstream task. While this definition for gggen-dimensions is quite general and could applicable to any object as long as there exist certain 1D eigen-lengths that are crucial and useful for checking the feasibility of accomplishing downstream task, we are assuming in our current experiments that having such sets of ID eigen-lengths are sufficient for the tasks, Therefore, our current setting would not apply to the tasks where having only such low-dimensional eigen-lengths is not sufficient, such as the tasks of geometric contour matching and object collision checking.
E.2 BROADER IMPLICATION OF THE STUDIED APPROACII FOR AI AND ROBOTICS
We believe the general approach we suggest can have very general applicability in AI and roboties, where the solution to downstream tasks suggests the emergence of generally useful geometric concepts such as length, height, width, and radius in unsupervised ways. As we described the introduction, learning such compact useful geometric eigen-lengths is beneficial in the ways that 1) they are highly interpretable, while most of the current learned representations in neural networks are opaque and learned black-box hidden features which may be unreliable or untrustworthy, 2) they could be shared and reused across different tasks, enabling fast adaptation to novel test-time tasks, and 3) the proposed learning formulation may discover novel yet crucial geometric Sige-llenghss that even unknown to us humans given the new test-time tasks. Furthermore, there could be more geometric concepts of great interest and importance that future work can explore in this direction. Examples can be 1) symmetry, as a result of trying to complete 3D shapes, 2) regular object arrangements and poses as tool for efficient search, and 3) tracking, as an essential capability for predicting the outcome of sports games. In other words, we want learning networks to invent the notions so symmetry, regularity, or tracking. If such capabilities could emerge from purely unsupervised learning, we no longer need to rely on black-box- bike neural networks and human annotations for this geometric information over 3D objects.
E.3 ROTATION OF OBJECTS DURING TASK EXECUTION
In our experiments, we are primarily concerned with translational motion during task execution. This setting stems from practical concerns: in many robotic manipulation scenarios, the rotation of the object is often given as the desired target to achieve by robot planners or unchangeable during robotic grasping and manipulation. For example, the robot gripper may only be able to grasp, hold and move the mug without spilling the content and with steady grasping in certain poses for pick-and -nd--lee place task and the robot may not be able to freely rotate the object as the arm kinematics may not allow.
That being said, our cylinder fitting task does allow rotation along the up-axis for the object and similarly, the sphere fitting task allows the full 5O(3)-space rotation, while for other tasks, in the case that multiple poses of the object are possible, we can simply pass the object in different poses into the same network for multiple times to query the joint fitting feasibility.
E.4 DETERMINING THE NUMBER OF EIGEN- LENGTTL TO LEARN
The number of eigen-lengths to learn, ie. S. is a hyperparameter of our learning framework and has to be set in advance. However, it should be interpreted as the upper bound on the number of
16

---

Under review as a conference paper at ICLR 2023
eigen-lengths the system can learn, and does not have to be the "groundtruth" number of relevant eigen-lengths. As shown in Sec. 4.3 and Sec. 5.2 when we set S for the countertop fitting task where only two eigen-lengths mattrr the extra "slot" either degenerates coincides with other slots. Such cases can be easily detected and filtered, and the actual number of relevant eigen-lengths can be discovered. Setting maximum number for unknown number of targets is also common practice in problems like object detection Redmon & Farhadi (2017). That being said, a more flexible mechanism that allows an arbitrary number of eigen-lengths would be desirable, especially for objects with complex compositional structures like robotic arms or closets with many drawers. We leave this as future direction.
NEGATIVE SOCIAL IMPACT
Our work joins the initial efforts of eigen-length emergence in unsupervised learning settings with many other works along this direction. There could be several bias in the data and training objectives, but this is general concern shared by most works in this field. Our work may also share the controversial arguments with other works that the future Al agents may have the capability of thinking by themselves causing threats for human beings. However, the results demonstrated in our work are way far from that. Other than these potential issues mentioned above, we do not see any other major concerns our work particularly introduces.
17

---

Under review as a conference paper at ICLR 2023
front vilew
top view
Instanca
(8)
(b)
(d)
front vlow
top vilw
Instomce
(a)
(h)
front view
top viow
Instonce
front view
top view
Instance
(p)
(0)
(8)
Figure 13: Additional qualitative results in Container Fitting. We show eigen-lengths in two views together with the underlying object following Fig, (d). (a)-(o) are successful cases where the learned planes correctly separate out the largest cavity in the object. (p)-(t) show failure cases.
18

---

Under review as a conference paper at ICLR 2023
R2 0.0089
R2 - 0,8870
R2 0.5404
R2 0.9683
R2 0,9972
Prddcttnn
Prodction
Predecton
Prediction
-2
1.0
2.0
1.5
0.8
1.0 1.5
0.95 1.00
(4) Tidsa
(4) Tubw 2
(b) Cylindir
(o) Cyllnder
14) Sphwnu R
R22 0.9978
R2 0,0067
R2 -0.9788
R2 0.5490
R2 12- 0,9830
Siple yomper
Predicoo22
Paadkton2
Protctio2
Predecton2
Preteten
1
-10
-10
22
2
0.5
1.0
1,5
0.8
0.5 1.0 1a
0.5 1.0
1.5
(1) Tube Y
[N]
Tube 2
(b)
Cylinder
(b) Cyllnder H
0 Mug Holder
R2 0.5221
R2 0.5152
R2 0.8838
R2 O 2.07
R2 0.9380
10
Predrtoon
Prdddonn
Predcteon
Prodicton
Pruditon
1. (d) O
0.5
1.0
(6) Osject
(4) Object 2
(4) Top, Object R2 0,9118
(e] Top. ObjectY
R2 0,4509
R2 0,8890
R2 -04457
R2 12-30 0,3198
Predcton?
Predicion2
Prodicio22
Prodicion2
Prudicton2
-20
0.0
10
6.5
1.
(4) Fit, Orject
(4) Orject
Fl, Object 2
(4) Top, Object R2 0.6486
[e] Top, Ooject R2 = 0.5940
R2 0.8814
R2 0.5679
R2 0.5267
Protetion3
Protecion
Presection
-10
0.0
100
1.5
0.6 1.0
1.0 [o] Top. ObjectY
(d)
O
(6) Object
(4) Object
(4) Top. Object
Figure 14; Full correlation plots and respective R2 values between ground truth measurements and predicted eigen-lengths from
R2
0.9988
F2 = 0.4781
R2 0.4168
R2 0,8962
R2 = 0.9988
1.5
1.6
1.5
Predictonn
Preditton1
Predicioon
Prediction
1.0
1.0
0.0
1.0
0.5
0.0
0.0
0.0
0.5
1.0
15
0.5
1.5
0.5
1.0
1.5
0.5
Fit, Oxject
(2) FR. Coject Y
(d) Fit, object
(c) Top, Oiject
(e) Top. Object
R2 0.4722
R2 0.9858
R2 0,5152
R2 0.9993
R2 - 0.5886
2
2
2
2
Onject
(6) Onject
Fi) Object
(e) Top. Oijest
Object R2 0.5993
084
0.5082
0.9993
0.9894
1.5
Prodctoon
1.5
Predictoon
1.6
3rediction
Predicion
0.5
0.5
0.5
0.5
1.0
0.0
0.5
1.0
1.5
0.5
1.0
1.5
0.5
1.5
(6) FR, Ooject
Fit, Object
ObjectX
Object
Figure 15: Full correlation plots and respective R² values between ground truth measurements and predicted eigen-lengths from Eigen-Length-Gree Grounded.
19

---

Under review as a conference paper at ICLR 2023
RZ - 3.0156
000339
144
14
Free coen
04
08
RZ = 2.9052
B2 Q.0221
23
2.0
1.3
18
1.5
Predicion
R2
0.5404
R2 0.8919
R2 0.4326
1.75
1.75
1.75
1.50
3.50
1.50
1.25
3.25
Podictonn
1.25
1.00
2.00
1.00
0.75
0.75
0.75
0.25
0.25
0.25
1.0
(d) Fit Object
Object
Object
R2 0.5033
R2 0.4208
R2 0.9963
1.50
1.50
1.59
Prodicton
1.00
Prdiction2
1.00
2
0.75
0.75
0.75
0.50
0.50
0.59
0.00
1.0 Object
(d1 FRt Object
Object
R2 0.8939
R2 0.5818
R2 0.5496
1.75
1.75
1.75
1.50
1.25
1.25
1.25
Predetion3
0.75
0.75
0.75
0.25
0.25
0.25
1.0
(d) FRt ObjectX
Object
Object
Figure 16: Full correlation plots and respective R² values between human ments and predicted eigen-lengths in rotated Tube Passing and rotated Container Fitting, respectively. Correspondences between predicted eigen-lengths and suman--hyoothsized ones can be observed.
human-hypothesiie hypothssized measure-
20

---

Under review as a conference paper at ICLR 2023
Rotated Tube Passing
Rotated Container Fitting
Figure 17: Visualization learned geometry groundings rotated Tube Passing and rotated Container Fitting, respectively. Vectors are visualized as arrows, and planes visualized disks. For Container Fitting, we also show the underlying geometry (before rotation) for better reference. The learned vectors and planes roughly align with the rotated object. Regions of interest like drawers are also selected by planes.
Figure 18: Visualization of the eigen-lengths learned with OR-AND clauses. Green and yellow, purple and red eigen-lengths belong to the same AND-group. turns out that each group attends to one of the tubes and captures its width and height.
21


Reviews:
[{"review_id": "bwDFI3rsF-E", "reviewer": "Reviewer_rLUp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new problem setting that learns the geometric eigen-lengths for robot fitting tasks. The authors build a benchmark for this task, propose two learning methods (learning directly from binary supervision and learning with geometric primitive), and test the methods with single-task and multi-task(transfer learning) settings.", "summary_of_the_review": "This paper proposes an interesting topic, and it is clear to follow.\nHowever, the validity of the problem setting and the comparisons to relevant methods should be addressed more carefully.\n", "strength_and_weaknesses": "###### Pros\n- The idea of learning task-specific geometry lengths of the object and the environment is interesting.\n- This paper is written clearly, with the motivation and methodology easy to follow.\n- The authors build a comprehensive benchmark suite including 6 kinds of tasks, ~1200 object models from ShapeNet, the corresponding datasets, and evaluation metrics.\n- The findings of being able to learn length directly from binary supervision are interesting and impressive.\n\n###### Cons\n- To apply the eigen-length learning step to robot fitting or other robot manipulation problems, it seems that how to choose and define the geometric primitive and the creteria is essential to this problem. After that step, parameter learning is relatively easy. Therefore, whether the proposed problem setting and method are practical remains unclear. This method design is contrary to the goal of the paper, which aims to rely on the least human prior. \n- To predict geometric information, there are tons of ways, including traditional image processing methods like edge detection, deep learning methods over computer vision like 3D object detection[1], segmentation[2], 6 DoF pose estimation[3] and computer stereo vision system[4] with necessary pre-process and post-process operations. The proposed method for the robot fitting task is one of the extended applications of the geometric information prediction problem, and thus the novelty is limited.\n- There are no experimental comparisons to baselines. For example, other robot fitting methods (or simpler, binary-classification methods) that explicitly estimate or implicitly learn the geometric information should be compared; Comparisons of the low-dimensional eigen-length to the high-dimensional latent codes and structured representations discussed in the Introduction section should also be empirically verified.\n- The constraints of the problem setting, i.e., only translation (without rotation) of pre-aligned objects and environments is too strict to be feasible in practice.\n- For Table 2, should the proposed method be compared with other transfer learning methods instead of direct training without knowing the training tasks?\n\n\n###### Refs\n- [1] Zhou, Yin, and Oncel Tuzel. \"Voxelnet: End-to-end learning for point cloud based 3d object detection.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.\n- [2] Wu, Bichen, et al. \"Squeezeseg: Convolutional neural nets with recurrent crf for real-time road-object segmentation from 3d lidar point cloud.\" 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2018.\n- [3] Xiang, Yu, et al. \"Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes.\" arXiv preprint arXiv:1711.00199 (2017).\n- [4] Wikipedia: Computer stereo vision. https://en.wikipedia.org/wiki/Computer_stereo_vision", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Qp97WrGpd5y", "reviewer": "Reviewer_VXuJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This article is concerned with the notion of eigen-lengths of objects, canonically width, height and depth. The authors argue that those eigen-lengths are key to common-sense reasoning (eg, can object A fit in container B). The authors propose that eigen-lengths should be grounded in the scene geometry. \nThe paper attempts to ground the eigen-lenths using a numbe of manipulation tasks. \nExperiments demonstrate that the approach allowed for accurate estimation of the eigen-lengths. ", "summary_of_the_review": "In summary, I found this paper  interesting and pleasant to read. The approach is novel and tackling an important problem for the community. I did not find any major concerns on this paper, although some of the discussion could be improved to clarify the contribution (see weaknesses above)", "strength_and_weaknesses": "### Strengths\n- The paper is well written and pleasant to read. \n- This is an interesting problem. The extraction of intrinsic properties of objects such as eigen-dimensions is an interesting and important avenue of research. \n- The use of the multi-task learning and task adaptation is well adapted to the problem.\n### Weakness\n- Although the general concept of eigen-dimensions is intuitive and several examples are discussed, the article lacks a formal definition of eigen-dimensions and their meaning for a range of objects would be useful to clarify the applicability of the approach.\n- The broader implication of the results for AI and robotics should be made explicit. What is the possible usage outside of the narrow tasks described in this paper?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MhUbl0TpAw", "reviewer": "Reviewer_4TGL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an early attempt at learning geometric lengths from trails and explores how to transfer the learned geometric knowledge to solve a different task. Besides offering a problem formulation, this paper presents an evaluation framework based on finding evaluation correlations against ground truth lengths. ", "summary_of_the_review": "Overall this is initial research. My main criticism is the lack of exploration or at least discussion on how to use this idea in more complex problems. I would like to see at least some discussion in this regard.", "strength_and_weaknesses": "Strengths\n----------\n\nIdeas are clearly elaborated through the paper with adequate support of intuition in explanations.\n\nElaboration of experimentations is adequate for an early exploration of learning object lengths. \n\nInitial results are promising on the capability of learning lengths from trials.\n\n\nWeaknesses\n-------------\nThis paper is an early exploration of the topic. Consequently, the experimentation is limited to a single dataset and simplistic tasks. Can this methodology be used to improve challenging tasks such as relative pose estimation and active visual navigation?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Fhjuuk7b-H", "reviewer": "Reviewer_rqrT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper explores an under-investigated problem, that of automatically discovering geometric quantities that indicate the potential success or failure of fitting tasks. These include passing an object through a tube-like container, putting an object in a drawer or a sphere, etc. The authors use the term eigen-lengths for these measurements and explore three approaches for learning them from data, with the help of some manual annotation in the form of specifying the number of relevant eigen-lengths. Several tasks accompanied with relevant data are defined and quantitative results are presented. (Presumably, the data and test suite will be released if the paper is accepted.)", "summary_of_the_review": "On one hand, accepting the paper may inspire other researchers to make progress on an interesting problem, surpassing the baseline algorithms presented here. On the other hand, some of the limitations listed above are serious. I recommend borderline accept for now to give the benefit of the doubt to the authors, but justification for the chosen representations, the use of PointNet, and the restriction to translation-only motion should be provided in the rebuttal. Section 6 should be explained more clearly.\n\n++++++++++++ Dec. 11 update ++++++++++++++++++\n\nAfter the discussion among the reviewers and area chair and the latest author responses, I am leaning towards recommending rejection. Even though the idea is interesting, I had undervalued the limitations, which outweigh the contributions. The handling of rotation in particular is unsatisfactory. According to the authors’ latest response, rotations in rectangular environments cannot be handled. This is an important limitation. I also do not understand the value of rotating spheres or sphere-shaped environments when the representation is purely geometric. Moreover, the response on the ordering of eigen-lengths so that they correspond across the object and environment is unsatisfactory. I do not see how the network can learn that based on the loss function.", "strength_and_weaknesses": "Assessing the strengths and weaknesses of this paper is difficult because the problem addressed is under-explored in the literature and the solutions presented here are not mature yet. This seems to be an interesting direction for research, but there are several shortcuts and limitations in the formulation. While this is partially understandable, I have doubts whether this is the right direction and whether extensions of this formulation are likely to be adopted by other researchers.\n\nStrengths\n\nThis is an interesting problem and a thought-provoking paper presenting a novel take on it.\n\nThe claim that the eigen-lengths are interpretable and universal is true, but also self-evident. Determining if an object fits in different objects depends on it dimensions. I do not consider this a major strength.\n\nThe concept of grounding, as defined in the paper, leads to improved performance. \n\nThe comparison of the direct, implicit and grounded methods is informative, but see below about my concerns on the representation.\n\nWeaknesses\n\nThe restriction of object motion to translation only appears reasonable for a first attempt at the proposed formulation, but reduces the search for “eigen” lengths to essentially the selection of relevant dimensions of the bounding box. Since both objects and containers/tubes are axis-aligned, the problem is greatly simplified and does not require the learning of affordances other than the concept of support surfaces in some cases. (It is not clear to me if points of contact are detected in the sphere fitting task, for example.)\n\nSuccess or failure of a given task is determined based on the existence of suitable final placement of the object in the environment. In some cases of interest, the existence of a feasible trajectory is also needed. An object may fit in a cabinet or drawer but there may be no way to put it in there.\n\nThe fact that the number of relevant eigen-lengths, S, has to be set manually is a limitation. \n\nThe chosen representation of the objects and environments requires further justification. PointNet descriptors are not well suited for capturing object properties such as dimensions. It seems that some other representation, possibly in the form of random projections, would be more effective.\n\nSection 6.1 is hard to connect to a practical scenario. (It is possible that I am missing something.)\n\nThe test set in Table 4 is dominated by jars. I do not see any reason to construct an imbalanced dataset for an unexplored problem.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject