T$^2$SQNet: A Recognition Model for Manipulating Partially Observed Transparent Tableware Objects
Authors: Young Hun Kim
Seungyeon Kim
Yonghyeon Lee
Frank C. Park
Venue: CoRL 2024 (2024)
Keywords: Transparent objects
Shape recognition
Object manipulation
TL;DR: This paper proposes a novel framework for recognizing and manipulating partially observed transparent tableware objects.

Abstract: Recognizing and manipulating transparent tableware from partial view RGB image observations is made challenging by the difficulty in obtaining reliable depth measurements of transparent objects.  In this paper we present the Transparent Tableware SuperQuadric Network (T$^2$SQNet), a neural network model that leverages a family of newly extended deformable superquadrics to produce low-dimensional, instance-wise and accurate 3D geometric representations of transparent objects from partial views.  As a byproduct and contribution of independent interest, we also present TablewareNet, a publicly available toolset of seven parametrized shapes based on our extended deformable superquadrics, that can be used to generate new datasets of tableware objects of diverse shapes and sizes. Experiments with T$^2$SQNet trained with TablewareNet show that T$^2$SQNet outperforms existing methods in recognizing transparent objects, in some cases by significant margins, and can be effectively used in robotic applications like decluttering and target retrieval.

T'sQNet: A Recognition Model for Manipulating Partially Observed Transparent Tableware Objects
Young Hun Kim Seungyeon Kim*.l Yonghyeon Lee² Frank Chongwoo Park
'Seoul National University, "Korea Institute For Advanced Study (yhun, ksy Grobotics... snu. ac. kr, ylee@kias.re re. kr, fcpesnu. ac.kr
ac.kr,
Abstract: Recognizing and manipulating transparent tableware from partial view RGB image observations is made challenging by the difficulty in obtaining reliable depth measurements of transparent objects. In this paper we present the Transpar- ent Tableware SuperQuadric Network (T'SQNet), a neural network model that leverages a family of newly extended deformable superquadries to produce low- dimensional, instance-wise and accurate 3D geometric representations of trans- parent objects from partial views. As byproduct and contribution of indepen- dent interest, also present TablewareNet, a publicly available toolset of seven parametrized shapes based on our extended deformable superquadries, that can be used to generate new datasets of tableware objects of diverse shapes and sizes. Experiments with TeSQNet trained with TablewareNet show that TesQNet out- performs existing methods in recognizing transparent objects and can be effec- tively used in robotic applications like decluttering and target retrieval. The code is available https: /// gi thub. cwm com/ssungyeon-eeeeoe waaereoonnn/ttsStttaaaaaa ctt/////eunyyoorrt////sstttteeee
Keywords: Transparent objects, Shape recognition, Object manipulation
Introduction
Recognizing and manipulating transparent tableware objects is made challenging by the difficulty in obtaining reliable depth measurements of the transparent objects [1]. The task is made even more difficult when such transparent objects are placed on shelves or against walls, offering only partial views of the object. This paper addresses the recognition of transparent objects from partial view RGB images. Specifically, we emphasize the need for a good representation for manipulation, one that accounts for object 3D geometries and instances. The first criterion is vital to avoid collisions and ensure effective grasping, while the second is necessary for target-driven manipulation.
Recently, methods using Neural Radiance Fields (NeRF) [2], which attempt to fit 3D field models to multi-view RGB images via differentiable rendering without using depth images, have been in- troduced for recognizing and grasping transparent objects [3, 4, 5]. However, since NeRF assumes the availability of all-around views, when only partial views are available, the performance of these methods may not be guaranteed.
Learning from data is essential to infer complete geometries from partial view information (c.g., [6] and [7]). Given the high cost of real-world datasets, simulations are often used, emphasizing the need for models robust to the sim-to-real gap. However, [6] produces representations that lack instance information and shows limited robustness to the sim-to-real gap. [7] generates instance wiiee 3D) representations, but these rely on depth images and are thus unsuitable for transparent objects.
In this paper, we present novel deep learning framework that tackles all the aforementioned chal- lenges: (i) object transparency, (ii) partial views. (iii) achieving good representations for manipu- lation, and (iv) bridging the sim-to-real gap. Our main contribution is twofold. Firstly, we extend
Equal contribution
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Segmentation
Voxel carving
Shape prediction
Mull
SAM
DETR3D
ResNet3D
Jottle
Cameras
Bounding boxes
Smoothed visual hull
Recognized shapes
RGB images
Mask images
Figure I: Model architecture of Te including: (1) pretrained SAM for predicting object
mask images [8]; (2) DETR3D for detecting object bounding boxes [9]; (3) a voxel carving mod- ule [10]; and (4) tableware parameter prediction model based on ResNet3D [11].
the deformable superquadries, a family of geometric shape primitives recently introduced in [7], by incorporating shearing and superparaboloids. This extension enables us to represent wider range of tableware objects, illustrated in Figure 2. By combining these primitives, templates for tableware that meet the criteria for good representations in manipulation tasks.
define shape
The second contribution is the Transparent Tableware SuperQuadric Network (QNNe), a neural network model that infers the geometries of tableware objects by representing each object as a union of the extended deformable superquadries. Our model comprises four key steps: prediction of 2D masks, 3D bounding boxes, visual hulls, and superquadric parameters (illustrated in Figure 1). There two novel elements to approach. First, leverage SAM mask images [8], which is critical in enhancing robustness to the sim-to-real gap. Secondly, we devise representations based on a set of visual hulls - one for each object as inputs to the shape prediction network. We have observed that this approach significantly improves performance compared to models that directly estimate superquadric parameters in an end-to-end manner.
As second contribution obtained byproduct of our approach, but also of independent interest, introduce TablewareNet, family of continuously parameterized shapes representing everyday tableware objects across seven classes, using extended deformable superquadrics and their combi- nations as shown in Figure 2. Users can easily generate new dataset with diverse sizes and shapes within each class; even a new class of objects can be constructed using our tools.
Our experiments demonstrate that TesQNet trained with TablewareNet outperforms existing state- of-the-art art methods in predicting the geometries of transparent objects. This superior performance extends beyond the test set from TablewareNet include unseen data from the large-scale trans- parent object dataset named TRansPose [12]. Furthermore, we show the practical applicability of our method in two downstream tasks including sequential decluttering and object rearrangement for target retrieval, highlighting the advantages of our representations.
TablewareNet: Dataset for Cluttered Transparent Tableware
In this section, we first extend the deformable superquadrics introduced in [7, 13] represent non- convex shapes such bowls and wineglasses. We then propose TablewareNet, family of contin- uously parametrized shapes representing tableware objects, where each shape is defined as proper combination of our extended deformable superquadrics.
2.1 Extended Deformable Superquadrics
Superquadrics, parametrized by only few parameters, can represent a relatively wide range of geometric shapes. We employ two kinds of superquadries: superellipsoids, which have been used for object manipulation [14, 15, 16, 7, 17, 18, 19], and superparaboloids, which newly introduced. Superellipsoids and superparaboloids are implicit surfaces with the following implicit functions with size parameters (a1, a2 .s) E R and shape parameters (e1, e2) E RR: for x (I, 2),
Superellipsoid
Superparaboloid
2
Vg
y
y
fae (x)
1, (xp
a1
Q2
03
ai
a22
03
By adjusting the parameters, various surfaces can be represented as shown in the upper of Figure 2. For the superparaboloid, only the region where < 0 is used. Deformable superquadrics extend

---

Super- paraboloids
Super- ellipsoids
Box
Cylinder
Ellipsoid
Octahedron Bicone
c
0.2 F1 = 0.2 e
1.0 e;
2.0 6 2.0
6 0.2 e = 0.2
6
1.0
6
1.0 6 0.5
0.2 = 1.0
1.0 E2 0.2 62
1.0
C2 0.2 e2
1.0 = 1.0
0.2 1.0
Global deformation
Tapering
|k| increase
Bending
h increase
Shearing
s increase
Wine glass
Bowl & Dish
Bottle
Beer bottle
Handless Cup
Mug
TablewareNet
Figure 2: Upper: Examples of superquadries, including superellipsoids and superparaboloids, and deformable superquadrics. Lower: TablewareNer objects as unions of deformable superquadrics.
superquadrics by incorporating global deformations. Existing works, such as [7]. have employed traditional deformations including tapering and bending. Tapering transformation where the shape gradually narrows or widens along a certain direction. Bending an axis turns into circular section. Additionally, we employ a shearing transformation, where an object is skewed along specified axis. Each transformation used in this paper expressed in the following formula (deformations applied in the z-axis direction):
transformation where
Tapering
Bending
Shearing
Tt(2)a7
x + (R r) cos
x
y
Dz(x) = 1(=)y Dx(x)
= x -R-r) sin
Dz(x) =
(b - sin
Here, motivated by [7], we use a linear tapering function ((=) = (x) == #= + with a single parameter For bending. b and a represents the degree and the direction of bending in the xy plane, and other values are computed from these parameters; r cos atan2(y, x)) x)) + + R = b- (b-1 r) cos 7 Y 2b. In shearing, leverages the intensity of shearing. These deformations are concatenated in an obvious way, i.e., D = D. Da o D,; we note that the deforma- tions are not commutative. The deformable superquadric function is then given by fD = f D-1, where and f are a concatenated deformation and a superquadric, respectively. We can position the implicit function using an object pose T e SE(3) with the following expression: (D = 1.
2.2 Tableware Templates and Synthetic Dataset Generation
Tableware Templates. We combine deformable superquadrics to define templates representing seven types of tableware: wine glasses, bottles, beer bottles, bowls, dishes, handleless cups, and mugs, see the lower of Figure 2. Let T be tableware object constructed using deformable superquadrics {S = 1,..., ,n.. Each superquadric S; includes function type fi E {f fap pose T, E SE(3), size a, E R shape parameters e; IR? and deformation parameters d, = (k2, b,, as 8.). For T to represent an appropriate shape, constraints among these parameters are necessary. For example, to represent an appropriate wineglass, constraints on the positions of the head and the base, well the length of the pillar, needed. To easily assign these constraints, we re-parametrize the above parameters to low-dimensional tableware parameters; note that there is one-to- one mapping between tableware parameters and the superquadric parameters fi, T, a, c, d, for i = 1,...,, n.. More details are found in Appendix B.I.
Synthetic Dataset Generation. We then generate the TablewareNet dataset as shown Figure 3. By adjusting tableware parameters, we can create diverse 3D tableware meshes. Spawning these meshes in user-defined environment (e.g., table or shelf) within PyBullet [20]. physics simulator, allows us generate cluttered scenes. Using Blender [21], photorealistic renderer, with transpar-
3

---

Scene
Camera
Blender
Figure 3: Examples of the TablewareNet dataset, including 3D geometries of transparent objects a table or shelf (Upper) and RGB images rendered from Blender (Lower).
textures, obtain RGB images of the scenes from arbitrary poses. For data generation, first randomly spawn one four tableware objects. Each dataset includes the poses, tableware pa- rameters, bounding boxes of the objects, RGB images, mask images, and depth images from seven camera views. Detailed statistics can be found in Appendix B.2. Note that this dataset can be used not only to train our model but also for other data-driven recognition models. Additionally, users can create customized datasets by modifying the compositions of superquadries.
Tet:Net: Transparent Tableware SuperQuadric Network
In this section, we propose novel framework for recognizing the 3D shapes and locations of trans- parent objects using only partial multi-view RGB observations. Overall, method consists of four steps: (1) prediction of masks in 2D images, (2) prediction of 3D bounding boxes, (3) computation of smoothed visual hull through voxel carving, and (4) prediction of tableware parameters (i.e., set of parameters of the superquadrics); see Figure 1. We sequentially apply these modules during inference, which can accumulate prediction errors. Therefore, we develop techniques to train each module accurately and robustly against noise and sim-to-real gaps, which we explain in detail below.
Mask Prediction. Simulated RGB images often non-photorealistic, leading to significant sim- the pretrained segmentation network SAM [8], trained
to-real discrepancies. To mitigate this, on real images, to generate masks for the next module, the bounding box estimator. Specifically, we use SAM with a "tableware" prompt to generate masks. However, SAM sometimes incorrectly predicts parts of tableware objects as background. To address this, we apply random color- ditterjg five times to an image, obtain five corresponding SAM output mask images, and compute their union derive the final mask. Empirically, this approach significantly improves mask predictions.
3D Bounding Box Prediction. The mask images are then input into 3D bounding box prediction model, the DETR3D [9]. This model outputs the class probability along with the central coordinates and sizes of the bounding boxes along the æ. y, and axes for tableware objects. We apply random cut-out augmentation to randomly selected mask image to prevent the model from relying solely information from specific image among the multi- view mask images. This encourages the model to ex- tract information evenly from the multi-view masks for predictions of bounding boxes.
J
View
Ø
View
View
S
View
E View
Mask
Visualization of
Corrupted visual hull
GT shape
smoothed visual
20
20.2
20.4
2066
2088
2.0
Figure 4: Visualization of the corrupted vi- sual hull and smoothed visual hull when one mask image (view 3) is corrupted.
Smoothed Visual Hull. Voxel carving constructs visual hull by removing voxels from volumetric grid that do not match object silhouettes captured from multiple camera views [10]. Voxel carving is applied to a volumetric grid within each detected bounding box using the predicted masks as object silhouettes. We found that, although rarely, the predicted masks can confuse objects with the background, leading to significant distortion in the resulting visual hull when even one mask fails to identify object pixels and removes corresponding voxels, as shown Figure 4. construct more robust 3D representations, we count the number of images whose mask pixels include the projection of each corresponding 3D voxel instead of storing binary values. We then store normalized values representing the frequency of voxel appearances, which refer as the smoothed visual hull.

---

Grasp poses for tableware objects
Collision-free grasp poses
Figure 5: Left: 6-DoF grasp sampling results for various tableware objects. Right: Collision-free grasp poses for a target object A (blue wine glass); green grippers indicate collision-free grasp poses, while red grippers indicate poses with collisions.
Tableware Parameter Prediction. The smoothed visual hulls are input into neural networks based on ResNet3D [11 to predict the tableware parameter. Since each tableware class has different voxel resolutions and varying number of shape parameters, separate predictors are assigned to each class, After our bounding box predictor outputs the position, size, and class of a bounding box, the parameter predictor for the corresponding class takes as input the smoothed visual hull carved within the predicted bounding box. To train these networks, ground truth point cloud and point cloud sampled from the predicted superquadries in differentiable manner. Lastly, to enhance the model's robustness to bounding box prediction errors, we apply random perturbations to the location and size of the bounding box to generate a perturbed smoothed visual hull, which is then used for robust training. Further implementation details on our overall methods can be found in Appendix B.4.
use the Chamfer loss between the
Geometry-Aware Object Manipulation with TeSQNet
TesQNet offers many practical advantages for downstream object manipulation tasks. enables faster recognition compared to approaches that require real-time optimization, such NeRF vari- ants [3, 4J. Moreover, its instance-wise object recognition facilitates target-driven manipulation.
distinctive advantage of TesQNet arises from its use of extended deformable superquadrics for output representations. First of all, the implicit function representations for their surfaces enable rapid collision checks. Suppose the surface meshs of the robot's links and gripper are represented by surface points. We can determine if the robot is in collision with the recognized objects by assessing whether the implicit function values of the surface points are greater than or equal to zero.
Additionally, we can easily design an effective 6-DoF grasp sampler (see the left of Figure 5). Given the superquadric representations, we can intuitively design grasp pose distributions, which adapt continuously (even differentiably) almost everywhere as the superquadric parameters change. As demonstrated with the wine glass examples in the left of Figure (the top part of the wine glass on the far left can be grasped, whereas that of the other wine glass cannot), the distributions may not change continuously some shapes. Nonetheless, for the most part, the distributions change differentiably, offering the potential for applying gradient-based grasp pose modulation.
We conclude this section with our graspability aware object rearrangement algorithm, which demon- strates the effective use of the grasp pose sampler and rapid collision checking in downstream robotic tasks, Consider a scenario in which a target object A is initially not graspable, surrounded by a set of objects. Our objective is to determine how to reconfigure the surrounding objects through pick-and- place actions to make A graspable. To achieve this, we explore the action space to identify an action or sequence of actions that result in AA becoming graspable. This process requires rapid grasp pose sampling and computation of the graspability of A, which we accomplish by assessing collisions between grippers at sampled grasp poses and the surrounding objects; see the right of Figure 5. We refer further details including grasp sampler and object rearrangement to Appendix B.5.
Experiments
In this section, we empirically demonstrate that (i) our proposed model, TeSQNet, surpasses existing state-of-the-ar models in recognizing transparent objects, and (ii) method is highly effective in various robotic applications, including sequential decluttering and target retrieval tasks.
5

---

RGBs GT
NeRF
Dex-NeRF
DVGO
Mask DVGO
GraspNeRF
TisQNet
Depth
Error
Low
High
Occ.
Figure 6: Recognition results from RGB images from test set Tableware dataset. Upper: Re- constructed depth images, Middle: Depth reconstruction errors within object masks, Lower: Recon- structed occupancy maps; for SQNe, the occupancy grid is colored by the instance.
Environment. We use the 7-DoF Franka Emika Panda robot equipped with parallel- jaw gripper and an Intel RealSense D435 cam- era mounted on gripper. The raw visual in- put consists sequence of seven RGB images captured from different robot poses. Figure illustrates the real-world robot environment, in- cluding the robot, camera, table, shelf, and the transparent objects used in the experiments.
Robot environment
Transparent objects
Figure 7: Real-world robot environment setting and various transparent objects.
Baseline Methods. We compare SeNN with the following baseline methods: NeRF [2), Dex- NeRF [3], DVGO [22], DVGO using segmentation masks (denoted as Mask DVGO) adopted from [5], and GraspNeRF [6]. The differentiable rendering-hased methods, including NeRF, Dex-NeRF, DVGO, and Mask DVGO, input multi-view RGB images fit 3D opacity field, which is then pre-pro pr-cooeessed into other representations (e.g., depth image or occupancy map) as needed. For the segmentation mask for Mask DVGO, we use the SAM [8]. GraspNeRF is a generalizable NeRF- based model that inputs multiple RGB images from random view-poses and predicts a voxelized truncated signed distance field (TSDF). Detailed implementations can found Appendix C.I.
5.1 Transparent Object Recognition Performance
In this section, we demonstrate the performance advantages of T SQNet over existing methods. We adopt two metrics evaluate 3D recognition quality: depth accuracy and volumetric occupancy IoU. Depth accuracy is the ratio of predicted depth values that fall within certain threshold of the ground-truth depth values in depth image. This accuracy is measured only for pixels where the ground-trune truth object is present, using depth images obtained from the view new-pssess used during train- ing. Volumetric occupancy loU is is the ratio of the intersection to the union between ground-truth occupancy voxel grid and predicted occupancy voxel grid. Further details for calculating evaluation metries for each method can be found in Appendix C.I.
We compare the recognition performances of SQNet, trained with TablewareNet, and the bascline methods using (i) 40 scenes from the test set of TablewareNet, with 10 scenes each containing one to four objects, and (ii) 40 scenes from the TRansPose dataset [12], with 10 scenes each containing one to four tableware objects. Figure shows the recognition results, including reconstructed depth images (with depth reconstruction error maps) and occupancy grids on test set of Tableware T"sQNet exhibits the lowest depth error and demonstrates the best performance in terms of 3D occupancy, accurately delivering instance information.
Table shows the quantitative recognition results. Differentiable rendering- methods struggle due to sparse partial views. Specifically, the opacity field seems to concentrate near the camera lens, resulting in lower depth accuracy in Dex-NeRF, which measures depth based on the distance to an opacity exceeding certain threshold, compared NeRF. Among these methods, Mask DVGO has the best performance but still shows relatively low performance from an occupancy IoU perspective. GraspNeRF achieves higher volumetric IoU performance than other differentiable rendering- based methods, likely due to its use of supervised learning to train ground truth TSDF values. How-

---

Table Depth and occupancy reconstruction accuracy on TablewareNet and TRansPose dataset.
TablewareNet
TRansPose [12]
Depth
Occ.
Depth
Occ.
METHOD
00.05
00.10
00.20
loU
00.05
6p.10
00.20
loU
NeRF [2]
0.068
0.134
0.310
0.000
0.062
0.122
0.279
0.000
Dex-NeRF [3]
0.061
0.115
0.278
0.000
0.018
0.037
0.131
0.000
DVGO [22]
0.144
0.288
0.664
0.029
0.158
0.307
0.646
0.017
Mask DVGO [5]
0.697
0.885
0.963
0.394
0.785
0,928
0.962
0.487
GraspNeRF [6]
0.771
0.827
0.888
0.554
0.716
0.778
0.881
0.551
SQNe (ours)
0.944
0.958
0.968
0.740
0.918
0.936
0.961
0.636
ever, GraspNeRR faces challenges in training due to its high-dimensional voxel output representation compared TesQNet. TeSQNet, with its lower-dimensional output, benefits from easier training and better generalization performance. resulting in the best overall performance. Additionally, both GraspNeRF and SQNet show competitive results the TRansPose dataset, indicating that Table- wareNet includes a sufficiently diverse and reliable set of tableware shapes. Further experimental results with more example figures are provided in Appendix D.1.
5.2 Object Manipulation Performance
In this section, we demonstrate the effectiveneess of our model, TeNNee, on two object manipulation tasks: (i) sequential declutter, which involves sequential grasping cluttered environment, and (ii) target retrieval, which involves object rearrangement planning to retrieve an initially non-graspable target object?. The target object is indicated by target tableware class name (e.g., wineglass).
Sequential Declutter. For sequen-
Table
Sequential declutter
tial declutter, TesQNet first recog- nizes the geometries of the tableware objects from input RGB images, as shown on the left of Figure 8. Grasp poses are then generated for each ob- ject using 6-DoF grasp sampler, as described in Chapter 4. Among the generated grasp poses, those for which inverse kinematics are solved are selected, and the grasp pose with the least likelihood of collision is chosen, detailed in Chapter 4. Af- ter grasping and removing object, the next grasp pose is generated to ex- ecute sequential grasping. Figure 8. shows an example of sequential declutter experiment results. Based on the accurately predicted geometries of the objects, our method succeeds sequentially grasping the objects without re- recognition, while avoiding collisions with other objects and the environment.
Recog Observe
Shelf
Sequential declutter
Recog. Observe
Figure 8: Examples of sequential declutter experiment re- sults a table and shelf.
We compare the sequential declutter perfor- mances of TSSQNet, Mask DVGO (which shows the best recognition performance among differ- entiable rendering-basee based methods), and GraspN- eRF. We use the depth image-based pretrained FC-GQ-CNN [23] and the TSDF-based Volumet- ric Grasping Network (VGN) [24] for grasp pose generation in Mask DVGO and GraspNeRF, re- spectively. The experiments are conducted five times for each scenario: one object (single) or four objects (cluttered) within the scene, for both table and shelf environments. Success is defined as
Table 2:
Real-world declutter success rates in single (S) and cluttered (C) environments for ta- ble and shelf.
Shelf
Table
METHOD
S
C
S
C
Mask DVGO [5]
1/5
0/5
0/5
0/5
GraspNeRE [6]
3/5
1/5
1/5
0/5
T2 (ours)
5/5
4/5
4/5
3/5
the robot successfully sequentially grasping all objects in the scene. The experimental details for
'The real-worke manipulation videos can be found https//yoouulkwwm3777tttsuu

---

sequential clutter can be found in Appendix C.2. Table shows the sequential declutter success rates in real-world experiments and demonstrates that our method outperforms the other baseline meth- ods by significant margins. Failure of baselines mainly results from inaccurate recognition results. Further experimental results and discussions are provided Appendix D.2.
Target Retrieval. For target retrieval, we first define graspability function, which is set to 1 if the object is graspable and 0 otherwise. Given target object, we sample grasp poses for the recognized target and check for collisions of the grippers with the environment (e.g., shelf) and surrounding other objects. The graspability function value is then set to if at least one collision-free grasp pose exists, and otherwise. Further implementation details are provided in Appendix B.4.
Figure shows an example of target retrieval experiment results. T'SQNet recognizes the tableware objects and checks whether there is exactly one object of the target class; in this case, the target object is a wineglass. To make the target ob- ject graspable, sample some pick- and-place actions and select action sequences that maximize the graspa- bility function. The experimental details can be found in Appendix C.3. TestNee-based method successfully rearranges the surrounding objects and finally retrieves the target wineglass. Note that since we currently use a sampling- terms of the number of actions (i.e., unnecessary rearranging actions appear). but this can be further improved with advanced object rearrangement methods [25, 26, 27, 18]. Our method achieves 80% success rates in the five trials. More details provided in Appendix D.3.
Shelf
Target retrieval
Recog. Observe
Wineglass
Tick-and-place actions
Retrieval
Figure 9: Examples of target retrieval experiment results on shelf (the target object the wineglass).
method and a discrete reward function, is not optimal in
5.3 Limitations and Future Directions
First, TeSQNet currently cannot recognize objects from unknown classes because it relies on known shape templates. One future direction is to exploit research that attempts to represent complex objects as multiple superquadries without predefined templates [28, 7, 29, 30]. Second, we need to use fixed set of camera view poses for both training and testing because DETR3D, which we use detect bounding boxes the objects, requires initially fixed camera view poses. To overcome this limitation, need to develop model learning technique capable of handling various camera poses, like the method used in GraspNeRF that aggregates 2D image features in 3D volume space to make the model capable of arbitrary views [6]. Lastly, when multiple objects extensively overlap in images, the performance of 3D bounding box estimation is likely to decline since masks alone may not sufficiently reveal object boundaries. Utilizing instance segmentation information is thus an interesting direction for future work.
Conclusion
We have introduced novel model, 7 SQNer and a novel dataset, TablewareNet, designed to rec- ognize and manipulate transparent tableware objects from partial-view RGB images. By leveraging deformable superquadries and TablewareNet dataset, TeSQNet achieves accurate, instance- wise ge- ometric representations essential for effective manipulation tasks. Our experiments demonstrate that TesQNet outperforms existing state-of-the methods in predicting the geometries of transparent objects and excels in sequential declutter and target retrieval tasks.

---

Acknowledgments
Y. H. Kim, S. Kim, and C. Park were supported in part by IITP-MSIT grant RS-2021-1121206 (SNU AI Innovation Hub), IITP-MSIT grant 2022-220480, RS RS-2022-1122204 (Training and In- ference Methods for Goal Oriented Al Agents), IITP-MSIT grant RS-2024-00436666 (Collabora- tive Research Projects with Microsoft Research) under the Global Research Support Program in the Digital Field program, KIAT grant P0020536 (HRD Program for Industrial Innovation), SRRC NRF grant 25-2202000020052, SNU-AIIS, SNU-IPAI, SNU-IAMD, SNU BK21+ Program in Me- chanical Engineering. SNU Institute for Engineering Research, and Microsoft Research Asia. Y. Lee was the beneficiary of an individual grant from CAINS supported by a KIAS Individual Grant (AP092701) via the Center for Al and Natural Sciences at Korea Institute for Advanced Study.
References
[1] S. Sajjan, M. Moore, M. Pan, G. Nagaraja, Lee, A. Zeng, and S. Song. Clear grasp: 3d shape estimation of transparent objects for manipulation. In 2020 IEEE international conference rubotics and automation (ICRA), pages 3634-3642. IEEE, 2020.
[2] B. Mildenhall, P. Srinivasan, M. Tancik, T. Barron, R. Ramamoorthi, and Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
[3] J. Ichnowski, Y. Avigal, J. Kerr, and K. Goldberg. Dex-nerf: Using a neural radiance field to grasp transparent objects. In Conference on Robot Learning, pages 526-536. PMLR, 2022.
[4] J. Kerr, L. Fu, H. Huang, Y. Avigal, M. Tancik, J. Ichnowski, A. Kanazawa, and K. Goldberg. Evo-nerf: Evolving nerf for sequential robot grasping of transparent objects. In 6th annual conference on robot learning, 2022.
[5] J. Lee, S. M. Kim, Y. Lee, and Y. M. Kim. Nft: Normal field learning for 6-dof grasping of transparent objects. IEEE Robotics and Automation Letters, 2023.
[6] Q. Dai, Y. Zhu, Y. Geng, C. Ruan, J. Zhang, and H. Wang. Graspnerf: Multiview-based 6-dof grasp detection for transparent and specular objects using generalizable nerf. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 1757-1763. IEEE, 2023.
[7] S. Kim, T. Ahn, Y. Lee, J. Kim, M. Y. Wang, and F. C. Park. Dsqnet: deformable model- based supervised learning algorithm for grasping unknown occluded objects. IEEE Transac- tions on Automation Science and Engineering, 2022.
[8] A. Kirillov, E. Mintun, N. Ravi, Mao, Rolland, L. Gustafson, Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick. Segment anything. arXiv:2304.02643, 2023.
[9] Y. Wang, V. C. Guizilini, T. Zhang, Y. Wang, H. Zhao, and J. Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In Conference on Robot Learning. pages 180-191. PMLR, 2022.
[10] W. N. Martin and J. K. Aggarwal. Volumetric descriptions of objects from multiple views. JEEE transactions on pattern analysis and machine intelligence, (2):150-158, 1983.
[11) H. Kataoka, T. Wakamiya, K. Hara, and Y. Satoh. Would mega-scale datasets further enhance spatiotemporal 3d cnns? arXiv preprint arXiv:2004.0490 2020.
[12] J. Kim, M.-H. Jeon, S. Jung, W. Yang. M. Jung, J. Shin, and A. Kim. Transpose: Large-scale multispectral dataset for transparent object. The International Journal of Robotics Research, page 02783649231213111, 2023.
[13] Solina and Bajesy. Recovery of parametric models from range images: The case for superquadrics with global deformations. IEEE transactions pattern analysis and machine intelligence, 131-1,7, 1990.

---

[14] A. Makhal, F. Thomas, and A. P. Gracia. Grasping unknown objects in clutter by superquadric representation. In 2018 Second IEEE International Conference on Robotic Computing (IRC), pages 292-299. IEEE, 2018.
[15] G. Vezzani, Pattacini, and L. Natale. grasping approach based on superquadric models. In 2017 IEEE International Conference Robotics and Automation (ICRA), pages 1579-1586. IEEE, 2017.
[16] G. Vezzani, U. Pattacini, G. Pasquale, and L. Natale. Improving superquadric modeling and grasping with prior on object shapes. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 6875 6882. IEEE, 2018.
[17] Kim, Lim, Y. Lee, and Park. Se (2)-equivariant pushing dynamies models for tabletop object manipulations. In Conference on Robot Learning, pages 427-436. PMLR, 2023.
[18] S. Kim, Y. H. Kim, Y. Lee, and F. C. Park. Leveraging 3d reconstruction for mechanical search on cluttered shelves. In Conference on Robot Learming, pages 822-848. PMLR, 2023.
[19] Kim. Learning for Vision-Bact Based Object Manipulation: A Shape Recognition-Bacs Ap- proach. PhD thesis, Seoul National University, Seoul, South Korea, 2024.
[20] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. 2016.
[21] B. O. Community. Blender a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. URL http: / /www. blender.org
[22] C. Sun, M. Sun, and H.-T. Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 5459-5469, 2022.
[23] V. Satish, J. Mahler, and K. Goldberg. On-policy dataset synthesis for learning robot grasping policies using fully convolutional deep networks. IEFE Robotics and Automation Letters, 2019.
[24] M. Breyer, J. Chung, L. Ott, S. Roland, and N. Juan. Volumetric grasping network: Real-time dof grasp detection clutter. In Conference on Robot Learning, 2020.
[25] Lee, Y. Cho, C. Nam, Park, and C. Kim. Efficient obstacle rearrangement for object manipulation tasks in cluttered environments. In 2019 International Conference Robotics and Automation (ICRA), pages 183-189. IEEE, 2019.
[26] J. Lee, C. Nam, J. Park, and C. Kim. Tree bearch-based task and motion planning with pre- hensile and non- non-prenieee prehensie manipulation for obstacle rearrangement in clutter. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 8516-8522. IEEE, 2021.
[27] E. R. Vieira, D. Nakhimovich, K. Gao, R. Wang, J. Yu, and K. E. Bekris. Persistent homology for effective non-prehensile manipulation. In 2022 International Conference on Robotics and Automation (ICRA), pages 1918-1924. IEEE, 2022.
[28] D. Paschalidou, A. O. Ulusoy, and A. Geiger. Superquadrics revisited: Learning 3d shape parsing beyond cuboids. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10344-10353, 2019.
[29] W. Liu, Wu, S. Ruan, and G. S. Chirikjian. Robust and accurate superquadrie recovery: probabilistic approach. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2676-2685, 2022.
10

---

[30] W. Liu, Y. Wu, S. Ruan, and G. S. Chirikjian. Marching-primitives Shape abstraction from signed distance function. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8771-8780, 2023.
[31] J. Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg, Dex- net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. arXiv preprint arXiv: 703.09312, 2017.
[32] A. Zeng, S. Song, K.-T. Yu, E. Donlon, F. R. Hogan, M. Bauza, D. Ma, O. Taylor, M. Liu, E. Romo, al. Robotic pick-and-place of novel objects in clutter with multi-afforddace grasp- ing and cross-domain image matching. The International Journal of Robotics Research, 41(7): 690-705, 2022.
[33] Y. Tang, J. Chen, Z. Yang, Z. Lin, Q. Li, and W. Liu. Depthgrasp: Depth completion of transparent objects using self-attentive adversarial network with spectral residual for grasping. In 202/ IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5710-5716. IEEE, 2021.
[34] H. Fang, H.-S. Fang. S. Xu, and C. Lu. Transeg: A large-scale real-world dataset for transpar- ent object depth completion and grasping baseline. IEEE Robotics and Automation Letters, 7(3):7383-7390, 2022.
[35] Q. Dai, Zhang. Q. Li, Wu, H. Dong, Liu, P. Tan, and H. Wang. Domain randomization- enhanced depth simulation and restoration for perceiving and grasping specular and transparent objects. In European Conference Computer Vision, pages 374-391. Springer, 2022.
[36] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF ternational conference on computer vision, pages 10012-10022. 2021.
[37] Z. Li, Y.-Y. Yeh, and M. Chandraker. Through the looking glass: Neural 3d reconstruction of transparent shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1262-1271, 2020.
(38) E. Xie, W. Wang, W. Wang, M. Ding, C. Shen, and P. Luo. Segmenting transparent objects in the wild. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIII 16, pages 696-711. Springer, 2020.
[39] H. Xu, Y. R. Wang, S. Eppel, A. Aspuru-Guzik, F. Shkurti, and A. Garg. Seeing glass: joint point cloud and depth completion for transparent objects. arXiv preprint arXiv:21 10.00721 2021.
10.00087,
[40] X. Chen, H. Zhang, 7. Yu, A. Opipari, and O. Chadwicke Jenkins. Clearpose: Large-scale transparent object dataset and benchmark. In European conference computer vision, pages 381-396. Springer, 2022.
[41] Y. R. Wang, Y. Zhao, H. Xu, S. Eppel, A. Aspuru-Guzik, Shkurti, and A. Garg. Mv- trans: Multi-view perception of transparent objects. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3771-3778. IEEE, 2023.
[42] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[43] T.Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117-2125, 2017.

---

Appendix
A Related Works
A.1 NeRF-based Transparent Objects Recognition for Manipulation
NeRF [2] and its variants use differentiable rendering to find optimal 3D vision models (e.g., 3D color fields, occupancy fields) from multiple 2D RGB images. Due to NeRF's ability to learn 3D structures without using depth images, several studies [3, 4] have attempted to use NeRF for grasping transparent objects. Dex-NeRF [3] has first proposed transparent object recognition and grasping using NeRF, introducing depth rendering technique suitable for grasping differing from traditional NeRF approaches. This work utilizes the rendered depth images to grasp objects using a pre-trained Dex-Net [31]. Building on this, Evo-NeRF [4] has enhanced performance by improving the time efficiency of the training process and adding a geometry regularizer term. Recently NFL [5] has employed pre-trained mask estimators and 2D normal field estimators to more accurately capture 3D geometry.
The primary limitation of these NeRF-based methods is their dependency on having access to all- around, hemispheric views. When only partial views are available, these methods experience catas- trophic failures. Furthermore, while NeRFs effectively capture the overall geometry of a scene, they do not provide details about specific object instances. This absence of instance-specific information poses significant challenges for tasks requiring target-driven manipulation.
A.2 Learning-based Transparent Objects Recognition for Manipulation
In this section, we summarize learning-baad -base methods developed for transparent object recognition and manipulation. One of the dominant approaches attempts refine corrupted depth image using information from an RGB image, and then apply existing depth image-based grasp pose generation algorithms [32]. For instance, Cleargrasp [1] has proposed method that takes an RGB-D image of scene containing transparent objects, uses the RGB portion to estimate surface normals, detect boundaries, and segment objects with a neural network, and then globally optimizes these outputs to restore the damaged depth image. DepthGrrasp [33] has developed a GAN-based generator to directly produce a completed depth image from raw RGB-D input. Similarly, TransCG [34] has provided a dataset of raw and refined depth images of real-world transparent objects and train a U-Net structure neural network predict refined depth images from RGB images and incomplete depth images. Moreover, SwinDRNet [35] has proposed depth image completion using two-stream Swin Transformer [36), introducing domain randomization to tackle sim-to-real domain shift issues.
However, the grasp pose generation methods based on depth images have fundamental limitations. First, full 6-DoF grasping is not possible, resulting in reduced diversity of grasp poses. Second, there is often lack of information on the 3D geometry of objects, making it difficult to generate collision-free grasping trajectories. Third, most cases do not involve instance-wise representation, which complicates target-driven manipulation.
GraspNeRF [6] predicts TSDF values from multiple RGB images and trains a model to predict grasp poses using VGN [24]. This method overcomes some of the disadvantages mentioned above. Since directly generates grasp poses, it can also be trained to create 6-DoF grasp poses. Additionally, because it outputs TSDF values, it has information on 3D geometry, enabling the generation of collision-free trajectories. However, since the current version does not provide instance information, target-aware manipulation is challenging. One of the most significant differences in our research is that we use extended deformable superquadrics for 3D scene representation, which, compared to TSDF, is more memory-efficient and allows for much faster collision checks and grasp sampling.
We would like to emphasize that all the learning-hasee methods mentioned above face a sim-to-real issue. Because RGB images in simulation differ from real images, models trained on simulated RGB images commonly experience significant performance drop when applied to real images. It would be ideal to use real-world data, but collecting it is quite challenging. Although various methods
12

---

have been introduced in previous studies [35, 6] to enhance robustness to the sim-to-real gap, none have been particularly effective. In our research, we developed a model that uses mask images as input, which we found to be significantly more robust to the sim-to-real gap compared to models that directly take RGB images as input.
Additionally, [37] combines rendering- based techniques for calculating light refraction and reflec- tion on transparent surfaces with learning methods for normal estimation and shape refinement. Such research excels in providing highly detailed surface representations of objects without shape prior when (i) the environment map is fully known, (ii) the object's center position is known, and (iii) all-around views are available from the real-world. Meanwhile, our method, while offering relatively constrained surface expressiveness using extended deformable superquadries, can predict the shape of objects in unknown poses within unknown environments, with only partial views.
13

---

B Implementation Details for Our Methods
B.1 Details for TablewareNet Objects
In this section, describe the constraints the superquadric parameters and poses for each table- ware object mentioned in Section 2.2. wine glasses, bottles, beer bottles, bowls, dishes, handleless cups, and mugs and explain how these constraints are represented using tableware parameters.
Notations. Following the notation used in Section 2.2., each tableware object is composed of superquadrics {i ........,., ..). with their function type, pose, size parameter, and shape parameter denoted St, T; at, and e;, respectively. Additionally, their deformation parameters d, (k,, bi, ag, s,). If there is deformation, d, (0, 0, 0, 0); denote the absence of bending as b, 0 for convenience, as bending diminishes when b approaches zero. In actual implementation, deformation Dú is not computed when b; 0.
denoted as
Wine Glass. The wine glass template encompasses wide range of wine glass and champagne glass shapes. This template consists of three superquadries: two superellipsoids Si, S2 for the foot and stem, and superparaboloid S3 for the bowl. The tableware parameters and the constraints the superquadrics are given in Figure 10 and Table 3.
2t3
e,- (ty, 1)
Parameter
Desc.
Range
d2 (4g. 0,0,0)
Pose
E SE
t
Foot thickness
t1 [0.003, 0.006
t2
Stem radius
t2 E [0.003, 0.006
t3
Bowl radius
f3 E [0.03, 0.057]
e, (0.2, 1)
14
Wine glass Height
ta E [0.155, 0.264
d, = 0,
15
Bowl height ratio
ts (0.4,0.0
202
t6
Foot/bowl radius ratio
ts E 0099, 1.1
17
Bowl ej
ty 00.6,200
(T)
e (0.2, 1)
88
Bowl A
ls E 0.00
d, 00,0,0,0)
Table 3: Tableware parameters of wine glasses.
26346
Figure 10: Description for the wine glass parameters.
Bowl. The bowl template consists one superquadric: a superparaboloid S₁ for the bowl. The tableware parameters are given in Figure 11 and Table 4.
Parameter
Desc.
Range
21z
T
Pose
T E SE
t
Bowl d1
t1 E [0.04, 0.15]
(T)
la
Bowl 02
E [0.04,
Bowl 03
la E [0.02,0.1
24g
2t;
La
Bowl
14 [0.01,0.3]
(t,,
ts
Bowl eg
t; E [0.1,
(C,, 0, 0,
t6
Bowl A
to E [-0.1,0.3)
Figure 11: Description for the bowl pa- rameters.
Table 4: Tableware parameters of bowls.
14

---

Bottle. The bottle template consists of three superquadrics: superellipsoids S1, S2, Sy for the body, shoulder and finish. The tableware parameters and the constraints on the superquadries are given in Figure 12 and Table 5.
2t
e3 (0.2,1.0)
Es
d3 (0,0,0,0)
e2 (0.2,
ta
d 1/ + 4
0,0,0)
Parameter
Desc.
Range
T
Pose
T E SE(3)
t
Body radius
t1 E [0.03, 0.055
12
Body height
12 [0.10, 0.23
13
Shoulder height
by E [0.03, 0.05
f2
f4
Finish radius
E [0.008, 0.012]
e₁ (0.2,4))
ts
Finish height
to E (0.01,0.02)
d1 = 00,0,0.0)
Overall 62 tB E [0.2, 1.0]
(T)
Table 5: Tableware parameters of bottles.
24
Figure 12: Description for the bottle rameters.
Beer bottle. The beer bottle template consists of three superquadries: superellipsoids S1, S2, S3 for the body, shoulder and neck. The tableware parameters and the constraints the superquadries given in Figure 13 and Table 6.
2t
Is
e, (0.05, L.0)
d, (tg. 0,0,0
Parameter
Desc.
Range
e, (0.05, 1)
Pose
T E SF(3)
Eg
G11 [2)
Body radius
t1 E [0.025, 0.05)
d2 4,11 (2]1-6))+
0,0,0)
f/2
Body height
12 [0.12, 0.19
13
Shoulder height
t3 [0.01, 0.07
82
e, (0.05, 1)
L4
Neck radius
4 [0.014, 0.016)
d, 00,0,0,0
1s
Neck height
Is E 10.07, 0.1]
t6
Neck &
to E [-0.2,
Table Tableware parameters of beer bottles.
Figure 13: Description for the beer bot-
tle parameters.
15

---

Handless Cup. The handless cup template consists of one superquadrie: superparaboloid Sj. The
tableware parameters are given in Figure 14 and Table 7.
Parameter
Desc.
Range
e, (tz, 1)
T
Pose
T E SE(3)
d, (4,, 0,0,0)
t
Cup radius
t E [0.025, 0.05
Cup height
f2 [0.05, 0.22]
fg
Cup e1
t3 E [0.01, 0.3]
Cup k
t4 E [0.0, 0.3
(T)
Table 7: Tableware parameters of handless cups.
2t,
Figure 14: Description for the handless
cup parameters.
Mug. The mug template consists of superquadrie: superparaboloids S1, S2 for the cup and handle. The tableware parameters and the constraints the superquadries given in Figure 15 and Table 8.
Parameter T
Desc.
Range
Pose
T SE
a (s. 1st. 12lr)
π
Cup radius
41 E [0.025, 0.05
t2
Cup height
f2 [0.08, 0.12]
e2 = (0.2, 4p)
f2
t
2
ty
Cup 61
t3 E (0.01, 0.3
d2 (0, 1277 0,1t)
t4
Cup k
t4 E [-0.2, 0.2
(T)
e, = (t,, 1)
15
Handle 6
Is E [0.002, 0.003)
d = (4, 0,0,)
16
Handle 02/a /a
lg [1.0,
20g
17
Handle length ratio
17 E 0.5,0.7
18
Handle e2
8g 0.0
Figure 15: Description for the mug pa- rameters.
tg
Handle s
tg E f 0.5, -0.0001
Table 8: Tableware parameters of mugs.
Dish. The dish template consists of one superquadric: superparaboloid Si. The tableware param- eters are given in Figure 16 and Table 9.
= (tg, tg) d, (tg, 0,
Parameter
Desc.
Range
T
Pose
T SE(3)
t1
Dish radius
t1 E [0.08, 0.14
+(T)
t2
Dish height
E (0.015,0.08
to
Dish é1
f3 E [0.01, 0.3]
2t,
t4
Dish
f4 [0.5, 1.0
to
Dish
ts E [0.3, 0.6
Figure 16: Description for the dish pa- rameters.
Table 9: Tableware parameters dishes.
16

---

B.2 Details for Tableware Dataset
Our dataset employs uniform random sampling for each object class, offering infinitely continuous variations in object shapes, unlike other datasets with finite number of fixed shapes. We generate TablewareNet objects using uniformly random sampled shape parameters and spawn them in the physics simulator for each scene. Since the parameters of objects in all scenes are sampled from a uniform distribution, is rare for exactly the same object to be spawned. For each scene, we provide mask images, depth images, and RGB images from seven different camera poses using the synthetic camera parameters of the RealSense D435. Additionally, each scene includes 3D geometry information including object poses, tableware parameters, class labels, bounding boxes, meshes, and TSDF values. Currently, The dataset features two versions: with transparent objects shelf and another with on table. Each dataset includes 128,000 scenes, with 32,000 scenes each containing 1, 2, 3, or 4 objects. The training, validation, and test data consist of 120,000, 4,000, and 4,000 scenes, respectively.
Comparison with Existing Dataset. Here we provide brief comparisons between the Tableware Dataset and existing datasets containing transparent objects. TRans [38] consists of real RGB images of transparent objects with segmentation masks generated by manual annotation, but it only includes segmentation information and lacks 3D information such as pose annotations or object CAD models. Cleargrasp [1] and TODD [39] provide real-world images of transparent objects along with annotated pose information for those objects with 3D CAD models, but they include only a small number of object types, no more than 10. ClearPose [40] and TRansPose [12] offer annotation-rich, large-scale real-world datasets containing relatively many transparent objects, up to 100. However, the number of objects used in data generation is still small for recognizing wide variety of transparent objects.
Recently, Syn-TODD [41] has provided a large-scale dataset with diverse object sets using synthetic objects and photorealistic renderer named Blender. Our dataset is also in the spirit Syn-TODD, as it aims to generate dataset containing various objects using synthetic objects. The main differ- ences between TablewareNet and Syn-TODD is the procedure of generating synthetic objects. The authors of Syn-TODD generate synthetic objects using combinations of linear, polynomial, and si- nusoidal functions with various parameters. This procedural generation provides variety of shapes but can also produce unrealistic transparent objects, and objects with multiple parts (e.g., mugs with handles) are difficult to generate. Mcanwhile, we have generated transparent objects that are realis- tie and can have multiple parts by utilizing extended deformable superquadrics. However, due to the smoothness the superquadric, it difficult to express surfaces with many curvatures, such as those of a cup. We believe that the two transparent object generation methods have different advantages and disadvantages and that they can complement each other in the future.
B.3 Examples of Class Supplementation Using Deformable Superquadrics
In this section, we present examples of additional objects that can generated using deformable superquadrics, which are not covered in our dataset. Perfume Bottle. Perfume bot- tles come in various shapes and are transparent objects com- monly seen in daily life. However, they are not included in our dataset because they are not tableware. The perfume bottle template consists of two superquadries: superellipsoids S1, S2 for the body and cap. Plastic Cup with Spherical Lid. Plas- tie cups with spherical lids are casily seen in the most coffee shops. This template consists of two superquadries: superel- lipsoids Si, S2 for the cup and lid. The example of perfume bottle and cup with spherical lid represented by deformable superquadrics can be found in Figure 17.
Perfume Bottle
Cup with Spherical Lid
Figure 17: Perfume bottle and plastic cup with spherical lid represented by deformable perquadrics.
17

---

B.4 Details for Model Architecture and Training Process
B.4.1 Bounding Box Prediction Model Architecture
In this section, we describe the structure of the neural network used for bounding box prediction. We utilize the architecture of DETR3D [9], which takes fixed multi-view RGB images as input and outputs multiple 3D bounding boxes. DETR3D first obtains 2D features from the multi-view images using ResNet [42] and FPN [43]. It employs Transformer architecture, where each layer decodes candidate positions for bounding box centers using a sub-network for object queries. These decoded positions are projected onto each image plane, and the corresponding image feature values are in- corporated into the object queries using multi-head attention. The refined object queries are then used as input for the next layer. From each object query passing through the layers, the bounding box center, size, and class are predicted using an MLP structure for training. During inference, the bounding boxes and classes are predicted from the object queries of the final layer. For more details, refer [9].
We adopt the DETR3D structure with a few modifications. Instead of using a classifier to predict the object class for each object query, we pre-assign classes to the object queries and use a confidence predictor to predict a value in [0, 1] whether the query corresponds to a present object. For example, there are 3500 object queries and 7 classes, the first 500 queries represent the first class, and the confidence predictor assigns high confidence (we use a threshold value of 0.75) one of these queries, it is predicted that bounding box for an object of that class exists. Unlike the original DETR3D using RGB image inputs, we use binary mask images inputs.
B.4.2 Bounding Box Prediction Model Training
Matching and Loss Calculation. As in the original DETR3D, we use bipartite matching to cal- culate the loss. Since we have pre-assigned classes each query, the matching occurs within each class. We use binary cross-entropy for the confidence scores and L, loss for the bounding box ordinates and size. The weights between the binary cross-entropy loss and the LI loss are set to 5:1. After calculating the loss for the optimal matching within each class, we sum these losses across all classes to obtain the final loss value.
Training Process. We train the network using 12,000 scenes from the TablewareNet dataset; al- though our model can utilize the 120,000 scenes, we use only the 12,000 samples that include RGB images for comparison with other CGB-based baselines. We employ the Adam optimizer with an initial learning rate of 0.00005. A cosine annealing scheduler is used, and the training process spans 200 epochs. Using a validation set composed of 400 scenes, we defined the best model as the one with the highest mAP, calculated with a bounding box loU threshold of 0.75.
Cutout Augmentation. For cutout augmentation, the number of holes per mask image is determined by uniformly sampling an integer between 0 and 2. The size of each hole is determined by uniformly sampling integer between 50 and 100 for both the width and height; note that mask image size is (240,324 320). The position of each hole is also uniformly sampled within the image pixels.
B.4.3 3D Voxel Representation of Smoothed Visual Hull
We need to voxelize the predicted bounding boxes with fixed voxel size L,, to create the raw 3D space to be carved. Here, i represents the class index. Given that bounding boxes can vary in size, voxelizing them directly would result in different resolution of voxels, disabling the inference of the 3D CNN and FCN architecture. To address this, we follow a specific procedure to standardize the raw voxel representation.
First, we inspect the all size of the ground truth bounding boxes of class i and store the maximum size (W iun Himas i Dimm). Using this maximum size, create standardized 3D voxel space centered on the predicted bbox center. This ensures that all voxel representations have consistent resolution (W tm Lt, H i Li, D Dimal Le).
18

---

Next, we calculate the smoothed visual hull within this standardized voxel space. To further refine the representation, we add a channel to the voxel grid, where voxels inside the predicted bounding box are assigned a value of I, and those outside are assigned a value of O. The final voxel input to the network thus takes the shape (w... Li, Hi Li; Di Li, 2), with the first channel repre- senting the smoothed visual hull and the second channel indicating bounding box occupancy. This standardized approach ensures consistent and accurate input for the 3D CNN, facilitating reliable network inference.
B.4.4 Tableware Parameter Prediction Model Architecture
We employ ResNet3D [11] FCN architecture to predict the tableware parameters, including the pose T. using the smoothed visual hull voxel as input. Given that each tableware class has different voxel resolutions and the number of parameters, we train separate predictors for each class. In other words, there are as parameter predictors as the number of the classes, with each predictor dedicated to estimating the parameters of a single class of tableware objects.
In addition, assume that all objects are upright. Therefore, instead of predicting the entire SE(3) matrix for the object pose T, the model only predict the relative position E IR3 to the bounding box center and the rotation around the 2-axis, 0 E [0, 2x).
B.4.5 Tableware Parameter Prediction Model Training
Chamfer Distance as Loss function. The tableware parameters output by the parameter predictor could be directly trained via supervised learning using the L1 or L2 distance from the ground truth tableware parameters. However, this approach may not yield zero loss value even when the pre- dicted object shape matches the ground truth shape due to symmetrical ambiguities. For instance, orientations of objects with circular symmetry such as wine glasses, beer bottles, handless cups, and dishes, do not affect their shapes. In the case of bottle, rotating the object by 90 degrees results in the same shape, thus the ground truth shape has four possible orientations. For bowls, swapping the width i and length l2 and rotating the orientation by 90 degrees along the z-axis results in the same shape. We want a loss function that yields a zero value when the predicted shape matches the ground truth shape, regardless of these symmetrical ambiguities.
To address this, we use the Chamfer distance between point clouds sampled from the predicted and ground truth object surfaces the loss function. This requires differentiable point cloud sampling method from deformable superquadric parameters, which will discuss later. For a tableware object composed of multiple superquadries , we compute the Chamfer distance for each deformable superquadric part. Specifically, if Pi.pred and P..gt are the point clouds sampled from the predicted and ground truth i-th deformable superquadric Si.pred and Si.gt respectively, the Chamfer loss defined as:
chamfer Ps.pred, Pi.ge)
Differentiable Point Sampling on Deformable Superquadries. For superquadrics without defor- mation, there exists an explicit parameterization as follows;
x [a cos 0 " o a2 cos A sin 22 &
a1 cos Φ azh sin 22 ④ as (h2/al - -1)]
Xsc =
y
=
Xap =
y =
2
ag sin $1 0
where - 7/2 < 0 < π 7/2. - の S h and 0 < h < 1, and cos" 0 = sgn(cos 0) cos 0J= and
sin e 0 = sgn(sin (sin 0)] sin $
We utilize uniform grid in 0, Φ. and h coordinates to sample points on the superquadric surface using this explicit parameterization. This method is differentiable with respect to the superquadric parameters.
19

---

When dealing with deformations, we apply the transformations D,. Db. and D, described in Section 2.1 to these sampled points. These transformations are also differentiable with respect to deforma- tion parameters.
During training, observe empirically that adding L, loss for tableware param- eters (excluding the object's 's pose) and MSE loss for position regularizers accelerate the training process. We do not use a regularizer for orientation due to symmetry issues previously discussed. The weights used between the position MSE loss, parameter L1 loss, and Chamfer loss were set to : 0.1: 1. The model is trained with 1 12,000 data which include RGB images. The input smoothed visual hulls are generated using ground truth bounding boxes and mask images for each scene. Sim- ilar to the bounding box predictor training. we employed the Adam optimizer with an initial learning rate of 0.00005, using cosine annealing scheduler. The training process spanned 200 epochs. Us- ing validation set composed of 400 scenes, defined the best model as the with the lowest chamfer distance.
Training Process.
Data Augmentation with Bounding Box Perturbation. During inference, errors in the bounding box predictions may lead to low performance of the parameter predictor if it is trained solely on visual hulls generated from ground truth bounding boxes. To address this, we add noise to the bounding box's position and size to create perturbed visual hull during training. Specifically, as mentioned in B.4.3., while generating the voxel representation, we apply perturbations to the center and size of the bounding boxes when creating the second channel representing the bounding box occupancy. This data augmentation strategy ensures that the parameter predictor remains robust to prediction errors from the bounding box predictor, enabling Soet to maintain high performance even when bounding box predictions are not perfectly accurate.
B.5 Details for Geometry-Awan Object Manipulation
Collision Checking. The implicit function representations for deformable superquadrics enable precise and rapid collision checks. Suppose the surface meshes of the robot's links and gripper are represented by surface points P, c R In practice, we represent the last two links (i.e., links and 7 of the 7-DoF Franka Emika Panda robot) and the grippers of the robot, where collisions likely to occur, as point cloud P, {xj E RR رリر where nt, is the number of points of the point cloud. Let a recogmized object be composed of n extended deformable superquadrics {S.} Each deformable superquadric S is represented by an implicit function S,(x) = fs o D; 1T, 1, where fi E {fse. } is the superquadric equation of the i'th part defined by superquadric parameters, T; is the pose of the i'th part, and D, the deformation of the i'th part.
=
The main idea to check for collision is that point E R is inside the i'th part S, when the value S,(x) is less than 1; otherwise, it is outside. Using this fact, we can determine if the robot is in collision with the recognized objects by assessing whether the implicit function values of S; for the surface points are greater than or equal to zero. The collision can be determined by
(min S:(x)) > 1) 1.3
where i = 1, n is the object index, j = 1, ..., n, is the point index of the robot point cloud P,. and I()) is the indicator function. If the value 1, there is no collision between the robot and the recognized object; if 0, there is collision.
6-DoF Grasp Sampler. After obtaining the deformable superquadric representation of the transpar- ent objects, we sample feasible grasp poses from the obtained shapes. While it is possible to find the antipodal point by utilizing the implicit function of the deformable superquadric and its closed-form normal vector [7], we manually design a faster and more diverse grasp sampler in this paper. Since each object consists of multiple deformable superquadric parts, we first generate grasp poses for each part. Inspired by previous works [17], we manually generate top-down and side grasp poses for the superellipsoids according to their shapes. For superparaboloids, we generate grasp poses that grasp the edge. Example grasp poses can be found in Section 4. After generating grasp poses for each part, we check whether the grasp poses avoid self-collision with the object. Grasp poses with
20

---

After generating the grasping trajectories, we should check whether the robot following the trajec- tory collides with the surrounding tableware objects or the environment (e.g., table or shelf). For the k'th grasping trajectory, we obtain an afterimage mesh of the robot and the gripper and obtain the a distance between the antipodal points greater than 7.5 em are removed from the candidates, as the maximum gripper width of the Franka gripper is cm.
Details for Sequential Declutter. Suppose that no objects are recognized and the i'th recognized object is composed of n, extended deformable superquadrics {S We additionally represent the environment (c.g., table or shelf) using superquadrie implicit functions and consider the envi- ronment as the (n, + 1)'th object; for example, shelf can be composed of five boxes, so it is represented by five superquadric implicit functions ()= Rng+1
For sequential declutter, we first sample the grasp poses using the 6-DoF grasp sampler for all recognized objects. For each recognized object, we sample up to 30 grasp poses. For each grasp pose, manually design the grasping trajectory, in which the gripper approaches about 20 along the z-direction of the gripper. Among the grasping trajectories, inverse kinematics cannot be solved. We denote each grasping trajectory by index k.
reject those for which
point cloud sampled from the afterimage mesh; denote the point cloud as Pa {xf E R 11= where n, 2048. We can determine that the k'th ''th grasping trajectory does not collide with the recognized objects and the environment if the value
1 min min Siz(x)) > 1)
is one for i 1, 1....,,, no -, 1,.., ni and 1,....... ...... Myy Sometimes there may be multiple grasping trajectories that do not result collision. In this case, the grasping trajectory with the largest value of min, miny, (xt) for is selected. Since all these computations are parallelizable, grasp planning for sequential declutter can be performed in real-time.
Details for Target Retrieval. Suppose that objects excluding the target object are recognized and the i'th recognized object is composed of n, extended deformable superquadrics {S.g 1= We also represent the environment using superquadric implicit functions and consider the environment as the (n, + 1)'th object.
For target retrieval, define graspability function, which is set to if the object is graspable and 0 otherwise. To compute the graspability function, we first sample ny kinematically feasible grasping trajectories for the target object. As in the case of sequential decluttering. we denote the afterimage point cloud of the robot and the gripper for the k'th grasping trajectory as Pk = {x/ E )= where n, = 2048. The target object defined to be graspable there is at least one collision free grasping trajectory. Therefore, the graspability function can be calculated by:
(max min min Siy > 1),
k
3.1
for 1, .,..., no + 1, 1, 1,.... ny. 1 1, Br. and h: 1, Mg. This function can also be calculated in real-time through parallel computation after recognition is performed.
To make the initially non-graspable target object graspable, we maximize the graspability func- tion using pick-and-place actions with a sampling-based model predictive control (MPC) approach. Since the dynamics of the objects after the pick-and-plact actions are precisely known, we do not use an additional learning-based model predict the dynamics. When sampling pick-and-place actions, it is important not only to find collision collision-free trajectory for grasping the object to be picked-ande placed but also to identify placeable locations for the object. Therefore, it is crucial to check whether the robot collides with the recognized objects and the environment. The collision checking using superquadric representation makes this possible precisely and rapidly as described above. Among these sampled pick-and place actions, we find the action that maximizes the function the most, and then repeat the same process to perform pick-and-plan actions until the target object becomes gras- pable.
21

---

C Experimental Details
C.1 Additional Details for Recognition Experiments
Baseline Implementations. NeRF and Dex-NeRF are trained using the seven partial views available in TablewareNet and use the same views for depth rendering. We use the model from the final epoch of training for evaluation. We follow the approach described in [3] for depth rendering of NeRF and Dex-NeRF, with a a threshold of 15 for Dex-NeRF. To generate occupancy voxel grids, we render depth images from uniformly sampled view poses over hemisphere using the trained NeRF and Dex-NeRF models. These depth images are converted to TSDFs, and regions with negative TSDF values are marked as occupied. DVGO and Mask DVGO are also trained using the seven partial views. For depth rendering, both methods employ the depth rendering technique used by NeRF in [3]. Predicted occupancy voxels defined as those with an occupancy probability exceeding threshold of 0.1, based on the trained DVGO and Mask DVGO models. For GraspNeRF, we train the model with the TablewareNet dataset, where the training and validation data consist of 12,000 and 100 samples, respectively. We use rendering loss and TSDF loss, where ground truth TSDF values are used to derive the occupancy voxels directly from the predicted TSDF values. In detail, GraspNeRF is trained on TablewareNer dataset with slight modifications: we fix the camera view poses during both training and testing to match the input view poses of Te We do train the volumetric grasp detection module simultaneously, nor do we use the Eikonal regularization term in the geometry loss. Depth images directly rendered from the reconstructed TSDF values. For our model, SQNet, we train the model with the TablewareNet dataset, where the training and validation data consist of 120,000 and 1,000 samples, respectively. We obtain depth images and occupancy voxels using the implicit function representation of the superquadrics of the predicted tableware objects.
Test Dataset.
The test dataset consists of (i) 40 scenes from the test set of TablewareNet, with 10 scenes each containing one to four TablewareNet objects, and (ii) 40 scenes generated from TRansPose objects, with 10 scenes each containing one to four transparent objects. For test dataset generation using TRansPose objects, we first select some transparent objects from the TRansPose dataset [12]. The TRansPose objects we use are shown in Figure 18. Using the TRansPose objects, we follow the same process as when generating the TablewareNet dataset, as described Section 2.2. We first spawn random objects from the TRansPose objects within PyBullet and then render RGB images of the scenes using Blender with transparent textures.
Figure 18: TRansPose objects used for test dataset [12].
Runtime of TeeQNet. We recall that the SoNee consists of four steps: (1) prediction of masks in 2D images, (2) prediction of 3D bounding boxes, (3) computation of smoothed visual hull
22

---

through voxel carving, and (4) prediction of tableware parameters (i.c., set of parameters of the superquadrics).
(1) The average inference time for one RGB image using the segmentation model (i.e., SAM) is 0.69 seconds. use a total of camera views and infer a total 35 images by performing color jittering times on each camera view. Therefore, the average time takes to obtain the 2D mask is 24.2 seconds.
It takes an average of 0.28 seconds to infer 3D bounding boxes using mask images as input with DETR3D.
(3) It takes an average of 0.15 seconds obtain a smoothed visual hull from the bounding box of each object through voxel carving.
(4) It takes an average 0.004 seconds to predict tableware parameters via ResNet3I on the smoothed visual hull of each object.
The average total runtime of SONe,t, when there objects in and it takes 7 RGB images input to output the superquadries of all objects, is 25 seconds. If SAM could be inferred in batches, the overall runtime could be greatly reduced. All computation times are measured with a GeForce RTX 3090 and a Gen Intel(R) Core(TM) i9-1 19-11900K @ 3.50GHz.
C.2 Additional Details for Sequential Declutter Experiment
We generate five scenarios for each number of objects, one (single) and four (cluttered), and for each environment type, including shelf and table, resulting total of 20 scenarios. The initial settings for all scenes, including the configuration and poses of objects, testing the three baselines, including Mask DVGO, GraspNeRF, and Te place the same objects in the same pose as consistently possible for each scene.
be seen in Figure 19. When
Shelf Single
Shelf Cluttered
Table
Single
Cluttered
Table
Figure Initial scene settings for sequential declutter experiments.
23

---

C.3 Additional Details for Target Retrieval Experiment
We generate five scenarios with designated target objects in a shelf environment. The initial scene settings, including the target tableware class name (c.g., wine glass), for all scenarios can found in Figure 20. For the target retrieval experiment, we place the target object so that is not initially graspable.
Wineglass
Wineglass
Bowl
Mug
Mug
Figure 20: Initial scene settings for real-world target retrieval experiments.
24

---

D Additional Experimental Results
D.1 Additional Results for Recognition Experiments
Additional examples of transparent object recognition results shown in Figure 21 and Figure 22. The trends of these additional results are similar to the representative example in Section 5.1. For the test sets of the Tableware dataset in Figure 21, TeSQNet succeeds in recognizing accurate 3D geometries of the transparent objects while also delivering instance information. GraspNeRF per- forms best among the baselines but predicts less accurate results than TesQNet. Although TONNe has slightly lower performance on the Tableware dataset compared TRansPose, succeeds in predicting somewhat accurate instance-wise geometries, as shown Figure 22.
RGBs
GT
NeRF
Dex-NeRF
DVGO
Mask DVGO
GraspNeRF
T'SQNet
Depth
Error
Low
High
Oct.
Depth
Error
Low
High
Occ.
Depth
Error
Low
High
Occ.
Depth
Error
Low
High
Occ.
Figure 21: Recognition results from RGB images from test sets of Tableware dataset.
25

---

RGBs
GT
NeRF
Dex-NeRF
DVGO
Mask DVGO
GraspNeRF
T'SQNet
Depth
Error
Low
High
Occ.
Depth
Error
Low
High
Occ.
Depth
Error
Low
High
Oct.
Depth
Error
Low
High
Occ.
Figure 22: Recognition results from RGB images from TRansPose dataset.
26

---

Additional Results for TeSQNet on TRansPose Objects. We provide additional inference results of TeSNNet on TRansPose objects to (i) more closely compare the inference results of Te with the ground-truth surface shapes and (ii) assess how well TºSQNet generalizes to common transparent tableware objects. Figure shows the ground-truth shapes the transparent TRans- Pose objects alongside the inferred implicit surfaces from Tette Although capturing surface details such as the curvature of water bottle is challenging due to the nature of superquadric sur- faces, can confirm that TetQNet infers the overall shapes to considerable extent. In particular, the predicted shapes appear accurate enough to perform manipulation tasks, including sequential declutter and target retrieval, which highlights the high performance of our method in real-world object manipulation scenarios.
RGB image
Ground-truth
T'SQNet
Figure 23: Recognition results of TSQNNet TRansPose objects.
27

---

Additionally, we provide some failure cases of Tett on TRansPose objects, as shown in Fig- ure 24. The first case (upper row of Figure 24) involves an error in the segmentation mask. this instance, the long bowl was predicted to be short because the back part of the long bowl was not de- tected in the segmentation mask. The second case (middle row of Figure 24) involves an error bounding box predictor. The bounding box predictor fails predict the bounding box of one bottle, so Net does not predict any shape for the corresponding bottle. The third case is interesting failure case for the bounding box predictor, where it gets confused between two object classes and predicts two shapes for one object. This suggests a potential improvement for the current ToNNN Currently, when learning a bounding box predictor, bipartite matching is performed only within class, and loss function is provided. However, if we generalize this approach and perform bipartite matching between classes, we can avoid duplicate predictions. This an interesting direction for our future work.
RGB image
Ground-truth
T'SQNet
Figure 24: Failure cases in the recognition of TS on TRansPose objects.
28

---

Detailed Results and Analysis for Rendering-basee based Methods. We describe the recognition results of rendering-basee based methods when provided with limited partial views and analyze the underlying reasons for their failure to predict the correct depth images. NeRF and Dex-NeRF are trained using seven partial camera view poses over a total of 200,000 iterations, and the results are shown in Figure 25, DVGO and Mask DVGO are also trained using the seven partial camera view poses. DVGO employs a two-stage training process consisting of coarse stage with 5,000 iterations and fine stage with 20,000 iterations. The results of DVGO and Mask DVGO are shown in Figure 26 and Figure 27.
Iteration
500
3000
11000
100000
200000
GT
Rendered RGB
Reconstructed depth
NeRF
Dex-NeRF
Figure 25: Results of RGB rendering and depth prediction for NeRF and Dex-NeRF trained on
partial views across training iterations.
Coarse iteration
Fine iteration
1000
5000
1000
10000
20000
Depth Rendered RGB
Figure 26: Results of RGB rendering and depth prediction for DVGO across training iterations.
As illustrated in the figures, the rendering-based methods, except for Mask DVGO, fail predict accurate depth images even for the training views, despite successfully reconstructing RGB images for these views. Mask DVGO shows some success in predicting depth for the training views (e.g.. the front view) but performs poorly in novel views (e.g., the side view).
The core issue is that the ultimate goals of NeRFs are to match the rendered images to the ground truth images. In other words, any radiance field that can generate the ground truth image can final solution, and there is no method select which of these solutions accurately represents real- world geometry based solely the given RGB information. Note that with sufficient view poses sampled from a hemisphere, the solution set can be expected to approximate real-world geometry; this has been demonstrated by Dex-NeRF and its subsequent study [3, 4J. However, the problem our work targets involves situations where only partial views are accessible, without the expectation of such extensive camera poses.
The figures above show how the problem of infinite solutions for generating the given RGB images becomes more severe with partial views. As shown in Figure 25, NeRF misinterprets the floor and
29

---

Coarse iteration
Fine iteration
1000
5000
1000
10000
20000
Reconstructed depth Rendered RGB
Side
Front
Front view
Side view
Figure 27: Results of RGB rendering and depth prediction for Mask DVGO across training itera- tions. The first two rows show the rendered results from the front view. The last row shows the rendered depth images from the side view, obtained by rotating the camera pose of front view 100 degrees around the z-axis
world frame passing through the workspace center.
table surface depths as similar. Although the scene consists of a dark brown table placed on a light- colored floor, NeRF perceives the table as part of the floor with a darker brown area. The issue more pronounced with DVGO, where most voxels are initially generated near the camera lens. The RGB values of these voxels are trained match the real images the lens proximity, resulting in very shallow depth values for all pixels, observed in the Figure 26.
Mask DVGO, by utilizing mask information in addition to RGB, succeeds in predicting depth in training views. This success attributed to the regularization term that forces points in rays outside the mask region to have zero opacity, effectively constraining opacity to exist only within the visual hull obtained from training mask images and view poses. While this visual hull functions well for depth observed near training view poses, fails accurately represent geometry occluded or shadow regions, these areas cannot be carved out. As shown in Figure 27, rendering side-view depth from Mask DVGO results in high opacity and depth in regions occluded from front views, indicating inaccurate geometry. One might argue that the accurate front view depth is sufficient for obtaining grasping poses using works like DexNet [31]. However, this is unsuitable for the target retrieval conducted in this paper, where reliable geometry information occluded regions is crucial.
30

---

D.2 Additional Results for Sequential Declutter Experiments
More examples of sequential decluttering with TeNQ on shelves and tables are shown in Fig- ure 28 and Figure 29, respectively. Generally, our method succeeds in sequentially grasping objects without re-recognition, while avoiding collisions with other objects and the environment based on the accurately predicted geometries of the objects. However, there are several failure cases: (i) slightly incorret shape leads to unstable grasp pose, as shown in the third example of Figure 28, (ii) some objects are not recognized by TNNeet, as shown in the first example of Figure 29, and (iii) an inverse kinematics solution does not exist, as shown in the third example of Figure 29. The first and second failure cases can be resolved through a more accurate recognition model, as described in the future works section (Section 5.3). The third failure case can be addressed by designing a more diverse 6-DoF grasp sampler.
Shelf
Sequential declutter
Observe
Recog.
Observe
Recog.
Observe
Failure
Recog.
Observe
Recog.
Figure 28: Examples of sequential declutter experiment results on shelves.
31

---

Table
Sequential declutter
Observe
Recog.
Recognition fail
Observe
Recog-
Observe
Recog.
No IK solution exists
Observe
Recog-
Figure 29: Examples of sequential declutter experiment results on tables.
32

---

D.3 Additional Results for Target Retrieval Experiments
Additional results for target retrieval with Teet are shown in Figure 30. In the three examples above, TesQNet-based method successfully rearranges surrounding objects and retrieves target ob- jects through appropriate pick-and-place actions. In the last example, TeNe fails recognize wine glass; consequently, the robot performs action of directly retrieving the target object, the mug, and result, it grasps both the wine glass and the mug together.
Shelf
Target retrieval
Observe
Recog.
Bowl
Pick-and-place
Success
Observe
Recog.
Wineglass
Pick-and-play -lacce
Pick-and-
Success
Observe
Recog.
Mug
Pick-and-play
Success
Observe
Recog.
Mug
Failure
Figure 30: Examples of target retrieval experiment results shelves.
33

---

D.4 Comparison of Te with an End-to-End Method
To validate the effectiveness of our Teet framework, which combines several separate modules, we developed and trained a simple end-to-end model for comparison. The end-to-end model struc- ture is as follows: utilizes our modified DETR3D structure (where queries are pre-assigned classes and confidence estimation is incorporated) but replaces the bounding box predicting FCN with an FCN that predicts tableware parameters. Given that the dimension of tableware parameters varies by object class, separate FCNs are used for each class to predict the parameters. These class-specific FCNs take as input the queries assigned to the same class and output the corresponding tableware parameters. The learning loss for these parameters employs the same position regularizer, parameter regularizer, and chamfer loss as TONNeet fidence loss, position regularizer, parameter regularizer, and chamfer loss to 1:1:1:0.1, respectively. The training results are shown in Figure 31. We observed that training this end-to-end model considerably challenging, and hypothesize the following reasons:
this end-to-end method, we set the weights for con-
ground truth
T'SQNet
End-to-end
Figure 31: Visual comparison of SQNet output with end-to-end method output on validation set
Chamfer Loss Scale. As the centers of two objects diverge, the chamfer loss scale increases quadratically. During the initial training phases, failing to align the object's position significantly inflates the chamfer loss scale relative to the confidence loss and position loss. Conversely, as ob- jects come closer, the chamfer loss scale becomes much smaller than the confidence loss and other loss terms. This large variation in loss scale during training leads inconsistent bipartite matching results, hindering significant training progress.
Positional Constraints. In TesQNet, the object center position is constrained to remain within the bounding box, providing structured framework for the output. However, the end-to-end method bypasses the bounding box prediction process, resulting in the lack of such positional constraints. Consequently, the predicted object can be located anywhere within the workspace during the initial training stages, making it difficult to resolve the aforementioned issues.
In conclusion, the large scale variation of the chamfer loss during training makes it challenging to balance the losses, leading to unstable bipartite matching results and ultimately hindering the learning process. While the end-to-end method theoretically streamlines the process, the practical challenges in training and the inherent issues in loss scaling and positional constraints highlight the advantages of our modular T"SQNet framework.
34


Reviews:
[{"review_id": "T64QdpT0KR", "reviewer": "Reviewer_3VBo", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper deals with the manipulation of partially transparent objects. The core contribution is on the perception part, where the authors proposed a framework to reconstruct the geometry and shape of the 3D transparent objects. There is also a synthetic dataset generated. Real robot experiments are shown.", "summary_of_recommendation": "Not sure if object shape parameterization is a good way to approach the problem. This severely limits the scope of application of this method. This is also the reason that the comparison against prior work might be unfair..", "technical_quality": 3, "clarity_of_presentation": 5, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- How well can this method generalize to transparent objects that are never seen during training?\n- The authors may want to compare their method to [A] in terms of 3D reconstruction quality.\n- The authors can reply to other comments and issues raised in the \"Weakness\" part of my review above.\n\n[A] \"Through the Looking Glass: Neural 3D Reconstruction of Transparent Shapes\", Zhengqin Li, Yu-Ying Yeh, Manmohan Chandraker, CVPR 2020", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8nlxOP07Ow", "reviewer": "Reviewer_p6K4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper studies reconstruction and manipulation of transparent tableware objects from multi-view images. It proposes an extension to the superquadrics implicit function representation for representing non-convex surfaces, dubbed superparabaloids, that includes deformation relevant to common tableware, then implements a set of tableware objects as a collection of these surfaces. The authors come up with a generation function for sampling each category of object, and use this to generate a space of test objects which they provide as a synthetic dataset. They then propose a soft voxel carving method for estimating the geometry of objects from an input set of multi-view masks, obtained from SAM. Given this voxel grid and 3D bounding box, they pass the grid through a 3D convnet to obtain a final set of category-specific super-quadric parameters, which is trained on the proposed synthetic dataset. Synthetic experiments evaluate the reconstruction ability of the system, and physical experiments evaluate its usefulness for multi-stage decluttering tasks involving planning trajectories to remove objects from a scene (shelf or table).", "summary_of_recommendation": "The paper and Appendix addresses an important problem in robotics.  Overall the material is well executed and thorough, so I would recommend weak acceptance. My reservations are mainly about the experiments and understanding precisely why this method works better than prior methods, and the technical motivation for the algorithm, because at its core the method seems relatively hard to scale and constrained to very specific categories of objects.", "technical_quality": 4, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "-The paper states that NeRF-based approaches “overfit and producing highly inaccurate geometries”.  It would be very helpful to provide MUCH more intuition and detail on where prior methods fail.  The examples in the paper and Appendix are very hard to see and it would be better to go though one example in much more detail.\nCan the authors add inferences from the network of the output superquadrics, I only see occupancy visualized (except maybe figure 1). It would be useful to see the actual implicit surfaces themselves and compare against the ground truth.\n- Details on the runtime of a single scene would be appreciated, for example time from when images have been captured to final output superquadrics.\nPlease address the concern above that the examples in the paper and video seem to be somewhat contrived in that the objects are within shelves and do not appear to exhibit reflections and specularities.\n- Please clarify the details of the tableware parameter prediction. Because there can be a different number of tableware parameters per tableware category, does the Resnet3D predict also the category of object each is? Or is a resnet trained per-category (wineglass, bowl, etc), and the semantics from DETR3D are used to choose which resnet to use?\nPlease also say more about the limitation mentioned in the Limitations section, about the fixed set of 7 camera viewpoints.  Is this required for both training and testing?  Please show how the method performs if those 7 are not available.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dQ9qroLCxz", "reviewer": "Reviewer_hYHE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The task of identifying and manipulating clear tableware from partial view RGB images is indeed challenging due to the difficulty in obtaining reliable depth measurements of transparent objects. The Transparent Tableware SuperQuadric Network addresses this by using deformable superquadrics to create low-dimensional, instance-specific, and accurate 3D geometric representations of clear objects from partial views. Superquadrics are a parametric family of surfaces that can describe various shapes such as cubes, cylinders, and spheres using a continuous parameter space, making them a natural choice for geometric primitives due to their ability to model various shapes with few parameters. Additionally, TablewareNet, a toolset of seven parameterized shapes based on these extended deformable superquadrics, allows for the creation of new datasets of tableware objects of various shapes and sizes. Experiments with T^2SQNet trained with TablewareNet have shown that T^2SQNet surpasses existing methods in identifying clear objects, making it highly effective for robotic applications such as decluttering and target retrieval.", "summary_of_recommendation": "The task of identifying and manipulating clear tableware from partial view RGB images is indeed challenging due to the difficulty in obtaining reliable depth measurements of transparent objects. The Transparent Tableware SuperQuadric Network addresses this by using deformable superquadrics to create low-dimensional, instance-specific, and accurate 3D geometric representations of clear objects from partial views. Superquadrics are a parametric family of surfaces that can describe various shapes such as cubes, cylinders, and spheres using a continuous parameter space, making them a natural choice for geometric primitives due to their ability to model various shapes with few parameters. Additionally, TablewareNet, a toolset of seven parameterized shapes based on these extended deformable superquadrics, allows for the creation of new datasets of tableware objects of various shapes and sizes. Experiments with T^2SQNet trained with TablewareNet have shown that T^2SQNet surpasses existing methods in identifying clear objects, making it highly effective for robotic applications such as decluttering and target retrieval.  The idea for using deformable superquadric to generate diverse transparent objects is novel compared to existing methods. Preview methods either use existing cad model or procedure generation to create rendered scenes. deformable superquadric allows for better diversity and more complex object shapes. But I think discuss the difference between TablewareNet and existing dataset with procedure generation is needed, for example SynTODD. And the ability to not only recover 3d bounding box or depth but the complete object shape is very help for robot manipulation task, which is shown in the paper. However, the novelty for the T^2SQNet architecture is questionable. Given that pretrained SAM, DETR3d and resnet3d are all published work, using them to form a pipeline is not novel in my opinion.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "1. justify the novelty of T^2SQNet pipeline, what is authors contribution over SAM DETR3D and ResNet3D?\n2. compare deformable superquadric object generation with procedure generation method used in SynToDD", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Thank you for your submission to CoRL 2024. The reviewers appreciated many aspects of the paper, particularly the real-robot experiments and high-quality reconstructions, good figures, videos, explanations, comparisons and baselines, and the novel idea of using super quadratic parameterization to generate a diverse set of objects. Some reviewers also pointed out that the tableware dataset makes a nice contribution on its own.

Reviewers raised concerns about the method's ability to generalize outside the training set and its reliance on object parameterization, which may limit its ability to work on complex objects in realistic settings. They also raised concerns about the novelty of the architecture and the dismissiveness of prior related work. A reviewer suggested that seeing it applied to a robot manipulation setting would strengthen the paper.

While reviewers appreciated the authors' responses, they were not sufficiently convinced to raise their recommendation. This suggests that there is still more work to do, and as such, we encourage the authors to continue improving the paper for the camera-ready version.
Decision: Accept