HANDLOOM: Learned Tracing of One-Dimensional Objects for Inspection and Manipulation
Authors: Vainavi Viswanath
Kaushik Shivakumar
Mallika Parulekar
Jainil Ajmera
Justin Kerr
Jeffrey Ichnowski
Richard Cheng
Thomas Kollar
Ken Goldberg
Venue: CoRL 2023 Oral (2023)
Keywords: state estimation
deformable manipulation

Abstract: Tracing – estimating the spatial state of – long deformable linear objects such as cables, threads, hoses, or ropes, is useful for a broad range of tasks in homes, retail, factories, construction, transportation, and healthcare. For long deformable linear objects (DLOs or simply cables) with many (over 25) crossings, we present HANDLOOM (Heterogeneous Autoregressive Learned Deformable Linear Object Observation and Manipulation) a learning-based algorithm that fits a trace to a greyscale image of cables. We evaluate HANDLOOM on semi-planar DLO configurations where each crossing involves at most 2 segments. HANDLOOM makes use of neural networks trained with 30,000 simulated examples and 568 real examples to autoregressively estimate traces of cables and classify crossings. Experiments find that in settings with multiple identical cables, HANDLOOM can trace each cable with 80% accuracy. In single-cable images, HANDLOOM can trace and identify knots with 77% accuracy. When HANDLOOM is incorporated into a bimanual robot system, it enables state-based imitation of knot tying with 80% accuracy, and it successfully untangles 64% of cable configurations across 3 levels of difficulty. Additionally, HANDLOOM demonstrates generalization to knot types and materials (rubber, cloth rope) not present in the training dataset with 85% accuracy. Supplementary material, including all code and an annotated dataset of RGB-D images of cables along with ground-truth traces, is at https://sites.google.com/view/cable-tracing.

HANDLOOM: Learned Tracing of One-Dimensional Objects for Inspection and Manipulation
Vainavi Viswanath*1,
Kaushik Shivakumar* Mallika Parulekar"', Jainil Ajmera Justin Kerr', Jeffrey Ichnowski Richard Cheng,, Thomas Kollar", Ken Goldberg
equal contribation, equal contribution
vainaviv@berkeleyy edu, kaushhss@berkeeeyy.yee
Tracing - estimating the spatial state of - long deformable linear ob- jects such as cables, threads, hoses, or ropes, is useful for broad range of tasks homes, retail, factories, construction, transportation, and healthcare. For long de- formable linear objects (DLOs or simply cables) with many (over 25) crossings, we present HANDLOOM (Heterogeneous Autoregressive Learned Deformable Linear Object Observation and Manipulation) learning- based algorithm that fits a trace to greyscale image of cables. We evaluate HANDLOOM on semi-planar DLO configurations where each crossing involves at most 2 segments. HAND- LOOM makes use of neural networks trained with 30,000 simulated cxamples and 568 real examples to autoregressively estimate traces of cables and classify crossings. Experiments find that in settings with multiple identical cables, HAND- LOOM trace each cable with 80% accuracy. In single-cable images, HAND- LOOM can trace and identify knots with 77% accuracy. When HANDLOOM is incorporated into bimanual robot system, it enables state-based imitation of knot tying with 80% accuracy, and it successfully untangles 64% of cable con- figurations across 3 levels of difficulty. Additionally, HANDLOOM demonstrates generalization knot types and materials (rubber, cloth rope) not present the training dataset with 85% accuracy. Supplementary material, including all code and annotated dataset of RGB-D images of cables along with ground-truth traces, is at https://sites..ooglee wttppo/ww...o/ewww//beeeeeeeeeee
Abstract:
Keywords: state estimation, deformable manipulation
Introduction
Tracing long one-dimensional objects such as cables has many applications in robotics. However, this challenging task as depth images are prone to noise, and estimating cable state from greyscale images alone is difficult since cables often fall into complex configurations with many crossings. Long cables can also contain a significant amount of free cable (slack), which can oc- clude and inhibit the perception of knots and crossings.
Prior cable tracing research use combination learned and analytic methods to trace cable, but are limited to most 3 crossings [1, 2]. Previous cable manipulation work bypasses state estimation with object detection and keypoint selection networks for task-specific points based on geometric patterns [3, 4, 5, 6, 7, 8]. Sundaresan et al. [9] employ dense descriptors for cable state estimation in knot tying but focus on thick short cables with loose overhand knots. However, this work tackles the challenges of long cables in semi-planar configurations (at most 2 cable segments per crossing), which often contain dense arrangements with over 25 crossings and near near-parallel cable segments. Although works like Lui and Saxena [10] and Huang et al. [11] also perform cable state estimation,
UUniversity of California, Berkeley. "Carnegie Mellon University. "Toyota Research Institute.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Trace Prodictor
Crossing Classifier
Prev
Next
Training
Over
Loss
BCB
Under
Lose
RCB
Over
Under
Inference
Figure 1: Example of HANDLOOM single knotted cable with 16 crossings: HANDLOOM includes two networks. One network (pink at top) is trained predict next point context window, and the other network (blue at the top) is trained to classify over- and under-crossings. During inference, HANDLOOM 1) uses the pink network to autoregressively find the most likely trace (illustrated with rainbow gradient, from violet to purple, depicting the path of the cable) and 2) performs crossing recognition using the blue network obtain the full state of the cable (3), where red circles indicate overcrossings and blue circles indicate undercrossings. 4) The state estimate from HANDLOOM can be used for inspection and manipulation.
trace given a prior
not address challenging cases mentioned above which we consider. While analytic methods require heuristics to score and select from traces, learning to predict cable traces directly from images avoids this problem.
they use analytic methods
This work considers long cables (up to 3 meters in length) in semi aamn-alaaar configurations, i.e. where each crossing includes at most 2 cable segments. These cable configurations may include knots within a single cable (e.g. overhand, bowline, etc.) or between multiple cables (e.g. carrick bend, sheet bend, etc.). This paper contributes:
1. Heterogeneous Autoregressive Learned Deformable Linear Object Observation and Ma- nipulation (HANDLOOM), shown in Figure 1: A tracing algorithm for long deformable linear objects with up to or more crossings.
2. Novel methods for cable inspection, state-based imitation, knot detection, and autonomous untangling cables using HANDLOOM.
3. Data from physical robot experiments that suggest HANDLOOM can correctly trace long DLOs unseen during training with 85% accuracy, trace and segment a single cable multi- cable settings with 80% accuracy, and detect knots with 77% accuracy. Robot experiments suggest HANDLOOM in physical system for untangling semi-planar knots achieves 64% untangling success in under 8 minutes and in learning from demonstrations, achieves 80% success.
4. Publicly available code and data (and data generation code) for HANDLOOM:
2 Related Work
2.1 Deformable Object Manipulation
Recent advancements in deformable manipulation include cable untangling algorithms [5, 6, 7, 10], fabric smoothing and folding techniques [12, 13, 14, 15, 16, 17, 18], and object placement in bags [19, 20]. Methods for autonomously manipulating deformable objects range from model-free to model-based the latter of which estimates the state of the object for subsequent planning.

---

Model-free approaches include reinforcement or self-supervised learning for fabric smoothing and folding [21, 22, 23, 24] and straightening curved ropes [22], or directly imitating human ac- tions [12, 25]. Research by Grannen et al. [6] and Sundaresan et al. [7] employs learning-baat -baee keypoint detection to untangle isolated knots without state estimation. Viswanath et al. [3] and Shiv- akumar al. [4] extend this approach long cables (3m) with learned knot detection pipeline. However, scaling to arbitrary knot types requires impractical amounts of human labels across many conceivable knot types. Our study focuses on state estimation to address this limitation.
Model-based methods for deformable objects employ methods to estimate the state as well as dynamics. Work on cable manipulation includes that by Sundaresan et al. [9], who use dense de- scriptors for goal-conditioned manipulation. Descriptors have also been applied to fabric smoothing [14), as well visual dynamics models for non-knotted cables [26, 27] and fabric [15, 26, 28]. Other approaches include learning visual models for manipulation [29], iterative refinement of dy- namic actions [30], and using approximate state dynamics with learned error function [31]. Fusing point clouds across time has shown success in tracking segments of cable provided they are not tan- gled on themselves [32, 33). Works from Lui and Saxena [10] and Huang et al. [11] estimate splines of cables before untangling them, and we discuss these methods among others below.
2.2 Tracing Deformable I Linear Objects
Prior tracing methods for LDOs, including in prior work [4], primarily employ analytic ap- proaches [2, 10, 11, 34]. Other works, Jackson et al. [35] and Padoy and Hager [36], optimize splines to trace surgical threads. Very recently, Kicki et al. [2] use primarily analytic method for fast state estimation and tracking of short cable segments with 0-1 crossings. In contrast to these works, this work focuses on longer cables with a greater variety of configurations including those with over 25 crossings and twisted cable segments. Other prior work attempts to identify the loca- tions of crossings [37] in cluttered scenes of cables but does fully estimate cable state.
Several prior works approach sub-parts of the problem using learning. Lui and Saxena [10] use learning to identify weights on different criteria to score traces. Huang et al. [1 I] focus on classifying crossings for thick cables. Song et al. [8] predict entire traces for short cables as gradient maps but lack quantitative results for cables and evaluation of long cables with tight knots. Yan et al. [38] use self-supervised learning to iteratively estimate splines in a coarse-to-fine manner. In very recent work, Caporali et al. [1, 39] use learned embeddings to match cable segments on opposite sides of analytically identified crossings, in scenes with 3 or fewer total crossings, while we test HANDLOOM on complex configurations containing over 25 crossings.
3 Problem Statement
The workspace is defined by an (z, y. 2) coordinate system with a fixed overhead camera facing the surface that outputs grayscale images. We assume that 1) the greyscale image includes only cables (no obstacles), 2) each cable is visually distinguishable from the background, 3) each cable has least one endpoint visible, and 4) the configuration is semi- planar, meaning each crossing contains at most cable segments. We define each cable state for i = 1,.., cables to be (s()) = ((x(s), v(s), 2(s))} where is arc-length parameter that ranges [0, 1]. representing the normalized length of the cable. Here, (a(s),, v()), is the location of a cable point at a normalized are length of s from the cable's first endpoint. We also define the range of ((s)-that is, the set of all points on a cable at time t-to be C. We assume all cables are visually distinguishable from the background and that the background monochrome. Additional assumptions we make for the manipulation tasks are stated in Section 5.4.2.
Workspace and Assumptions:
Objective: The objective of HANDLOOM is to estimate the cable state pixel-wise trace function of of one specified cable indicating all over- and under-crossings.
4 Algorithm
HANDLOOM (Figure 1) includes learned cable tracer that estimates the cable's path through the image and crossing classifier with correction method that refines predictions.
3

---

Sim
Real
Figure 2: Cropped images of simulated and real cables for training HANDLOOM: On the left are simulated cable crops augmented with Gaussian noise, brightness, and sharpening match the real images (right).
4.1 Learned Cable Tracer Model
We break the problem of cable tracing down into steps, where each step generates probability distribution for the next point based on prior points. To achieve this, we employ learned model that takes an image crop and trace points from previous iterations and predicts a heatmap representing the probability distribution of the next pixel's location. By operating on local information within crops, the model mitigates overfitting and facilitates sim-to-real transfer, focusing on local characteristics rather than global visual and geometric attributes like knots.
More formally, representing the grayscale image as 7, and each trace 8i,tot in the image sequence of pixels 51,0: $1,1, we break the probability distribution over traces conditioned on the image into smaller, tractable pieces using the chain rule of probability. fe is learned neural network, crop(I, p) is a crop of image centered at pixel p. and is the context length.
P(st.to||Sipp 1)
II j=1
II 3=1
folsi. Siiiesssssj-e bi..--... crop(l, si, 81g--1)
4.1.1 Dataset and Model Training
To train the crossing classifier, we simulate a diverse range of crossing configurations to generate dataset. We Blender [40] create 30,000 simulated grayscale images that closely resemble real observations (Fig. 2). Cable configurations are produced through three methods with random Bezier curves: (1) selecting points outside small exclusion radius around the current point, (2) intentionally creating near- par-aaalllll segments, and (3) constraining specific cable segments to achieve a dense and knot-like appearance in certain spatial regions. The curves are colored white and have slightly randomized thicknesses.
We randomly sample image crops along the cable of interest, with a focus on cable crossings (rep- resenting 95% of samples). The simulated images augmented with pixel-wise Gaussian noise with standard deviation of 6, brightness with standard deviation of 5, and sharpening to imitate the appearance of real cables. Additionally, we include a smaller dataset of 568 crop images, sampled such that it comprises approximately 20% of the training examples. Training employs the Adam optimizer [41] with pixelwise binary cross-entropy loss, using a batch size of 64 and learning rate 10-5
real cable
4.1.2 Model Architecture and Inference
We use the UNet architecture [42]. We choose trace points spaced approximately 12 pixels apart, chosen by grid search, balancing between adding context and reducing overfitting. We further bal- ance context and overfitting by tuning the crop size (64 x 64) and the number of previous points fed into the model (3) through grid search.
Naively training the model for cable tracing in all possible initial directions leads to poor perfor- mance, as it would require learning rotational equivariance from data, reducing data efficiency. To overcome this, we pre-rotate the input image, aligning the last two trace points horizontally and ensuring the trace always moves left to right, optimizing the model's 's capacity for predicting the next trace point. We then rotate the output heatmap back to the original orientation.

---

During inference, initializing the trace requires an input of single start pixel along the cable (in practice, one endpoint). We use an analytic tracer as in [4] to trace approximately 4 trace points and use these points to initialize the learned tracer, which requires multiple previous trace points to predict the next point along the trace.
The tracer autoregressively applies the learned model to extend the trace. The network receives cropped overhead image centered on the last predicted trace point (64 x 64 pixels). Previous trace points are fused into gradient line (shown in Figure 1), forming one channel of the input image. The other two channels contain an identical grayscale image. The model outputs a heatmap (64 x 64 x 1) indicating the likelihood of each pixel being the next trace point. We greedily select the argmax of this heatmap as the next point in the trace. This process continues until leaving the visible workspace reaching endpoint, which is known using similar learned detectors Shivakumar al. [4].
4.2 Over/Undererossing Predictor
4.2.1 Data and Model Input
We use 20 x 20 simulated and real over/undercrossingg crops. The 568 real images are oversampled such that they are seen 20% of the time during training. Augmentation methods applied to the cable tracer are also used here to mimic the appearance of real cable crops. The network receives 20 x 20 3 input crop. The first channel encodes trace points fused into line, representing the segment of interest. To exploit rotational invariance, the crop is rotated to ensure the segment's 's first and last points are horizontal. The second channel consists of a Gaussian heatmap centered at the target crossing position, providing positional information to handle dense configurations. The third channel encodes the grayscale image of the crop.
4.2.2 Model Architecture and Inference
We use ResNet-34 classification model with sigmoid activation to predict scores between 0 and 1. The model is trained using binary cross-entropy loss. We determine the binary classification using threshold of 0.275 (explanation for this number is in the appendix Section 7.1.1). The algorithm uses the fact that each crossing is encountered twice to correct errors in the classifier's predictions, favoring the higher confidence detection and updating the probability of the crossing to (1- the original value), storing the confidences for subsequent tasks. The learned cable tracer, over/undderroosing predictor, and crossing correction method combined together result in a full cable state estimator: HANDLOOM.
4.3 Using HANDLOOM in Downstream Applications
Tracing in Multi-cable Settings: We apply HANDLOOM to inspection in multi-cable settings for tasks like locating the power adapter of a cable tangled with other visually similar cables given the endpoint. HANDLOOM is fed an endpoint and returns a trace to the adapter connected to it.
Learning from Demonstrations: We HANDLOOM enable physical, state-based knot tying in the presence of distractor cables, which is much more challenging for policy operating RGB observations rather than underlying state. Demonstrations are pick-place actions, parameterized by absolute arc-length or arc-length relative to crossings. More details are in Appendix Section 7.3.
Robot Cable Untangling: For cable untangling, HANDLOOM is combined with analytic knot detection, untangling point detection techniques, and bi-manual robot manipulation primitives to create system for robot untangling. Appendix Section 7.2 contains more details.
Physical Experiments
We evaluate HANDLOOM on 1) tracing cables unseen during training, 2) cable inspection in multi- cable settings, 3) learning knot tying from demonstrations, 4) knot detection, and 5) untangling.
The workspace has 1 m 0.75 m foam-padded, black surface, bimanual ABB YuMi robot, and an overhead Photoneo PhoXi camera with 773 1032 x 4 RGB-D observations. HANDLOOM is fed only the RGB image, but the depth data is used for grasping. The PhoXi outputs the same values
5

---

Table I: Generalization of HANDLOOM Different Cable Types
Cable Reference
Length
Color
Texture
Physical Properties
HANDLOOM
TR (trained with)
2.74
White/gra
Braided
Slightly stiff
6/8
2,09
Gray
Rubbery
Slightly thicker than TR
7/8
4.68
Yellow with
Rubhery and plastic
Very stifT
8/8
2,08
Tan
Rubbery
Highly clastic
7/8
1,79
Bright red
Braided
Flimsy
6/8
4.61
White
Braided
Flinsy
0/8
across all 3, making the observations grayscale and depth. See Appendix, Section 7.4 for details on failure modes for each experiment.
5.1 Using HANDLOOM for Tracing Cables Unseen During Training
For this perception experiment, the workspace contains a single cable with one the following knots: overhand, figure-eight, overhand honda, or bowline. Here, we provide HANDLOOM with the two endpoints test the tracer in isolation, independent of endpoint detection. We report success if the progression of the trace correctly follows the path of the cable without deviating.
Results in Table 1 show HANDLOOM can generalize to cables with varying appearances, textures, lengths, and physical properties. Cables can also be seen in Appendix Figure 9. HANDLOOM performs comparably on cable TR (the cable it was trained with) as it does on the other cables.
5.2
Using HANDLOOM for Cable Inspection in Multi-Cable Settings
Tier A1
Tier A2
Tier A3
Figure 3: Multi-cable tracing: here 3 pairs of images. The left of each pair illustrates an example from the and the right the successful trace. The traces from left to right encounter 24, 13, and 28 crossings.
For cable inspection, the workspace contains a power strip. Attached to the power strip are three white MacBook adapters with two 3 m USB C-to-C cables and one m USB-C to MagSafe 3 cable (shown in Figure 3). The goal is to provide the trace of a cable and identify the relevant adaptor given the endpoint. We evaluate perception in multi-cable settings across 3 tiers of difficulty.
1. Tier A1: No knots; cables are dropped onto the workspace, one at a time.
2. Tier A2: Each cable is tied with single knot (figure-eight, overhand, overhand honda, bowline, linked overhand, or figure-eight honda) measuring 5-10 em diameter, and sub- sequently dropped onto the workspace one by one.
3. Tier A3: Similar to tier A2 but contains the following 2-cable knots (square, carrick bend, and sheet bend) with up to three knots in the scene.
Across all 3 tiers, we assume the cable of interest cannot exit and re-enter the workspace and that crossings must be semi-planar. Additionally, we pass in the locations of all three adapters to the tracer and an endpoint to start tracing from. To account for noise the input images, we take images of each configuration. We count success if majority (2 of the 3 images) have the correct trace (reaching their corresponding adapter); otherwise, report failure.
We compare the performance of the learned tracer from HANDLOOM against an analytic tracer from Shivakumar et al. [4] as a baseline, using scoring rules inspired by Lui and Saxena [10] and Keipoured et al. [34]. The analytic tracer explores potential paths and selects the most likely trace based on scoring metric [4], prioritizing paths that reach an endpoint, have minimal angle deviations, and have high coverage scores. Table 2 shows that the learned tracer significantly outperforms the baseline analytic tracer all 3 tiers of difficulty with a total of 80% success across the tiers.

---

5.3 Using HANDLOOM for Physical Robot Knot Tying from Demonstrations
Knot-tying demonstrations are conducted on a nylon rope length 147 em and a diameter of mm. We tume our demonstrations to work on plain backgrounds; however, during rollouts, we add distractor cables that intersect the cable to be tied. These distractor cables are of identical types to the cable of interest; thus, manipulating the correct points requires accurate cable state estimation from HANDLOOM. Although these cables are thicker, more twisted, and less stiff than the cable HANDLOOM is trained with, HANDLOOM is able to generalize, tracing the cable successfully in 13 out of 15 cases. We evaluate the policy executed by the YuMi bimanual robot on the following: Tier B1: O distractors, Tier B2: distractor, and Tier B3: distractors. We count a success when a knot has been tied (i.e. lifting the endpoints results in a knot's presence).
Table 4 show 86% tracing success and 80% robotic knot tying success across the 3 tiers, suggesting HANDLOOM can apply policies learned from real world demonstrations, even with distractors.
5,4 Using HANDLOOM for Knot Detection and Physical Robot Untangling
To test HANDLOOM applied to the knot detection and physical untangling task, we use single white, braided USB-A micro-USB cable.
5.4.1 Knot Detection
The details on the state-based knot detection method, which uses the sequence of over- and under- crossings identify knots, can be found the Appendix Section 7.2.2.
Tier C1
Tier C2
Tier C3
Tier D1
Tier D2
Tier D3
Figure 4:
Starting configurations for the tiers for HANDLOOM experiments and robot untangling experi- ments. Here is the crossing count for these examples from left to right: 5, 10, 6, 4, 9, and 14.
We evaluate HANDLOOM on 3 tiers of cable configurations, shown in Figure 4. The ordering of the categories for these experiments does not indicate varying difficulty. Rather, they are categories of knot configurations to test HANDLOOM on.
Tier C1: Loose (35-40 em in diameter) figure-eight, overhand, overhand honda, bowline, linked overhand, and figure-eight honda knots.
2. Tier C2: Dense (5-10 em in diameter) figure-eight, overhand, overhand honda, bowline, linked overhand, and figure-eight honda knots.
3. Tier C3: Fake knots (trivial configurations positioned to appear knot-like from afar).
We evaluate HANDLOOM on the following 3 baselines on the 3 tiers.
1. SGTM 2.0 [4] perception system: using a Mask R-CNN model trained on overhand and figure-eight knots for knot detection.
2. HANDLOOM (-LT): replacing the Learned Tracer with the same analytic tracer from Shiv- akumar et al. [4] as described in Section 5.2 combined with the crossing identification.
3. HANDLOOM (- (-CC): using the learned tracer and crossing identification scheme to do knot detection without Crossing Cancellation, covered the appendix (Section 7.2.3).
We report the success rate of each these algorithms as follows: a knot is present, the algorithm successful if it correctly detects the first knot (i.e. labels its first undercrossing); if there are no knots, the algorithm is successful it correctly detects that there are no knots.
As summarized in Table 3, knot detection using HANDLOOM considerably outperforms SGTM 2.0. HANDLOOM (-LT), and HANDLOOM (-CC) on tiers C1 and C3. SGTM 2.0 marginally outperforms HANDLOOM in tier C2. This is because SGTM 2.0's Mask R-CNN is trained on dense overhand and figure-eight knots, which are visually similar to the knots in tier C2.
7

---

Table 3: Knot Detection Experiments
Table 2: Multi-Cable Tracing
Her
SGTM 2.0
(-LT)
(-CC)
HL
Tier
Analytic
Learned
C1
2/30
14/30
20/30
24/30
A1
3130
26/30
28/30
8/30
21730
26/30
2730
23/30
A3
1/30
23/30
HL. HANDLOOM. HANDLOOM vatperforms the huseline and
Learned (HANDLOOM) compared to analytic tracer from Shivakumar al. [4].
ablations all tiers except tier C2. Explanation provided in Section
Table Robot Untangling Experiments (90 total trials)
Table 4: Learning From Demos
Tier D1
Tier D2
Tier D3
SGIM20
SGTM
HL
SGIM 2.0
HL
Knot Suce.
11/15
12/15
6/15
11/15
9/15
14/15
B1
5/5
515
Knot 2 Suce.
2/15
6/15
82
4/3
413
Verif. Rate
11/11
8/12
6/6
6/11
2/6
TTme (min)
1.10011
2.1::0.3
3.5
3.91111
Corr. Trace cnrrect trace, perception result Suce. perception and manipulation success.
Knot 2 Time (min)
3.121.22
7.5
Verif. Time (min)
5.7-0.9
6.1414
6.4:1..8
10.1::0.7
5.4
9.6 1.5
HL HANDLOOM, Across all tiers, HANDLOOM ounperforms SGTM 2.0 on untangling success. SGTM 2.0, bowever, octperforms HANDLOOM on verification.
Details provided in Section 5.4.2
5.4.2 Physical Robot Untangling
For the untangling system based on HANDLOOM, compare performance against SGTM 2.0 [4], the current state-of-the-ar algorithm for untangling long cables, using the same 15-minute timeout on each rollout and the same metrics for comparison. We evaluate HANDLOOM deployed on the ABB YuMi bimanual robot in untangling performance on the following levels difficulty (Figure 4), where all knots are upward of em in diameter:
Tier D1: Cable with overhand, figure-eight, overhand honda knot; total crossings S 6.
Tier D2: Cable with bowline, linked overhand, or figure-eight honda knot; total crossings E [6, 10).
Tier D3: Cable with knots (1 each from tiers DI and D2); total crossings E [10, 15).
Table 5 shows that our HANDLOOM-based untangling system achieves a higher untangling success rate (29/45) than SGTM 2.0 (19/45) across tiers of difficulty, although SGTM 2.0 is faster. The slower speed of HANDLOOM attributed to the requirement of a full cable trace for knot detection which is inhibited by the fact that the cable is 3 as long as the width of the workspace, Ieading to additional reveal moves before performing an action or verifying termination. On the other hand, SGTM 2.0 does not account for the cable exiting the workspace, benefiting speed, but failing to detect off- off-wooks knots, leading premature endings of rollouts without fully untangling.
Limitations and Future Work
HANDLOOM assumes cables are not occluded by large, non-cable objects, are semi- ammi-planar con- figurations, and distinguishable from the background, which is one uniform color. Additionally, HANDLOOM is not trained on cables with wide range of physical (e.g. thickness) and material (elasticity, bending radius) properties, such as loose string or rubbery tubing; and certain empirical experimentation suggests that for significantly thicker ropes and threads that form "kinks", perfor- mance of HANDLOOM suffers due to the out-of-distribution nature of these cases. Future work will aim to mitigate these problems and also generalize across more broadly varying backgrounds and properties of cable. Another area of work will involve sampling multiple likely traces from the model and using resultant uncertainty signal downstream means of mitigating noise.

---

Acknowledgments
This research was performed the AUTOLAB UC Berkeley affiliation with the Berkeley AT Research (BAIR) Lab. The authors supported in part by donations from Toyota Research Institute.
References
[1] A. Caporali, R. Zanella, D. D. Greogrio, and G. Palli. Ariadne+: Deep learnin-bbssee aug- mented framework for the instance segmentation of wires. IEEE Transactions on Industrial Informatics, 18(12):860-8860
2022. doi: 10. 109/THI 2000/TT.0222.154447777
[2] Kicki, A. Szymko, and K. Walas. Dloftbs fast tracking of deformable linear objects with b-splines. In 2023 IEEE International Conference Robotics and Automation, 2023.
[3] V. Viswanath, K. Shivakumar, J. Kerr, B. Thananjeyan, E. Novoseller, J. Ichnowski, A. Es- contrela, M. Laskey, J. F. Gonzalez, and K. Goldberg. Autonomously untangling long cables. Robotics: Science and Systems (RSS), 2022.
[4] K. Shivakumar, V. Viswanath, A. Gu, Y. Avigal, J. Kerr, Ichnowski, R. Cheng, T. Kollar, and K. Goldberg. Sgtm 2.0: Autonomously untangling long cables using interactive perception. arXiv preprint arXiv:2209. 13706, 2022.
[5] V. Viswanath, J. Grannen, P. Sundaresan, B. Thananjeyan, A. Balakrishna, E. Novoseller,
J. Ichnowski, M. Laskey, J. E. Gonzalez, and K. Goldberg. Disentangling dense multi-cable knots. Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), 2021.
[6] J. Grannen, P. Sundaresan, B. Thananjeyan, J. Ichnowski, A. Balakrishna, M. Hwang, V. Viswanath, M. Laskey, E. Gonzalez, and K. Goldberg. Untangling dense knots by learning task lask-relevant keypoints. Conference on Robot Learning. 2020.
[7] Sundaresan, J. Grannen, B. Thananjeyan, A. Balakrishna, Ichnowski. E. Novoseller, M. Hwang, M. Laskey, J. E. Gonzalez, and K. Goldberg. Untangling dense non-planar knots by learning manipulation features and recovery policies. Proc. Robotics: Science and Systems (RSS), 2021.
[8] Y. Song, K. Yang. X. Jiang, and Y. Liu. Vision based topological state recognition for de- formable linear object untangling conducted in unknown background. In 2019 IEEE In- ternational Conference on Robotics and Biomimetics (ROBIO), pages 790-795, 2019. doi: 00.100000810049422219999999 89911922.9.6
[9] Sundaresan, J. Grannen, Thananjeyan, A. Balakrishna, M. Laskey, K. Stone, J. Gon- zalez, and K. Goldberg. Learning rope manipulation policies using dense object descriptors trained on synthetic depth data. In Proc. IEEE Int. Conf. Robotics and Automation (ICRA), 2020.
[10] W. H. Lui and A. Saxena. Tangled: Learning untangle ropes with RGB-D perception. In 2013 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, pages 837-844. IEEF, 2013.
[11] X. Huang, D. Chen, Y. Guo, X. Jiang, and Y. Liu. Untangling multiple deformable linear objects in unknown quantities with complex backgrounds. IEEE Transactions on Automation Science and Engineering, pages 1-13, 2023. doi: 10.1 109/TASEE.023.333333333
[12] D. Seita, Ganapathi, Hoque, M. Hwang, Cen, K. Tanwani, A. Balakrishna, B. Thananjeyan, J. Ichnowski, N. Jamali, et al. Deep imitation learning of sequential fab- ric smoothing from algorithmic supervisor. In Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), 2020.

---

[13] т. Weng, S. M. Bajracharya, Y. Wang, K. Agrawal, and D. Held. Fabricflownet: Bimanual cloth manipulation with a flow-based policy. In Conference on Robot Learning, pages 192- 202. PMLR, 2022.
[14] A. Ganapathi, P. Sundaresan, B. Thananjeyan, A. Balakrishna, D. Seita, J. Grannen, M. Hwang, R. Hoque, J. E. Gonzalez, N. Jamali, et al. Learning to smooth and fold real fabric using dense object descriptors trained on synthetic color images. In Proc. IEEE Int. Conf. Robotics and Automation (ICRA), 2021.
[15] R. Hoque, D. Seita, A. Balakrishna, A. Ganapathi, A. K. Tanwani, N. Jamali, K. Yamane, S. Iba, and K. Goldberg. Visuospatial foresight for multi-step, multi-task fabric manipulation. In Proc. Robotics: Science and Systems (RSS), 2020.
[16] T. Kollar, M. Laskey, K. Stone, B. Thananjeyan, and M. Tjersland. Simnet: Enabling robust unknown object manipulation from pure synthetic data via stereo. In Conference on Robot Learning, pages 938-948. PMLR, 2022.
[17] B. Thamanjeyan, J. Kerr, H. Huang, J. E. Gonzalez, and K. Goldberg. All you need is luv: insuperrvised of labeled images using invisible uv fluorescent indicators, 2022. URL. org/ /tt./203.3... 04566.
[18) R. Hoque, K. Shivakumar, S. Aeron, G. Deza, A. Ganapathi, A. Wong, J. Lee, A. Zeng, V. Van- houcke, and K. Goldberg. Learning to fold real garments with one arm: A case study in cloud- based robotics research. In 2022 IEEF/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 2577 2022. doi: 100.1000000044471222222.99999999
[19] D. Seita, P. Florence, J. Tompson, E. Coumans, V. Sindhwani, K. Goldberg, and A. Zeng. Learning to rearrange deformable cables, fabrics, and bags with goal-conditioned transporter networks. In Proc. IEEE Int. Conf. Robotics and Automation (ICRA), 2021.
[20] L. Y. Chen, B. Shi, D. Seita, R. Cheng, T. Kollar, D. Held, and K. Goldberg, Autobag: Learning to open plastic bags and insert objects, 2022. URL https: arwxaaiivvvaaaaa www.aawwwwww.a.....rrrrrrreeeaee wtrggabb/2210... 17217.
[21] Matas, S. James, and A. Davison. Sim-to-real reinforcement learning for deformable object manipulation. In Conf. on Robot Learning (CoRL), 2018
[22] Y. Wu, W. Yan, T. Kurutach, L. Pinto, and P. Abbeel. Learning to manipulate deformable objects without demonstrations. arXiv preprint arXiv: 2019.
[23] R. Lee, D. Ward, A. Cosgun, V. Dasagi, P. Corke, and J. Leitner. Learning arbitrary- goal fabric folding with one hour of real robot experience. In Conf. on Robot Learning (CoRL), 2020.
[24] Y. Avigal, L. Berscheid, T. Asfour, T. Kröger, and K. Goldberg. Speedfolding: Learning efficient bimanual folding of garments, 2022. URL
wrtp:/s///////////// 10552. 10552.
[25] Schulman, J. Ho, C. Lee, and P. Abbeel. Learnnng from demonstrations through the use of non-rigid registration. In International Symposium of Robotics Research, 2013.
[26] W. Yan, A. Vangipuram, P. Abbeel, and L. Pinto. Learning predictive representations for deformable objects using contrastive estimation. In Conf. on Robot Learning (CoRL), 2020.
[27] A. Wang, T. Kurutach, K. Liu, P. Abbeel, and A. Tamar. Learning robotic manipulation through visual planning and acting. Robotics: Science and Systems (RSS), 2019.
[28] X. Lin, Y. Wang, Z. Huang, and D. Held. Learning visible connectivity dynamics for cloth smoothing. In Conference on Robot Learning, pages 256-266 PMLR, 2022.
[29] A. Nair, D. Chen, P. Agrawal, P. Isola, Abbeel, Malik, and S. Levine. Combining self- supervised learning and imitation for risien-based rope manipulation. CoRR, abs/1773.00218. 2017. URL http: / arxivv. org/ aa//ab// /tab///703..mm 02018.
10

---

[30] C. Chi, B. Burchfiel, E. Cousineau, S. Feng, and S. Song. Iterative residual policy for goal- conditioned dynamic manipulation of deformable objects. Proceedings of Robotics: Science and Systems (RSS), 2022.
[31] D. McConachie, T. Power, P. Mitrano, and D. Berenson. Learning when to trust dynamics model for planning in reduced state spaces. CoRR, abs/2001.11051, 2020. URL https: //axxiv.rg//
11051.
[32] J. Schulman, A. Lee, J. Ho, and P. Abbeel. Tracking deformable objects with point clouds. In 2013 IEEE International Conference on Robotics and Automation, pages 1130-113, 2013. doi: 100.111100/1CA22211..3.666
[33] T. Tang and M. Tomizuka. Track deformable objects from point clouds with structure preserved registration. The International Journal of Robotics Research, 41(6):599-614 2022. doi: 10.
1177002783349918444433 URL hattps://do org/10. 117011111177777777666999944433
[34] A. Keipour, M. Bandari, and S. Schaal. Deformable one-dimensional object detection for routing and manipulation. CoRR, abs/2201.06775, 2022. URL https://arxiv. /bs// 2201.06775.
[35] R. C. Jackson, Yuan, D.-L. Chow, W. S. Newman, and M. C. Çavuşoğlu. Real-time visual tracking of dynamic surgical suture threads. IEEE Transactions on Automation Science and Engineering, 15(3):107--1090, 2018. doi: 10011.11999AASS2200777766666
[36] N. Padoy and G. Hager. Deformable tracking of textured curvilinear objects. In Proceedings of the British Machine Vision Conference, pages 5.1-5.11. BMVA Press, 2012. ISBN 1-901725- 46-4.
[37] Parmar. Use of computer vision detect tangles in tangled objects. 2013 IEEE Second International Conference on Image Information Processing (ICIIP-2013), IEEE, dec 2013. 10t:1001109/iciip222111
[38] M. Yan, Zhu, N. Jin, and Bohg. Self-supervised learning of state estimation for manip- ulating deformable linear objects. IEEE Robotics and Automation Letters, 5(2):2372-235 2379, 2020. doi: 10.
[39] A. Caporali, Galassi, R. Zanella, and G. Palli. Fastdlo: Fast deformable linear objects instance segmentation. IEEE Robotics and Automation Letters, 7(4):9075-9082, 2022. doi: 1..1009/LAA.22223188777
[40] B. O. Community. Blender a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. URL http:://tatt blender org.
[41] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
[42] P. lakubovskii. segmentation_model. pytorch, 2019.
Segmentation models pytorch.
https://github com /ubvel/
[43] K. Reidemeister. Knot theory. BCS Associates, 1983.
7
Appendix
7.1 Details on HANDLOOM Methods
7.1.1 Over/Undererossing Predictor
Model Architecture and Inference: The binary classification threshold of 0.275 is determined
by testing accuracy on held-out validation set of 75 images threshold values in the range

---

x
P
Figure 5: Reidemeister Moves and Crossing Cancellation: Left of part depicts Reidemeister Move II. Right part depicts Reidemeister Move Part 2 shows that by algorithmically applying Reidemeister Moves and I, we can cancel trivial loops, even if they visually appear as knots.
[0.05, 0.95] at intervals of 0.05. Scores < 0.275 indicate undercrossing predictions and scores 2 0.275 indicate overcrossing predictions. We output the raw prediction score and a scaled confi- dence value (0.5 1) indicating the classifier's probability.
7.2 Details on Robot Untangling using HANDLOOM
7.2.1 Knot Definition
Consider pair of points P1 and P2 the cable path time with (P1,P2 Ci). Knot theory strictly operates with closed loops, so to form loop with the current setup, we construct imaginary cable segment with no crossings joining to P2 [43]. This imaginary cable segment passes above the manipulation surface to complete the loop between pl and ("p1 - P22 loop"). A knot exists between P1 and p2 at time no combination Reidemeister moves I, II (both shown in Figure 5), and III can simplify the p1 - P2 loop an unknot, i.e. a crossing- free loop. In this paper, we aim to untangle semi-planar knots. For convenience, we define indicator function k(s) (0, 1] - {0, 1) which is 1 if the point (s)) lies between any such points P1 and p2, and otherwise.
Based on the above knot definition, this objective is to remove all knots, such that f k(s)) = 0. In other words, the cable, if treated as a closed loop from the endpoints, can be deformed into an unknot. We measure the success rate of the system at removing knots, as well as the time taken to remove these knots.
7.2.2 State Definition
We construct line segments between consecutive points on the trace outputted by the learned cable tracer (Section 4.1). Crossings are located the points of intersection of these line segments. We the crossing classifier (Section 4.2) estimate whether these crossings are over/unndercoosings. We also implement probabilistic crossing correction with the aim of rectifying classification errors, as we describe in Section 4.2.2.
We denote the sequence of corrected crossings, in the order that they are encountered in the trace, by X' (cI,..., Cn), where the total number of crossings and 61, Cn represent the crossings along the trace.
7.2.3 Crossing Cancellation
Crossing cancellation allows for the simplification of cable structure by removing non-essential crossings, shown in Figure 5. allows the system to filter out some trivial configurations Rei- demeister moves maintain knot equivalence [43]. We cancel all pairs of consecutive crossings (c,, c++1) in X' for some j) that meet any of the following conditions:
Reidemeister I: c and C++1 are at the same location, or
Reidemeister II: e; and c are at the same set of locations Cj and Cj+1 (cj,044) X)). Additionally, C and c++1 are either both overcrossings both undercrossings. We also cancel (c,, Cj+1) in this case.
12

---

Cage
Pinch
Knot
Knot Detection & Topology
Cage-Pinch Points
Figure 6: Knot Detection and Cage Pinch Point Selection: The left image shows using crossing cancellation rules from knot theory, the knot detection algorithm analytically determines where the knot begins in the cable. The right image shows the survey process for selecting the cap pinch points.
We algorithmically perform alternating Reidemeister moves I and II described. We iteratively apply this step on the subsequence obtained until there are such pairs left. We denote the final subsequence, where no more crossings can be canceled, by X"
7.2.4 Knot Detection
We say that a subsequence of X", Kij = (ci,..., Cj), defines a potential knot if:
Cj is an undercrossing, and
Cj an overcrossing at the same location, and
at least one intermediate crossing, i.e. crossing in X' that is not o; or ej. an overcrossing.
The first invariant is result of the fact that all overcrossings preceding the first undercrossing (as seen from endpoint) removable. We can derive this by connecting both endpoints from above via an imaginary cable (as in Section 7.2.1): all such overcrossings can be removed by manipulating the loop formed. The second invariant results from the fact that cable cannot be knotted without closed loop of crossings. The third and final invariant can be obtained by noting that a configuration where all intermediate crossings are undercrossings reduces to the unknot via the application of the Reidemeister moves. Therefore, for a knot to exist, must have at least one intermediate overcrossing.
Notably, these conditions are necessary, but not sufficient, to identify knots. However, they improve the likelihood of bypassing trivial configurations and detecting knots. This increases the system's 's efficiency by enabling to focus its actions on potential knots.
7.2.5 Algorithmic Cage-Pinch Point Detection
As per the definition introduced in Section 7.2.4, given knot Kcj- = (C...., Cj), e, and c; define the segments that encompass the knot where c; is undercrossing and cj is overerossing for the same crossing. The pinch point is located the overcrossing cable segment, intended to increase space for the section of cable and endpoint being pulled through. The cage point is located on the undercrossing cable segment. To determine the pinch point, we search from crossing Cal to crossing C,2- C,1 the previous undercrossing in the knot closest in the trace to j. u2 > and 6, is the next undercrossing after the knot. We search in this region and select the most graspable region to pinch at, where graspability (G) defined by the number of pixels that correspond a cable within a given crop and a requirement of sufficient distance from all crossings c; To determine the cage point, search from crossing G, to c where i k and c is the next undercrossing in the knot closest in the trace to G. We similarly select the most graspable point. If no points in the search
13

---

space for either the cage or pinch point are graspable, meaning G < T where T is an experimentally derived threshold value, we continue to step along the trace from c2 for pinch and from Cy. for cage until G 2 T. This search process shown Figure 6.
7.2.6 Manipulation Primitives
We the same primitives in SGTM 2.0 (Sliding and Grasping for Tangle Manipulation 2.0) [4] to implement HANDLOOM as shown in Figure 7 for untangling long cables. We add perturbation move.
Cage-Pinch Dilation: use cage-pinch grippers as in Viswanath et [3]. We have one gripper cage grasp the cable, allowing the cable to slide between the gripper fingers but not slip out. The other gripper pinch grasps the cable, holding the cable firmly in place. This is crucial for preventing knots in series from colliding and tightening during untangling. The partial version of this move introduced by Shivakumar et al. [4] separates the grippers to small, fixed distance of cm.
First, we detect endpoints using a Mask R-CNN object detection model. both endpoints are visible, the robot performs an Endpoint Separation Move by grasping at the two end- points and then pulling them apart and upwards, away from the workspace, allowing gravity to help remove loops before placing the cable back on the workspace. If both endpoints not visible, the robot performs an Exposure Move. This is when it pulls in cable segments exiting the workspace. Building on prior work, we add a focus on where this move is applied. While tracing, if we detect the trace hits the edge, we perform an exposure move at the point where the trace exits the image.
Reveal Moves:
If endpoint or the cable segment near an endpoint has distracting cable segments nearby, making it difficult for the analytic tracer trace, perturb it by grasping it and translating in the x-y plane by uniformly random displacement in 10cm x 10cm square in order to separate it from slack.
Perturbation Move:
|Reystart
Dilation Knot
Detection
Perturb Endpoint
IOO
Uncertain
No Knots
Certein
Exposure Move
Partial Dilation
Anelyze Tapology
Tracer Intiiallzation
Left Workspace
Re Re-Ennountered Trace
Hit Endpoint
Figure 7: Untangling Algorithm with HANDLOOM: We first detect endpoints and initialize the tracer with start points. If we are not able to obtain start points, we perturb the endpoint and try again. Next, we trace. While tracing, if the cable exits the workspace, pull the cable towards the center of the workspace. If the tracer gets confused and begins retracing knot region, perform partial cage-pinch dilation that will configuration easier trace on the next iteration.
loosen the knot, intended make to successfully complete, we analyze the topology. there are no knots, we are done. If there are knots, we perform a ange-pinch dilation and return to the first step.
trace able
7.2.7 Cable Untangling System
Combining HANDLOOM and the manipulation primitives from Section 7.2.6, the cable untangling algorithm works as follows: First, detect endpoints and initialize the learned tracer with steps of the analytic tracer. HANDLOOM is unable to get these initialization points, perturb the endpoint from which we are tracing and return to the endpoint detect step. Otherwise, during tracing,
the cable leaves the workspace, perform exposure move. If the trace fails and begins retracing itself, which can happen in denser knots, perform partial cage-pinch dilation in [4]. If the trace completes and reaches the other endpoint, analyze the topology. If knots are present, determine the

---

Table Tracing Unseen Cables Results
Cable Reference
TR
1
2
Avg. 40/48=83%
Tracing Success Rale
6/8
7/8
8/8
7/8
6/8
6/8
Failures
(1)
(1)
(1I) (1) (III)
(II) (III)
cage-pinch points for it, apply a cage-pinch dilation move to them, and repeat the pipeline. If no knots are present, the cable is considered to be untangled. The entire system is depicted in Figure 7.
7.3 Details on Knot Tying Experiments
Tying Demo
Tying
Figure 8: Using HANDLOOM for Learning from Demos: The left (blue panel) displays the single human demonstration, indicating the pick and place points for tying overhand knot. The right (pink panel) shows this demonstration successfully applied to the cable in different configuration with other distractor cables in the scene. The first step the demonstration is achieved through an length relative action while the second step is achieved through a crossing relative action.
When performing state-based imitation, each the pick and place points p; from the demonstration parameterized in the following way: 1) find the point along the trace, T, closest to the chosen point p; with index j in T, 2) find the displacement system of p. 3) in memory, for point Pt, store d,, are length of (E T. TT-1), and the index value of the crossing in the list of crossings just before yi:.
= pi ps the local trace-aligned coordinate
When rolling out policy using this demonstration, there are two ways to do so: 1) relative to the are length along the cable, or 2) relative to the fraction of the are length between the crossing indices. The way do so is to find the point on the cable with the same are length as P, from the demo the fractional are length between the same 2 crossing indices, depending the type of demonstration. Then, apply d, in the correct trace-aligned coordinate system. An example demonstration is shown in Figure 8.
7.4 Experiments Failure Mode Analysis
7.4.1 Using HANDLOOM for Tracing Cables Unseen During Training
(1) Retraces previously traced cable (went in a loop).
(2) Missteps onto parallel cable.
(3) Skips loop.
Figure 9 shows the cables tested The most common failure mode is (I), retracing previously traced cable. This is commonly observed in cases with near parallel segments or in dense loop areas within a knot.
7.4.2 Using HANDLOOM for Cable Inspection in Multi-Cable Settings
(I) Misstep in the trace, i.c. the trace did not reach any adapter.
The trace reaches the wrong adapter.
15

---

Figure Cables for Tracing Unseen Cables Experiment
Table 7: Multi-Cable Tracing Results
Analytic
Learned
27/30
Tier A2
2/30
23/30
Lier A3
1/30
23/30
Failures
() (1) 45, (II)
(1) 14. (II) 1, (
IIII) The trace reaches the correct adapter but is incorrect trace.
The most common failure mode for the learned tracer, especially in Tier A3, is (I). One reason for such failures is the presence of multiple twists along the cable path (particularly in Tier A3 setups, which contain more complex inter-cable knot configurations). The tracer is also prone to deviating from the correct path encountering parallel cable segments. In Tier A2, we observe two instances of failure mode (III), where the trace was almost entirely correct in that it reached the correct adapter but skipped a section of the cable.
The most common failure modes across all tiers for the analytic tracer are (II) and (III). The analytic tracer particularly struggles in regions of close parallel cable segments and twists. As result of the scoring metric, 87 of the 90 paths that we test reach adapter, however, 45/90 paths did not reach the correct adapter. Even for traces that reach the correct adapter, the trace is incorrect, jumping to other cables and skipping sections of the true cable path.
7.4.3 Using HANDLOOM for Physical Robot Knot Tying from Demonstrations
(1) Trace missteps onto a parallel cable.
(2) Cable shifted during manipulation, not resulting in a knot at the end.
Failure mode (1) occurs when the distractor cable creates near parallel sections to the cable of interest for knot tying, causing the trace to misstep. Failure mode (2) occurs when the manipulation sometimes slightly perturbs the rest of the cable's position while moving one point of the cable, causing the end configuration to not be a knot, as intended.
7.4.4 Using HANDLOOM for Knot Detection
(A) The system fails to detect knot that is present false negative.
(B) The system detects a knot where there is no knot present-a a false positive.
(C) The tracer retraces previously traced regions of cable.
(D) The crossing classification and correction schemes fail to infer the correct cable topology.
( The knot detection algorithm does fully isolate the knot, also getting surrounding trivial loops.
The trace skips a section of the true cable path.
(G) The trace is incorrect in regions containing a series of close parallel crossings.
Table 8: Learning From Demos
Suce, Rate
Failures
Tier B
5/5
Tier B2
4/5
(
(2)
16

---

Table 9: HANDLOOM Experiments
SGTM
(ANDLOOM(-LI) HANDLOOM
HANDLOOM
Tier C1
2/30
14/30
20/30
24/30
Tier C2
28/30
8/30
21/30
26/30
12/30
Failures
(A) (B) 18
(D) 11, (F)
(B) 38, (C)..
(B) 11. (D) (F)
(G) 24, (1)) 11
(
Table 10: HANDLOOM and Physical Robot Experiments (90 total trials)
Tier DI
Tier D2
Tier D3
SGTM
HANDLOOM
SGIM2U
HANDLOOM 11/15
SGTM 2.0
HANDLOOM
Knot Succ.
11/15
12/15
6/1
9/15
14/15
Knot 2 Succ.
2715
615
Verif. Rate
II/I
8/2
6/6
611
1/2
2/6
Time (min)
111011
21100.3
39111
1.800,4
200:0.4
Knot Time (min)
3.11122
7,5+16
Verit Time (min)
5,74099 (7)
6.1414
6.42188
10.12:0.
5.4
9.6:115
Failures
(2)
(5)
22)
(1)3, (2).3))).555 (5)
(l) (2)
(1) 1
(5)
(5)
(6) (7)
(3) 1. (6)
(H) The tracer takes an incorrect turn, jumping to another cable segment.
For SGTM 2.0, the most common failure modes are (A) and (B), where it misses knots incorrectly identifies knots when they are out of distribution. For HANDLOOM (-LT), the most common failure modes are (F), (G), and (H). All 3 failures are trace-related and result in knots going undetected or being incorrectly detected. For HANDLOOM (-CC), the most common failure modes are (B) and (E). This is because HANDLOOM (-CC) is unable to distinguish between trivial loops and knots without the crossing cancellation scheme. By the same token, HANDLOOM (-CC) also unable to fully isolate knot from surrounding trivial loops. For HANDLOOM, the most common failure mode is (B). However, this is derivative of failure mode (D), which is present in HANDLOOM (-LT), HANDLOOM (-CC), and HANDLOOM. Crossing classification is a common failure mode across all systems and is a bottleneck for accurate knot detection. In line with this observation, we hope to dig deeper into accurate crossing classification in future work.
7.4.5 Using HANDLOOM for Physical Robot Untangling
(1) Incorrect actions create complex knot.
(2) The system misses a grasp on tight knots.
(3) The cable falls off the workspace.
(4) The cable drapes on the robot, creating an irrecoverable configuration.
(5) False termination.
(6) Manipulation failure.
Timeout.
The main failure modes HANDLOOM are (L), (2), and (6). Due to incorrect cable topology estimates, failure mode (1) occurs: a bad action causes the cable to fall into complex, irrecoverable states. Additionally, due the limitations of the cage-pinch dilation and endpoint separation moves, knots sometimes get tighter during the process of untangling. While the perception system is still able to perceive the knot and select correct grasp points, the robot grippers bump the tight knot, moving the entire knot and causing missed grasps (2). Lastly, we experience manipulation failures while attempting some grasps as the YuMi has a conservative controller (6). We hope to resolve these hardware issues in future work.
The main failure modes in SGTM 2.0 (5) and (7), Perception experiments indicate that SGTM 2.0 has both false positives and false negatives for cable configurations that out of distribution. (5) occurs when out-of-distribution knots go undetected. (7) occurs when trivial loops are identified as knots, preventing the algorithm from terminating.
17


Reviews:
[{"review_id": "TL_8sS_1VQd", "reviewer": "Reviewer_KUAt", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a novel learning-based method of tracing one-dimensional objects on grayscale images. This approach works in an autoregressive manner and traverses the one-dimensional object by identifying the points on the cable step by step using local image crops. The proposed approach allows for tracing very long objects in very challenging scenarios despite intersections with other identical objects and itself. The authors presented also the application of the proposed method to downstream tasks, such as cable untangling.", "summary_of_recommendation": "Due to the high quality of the manuscript, proposed method, and performed experiments I'm convinced that this paper should be accepted.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. What about non-semi-panar cases? What are the limitations of this method in these cases? Is it only a matter of relevant data, or there are more substantial problems with it?\n\n2. Do you expect your method to be robust to some segmentation artifacts that are common for not plain backgrounds?\n\n3. Running time of the method itself is not mentioned (only execution of the downstream tasks). How long does it take to trace a 3m cable?\n\n4. Why grayscale? Colors could help in finding the true continuation of the DLO.\n\n5. What is the impact of lighting? Have you tested and trained in the same lighting conditions?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a novel method and very impressive results in the task of detecting one-dimensional objects.\nPerformed evaluation is very thorough and shows the high accuracy of the proposed approach and high applicability potential in downstream tasks. The presented approach seems to be a significant contribution to the field of one-dimensional object tracing.\n\n\nStrengths\n\n1. Interesting approach.\n1. Extensive real-world evaluation.\n2. Public dataset and code.\n3. High success rate on very challenging benchmarks - very impressive.\n4. Presented application in downstream tasks.\n5. Clarity of the presentation.\n\nWeaknesses\n\n1. In terms of tracing one-dimensional objects many important related works are not mentioned [1, 2, 3]. \n\n2. Do you expect your method to be robust to some segmentation artifacts that are common for not plain backgrounds?\n\n3. Running time of the method itself is not mentioned (only the execution of the downstream tasks). How long does it take to trace a 3m cable?\n\n\n[1] A. Caporali, K. Galassi, R. Zanella and G. Palli, \"FASTDLO: Fast Deformable Linear Objects Instance Segmentation,\" in IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 9075-9082, Oct. 2022, doi: 10.1109/LRA.2022.3189791.\n\n\n[2] A. Caporali, R. Zanella, D. D. Greogrio and G. Palli, \"Ariadne+: Deep Learning--Based Augmented Framework for the Instance Segmentation of Wires,\" in IEEE Transactions on Industrial Informatics, vol. 18, no. 12, pp. 8607-8617, Dec. 2022, doi: 10.1109/TII.2022.3154477.\n\n[3] P. Kicki, A. Szymko and K. Walas, \"DLOFTBs – Fast Tracking of Deformable Linear Objects with B-splines,\" 2023 IEEE International Conference on Robotics and Automation (ICRA), London, United Kingdom, 2023, pp. 7104-7110, doi: 10.1109/ICRA48891.2023.10160437.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "DAEkxhgprz", "reviewer": "Reviewer_XRHn", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a deep-learning based scene-understanding method for entangled cables from 2D images. The methods is trained with images of white cables on a black background and learns first to segment and trace the cables from the background,  and then to detect line crossing and classify them as crossing-over or crossing-under. The tracing part takes an image with a part of the cable known and predict where the cable will continue. \n\nThe method is trained on a mix of real and simulated examples and then tested on real images, with a diversity of cables type showing the generalization of the method.\n\nFinally, the method is used with a dual-arm robot manipulator to disentangle cables on a table. \n\nThe approach is evaluated quantitatively and  compared to the state of the art. \n\nThe additional material includes dataset and will include a git repo. ", "summary_of_recommendation": "A very enjoyable paper which not only present an interesting concept but also suggest new ideas for other applications. ", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "As a minor question that could be discussed in the paper (and in the rebuttal): I did not get what the over/under classifier was using to take its decision. Does it rely on shadows from one cable on the other ? This question is related to the generalizability of this task to different applications where the traces to untangle are not the physical representation of a cable, but a hand drawing, or a trajectory in a tracking application. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is well presented, well structured and well evaluated. \nOne could regret that the methodology is presented too fast, but this is clearly a result of the page limit for the conference.\n\nThe overall approach is interesting and original. The performances are very convincing and the potential for applications very interesting. \n\nMinor typos could be fix to improve the paper. Specifically, figure 3 refers to a bottom row but there seems to be a single row in the figure.\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "4q0QP1nLyP1", "reviewer": "Reviewer_bwb5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a method to estimate the state of long and thin cables in semi-planar configurations from RGBD observations. The method consists of a cable tracer model and a crossing classifier. The tracer auto-regressively predicts the next coordinate on a cable, conditioned on the history of prior keypoints. The crossing classifier takes as input a small image crop centered on the crossing of two traced cables, and predicts which cable is on top. The models are trained on synthetic images of cables synthesized in Blender using curve-generation heuristics to produce diverse and interesting cable crossings. A small number of hand-labeled real images are included as well.\n\nThe paper includes experiments on 3 main tasks: tracing unseen cables, learning knot tying from demonstrations, and cable untangling. The paper shows compelling performance, including enabling better learning of untangling behavior compared to prior state of the art.", "summary_of_recommendation": "Overall I enjoyed the paper, learned an interesting method, and was sufficiently convinced that the proposed method works well at solving the stated task. I appreciate the transparency of the assumptions. My main concern is around the narrow applicability of the method (in particular the semi-planar assumption), and I was not sure if there is sufficient comparison to prior work (I honestly don't know).", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Are more comparisons to prior work possible, especially on the cable tracing and knot detection tasks? I am not very familiar with cable manipulation so I am not sure, but a response on this would help.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n- I really appreciate the transparency of the paragraph on \"Workspace and Assumptions\". It so clearly lays out limiting assumptions about the input images instead of obfuscating them in experimental results. It really helps to understand where this method can be applied.\n- Very clearly organized, easy to follow, and enjoyable to read.\n- Clear task formulation and a reasonable method. It is easy to see how and why the method solves the task, and how it works.\n- I am not an expert on cable tracing or knot tying, but the results really look quite good. The method traces cables correctly ~80% of the time, and each example consists of a lot of crossings in a pretty big mess.\n- The experiments are very extensive, showing multiple downstream uses. I appreciate the dedication. In learning untangling, the proposed method outperforms prior state of the art.\n\nCons:\n- Some of the motivation is a little far fetched and doesn't come across as intellectually honest.\n- The semi-planar assumption limits applicability in practical untangling scenarios, \nwhere cables would form 3D structures. It's not really a flaw of the paper though: the paper clearly states this assumption, which helps.\n\n\nMinor comments and errors:\n- The first paragraph in introduction is a bit of a stretch and a tad dramatic. I am sure there are more realistic solutions to messy cabling compared to deploying a robot. None of the works cited to substantiate this [1,2,3,4] actually study safety issues of messy cabling. [1] is a survey of methods, [2,3] are on surgical robotics, [4] tries to tie knots (not untie anything). I would remove this paragraph - it motivates the work with a shaky premise. This research has academic and intellectual value even without a clear application in mind. If the authors do wish to motivate the work practically, there are more realistic applications. For example, robots in automated testing for quality assurance might be applied to plug cables into devices to check that the connectors work.\n- Line 111 has a typo.\n- Figure 3 caption mentions 2 rows, but there is only 1 row. The second row sounds interesting, I would like to see it. But right now I don't understand the image.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method for estimating the configuration of deformable 1D objects (i.e. cables, ropes) using visual data. The method "traces" the object in the image to estimate the configuration, especially estimating whether crossings are "over" or "under". Several impressive results are shown with very tangled cables being correctly traced.

Strengths:
- The approach is extensively evaluated in the real-world, achieving a high success rate
- The dataset and code are open 
- The paper is generally clear and understandable

Weaknesses
- Some related work is overlooked and should be included (see reviewer comments)
- The method relies on effective segmentation. Though segmentation is not within the scope of the work, the authors could consider incorporating a method to deal with segmentation artifacts that will be inevitable.
- The results should present run-time statistics

Overall, the rebuttal did not significantly change the reviewers' opinions.
Decision: Accept (Oral)