Hierarchical Planning for Rope Manipulation using Knot Theory and a Learned Inverse Model
Authors: Matan Sudry
Tom Jurgenson
Aviv Tamar
Erez Karpas
Venue: CoRL 2023 Poster (2023)
Keywords: Manipulation
Robot Learning and Planning

Abstract: This work considers planning the manipulation of deformable 1-dimensional objects, such as ropes or cables, specifically to tie knots. We propose TWISTED: Tying With Inverse model and Search in Topological space Excluding Demos, a hierarchical planning approach which, at the high level, uses ideas from knot-theory to plan a sequence of rope configurations, while at the low level uses a neural-network inverse model to move between the configurations in the high-level plan. To train the neural network, we propose a self-supervised approach, where we learn from random movements of the rope. To focus the random movements on interesting configurations, such as knots, we propose a non-uniform sampling method tailored for this domain. In a simulation, we show that our approach can plan significantly faster and more accurately than baselines. We also show that our plans are robust to parameter changes in the physical simulation, suggesting future applications via sim2real.

Hierarchical Planning for Rope Manipulation using Knot Theory and a Learned Inverse Model
Matan Sudry, Tom Jurgenson, Aviv Tamar and Erez Karpas Technion - Israel Institute of Technology. Haifa, Israel (matansudry, tomj @écmmuss technion. il, (avivt, karpase @technion ac.
Abstract: This work considers planning the manipulation of deformable dimensional objects such as ropes or cables, specifically tie knots. We propose TWISTED: Tying With Inverse model and Search in Topological space Exclud- ing Demos, hierarchical planning approach which, at the high level, uses ideas from knot theory to plan a sequence of rope topological states, while at the low level uses a neural-network inverse model to move between the configurations the high-level plan. To train the neural network, we propose self-supervised approach, where we learn from random movements of the rope. To focus the random movements on interesting configurations, such as knots, we propose a show
non-uniform sampling method tailored for this domain. In simulation, that our approach can plan significantly faster and more accurately than baselines. We also show that our plans are robust to parameter changes in the physical sim- ulation, suggesting future applications via sim2real.
Keywords: Knot tying, Learning, Planning
Introduction
Deformable object manipulation important for many applications, such as manufacturing and robotic surgery. particular, manipulating I-dimensional (ID) objects such as ropes, cables, and hoses, is challenging and exciting research area that has drawn recent attention 21, 2, 3, 4, 5, 6, 7, 9, 10]. There are several challenges to ID object manipulation.
Representing the state of the object is difficult, as unlike rigid objects, the object may have infinite degrees of freedom [11, 12, 13]. Perception of a rope-like object is complex due to self-occlusions, the similarity between different rope parts, and self-loops [1, 14, 15, 16, 17, 18]. Planning typically requires an effective abstraction of the states and the actions, which may be difficult define [19, 20], and low-level control for executing plan must handle the flexibility and deformability of the rope all non-trivial control problems [3, 21, 22]. To the best of our knowledge, system that can generally manipulate 1D objects is beyond the capabilities of current technology. Our focus in this work is on the planning component ID manipulation, particularly rope manipulation and knot tying. As mentioned above, planning for rope manipulation non-trivial, as the state space may be large or infinite, and tasks like knot-tying essentially have 'needle haystack' characteristic and require exhaustive exploration reach desired states. Accordingly, most prior studies on rope manipulation relied on human demonstrations in lieu of automatic search [3, 23, 24, 25, 6, 26, 27].
This paper tackles the problem of rope manipulation planning without any demonstrations. Our main contribution a hierarchical search algorithm that exploits prior knowledge about knot-tying geometry for its high-level plan, with self-supervised learning of an inverse model for exccuting the low-level control, which we call Tying With Inverse model and Search in Topological space Excluding Demos TWISTED. TWISTED trained and evaluated in physical simulation. We demonstrate, however, that planning results robust to variations of physical properties such as friction. Thus, we believe that our planning approach can be integrated with real-robot perception and control for complete ID manipulation system in the future. We demonstrate that TWISTED
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

can tie various types of knots and can generalize to tie knots that were not seen during the training. To the best of our knowledge, this is the first demonstration of such a capability, which cannot be obtained by previous work that required a human demonstration of the knot to tie. Finally, while TWISTED tailored for knot tying by building on knot theory for high-level planning, our general methodology may be useful for other tasks where well-established theory may inform the high- level characterization of the problem, while learning- baasing-basedd method is used for low-level control.
2 Background
In our work, we build on knot- theory for high-level planning. In this section, we give a brief overview of knot theory. The most common way solve problems like knot-tying, with a high-dimensional and continu- ous state and action spaces, and long-horizon planning is to sep- arate it into a topological repre- sentation for high-level planning and geometric representation for low-level control, which is solved using learning [24]. We represent a rope as having L links, and denote by q E Q the rope configuration, with Q C R2L-5, The first seven coordinates describe the global position of the middle rope link (position (I, y. 2) and quaternion representation for the rotation), and the remaining L joints are each described by yaw and pitch values of the i link relative link i.
TrMaletate
Figure I: P-data Topological state representation: each column corresponds to an intersection along the rope of L links. Row one ordered from 1 L in ascending order. Row two, for each intersection, defines the other link in the intersection. The label "o"/"u" classifies the vertical arrangement at each intersection (over or under). Finally the last row identifies the "sign" see appendix Section 1. E.g. in the center state representation link 1 is over link 2 with si"sign
We follow Yan et al. [3], where the discrete topological representation for S is P-data (see appendix Section 9.1), and we denote Top:Q + S the mapping from a configuration to its topological state. The "complexity" of a topological state s E S is defined according to the number of crosses (link intersections, see Section 9.1) 1) represents figure 1. Knot theory [28] suggests Reidemeister moves actions that transition the rope between topological states. In this work, will use them as high-level actions during the search. We denote the space of Reidemeister moves Aς and Pn S Ax S as the transition function of topological states using Reidemeister moves. Reidemeister [28] proved that between any two topological states s' E S, there exists a sequence of actions that starts 8 and ends s', namely, Bao..... O* E Ar s.t. S' = Pn( Pa( Pn(s, ao), (1) .......... a*). The Reidemeister moves are (1) Reidemeister I (RI) which moves one segment to create a new loop, (2) Reidemeister (R2) which pulls the middle segment and creates a new intersection with opposite signs, and (3)the Cross (C) creates a new intersection between two segments. Examples of those actions can be visualized in Figure 4 in the appendix.
Considering the knot-tying problem as trajectory over topological states with Reidemeister moves as actions, translates the original problem of directly manipulating rope configurations to a problem of a shorter horizon and a "lower" branching factor. This approach has been adopted in different algorithms [3, 23, 5, 24, 25].
Finally, we use the topological motion primitives action space [3]: When manipulating rope with L links, an action is curve C, parameterized by the link to grab I E L]', endpoint in (r,y) 2) (in the workspace), and the maximal height =max- We denote the transition function for curves applied on configurations by f Q x + Q. Yan et al. [3] observed that the space of curves C' approximates well all the possible Reidemeister moves available from given topological state.
We associate a fixed point for every link.

---

High lvvi siame!
Suggested tightlevel plane
Selecledd plan
High- planner
S1
loxt plan
God 1OW
98
S1
S2
S3
S4
q1
99
Mujoco
LOM lovel plSTer
Figure 2: TWISTED: Given initial and goal topological states, we iteratively call high-level plan- to find plans follow (top row). The plan uses inverse model to transition between consec- utive topological states (bottom right). When following plan, new information integrated into a tree of all known configurations, which seeds the high-level planner with initial states (bottom left). Gray boxes are high-level states, green boxes are low-level states (rope configuration), blue boxes represent the inverse model, and our environment in Mujoco is red. See Section 4.
For this reason, in this work, we follow Yan et [3], and plan using Reidemeister moves while manipulating the rope with curves.
Related work
Manipulating deformable objects presents varying degrees difficulty [29]. For rope manipulation, recent studies focused on either learning from human demonstrations [2, 6, 3] solving short-term plans (e-g., changing the shape of the rope, but tying knot) through pick-and-place actions [7, 27]. Differently, in work tackle long-tern planning, such as knot tying, without demon- strations. To handle the challenging deformable-object dynamics of ropes, previous methods used self-supervised learning [10, 26, 17J. We also use self-supervised collected data to train our inverse model. Finally, some previous works attempted to learn rope manipulation using reinforcement learning (RL) methods [30, 31, 32]. However, as we show, our knot-tying tasks are too complicated for Several recent works studied the simpler task of rope un-tying [33, 34, 35, 15, 36). In particular, [33, 34] used graph-based search algorithms. Knot un-tying is simpler because the end goal is al- ways the same (untangled rope), while in our work we specifically focus on a large space of possible goals, which renders un-tying methods inapplicable.
RL algorithms (see Section 5).
4 TWISTED
In this section we describe the components that comprise our solution TWISTED. We start with the description of the simulated environment in Section 4.1, we follow with the description of the al- gorithmic components in Sections 4.2, 4.3, and 4.4, and finally describe our data collection method- ology 4.5. See Figure 2 for an overview.
4.1 Simulated Environment
The environment used to learn and test TWISTED was created by the free, open-source sim- ulation environment of Mujoco [37]. It includes the default rope of Mujoco and the end-effector moving the rope. We used free-moving end-effector to focus on the complexity of tying knots, ignoring the additional complexity of controlling robot manipulator It is crucial to mention that
2 Although non anivriiial, we expect that common motion planning solutions could be utilized in order to bridge the gap from free- freemooving end-effector to complete robotic manipulator.

---

during planning, we run actions in the simulation itself, meaning that both evaluation and search use the same Mujoco environment (i.e., the search acts with a perfect world model).
4.2 Planning Algorithm
TWISTED' is best summarized as iterative algorithm that given an initial rope configuration Ginit and goal state 89 repeats three steps: (1) starts searching from a known reachable configuration (2) plans high level trajectory whose states are in S, and actions in Ap. and (3) uses low- level planner to follow subsequent states in the selected high-level plan. The iterative process of TWISTED repeats until $g is reached (success) or a pre-specified timeout expires (terminating in failure). See Algorithm
Data structures: maintain two data struc- tures, a tree of known reachable configu- rations and set of high-level plans. The tree
Algorithm TWISTED algorithm
Input qinit Initial configuration state and 8n topo- logical goal state Output Low-level plan if found
known reachable configurations, whose vertices are configurations of the rope with their corresponding topological states (q, Top(a)) and the edges are low-Ievel ac- tions in C. Initially, T contains only root node the rope's initial configuration Bnit and its topological state Sinit. We also maintain a list of high level plans, P
I: init: T.P
see data structures
2: 8init = Top((iiit)
3: populate P with plans from Sinit
4: while Not timeout do
5: Selct Select = opologicaoooll
6:
Potet = Selectllan( (slat)
7:
qaict SelectCon o fig(S
{P, = (so: $1,...51, 5t, = sg)}s from cur- rently reachable topological states (see Sec- tion 4.3). When topological state s' is dis- covered for the first time by the low-level
8:
Plam Found= Follow Plannlsee. Patct)
if Plan Found then
10:
Return
11:
else
planner (Section 4.3), we run the high-level
12:
RandomErpande
13:
end if
planner from s' and store the results into P.
14: end while
Plan selection: At the start of each iteration, we need to select a plan to execute from and a configuration start executing the plan from. One naive heuristic is select random configuration from the reachable configurations, However, due to the problem's 's sparsity. are exponentially fewer than those with fewer crosses. We thus seek an approach that promotes con- figurations corresponding to topological states with higher crosses. We therefore run the following three sub-procedures in sequence:
observe that during the search, configurations in T with more crosses
Select] (ooicdddoooogiiSSSttt((]]] identifies the reachable topological states in T that have a high-level plan to the goal. Samples one such topological state s according to one of two heuristics; random, which the uniform distribution", and prioritize-croxxss, which assigns 8 probability that proportional to Cross(s) (i.e. prefers topological states with more crosses motivating to search deeper than the random heuristic).
Select SelectPlan(ss) (s)) a high-level plan (sequence of topological states) P 8, 81 selected from all the high-level plans that start in s.
is randomly
SelectCon figiooiioiurrsrsss randomly select configuration from all configurations belonging to
Plan execution: Next, in Follow Plan, follow the high level plan P Su, $ St, where 80 8 and 81 = 8g. starting in 8, and incrementally try to reach 8:>0 until 8, is reached. To transition from s; to 8i+1, we use the low-level planner (Section 4.3) that uses the learned inverse model (Section 4.4) to predict curves. The low-level planner applies multiple curves {c,}, to the current configuration qi- Let of = f(g, cý), and s'j = Top(o)). If the transition for сý reaches a configuration with more crosses, we add this information T, and note that could be the case that
} 'https://github comeb/..ommmaatt/ nat aataaatttuuuuusussteeeeeeee 'Even the random heuristic prioritizes complex topological states, as sampling a random topological state induces different distribution than sampling from all reachable configurations in I directly.

---

s'3 * 8,41 because the inverse model is not perfect. Nevertheless, this is an executable low-level action; thus, we add it to our search tree.
Completeness guarantee: Finally, for completeness of the algorithm, after every iteration of TWISTED, with probability p (hyper-parametter, with a default value of 0.05), we sample ran- dom reachable configuration q E T. execute & 100 random actions and add them to T if the action transitions to topological state with a greater or equal number of crosses (same conditions as in the "plan execution" above). This ensures that given enough time, our algorithm is guaranteed to find a solution. We denote this sub-routine as Random Expandd
4.3 Planning and Search
TWISTED is composed of two levels of plamning, high-level and low-level planning, that are called as sub-procedures by the algorithm. We now describe the two planners with their states and actions.
High-level planner: with state space S and Reidemeister moves Ag as actions, finds all paths from currently reachable topological states E T (not necessarily Sinitial) to 8g E S using Breadth- first search (BFS). Our BFS prunes new states s' with Cross(s') > Cross and returns a set (possibly empty) of all paths that start in s and terminate in Sg.
Low-level Planner: Given and s' two consecutive topological states in the high-level plan, we search for curve C that traverses from s'. To successfully and efficiently find such a curve, we utilize our inverse model (Section 4.4) and generate curves {cr E CJ (K = 6). If any of the newly found topological states are s', we return success (if more than one action succeeds we use the first one found), and the plan execution will move to the next topological state in the high-level path. Otherwise, we return failure, and the iterative process of TWISTED repeats.
4.4 Inverse model
An action generator is crucial in knot-tying as the proportion of curves that transition the rope to given topological state could be extremely small (See Section 5.1 where we show the data collection difficulties). This makes it unlikely that small set of randomly selected curves could be found to satisfy the required transition between topological states. Thus, we trained an inverse model to generate action candidates that are likely to satisfy the required transition. The inverse model is an ruto-regressive model [38] predicts elements as follows: link to pickup, the height of the curve Amax. destination X position, and destination Y position. The link is categorical and modeled a multinomial distribution, and the other elements are continuous and modeled with Normal distribution. Every element is predicted with independent sub-network, whose inputs are: (I) the current configuration, (2) the current (z, 2) coordinates of each of the L links, (3) the next topological state x', and (4) all the elements before the current element (e.g. Zmaz gets the link index as input). See Figure 5 in appendix section 9.4.
Training: we collected data generated from random actions train the inverse model (see Sec- tion 4.5). The data D contains transitions (s, s', q, c), and s' are current and following topological states, is the current configuration, and is the curve taken. Since the data collection is time- consuming, we follow previous work of Yan et al. [3] and apply the Mirror and Reverse augmenta- tions to our data. We train the model via a maximum likelihood objective on D (predicting c).
Inference: during inference, we follow the standard ancestral sampling scheme for auto-regressive models [38]; we predict a distribution for every element, sample from it, and feed the result to predict the next element in the sequence.
4.5 Data collection
To train the inverse model, we must collect data that represents movements typically encountered when tying knots. The problem, however, is that without controller that knows how to tie knots, nor human demonstrations, rollouts simply by executing random walks of curves. However, in doing so, found very low
not clear how collect such data, Initially, we tried collect
In Yan et al. [3] these augmentations were applied over manual demonstrations. In our work, we apply them randomly collected data.
5

---

number of topologiaal states with three crosses (only 27 states per CPU core per hour), demonstrat- ing that applying random actions to the rope typically does not lead to knot-like configurations. We, therefore, designed a collection scheme that selectively resets the environment. Using our scheme, we collected 537 successful transitions per hour per CPU core. We used that to collect data set of 1,670,000 data points. For full details see Appendix 9.3.
5 Experiments
The experiments aim address the following items: (1) How sensitive is knot-tying planning to the action space, and is continuous action space necessary? (2) Comparison of TWISTED with baselines (3) How sensitive is TWISTED to changes in the physical simulation? (4) How well does TWISTED generalize to unseen knots?
5.1 What makes knot-tying difficult?
One difficulty of our knot-tying problem is that it requires very accurate actions to solve. To demon- strate this, we verify that even a fine discretization of the problem leads to significantly different outcomes. In this experiment, we measure sensitivity to discritization of curves, i.e. test if using a discretized curve reaches the same topological state as the next topological state (obtained by exe- cuting the original non-discretized eurve). Formally, given curve, C (1,, Emax, 2; V) E C, which includes three continuous elements (zmax. I, and y), convert it to a discrete curve where each element is rounded. Zmax, is discritized in steps of 0.001, and x and in steps of 0.01. Notice that the size of the discretized curve space is 21 x 70 x 100 x 100 = 14, 700, 000, already rather large. We measure the accuracy of the resulting topological states. If the accuracy is high, there little difference discretizing the action space, suggesting that knot tying could be solved using off-the- shelf discrete planners [39, 40]. We ran over 600k data points transitions from topological states of two crosses topological states of three crosses, and got accuracy of 82%. These results show knot tying is sensitive to discretization as very small changes in the actions can lead to different topological states. As the space of available discretized actions is already rather large (larger action spaces would make planning even more difficult) we conclude that discretization of the action space is not a suitable approach to the knot-tying problem.
5.2 Success Rate of Different Algorithms
In this experiment, we compare several algorithms, including TWISTED and its ablations.
Low level only: We modify TWISTED to not use any high-level information. Essentially, using random search over configurations with curves as actions. As there is no notion of topological states, there is no way to use the inverse model here. Instead, we sample random curves. It is important to notice that for this baseline, the search does not get feedback along the trajectory (in TWISTED, we do, for instance, count the number of crosses). We use this bascline to demonstrate how crucial high-level information is for knot tying.
Low+high level: We modified TWISTED not to use the inverse model. Instead, we sample random actions replacing those suggested by the inverse model. Unlike the previous baseline, we do try to follow high-level plan. This baseline demonstrates the trade-off between intensive but more accurate action prediction (inverse model) vs. an approach of guessing many random actions and seeing if any suffice.
SAC+HER: In this baseline, we learn a stochastic policy using the Soft Actor-Critic (SAC)[41), with Hindsight Experience Replay (4E))[42), and after training we replace our inverse model with the policy. The objective of this baseline is establish the performance of model-free RL methods and the challenging problem of knot-tying.
TWISTED, RND: TWISTED using the random heuristic for topological state selection.
TWISTED, CRS: TWISTED using the priorifize-crosser heuristic for topological state selection.
Evaluation protocol: To evaluate the performance on different difficulty levels, we split our col- lected data D (Section 4.5) into three levels: easy, medium, and hard. To classify the problems (topological goal states), we counted the frequency each topological state has been recorded. Easy,

---

medium and hard are the 33%, 66%, and 100% percentiles appearing the most in the data. From every class of problems, we sampled ten representatives.
Results: None of the algorithms solve medium or hard in the time limit of 1800 seconds, demon- strating the hardness of the knot-tying problem. Figure 3 (a) shows the number of solved tasks vs. the running time for easy problems. First, observe that "low level only" is barely able to solve two out of the ten problems. This validates our earlier hypothesis in Section 4.5 that the problem is too sparse to solve without prior knowledge of the problem structure (such as our high-level search). Surprisingly, the model-free RL baseline is barely better than the random search. We observed that during training, it did manage to consistently solve all I-cross problems, but already for 2-cross problems success rate was near zero. This suggests that knot-tying a hard task to learn end-to-end without proper domain knowledge. We hypothesize that the main reason this baseline fails is due to the discrete nature of the topological states in such cases algorithms cannot generalize between "similar" states because categorical variables, there is no notion of similarity, only the relation of equality. Even a well utilized exploration method such as HER does little to mitigate this problem, because it can only reinforce patterns for goals we reach, and as we saw when acting randomly, like RL agents do at the beginning of training, there is little chance to advance to topological states with many crosses (see Section 4.5). Finally, regarding the baselines, we see that because "low + high level" so inferior to the full TWISTED versions, the inverse model is a valuable component of our full solution. Comparing "TWISTED, RND" and "TWISTED, CRS", we observe that the results are not conclusive. To identify the better model, we sampled 15 additional easy goals to get a statically significant separation on which is the better variant of TWISTED. The "TWISTED, CRS" solved a total of 24/25, and the "TWISTED, RND" solved only 19/25. Under a Z-test the "TWISTED, CRS" has a statistical significance of being better than the "TWISTED, RND" (using a-level of 0.05), showing that planning deeper and utilizing prior knowledge (number of crosses) is preferable. Therefore, in our next experiments, we use the "TWISTED, CRS" version.
5.3 Sensitivity Analysis
To motivate the usage of TWISTED in real-world applications we test what happens if the model of the world in the planning computation is mismatched with the evaluation environment. In our experiments, we focus on the friction coefficient. First, we validate that friction indeed significantly impacts rope tying. To measure this, we compare the next topological state observed when applying the same action from the same low-level state, under different friction coefficients. We evaluated over 600,000 actions, and only 82% curves had the same topological state the original friction value. Next, we evaluate the performance of TWISTED trained with single friction coefficient on simulated environments with different frictions of the rope.
Variants: 100% friction denotes the default Mujoco friction, and the one we use for TWISTED. The 95% and 105% variants, denote decreasing and increasing the friction by 5%.
Results: The performance of TWISTED is well- well-maintained with the different friction coefficients (Figure (b)). This asserts that TWISTED can handle some variations in the environment's physics such as friction (even though the resulting trajectories might be different than the original ones).
5.4 Generalization to Unseen Topological States
The number of available topological states for states with 3 or 4 crosses above 500 and almost 8000 correspondingly. Naturally our data does not contains all of them because it hard to sample topological states of higher crosses (see Section 4.5 for analysis). Thus, we require our algorithm to handle unseen topological states. We evaluate whether TWISTED can tie knots where the goal state was not represented in D. For this, we take topological states with three crosses not seen in D, and topological states with four crosses. In these out-of-distributtin cases we expect the inverse model to contribute less than in well represented states, and we expect that the planning components in TWISTED will compensate for this distribution shift. For this reason, we extend search time by a factor of 4x. Results are shown in figure 3 (c). Figure 3 (c) shows that TWISTED solved two out of eight problems with unseen three cross states. Those states are harder to reach because they were never seen in D during data collection. In figure 3 (c) problems with unseen states with four crosses. We recall that our data contain only one, two, and
that TWISTED solved three out of ten
7

---

TWITED,
13
Unsesn sletes with four usoers states
TWSTED,
Law local
Salved problems
Soleed problems
seves problems
SHR+
Ficion 085
ciaan 105
500
1000
320
1000
1500
Run
Runcima
(a)
(b)
(C)
Figure 3: Anytime Success Rates for Different Settings. The X-axis the algorithm run time and
the Y-axis is the number of solved problems. (a) Differnt knot tying algorithms, (b) "TWISTED, CRS" with different ropes and different friction to knot tying, and (c) "TWISTED, CRS" on unseen states with three crosses and four crosses, with eight and ten problems respectively
Algorithm
Solved problems
Friction
Solved prohlems
Goal states
Solved problems
Low level only
95%
3 crosses
3/10
Low+high level
100%
10
4
2/8
SAC+HER
105%
TWISTED, RND
10
TWISTED, CRS
10
Table I:
Experiments Summary: (a) Success Rate of Different Algorithms, (b) Sensitivity Anal-
ysis, and (c) Generalization to Unscen Topological States
three crosses, and therefore these results show that TWISTED not only memorizing the data, but can generalize to some degree unseen goal states.
Limitations
Our work has several important limitations that need to be addressed in order to make it more practical and useful in real-world applications. First, simulation accelerates the training process but introduces sim2real gap between the simulation and real- weal-world performance. This gap should be tested on a real robot using real ropes. Furthermore, in this work we also simplified the problem; we control free-moving end-effector instead of controlling manipulator (which might make some curves unfeasible some scenarios), and we get perfect representation of the rope, where in reality would need first estimate one. Finally, better performance on frequent data from the easy problems, but its performance decreases when trying to solve more rare or unseen goals from medium and hard.
experiments demonstrate, TWISTED has shown
7 Outlook
We presented TWISTED hierarchical planning algorithm for knot tying. that relies on knot- theory and a learned inverse model to automatically solve problems that previously required access to human demonstrations. TWISTED outperforms various baselines, including model-free deep RL agent, and we demonstrated robustness to simulation parameters such as friction, and general- ization problems seen during training (even problems of greater complexity). We see this important step towards general 1D object manipulation, and to the best of our knowledge, this the first work that manages to tie knots using random data instead of demonstrations.
An exciting area for improvement would be to utilize TWISTED as demonstrations provider to generate "valuable" data for an off-policy RL algorithm, either by distilling the planner into pol- icy [43] or by combining RL with imitation learning [44]. This could be the missing prior knowledge that RI. methods lack for knot-tying tasks (ef. Section 5). A different interesting future direction might be to improve the data collection process to seek novel states instead of relying random actions. Collecting data from states where the system is less capable, could ultimately provide data of higher quality and improve the performance of our learned inverse model. Finally, it would be interesting to test TWISTED on a real system overcoming challenges such as estimating the rope configuration and executing precise rope manipulation actions.

---

8 Acknowledgement
Supported by a grant from the Israeli Planning and Budgeting Committee. This work received funding from the European Union (ERC, Bayes-RI., Project Number 101041250). Views and opinions expressed are however those of the author(s) only and do not nec- essarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.
References
[1] Sundaresan, J. Grannen, Thananjeyan, A. Balakrishna, M. Laskey, Stone, J. Gon- zalez, and K. Goldberg. Learning rope manipulation policies using dense object descriptors trained synthetic depth data. In 2020 IEEE International Conference Robotics and Au- tomation (ICRA), pages 9411-9418. IEEE, 2020.
[2] J. Van Den Berg, S. Miller, D. Duckworth, H. Hu, A. Wan, X.-Y. Fu, K. Goldberg, and Abbeel. Superhuman performance of surgical tasks by robots using iterative learning from numan-guided demonstrations. 2010 IEEE International Conference on Robotics and Au- tomation, pages 2074-2081. IEEE, 2010.
[3] M. Yan, G. Li, Y. Zhu, and J. Bohg. Learning topological motion primitives for knot planning. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9457-9464. IEEE, 2020.
[4] H. Mayer, F. Gomez, D. Wierstra, I. Nagy, A. Knoll, and Schmidhuber, A system for robotic heart surgery that learns to tie knots using recurrent neural networks. Advanced Robotics, 22 (13-14):1521-1537 2008.
[5] Y. She, S. Wang. S. Dong, N. Sunil, A. Rodriguez, and E. Adelson. Cable manipulation with a tactile-reactive gripper. The International Journal of Robotics Research, 40(12-14):1385 2021.
[6] Schulman, J. Ho, C. Lee, and P. Abbeel. Learning from demonstrations through the use of non-rigid registration. In Robotics Research, pages 339-354. Springer, 2016.
[7] Y. Wu, W. Yan, T. Kurutach, L. Pinto, and P. Abbeel. Learning to manipulate deformable objects without demonstrations. In 16th Robotics: Science and Systems, RSS 2020. MIT Press Journals, 2020.
[8] M. Yu, Zhong, and Li. Shape control of deformable linear objects with offline and online learning local linear deformation models. In 2022 International Conference on Robotics and Automation (ICRA), pages 1337-1343. IEEE, 2022.
[9] V. Lim, H. Huang, L. Y. Chen, J. Wang, J. Ichnowski, D. Seita, M. Laskey, and K. Goldberg. Real2sim2real: Self-supervised learning of physical single-step dynamic actions for planar robot casting. In 2022 International Conference on Robotics and Automation (ICRA), pages 8282-8289. IEEE, 2022.
[10] C. Chi, B. Burchfiel, E. Cousineau, S. Feng, and S. Song. Iterative residual policy: for goal- conditioned dynamic manipulation of deformable objects. arXiv preprint arXiv:2203.00663, 2022.
[11] W. Yan, A. Vangipuram, P. Abbeel, and L.. Pinto. Learning predictive representations for deformable objects using contrastive estimation. In Conference on Robot Learning, pages 564 574. PMLR, 2021.
[12] A. Wang, T. Kurutach, K. Liu, P. Abbeel, and A. Tamar. Learning robotic manipulation through visual planning and acting. In Robotics: science and systems, 2019.

---

[13] Y. Wi, P. Florence, A. Zeng, and N. Fazeli. Virdo: Visio-tactile implicit representations of deformable objects. In 2022 International Conference on Robotics and Automation (ICRA), pages 3583-3590. IEEE, 2022.
[14] A. Ganapathi, P. Sundaresan, B. Thananjeyan, A. Balakrishna, D. Seita, J. Grannen, M. Hwang, R. Hoque, J. E. Gonzalez, Jamali, et al. Learning dense visual correspondences in simulation to smooth and fold real fabrics. In 2021 IEEE International Conference on Robolics and Automation (ICRA), pages 11515-11522. IEEE, 2021.
[15] J. Grannen, P. Sundaresan, B. Thananjeyan, J. Ichnowski, A. Balakrishna, V. Viswanath, M. Laskey, J. Gonzalez, and K. Goldberg. Untangling dense knots by learning task-relevant keypoints. In Conference on Robot Learning, pages 782-800. PMLR, 2021.
[16] L. Yen-Chen, Florence, J. T. Barron, T.-Y. Lin, A. Rodriguez, and P. Isola. Nerf-supervision: Learning dense object descriptors from neural radiance fields. In 2022 International Confer- ence on Robotics and Automation (ICRA), pages 6496-6503. IEEE, 2022.
[17] M. Yan, Y. Zhu, N. Jin, and Bohg. Self-supervised learning of state estimation for ma- nipulating deformable linear objects. IEEE robotics and automation letters, 5(2):2372-2379 2020.
[18] X. Ma, D. Hsu, and W. S. Lee. Learning latent graph dynamics for visual manipulation of deformable objects. In 2022 International Conference on Robotics and Automation (ICRA), pages 8266-8273 8273. IEEE, 2022.
[19] B. Lu, H. K. Chu, and I.. Cheng. Dynamic trajectory planning for robotic knot tying. In 2016 IEEE International Conference on Real-time Computing and Robotics (RCAR), pages 180-185. IEEE, 2016.
[20] T. Osa, N. Sugita, and M. Mitsuishi. Online trajectory planning and force control for au- tomation of surgical tasks. IEEE Transactions on Automation Science and Engineering, 15(2): 675-691, 2017.
[21] B. Lu, H. K. Chu, and L. Cheng. Robotic knot tying through spatial trajectory with vi- sual servoing system. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 771667716. IEEE, 2017.
[22] Jin, C. Wang, and M. Tomizuka. Robust deformation model approximation for robotic cable manipulation. In 2019 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 558666593.. IEEE, 2019.
[23] Takamatsu, Morita, K. Ogawara, H. Kimura, and Ikeuchi. Representation for knot- tying tasks. IEEE Transactions on Robotics, 22(1):65-78, 2006.
[24] T. Morita, J. Takamatsu, K. Ogawara, H. Kimura, and K. Ikeuchi. Knot planning from ob- servation. In 2003 IEEE International Conference on Robotics and Automation (Cat. No. 03CH374222,, volume pages 3887-3892. IEEE, 2003.
[25] H. Wakamatsu, A. Tsumaya, E. Arai, and S. Hirai. Planning of one-handed knotting/raveling manipulation of linear objects. In IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004, volume 2, pages 1719-1725. IEFE, 2004.
[26] A. Nair, D. Chen, P. Agrawal, P. Isola, P. Abbecl, J. Malik, and S. Levine. Combining self- supervised learning and imitation for vision-based rope manipulation. In 20/7 IEEE interna- tional conference on robotics and automation (ICRA), pages 2146-21 2153.. IEEE, 2017.
[27] Y. Teng, H. Lu, Y. Li, T. Kamiya, Y. Nakatoh, S. Serikawa, and P. Gao. Multidimensional deformable object manipulation based dn-transporter networks. IEEE Transactions on In- telligent Transportation Systems, 2022.
10

---

[28] K. Reidemeister. Knot theory. BCS Associates, 1983.
[29] J. Matas, S. James, and A. J. Davison. Sim-to-real reinforcement learning for deformable object manipulation. In Conference Robol Learning, pages 734-743. PMLR, 2018
[30] X. Lin, Y. Wang. J. Olkin, and D. Held. Softgym: Benchmarking deep reinforcement learning for deformable object manipulation. In Conference Robot Learning, pages 332-448. PMLR, 2021.
[31] H. Han, G. Paul, and T. Matsubara. Model-based reinforcement learning approach for de- formable linear object manipulation. In 2017 13th IEEE Conference on Automation Science and Engineering (CASE), pages 750-755. IEEE, 2017.
[32] Y. Deng, C. Xia, X. Wang, and L. Chen. Deep reinforcement learning based on local gnn for conditondi deformable object rearranging. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1131-1138. IEEE, 2022.
[33] V. Viswanath, J. Grannen, P. Sundaresan, B. Thananjeyan, A. Balakrishna, E. Novoseller, J. Ichnowski, M. Laskey, J. E. Gonzalez, and K. Goldberg. Disentangling dense multi-cable knots. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2021.
pages
[34] K. Shivakumar, V. Viswanath, A. Gu, Y. Avigal, J. Kerr, Ichnowski, R. Cheng, T. Kollar, and K. Goldberg, Sgtm 2.0: Autonomously untangling long cables using interactive perception. arXiv preprint arXiv:2209.13700 2022.
[35] V. Viswanath, K. Shivakumar, J. Kerr, B. Thananjeyan, E. Novoseller, J. Ichnowski, A. Es- contrela, M. Laskey, J. E. Gonzalez, and K. Goldberg. Autonomously untangling long cables. arXiv preprint arXiv:2207.07813, 2022.
[36] Sundaresan, J. Grannen, B. Thananjeyan, A. Balakrishna, Ichnowski, E. Novoseller, M. Hwang, M. Laskey, J. E. Gonzalez, and K. Goldberg. Untangling dense non-planar knots by learning manipulation features and recovery policies. arXiv preprint arXiv:2107.08942, 2021.
[37] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
[38] K. Gregor, Danihelka, A. Mnih, C. Blundell, and D. Wierstra. Deep autoregressive networks. In International Conference on Machine Learning, pages 1242-1250. PMLR, 2014.
[39] G. M. J. Chaslot, M. H. Winands, H. J. v. d. Herik, J. W. Uiterwijk, and B. Bouzy. Progressive strategies for monte-carlo tree search. New Mathematics and Natural Computation, 4(03): 343-357, 2008.
[40] H. Finnsson and Y. Björnsson. Simulation-basee approach to general game playing. In Aaai, volume 8, pages 259-264, 2008
[41] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[42] M. Andrychowiez, F. Wolski, A. Ray, Schneider, R. Fong, Welinder, McGrew, J. To- bin, O. Pieter Abbeel, and W. Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[43] D. Silver, A. Huang, C. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schritwieser,
I. Antonoglou, V. Panneershelvam, M. Lanctot, al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.

---

[44] A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and P. Abbeel. Overcoming exploration in reinforcement learning with demonstrations. 2018 JEEE international conference on robotics and automation (ICRA), pages 6292-6299. IEEE, 2018.
12

---

Appendix
9.1 P-data, an abstract representation for rope states
A common way to abstract the state space in knot tying is using the P-data representation [24]. The P-data representation translates rope configuration to matrix of discrete values that depends on the number of link intersections. The P-data algorithm stages are: (1) project the 3D rope onto a 2D on the horizontal plane. (2) Select rope direction by defining the head and tail of the rope. (3) Move from head to tail and count the number intersections along the path, starting from to N. Those intersections are also called crosses. Finally, (4) each intersection gets over/under value based on which segment is over the other in the height dimension and also gets a sign plus/minus. The sign defined as
I Lover x xT under sign 17 over X Tander! C.y
where 6 is the unit normal of the horizontal plane, and lover and hunder the strands direc- tional vectors. Examples of P-data of projected knots in Figure 1.
9.2
Reidemeister Moves
The various Reidemeister Moves are depicted in figure 4.
9.3 Collection process
Our data collection process is split into two steps, the first is random sampling with resets and the second is noisy re-sampling.
The random sampling with resets works as follows: We maintain set of configurations we have already seen during data collection and their respective number of crosses, i.c. DQ = {(g (Toss))op( figuration sampled uniformly from gr ~ Dg. take random 100 curves of (for E [1... 100)), and reach new configuration li+1 with topological state s 81+1- We sample the curve parameters uniformly: the link is discrete and sampled from [1, 21]. The other three, I, y. and Zmazs are con- tinuous variables sampled from [-0.5, 0.5], [-0.5, 0.5], and [0.001, 0.07], respectively. If the number of crosses in Cross($)41) > Cross(s:) (sx) the transition (gt, St, ct a/-111100 i1+1) is added to D, and the configuration 4i+1 is added to Dq.
For every data collection iteration t, we load the simulation with a con-
noisy re-sampling, the goal is to increase the amount data using previously collected data. First, we sample transitions from our data (thus they are "interesting", i.c. move the agent to higher transition based on the same starting
number of crosses), add noise the action, and obtain configuration and the modified action. The noise distribution is uniform and characterized by four parameters that offset the current action: The link offset is sampled from 1-1, The y. and Zmar offsets are sampled from [-0.05, 0.05] but are clipped to stay within the limits of [0.001, 0.07] for Zmar and [-0.5, 0.5] for x and y. See Algorithm for more information.
9.4 Inverse model
Our inverse model architecture is detailed in figure 5.
- - 1--994
--0 -
-
(a)
(b)
(c)
Figure 4: (a) Reidemeister move one, (b) Reidemeister move two and (c) cross.
13

---

Algorithm Data Collection
while time < Time D'oneeBuugeeSStgeeOOnee do 2: state = GretStateFFoomDoleee
1:
D. Select state randomly
3: Potential Actions
Sample Random for action PotentialActions do
4:
5:
N extSttate RunnActionn (ew..cccooonnnSSmmamaaiii (Smim(attio))
6:
alidaStetel N ext then
& Checks if the action increases number of crosses
7:
Aclion
8:
else ResetState()
9:
10: end if
end for
12:
end while
while time < Time Stgge wo do
14:
state, action =
Select tuple randomly
noise = -acoole oise() Aewion aclion noise
16:
Nexl.Slate = Antion Action)
18:
if V ValidStete (NextState) then
Checks if the action increases number of crosses
19:
SaveAction (action)
20:
else
21:
ResetStaaee)
22:
end if
23:
end while
247H5 12
12x 1024
100-4:2049
2010x2046
20-0-2046
2013:2
(Ot, RappPPositinn)
Figure 5: Inverse model Auto-regressive Stochastic Network. The network predicts an action in an auto-regressive manner: first is predicts the link index E [1, L]. then the height of the curve Zmazs finally predicts the x and y coordinates the curve. All predictions are stochastic (Multinominal for link index, and Gaussian otherwise). Besides the previous elements, the input of each element includes the current configuration gt. the next topological state 81+1, and the link positions of all the rope links. The weights of the sub-components not shared.
14


Reviews:
[{"review_id": "zSv3n157ZT", "reviewer": "Reviewer_6Lwg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper propose a hierarchical planning algorithm for knot tying, which is built on knot-theory and doesn't require access to human demonstrations. The proposed algorithm first discretize the continuous state space of rope into topological states, and plan in this discretized space using graph search. To move between the configurations in the high-level plan, the authors utilize a learned inverse model, which predicts pick position and motion autoregressively. The authors show in simulation that the proposed method outperform various baselines.", "summary_of_recommendation": "Overall, I think it's a good paper that makes solid contribution in rope manipulation. The authors leverage knot theory to design a hierarchical planning method that tackles a really challenging task robotic manipulation task, knot tying. Although no physical experiment is performed, I think it's understandable since the focus is mainly on the planning side. And I do not expect all challenges (perception, planning, controls) are solved in a single paper. Therefore, I recommend weak accept and will raise my rating if all questions are clarified.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- In line 113, it says \"The known reachable configurations, is a tree T\", while in line 127, it says \"identifies the reachable topological states in the graph\". Can you clarify whether the known reachable configurations is stored as a tree or a graph?\n- Fig 1 is a bit confusing. In the bottom left,  topological state is represented by grey sphere, while in the rest of the figure, grey cube is used.\n- Why do we need to store all known reachable configurations? Is it because the transition function is not well defined under the topological representation and Reidemeister moves? If so, can I see it as some sort of probabilistic roadmap, where you build a roadmap (graph) made of topological states, and then use BFS to find high level plan at test time?\n- During planning, multiple (K=6) curves are generated by the inverse model (line 163). I'm wondering how do you determine which curve leads to the desired topological state s'.\n- How does the Low level only (line 223) work? Is it basically a random policy that samples in the low-level action space? If so, it doesn't seem to be a fair comprison. To show the advantage of hierarchical planning, you compare with low-level planner such as CEM or MPPI. \n- The ablation shows that a discretized actions space (discretized curve) hurts performance, and a continuous action is inferred by the inverse model. But I also notice that the picked point selection is discretized, i.e., rope is divided into multiple links. I wonder how much does this affect the performance. And how does the performance change with respect to the number of links L", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n- Knot tying is a very challenging task and the results shown in the video are truly impressive. The paper definitely makes a step forward in the field of rope manipulation.\n- The proposed hierarchical planning method seems really promising. It transforms the high-dimensional state space of rope into a fixed set of topological states based on the know theory, which significantly improves the efficiency of planning. The low-level inverse model is trained in a self-supervised manner and also works very well in tracking the high-level plan.\n- The paper conducts thorough experiments to evaluate the effects of different design choices. Although no physical experiments is conducted, the authors show in simulation that the proposed method is robust to different phyiscal properties.\n\nWeaknesses\n- The proposed method is only tested in simulation but not real world.\n- The paper makes a lot of simplifications in terms of perceptions and controls. It assumes perfect state estimation, which is very difficult in real world due to self-oclcusion. Also, it uses a floating gripper that can move freely. While in real world, a robot arm will introduce additional topological constraints that makes certain actions infeasible. Sometimes, the robot may need to release the gripper and regrasp the rope from the other side.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "s2nm2E10Ef", "reviewer": "Reviewer_fZRK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents TWISTED, a pipeline for tying knots without demonstration through hierarchical planning. The high-level planner plans a path with breadth first search in the space of topological states, and the low-level planner uses a learned auto-regressive model to transition between topological states. The proposed method is evaluated in simulation using Mujoco and outperformed the baselines. TWISTED was tested on different friction coefficient and yielded similar results.", "summary_of_recommendation": "Overall, I think the paper is quite interesting and the proposed idea could have impact. I am not familiar enough with rope manipulation to judge exactly how this paper stands relative to the literature, but it seems like a decent advancement. This could make a solid paper with improvements in execution and the presentation issues ironed out. However, in its current form, I cannot recommend it for CoRL.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1.\tThe sensitivity analysis under different friction coefficients needs to be justified further. The authors did not state how much the friction coefficient was changed to yield “82% curve has the same topological state as the original friction value.” The difference between the results in figure 3(b) differ by up to 2 trials at times, which does yield a 20% success rate difference. More trials are needed to support the sensitivity claim, i.e. improve similarity to significantly over 82%.\n\n2.\tWhat exactly is the setup for the SAC+HER baseline? The text says “after training we replace our inverse model with the policy.” That sounds like the high-level planner is still used. If so, doesn’t that just mean SAC+HER is significantly worse than the low-level action generator?\n\n3.\tI suggest moving the TWISTED Algorithm from the appendix to the main text. It was very useful for understanding the algorithm in my opinion.\n\n4.\tThe purpose of SelectTopologicalState is unclear to me, given that the rope is starting from a specific $q_{init}$. Is this just finding the next $s$ for the rope to goto? i.e. the full topological state trajectory of the rope should be $s_{init}, s_{selected}, …$\n\n5.\t“FollowPlan” and “RandomExpand” in the appendix are not defined.\n\n6.\tShould “FollowPlan” take in the current configuration of the rope as well? How many topological states is “FollowPlan” expected to traverse? If the plan following fails, for instance the rope did not end up in the next topological state in $P_{selected}$, is there any replanning?\n\n7.\tIn the abstract “…at the high level, use ideas from knot-theory to plan a sequence of rope configurations…”. Shouldn’t “rope configurations” be “topological states”? Also there should not be a hyphen in “knot-theory.”\n\n8.\tSection 4.2 Data structures: is the known configurations really a tree? The “reachable configurations” in Figure 1 contains cycles.\n\n9.\tSection 4.3 “The result of the BFS is a set of paths” Could you clarify how the BFS may return multiple paths?\n\n10.\tOn page 4 “SelectTopologicalState(): identifies…in the graph.” Is “graph” referring to the tree T?\n\n11.\tMisaligned “S1” in the “Hi-level planner” box in figure 1. \n\n12.\tOn page 4 “our aim is to increase the frequency in which topological states of higher complexity are utilized as the initial state in the high-level plan”. What is the reason behind this?\n\n13.\tShould “low-level states” in the caption of figure 1 be “rope configuration”?\n\n14.\tI suggest providing an animated visualization of the TWISTED algorithm. It would be helpful for the reader.\n\n15.\tThe texts in Figure 3 need to be larger.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The main idea of this paper is interesting and the method is supported with empirical results. Admittedly, I am not familiar enough with the rope tying literature to know if something similar has been attempted before. \n\nThe execution of the paper should be improved.  There should be more empirical data especially since this paper is simulation only. The presentation of the paper is unsatisfactory. Please refer to the “Questions for rebuttal” section for more details.\n\nStrengths:\n\n1.\tClear proposed idea\n\n2.\tFavorable empirical results\n\nWeaknesses:\n\n1.\tLimited empirical data\n\n2.\tUnpolished presentation\n\n3.\tNo hardware experiments", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "iQkSSAYG8D", "reviewer": "Reviewer_AJzJ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a pipeline for making it possible for a robot to tie knots and manipulate wires without relying on a large set of human demonstrations. This is done by splitting the approach into (i) a low-level part which involves learning how to represent states of the rope from perceptual data, and (ii) a high-level part that uses ideas from knot theory to reason about how the rope needs to be manipulated to bring it to the desired state. The approach is shown to generalize, demonstrating the ability to tie knots that were not seen during training. In total, the approach yields a system capable of handling knots in a category the authors label \"easy\" - a significant improvement on baselines such as deep RL, which can't even do that - in my view, fantastic progress.", "summary_of_recommendation": "This paper brings great improvements on a hard problem, using an approach with a significant amount of bells and whistles which are largely justified and make sense. I am happy with the results and evaluations. The main downside is that the paper does not describe enough detail about the setting or method in the main body, due to a combination of repetitive text, too much detail about prior work, and generally inefficient use of space. The paper would be significantly improved with a a rework of the writing, but I think is already interesting enough that I am more than happy to see it published now in spite of its flaws.\n* Since important parts of the algorithm such as the data-collection scheme of Section 4.5 are not described at all in the main text, it's also possible there are hidden problems that should have been mentioned as limitations have been withheld. I didn't detect such issues, but am very curious if other referees might have.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Why are knot-theoretic notions which are critically important to understanding this work not defined in the main text, whether intuitively or rigorously, and instead relegated to references and/or appendix?\n\nDoes the tree search algorithm you use - which, if I'm not mistaken, consists of picking a random node in the tree, and executing random actions out of that node - have a standard name?\n\nI presume Fig. 3(b) shows different kinds of rope with the same algorithm - which algorithm is used here?\n\nAre there any key difficulties or limitations involving the data-collection scheme of Section 4.5 or other parts of the paper that have not been described in detail in the main body?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is an important problem. Enabling robots to manipulate ropes and wires at the same level as a human being would be very valuable commercially, and would greatly increase our understanding of how to solve more general dextrous manipulation problems. This is particularly interesting because, unlike many manipulation problems involving rigid physics, ropes present a continuum of degrees of freedom, and appropriate understanding needs to be developed to handle this.\n\nI am not familiar with knot theory - and, I suspect, most readers will not be as well. Therefore, a more comprehensive review would have been helpful, in my opinion the background is too short. For example, the authors introduce the notion of \"Reidemeister moves\" which is an important notion because each such move represents steps in the high-level planner, but don't attempt to offer an explanation. At least something needs to be said about what those moves represent intuitively and why they are the appropriate knot-theoretic notion to use for high-level planning steps. If the authors think this is too complex to define properly - and I am skeptical this is the case - they should instead explain why this is too complex.\n\nThe writing is wordy, which makes certain sections feel like they take up a lot of space without containing a lot of content. The \"prior work\" section is arguably the worst offender. There is no need to repeatedly explain why the other papers in the literature are different from this one in point-by-point detail - this is a paper, not a review rebuttal. If this kind of detail is offered at all, this should be presented in the conclusion section after the reader has read about the paper's techniques. This section should therefore be cut to 1/3 its current length and the extra space used to better explain the background so that readers have the tools they need to understand the differences between this paper and other papers on their own. \n* For example, the \"knot un-typing paragraph\" could be replaced with \"[these authors] have examined the task of un-tying a knot in a given rope, with most approaches operating using high-level abstractions which mirror the ones we study\". There is no need to say that bringing a rope to an arbitrary state - rather than to an untied state - is more complex, as this is obvious.\n\nSection 4 could have a more informative title than \"Method\", which instead describes something about how the method works.\n\nSection 4 focuses on the steps taken by the planning algorithm, which is a tree-based algorithm where nodes are added based on picking a random node and executing random actions. One could easily generalize this to more general tree search algorithms, which might be faster at the cost of being more complex or more difficult to implement. It would be nice to formulate this in a general way, and say that a particular way of searching the tree is used by the authors because it works.\n\nThe data collection scheme is not described, but is critical to the algorithm's ultimate success. This part is therefore important enough that it needs to be described in the main body, not the appendix.\n\nFigure 3 fonts are too small, and because my eyesight is not great I cannot tell what algorithms are doing well and which ones are not. Please make this figure significantly larger, so that its fonts are at least the same size as the paper's text.\n\nIn the text, \"Figure 3\" is not capitalized consistently. Please make this consistent.\n\nThe CoRL-required \"limitations\" section is not used effectively, because the authors use it to describe limitations a second time that have already been stated earlier in the text. As mentioned earlier, this paper would be significantly better if more space was dedicated to describing the background and methods in more detail rather than moving critical information into the appendix, which means that repeating information here harms the paper. Please consider moving the limitations to a subsection of the methods section, and consider moving other parts of the text that describe limitations elsewhere into this section.\n* Additionally, the most important limitation of TWISTED is that - while a better algorithm than baselines - it also does not solve in the medium or hard levels. This should be mentioned in limitations, so that readers who don't read the rest of the paper can quickly see that this is the level of performance to be expected. Note that being upfront about this limitation does not devalue the work, since knot-tying is an important enough problem that even limited performance improvements are of good scientific value.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "buXB3hWDqH", "reviewer": "Reviewer_o22m", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors present a hierarchical approach to manipulating rope into a desired goal configuration. The hierarchy is broken down into a high-level solver with a discrete representation of the rope and a lower-level that uses a neural network to plan a sequence of robot actions between subgoals dictated by the high-level solver. The authors present several simulated experiments that investigate their approach against some simple baselines. ", "summary_of_recommendation": "The authors propose a compelling method to complete rope manipulation tasks. However, the clarity of presentation and the choice of experiments along with baselines needs to be addressed for the paper to show the advantages of their approach over other competitive methods. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- The paper needs to be thoroughly edited to rectify issues, for example, figure 2 isn't referenced in text (I do not believe); Figure 1 is on page 2 but not referenced until page 4; some non-intuitive phrases in lines 61-21, 59-65, 122-124. \n- What is the notation q supposed to represent? It is not clear in text. \n- Acronyms are used but not spelt out, i.e. BFS\n- What is meant by \"The difficulty in manipulating ropes arises from the limited number of actions to change the rope's topological state\"?\n- Is rope a 1-dimensional object? Is it not simulated as a 3D object with many degrees of freedom?\n- What is the inverse model based on? I would suggest adding the network architecture diagram into the text as this would be helpful for a reader trying to follow the text. \n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n\n- The authors present a novel approach to rope manipulation that is compelling.\n\nWeaknesses\n\n- The experiments the authors have chosen to highlight their proposed method and the research questions they proposed seem disconnected. \n- The authors do not validate their method against a competitive state-of-the-art background. I would suggest applying a hierarchical RL method. \n- The presentation of the results within the paper is poorly executed. A table would help a lot. Figure 3 is not referenced in the correct places and is not intuitive to follow (i.e. what is the y-axis).\n- The paper needs to be thoroughly edited to rectify writing issues.\n- Sim-to-real is a significant overheard for transferring this method to a real-world environment although the authors allude to \"real-world applications\" in their sensitivity analysis as if overcoming small changes in friction make it likely that this will naively transfer. \n- The format of the related work induces a lot of repetition.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a hierarchical planning algorithm, TWISTED, for solving the challenging problem of knot tying in rope manipulation. The proposed approach leverages concepts from knot theory and introduces a novel methodology that does not rely on human demonstrations. The algorithm involves two levels of planning: a high-level planner that operates in a discretized space of topological states and a low-level planner that employs an auto-regressive model to transition between these states. The results of the algorithm are evaluated in simulation using the Mujoco platform and compared against various baselines.
Strengths: The paper offers a novel approach to addressing knot tying in rope manipulation through the introduction of a hierarchical planning algorithm. The concept of utilizing topological states to represent the rope's configurations is innovative and demonstrates an effective way to reduce the complexity of the problem. The proposed method exhibits improved performance in simulation, surpassing the performance of various baselines, which underscores the efficacy of the algorithm. The paper demonstrates a systematic approach to evaluating the algorithm by conducting experiments to evaluate the impact of different design choices on its performance.
Weaknesses: While the simulation results are promising, the lack of real-world experimentation is a limitation. It is acknowledged that real-world conditions would present additional challenges, such as perception and control issues, which need to be considered for practical deployment. Some simplifications in terms of perception and control are made, including assuming perfect state estimation and the use of a floating gripper. These assumptions may not hold in real-world scenarios and should be addressed in future work.
The reviewers have raised valid concerns and provided constructive feedback on various aspects of the paper. The authors' responses indicate a thorough understanding of the issues and plans to address them. The revisions made in response to the reviewers' feedback are documented and are aimed at enhancing the clarity, presentation, and understanding of the proposed algorithm. While the absence of real-world experiments and the impact of simplifying assumptions are acknowledged limitations, the novelty and potential of the proposed method make it a valuable contribution to the field of robotic manipulation.
Decision: Accept (Poster)