TieBot: Learning to Knot a Tie from Visual Demonstration through a Real-to-Sim-to-Real Approach
Authors: Weikun Peng
Jun Lv
Yuwei Zeng
Haonan Chen
Siheng Zhao
Jichen Sun
Cewu Lu
Lin Shao
Venue: CoRL 2024 (2024)
Keywords: cloth manipulation
learning from demonstration
robot learning

Abstract: The tie-knotting task is highly challenging due to the tie's high deformation and long-horizon manipulation actions. This work presents TieBot, a Real-to-Sim-to-Real learning from visual demonstration system for the robots to learn to knot a tie. We introduce the Hierarchical Feature Matching approach to estimate a sequence of tie's meshes from the demonstration video. With these estimated meshes used as subgoals, we first learn a teacher policy using privileged information. Then, we learn a student policy with point cloud observation by imitating teacher policy. Lastly, our pipeline applies learned policy to real-world execution. We demonstrate the effectiveness of TieBot in simulation and the real world. In the real-world experiment, a dual-arm robot successfully knots a tie, achieving 50% success rate among 10 trials. Videos can be found on https://tiebots.github.io/.

TieBot: Learning to Knot a Tie from Visual Demonstration through a Real-to-Sim-to-Real
Approach
Weikun Peng Jun Lv², Yuwei Zeng', Haonan Chen', Siheng Zhao",
Jichen Sun?, Cewu Lu Lin Shao 1 School of Computing, National University of Singapore 2 Department of Computer Science, Shanghai Jiao Tong University á Department of Computer Science and Technology, Nanjing University
Abstract: The tie-knotting task is highly challenging due to the tie's high de- formation and long-horizon manipulation actions. This work presents TieBot, a Real-to-Sim-to-Real learning from visual demonstration system for the robots to learn to knot a tie. We introduce the Hierarchical Feature Matching approach to estimate sequence of tie's meshes from the demonstration video. With these estimated meshes used as subgoals, we first learn a teacher policy using privi- leged information. Then, learn student policy with point cloud observation by imitating teacher policy. Lastly, pipeline applies learned policy real- world execution. We demonstrate the effectiveness of TieBot in simulation and the real world. In the real-world experiment, dual-arm robot successfully knots a tie, achieving 50% success rate among 10 trials. Videos can be found on our website.
Keywords: Learning from Visual Demonstration, Real-to-Simm-t--Rall Cloth Manipulation
Introduction
Learning cloth manipulation holds great utility across a wide range of applications. One intriguing domain robotic tie knotting. Service robots must be adept at tasks like aiding the elderly or individuals with disabilities in dressing for certain social events. Teaching robots knot ties, special case of cloth manipulation, typically pushes the limits of robotic cloth manipulation. This offers valuable insights for tie knotting and the broader field of robotic cloth manipulation.
Cloth manipulation presents challenges for robots due to its high-dimensional state and complex dynamics. Extracting and modeling state information are difficult problems. In contrast, humans have accumulated extensive knowledge about cloth manipulation. These priors make learning from demonstration (LfD) a promising direction. LfD empowers a robot acquire policy from expert demonstrations, significantly reducing the need design task-specific reward functions manually. Consequently, LfD stands potent and efficient framework for instructing robots in the execution of complex skills.
However, existing LfD methods struggle with tie-knotting tasks. Kinesthetic demonstration or tele- operation suffers from the complexity of tie- tie-knotting tasks. Tie-knotting tasks require bi-manual operations, placing high demand on human operators skills and equipment. For instance, Zhang al use VR headsets for teleoperation
[1]. Thus, simple behavior cloning may be significantly labor- intensive. Learning from visual demonstration is usually an easier approach in terms of collecting
Correspondence to peng weikun ctc.uuuuu.ccc. and linshao @ @us..cu.uu....ggg
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

demonstration data. But this approach also leads to embodiment gaps. Therefore, researchers at- tempt to find some object-centric representations that robots can utilize to generate correct actions, overcoming embodiment gaps. Several methods attempt to learn a general visual representation of some simple pick-place skills via large-scale pre-training on actionless videos [2, 3, 4J. These works present strong generalizations on the learned visual representations, but none of them shows the abil- ity to learn dexterous manipulation skills that can knot tie. Other methods such as [5, 6, 7, 8, 9] try to leverage object trajectories or keypoints as representations to guide the policy learning. Such representations are indeed sufficient to describe simple object motions but fail to capture the tie's complex topology and subtle dynamics.
Compared to the existing LfD work mentioned in the previous paragraph, our insight is that mesh is the most suitable representation for tie-knotting tasks and other complex cloth ma- nipulation tasks. It captures accurate geomet- ric structures and physics properties of the tie, which crucial for tie-knotting tasks. It also disentangles irrelevant information the vi- sual demonstrations, such as environment back- ground, object colors, and so forth, enabling the learned policy to apply to different test set- tings. Therefore, inspired by [10], we propose a Real-to-Sim-to-Real LID framework. First, we propose a Hierarchical Feature Matching method iteratively estimate the tie's meshes with cloth simulation from the demonstrated video. We use cloth simulator called Dif- fClothAl [11] that supports intersection-free contact for cloth to maintain the tie's topo- logical structure during the estimation process. These estimated meshes from the demonstrated video are then used as subgoals. To learn where grasp the tie and where to pull the tie from point clouds observations in simulation, adopt teacher-student training paradigm similar to [12]. Lastly, our pipeline executes learned policy in real-world settings.
Demonstration
Real2Sim
Learn@Sim
Sim2Real
Maniulatioo
Our proposed TieBot performs tie- knotting task. We leverage cloth simulation to re- cover the cloth's 's state from human demonstration and learn a goal-condition policy to accomplish the tie-knotting task.
Figure 1:
In summary, we make the following contribu- tions: 1) We introduced systematic LíD framework for a dual-arm robot learn knot to tic. 2) We proposed Hierarchical Feature Matching approach estimate the tie's mesh with high de- formation from the demonstrated RGB-D video using cloth simulation. 3) With estimated meshes as subgoals, we presented teacher-student training paradigm to learn grasping points and placing points from point cloud observations in simulation. 4) We conduct experiments in simulation and the real-world to demonstrate the effectiveness and advances of our pipeline. To the best of our knowledge, this work is the first effort to develop robotic system that integrates perception, model- ing, and robot learning to train robots to tackle the task of tie-knotting particularly complex and underexplored area of cloth manipulation.
2
Related Work
2.1 Cloth Manipulation
Previous work mainly addresses short-horizon cloth manipulation tasks that only involve simple pick-place actions. There several approaches to learning cloth manipulation skills. One approach

---

is using model-free RL or learned dynamics model to learn cloth unfolding, rope rearranging, and dressing assistance tasks on raw sensor input [13, 14, 15, 16, 17]. Other approaches will collect and annotate data from images [18, 19] or generate demonstration trajectories in simulation [20] to learn policy. Because the short-horizon and simple actions features of tasks, also possible to infer correct actions from some visual representations, such flow between current observation and target images [21].
In contrast, tie tie-tnoting tasks require flipping or rotating a part of the tie, which makes it difficult to annotate robot actions or design action primitives. Therefore, collecting and annotating robot actions on observations is infeasible. It's also difficult to generate demonstrations or directly apply RL in simulation since the trajectories of tie- tie-knoting tasks are much longer and the possible state space much larger. Thus, in this work, choose to learn skills from human demonstration.
2.2 Learning from Visual Demonstration
One line of research explores pre-training neural representations from actionless videos [22, 23, 2, 24, 4, 25]. This approach aims at learning general representations for different actions, whereas none of them shows the ability to learn dexterous manipulation skills that can knot tie. Another line of research attempts to learn from visual priors extracted from visual demonstrations, such as object trajectories [5, 26], hand poses [27, 28), keypoints positions [29, 6], graph relations [7], or affordances [30]. The third approach is to learn a video or trajectory prediction model to guide policy 32, 33, 34, 8]. These approaches require in-domain demonstrations, placing restrictions
learning 32, on visual demonstrations. Moreover, the prediction model may suffer from the long-horizon feature of tie-knotting tasks. ORION the most closely related work, which builds graph representation from object motions that generalize across diverse test environments [35]. However, simple graph representations cannot capture the tic's complex topology and subtle dynamics during the tie- knotting process.
Consequently, we propose explicitly modeling the demonstration as a sequence of meshes. Mesh can accurately describe the tie's structure and dynamics, which is crucial to learning correct robot actions and generalizing them to different test scenarios.
2.3 Cloth State Estimation
One cloth state estimation method directly predicts cloth states using deep learning [36, 37, 38, 39]. Non-rigid point cloud registration methods such as coherent point drifting are also applied for linear deformable object tracking [40, 41, 42]. However, purely vision-based methods do not guarantee correct cloth topology due to the lack of physics prior. Huang et al. propose a method to reconstruct and track cloth state with dynamics model [43]. However, this method requires known actions, which cannot be accessed easily from human demonstration sometimes. Lv et al. use differentiable rendering to estimate the state of linear deformable objects [10]. However, tie is a 2D deformable object.
Therefore, we propose Hierarchical Feature Matching method to iteratively estimate the tie mesh in the demonstration video with cloth simulation. Cloth simulation provides important physics prior for state estimation, such as non-penetration, which is crucial for maintaining correct topology.
Technical Approach
This work presents a Real-to-Sime -oo--ii-ttt-elll LiD framework called TieBor to guide a dual-arm robot shown in Fig. knot the tie from an RGB-D demonstration video. overview of our proposed method is in Fig. 2. We first describe the procedure estimate the tie's mesh sequences from the demonstrated video (Sec. 3.1). Using the tie's mesh sequences as subgoals, introduce pipeline to generate robot actions to manipulate the tic, using teacher-student training paradigm (Sec. 3.2).
3

---

Real2Sim
Iterative Estimation
Feature Extraction
CT
Local Feature Matching. If D and x-_ are aligned and there are correspondences be- tween X and X.P we can build the cor- respondences from the tie's mesh to the next with Sim
Matching
Sim2Real
Learn@Sim
Where Pull?
Where Grasp?
Policy Execution
H
Gronp Candiddates
Manipulation
Reacher-Studont Training
TieBot utilizes simulation to estimate the tie's meshes from the demonstrated video. Then, using mesh sequences as subgoals, introduce how to generate the robot's actions to manipulate the tic. The pipeline finally executes learned policy in real world.
Figure 2:
3.1 Real2Sim
To better estimate the tie's mesh, we propose to integrate cloth simulation into our pipeline, which provides important physical prior such as non-penetration in the estimation process. We segment the tie in the demonstrated RGB-D video using Track-Anything [44] and transform the associated segmented depth images into point clouds. Meanwhile, a tie's 's mesh loaded into the DiffClothA. At time step RGB images and segmented point clouds denoted {I,²) and {X,", Real2Sim pipeline estimates tie's 's mesh sequences {)/*} with simulation. The pipeline manually aligns the mesh with the initial frame. We assume the initial mesh fully overlaps with the initial point cloud.
use the tie mesh's vertices denoted as D describe the tie's shape. From the
demonstrated point cloud X& and move the tie's 's vertices to align 2 towards XPP.
Here we adopt an off-the-shelf feature match- ing model called LoFTR [45] build up cor- respondences between two RGB images I,- and I, shown in Fig 3. Typically, LoFTR can provide more than hundred reliable correspondences between two images, which cover almost every visible part of the tic. From the correspondences between Z,-1 and I,, we can find the feature points on XP_ and their corresponding feature points on X;P Then, to control the mesh in Diff- ClothA/ to align with x,, we need to define several vertices on the mesh as control vertices V. Since y aligns well with XP_,, we map the feature points on XP_, to nearest vertices on y These vertices are assigned as control vertices Finally, we control V to move the positions of feature points X. to align ) towards X, in DiffClothAl.
Figure 3: Local Feature matching between two images. A hand caused gap along the length of the tie during the demonstration.
However, vanilla local feature matching cannot create correspondences in occluded regions, which is common in tie-knotting tasks. We lose the motion information due to occlusion, and the estimation will deviate. Therefore, we propose to add global keypoints information to amend this pipeline.
Keypoints Detection. Keypoints detection directly build correspondences between mesh ver- tices and the point cloud. Thus, it will not be affected by occlusion. We define five keypoints along the tie's surface and the corresponding five key vertices on the mesh, shown in Fig. 4. For each keypoint as the origin, we define the local frame as follows. The z direction is the surface normal from the tie's positive side to the negative side. The x direction is the direction of the tic's middle skeleton. The y direction is derived using the right-hand rule. These five keypoints, predefined order, play the role of the skeleton definition.

---

Then, we train Pointnet++ [46] to predict the keypoints and associated local frames on the demonstrated point clouds. However, the high- dimensional state makes it challenging to gen- erate sufficient training data to cover all the states encountered in the knotting procedure. A successful tie-knotting trajectory occupies only small portion of the whole state space of the tie. Thus, uniformly applying random actions on the initial tie's 's mesh in the simulation to produce training data fails to cover these states, In contrast, we generate training data based on the current estimated mesh. When we detect the chamfer distance between 2 and X, is larger than a threshold, we backtrack to the previous time step, gather the tie's shape y and apply random actions the tie's mesh at in the simulation to generate annotated training data and train the keypoints prediction network.
The oriented keypoints to represent the state of the tie. The x,y.z axis are represented by the red, green, blue arrow, respectively.
Figure 4:
Hierarchical Feature Matching (HFM), Finally, we combine them as Hierarchical Feature Match- ing (HFM) for state estimation. Control vertices assigned in local feature matching and key vertices will be used together to pull the mesh to target positions specified by local feature matching and keypoints detection. Local feature matching provides detailed motion of vertices, while global key- points indicate global tie's structure. This global structural information ensures the estimation won't deviate too much, alleviating the shorteomings of the local feature matching method. We use this method to estimate the tie's s meshes from demonstration and output sequence of meshes ()/g) ). The next part will use these meshes as subgoals to guide robot action generation.
3.2 Learn@Siim
Our pipeline begins to sequentially generate feasible robotic actions the simulation to guide the tie ( towards these subgoals. Since the tie-knotting task a long-horizon task with multiple grasp and pull actions, we aim learn where to grasp the tie and where to pull the tic.
For where to pull, we apply a simple strategy, Once we identify the grasping vertices, we pull these vertices to the positions of those vertices on the subgoal. For where to grasp, we adopt similar teacher-student training paradigm in [12] to ease policy learning. Directly learning from high dimensional observations such as point cloud is data-inefficient because the policy needs to simultaneously learn which features to extract from visual observations and what the high-rewarding actions are. On the contrary, learning a policy via RL from sufficient state information would be much easier, suggested by [12]. Therefore, first use privileged information learn teacher policy, and then train a student policy imitating teacher policy with point clouds as observations.
We first learn a teacher policy to select proper grasping points using privileged in- formation. The state s contains the previous tic's vertices positions and the point-wise displacement for each tie vertices to the subgoal. The action is one or two grasping vertices of all the tie mesh't vertices. The reward function R is defined in equation
Teacher Policy.
Note that we specify the action space as the discrete space (vertex index of the tie). Although there are multiple 6D poses of the robotic grippers to grasp one vertex position of the mesh the learned policy still reflects the overall grasping quality of these 6D poses associated with one vertex. In the engineering practice, we record each grasping pose offline so that once we figure out the grasping vertices on the tie's mesh each timestep, we can automatically produce the feasible grasping poses concerning specific hardware platforms using inverse kinematies.
C1, C2. C3- |1
if knotting-tic tie succeeds
R(s,a) a)
if fails to reach any subgoal along the trajectory Otherwise
yg
For the reward function, here C1 C2 C's are constant positive values. X; is the result tie mesh. The failure to reach the subgoal is due to the distance ! - Df || is larger than given threshold, the tic
5

---

could not be pulled close to the subgoal by grasping on the wrong selected vertex; otherwise, it will return C3 - 1 -/( for intermediate steps or C₁ for the final step.
Student Policy. To learn actions from point clouds, we train a student policy imitate teacher policy. We add some perturbations to the size and positions of the mesh and update the associated trajectories accordingly to generate training data in the simulation. We render point clouds from meshes in PyBullet [47] as the input of our policy network t and output the grasping points and placing points positions. We use Pointnet++ [46] as the policy network and train it in a supervised learning manner.
Experiments
In this section, we introduce our experimental setup and conduct quantitative and qualitative eval- uations to demonstrate the effectiveness of our approach. Our experiments focus on answering the following questions.
How do our pipeline and baseline methods perform tie- tie-knotting task?
Can our pipeline apply other cloth manipulation tasks?
How does HFM compare to other cloth state estimation methods?
Can our HFM apply to other cloth manipulation tasks?
Considering the complexity of the entire system, we provide additional experiment results, along with detailed explanations of submodules, in the supplementary materials and website.
4.1 Comparing TieBot and Baseline
We first evaluate the whole pipeline of TieBot and baseline method in tie-knotting task. We estimate sequence of meshes from one human demonstration video. Then, divide the whole trajectory into 6 parts with 6 subgoals. Our teacher policy learns to select proper grasping points using PPO [48], and student policy imitates the teacher policy to infer grasping points and placing points from the point cloud. We evaluate TieBot and the baseline method 10 times for each of the two different ties in DiffClothAl and evaluate TieBot on two real ties with dual-arm robot.
To illustrate our pipeline applies other cloth manipulation tasks besides the tie-knotting task, conduct experiments the towel-folding task in the real setting. The towel-folding process is shown in the last row of Fig. 7.
ATM ATM proposes to model tasks as points trajectories [8]. first learns a trajectory prediction model, and then learns policy with the learned prediction model using imi- tation learning. For tie-knotting task, following similar experiment settings in ATM, we collect 100 demonstra- tion videos in simulation to train the trajectory prediction module. Then, the 45 demonstration videos with ground truth action annotations train the policy network and test the policy in simulation. The action is the 3D offset of the grasping vertices. For towel-folding task, we collect 100 human demonstrations and 10 robot demonstrations in the real world. We train the ATM to predict the displacement of the two robot arms' end effectors cartesian space.
Success Rate / Average Achieved Subgoals
Ours
ATM
simulation: normal tie
60% / 5.1
0% 0.0
simulation: larger tie tie(unseen)
30% / 4.3
0% 0.0
real: real tie(softer)
50% 5.0
/
real: real tie(harder, unseen)
30% / 4.15
real: towel(unseen)
70% / 1.6 0% / 0.8
Table 1: Success rate and average achieved subgoals of pol- icy rollouts
Metrics. We compare the success rate between our pipeline and ATM [8]. For the tie-knotting task, in simulation experiments, if the distance of the final tie's 's mesh to the target tie's 's mesh is smaller than threshold, we consider it success. In real-worke experiments, if the little end of the tie is inserted

---

Step1
Step2
Step3
Stepd
Step5
Step6
Final
Human Demo
d
State
Estimation
I. Learned Policy
Real Execution
Figure 5: The results of TieBor at different stages. We show different sides of the tie red and blue and manipulation action in yellow better visualize.
into the hole, as shown in the final stage in Fig. 5 and Fig. C.9, we consider it a success. Since the tie-knotting task is long- oogghorizon, we also compute the averaged number of achieved subgoals for further evaluation. For the towel-folding task, We consider the state to be successful if the towel stays totally on one side the folding line, as shown in Fig. D.12.
Experiment Result. We test TieBot on two real ties that differ in materials: one is softer, and the other one is harder. We tested each of them 10 times. We also test TieBot and ATM two ties with different sizes in simulation 10 times for each. The quantitative results are shown in Tab. 1, and qualitative results of TieBot are shown in Fig. 5. This comparison suggests that object trajectories are insufficient to represent subtle dynamics and topology of the tie in tie- tie-tnotingg tasks. ATM quickly deviates from the correct trajectory since it cannot capture the subtle dynamics of the tic. Therefore, fails to achieve even one subgoal. Hence, explicitly modeling the tie in meshes necessary. For qualitative evaluation, in Fig. 5, we can see that although the tie in the demonstration video, the mesh in the simulation, and the tie used for real robot manipulation overcome these gaps and learn feasible robot policy. For the towel-folding task, we find that ATM can learn the first folding action but struggle with learning the latter action.
different, pipeline can
4.2 Evaluating Hierarchical Feature Matching (HFM)
Real2Sim is the most important part of our pipeline. Without accurate state estimation, particularly estimating the correct topology for the tie, it's impossible learn a feasible policy to accomplish the task. To illustrate the importance of different components of HFM and its performance against other cloth state estimation methods, design three experiments in simulation test baseline methods and the ablation versions of HFM.
Coherent Point Drift. Coherent Point Drift (CPD) [49] is a non-rigid point cloud registration algorithm. We employ the CPD to predict the target positions of the mesh vertices in the target point cloud and directly align the mesh the target positions.
Ablated Version. Ours w/o KP stands for only using local feature matching; Ours w/o LF stands for using local feature matching and the predicted keypoints positions; Ours w/o FM stands for only using predicted keypoints positions and local frames.
L2 Distance
Ours
Ours w/o FM
Ours w/o KP
Ours w/o LF
CPD
expl
0.0248
0.0732
0.2424
0.0512
0.1384
exp2 exp3
0.0053
0.0107
0.0123
0.0088 0.0049
0.0661
0.0032
0.0093
0.0053
0.1049
Table 2: Quantitative results of ablation study and comparison to CPD simulation.
7

---

initial
larget
Ours
Ourswo FM
Ourswo KP
OurswoLF
CPD
exp
esp2
eqp
Figure 6: The visualization of the ablation study of HFM in simulation. We put cross sign in the image's bottom-right corner to indicate failures estimating the correct target state, according to human evaluation. Red and blue colors represent different sides of the mesh.
Experiment Result. The qualitative results are shown Fig. 6. can find that either CPD or ablated versions of HFM cannot estimate the target mesh correctly among these three experiments. We also compute the L2 Distance between the vertices of the target mesh and estimated mesh as quantitative evaluation shown in Tab. 2. It also suggests that the performance will degrade if cancel some parts of HFM, while CPD deviates lot from the correct states.
4.3 Apply HFM on Other Cloth Manipulation Tasks
We demonstrate that HFM can applied to other cloth manipulation tasks. One is different way knot tie. The other one to fold a towel. We visualize the estimation results in Fig. The results show that HFM can be applied different cloth manipulation tasks.
Figure 7: The visualization of a different way to knot a tie and the towel folding. The first row is the human demonstration of tie-knotting, the second row the estimated states in simulation, and the third row is towel folding. We show the manipulation action in yellow dots and arrows.
5 Conclusion
This paper presents TieBot, novel system designed to teach robots perform the tie- tie-knotting task through a Real-to-Sime Seie-Siittt--ee Real learning approach. The framework leverages visual demonstrations and integrates a mesh of the tie to capture its complex structure and subtle dynamics, which are es- sential for effective manipulation. Through the proposed Hierarchical Feature Matching method, we estimate tie meshes from demonstration videos and use them as subgoals for policy learning in sim- ulation. Our teacher-student training paradigm enables robots to learn from point cloud observations and execute the learned policy in real-world settings. TieBot demonstrates promising results both in simulation and real-world experiments, achieving 50% success rate for tie-knotting. Additionally, showcase its potential for other cloth manipulation tasks, such as towel folding, indicating the broader applicability of our approach. This work marks a significant step forward in robotic cloth manipulation, particularly for long-horizon, complex tasks such as tie-knotting.
Nonetheless, our pipeline has some limitations. First, our Real2Sim module requires training key- points detection models iteratively, which computationally intensive. Second, due the hardware limits, the last step in the real-world experiments shown in Fig. 5 is hardcode action. Better video tracking methods and more dexterous robot arms may alleviate these issues.

---

Acknowledgments
The authors would like thank Zihao Xu from National University of Singapore for setting up the tie-knotting experiment in the real setting, Zhixuan Xu and Haoyu Zhou from National University of Singapore for the support of computation resources, Hongjie Fang from Shanghai Jiao Tong University and Flexiv Robotics for helping with towel-folding experiment in the real setting.
References
[1] T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel. Deep im- itation learning for complex manipulation tasks from virtual reality teleoperation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 5628-5633 2018. 11:11..11/1IRR.2200 2499.
[2] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual represen- tation for robot manipulation. arXiv preprint arXiv:2203.12600 2022.
[3] I. Radosavovic, T. Xiao, S. James, P. Abbeel, J. Malik, and T. Darrell. Real-world robot learning with masked visual pre-training. In Conference Robot Learning, pages 416-426. PMLR, 2023.
[4] K. Shaw, S. Bahl, and D. Pathak. Videodex: Learning dexterity from internet videos. In Conference Robot Learning, pages 654-665. PMLR, 2023.
[5] D. Seita, Wang, Sheuty, Li, 7. Erickson, and Held. ToolFlowNet: Robotic Ma- nipulation with Tools via Predicting Tool Flow from Point Clouds. In Conference on Robot Learning (CoRL), 2022.
[6] N. Das, S. Bechtle, Davchev, D. Jayaraman, A. Rai, and F. Meier. Model-based inverse reinforcement learning from visual demonstrations. In Conference on Robot Learning, pages 1930-1942. PMLR, 2021.
[7] S. Kumar, J. Zamora, N. Hansen, R. Jangir, and X. Wang. Graph inverse reinforcement learning from diverse videos. In Conference on Robot Learning, pages 55-66. PMLR, 2023.
[8] C. Wen, X. Lin, J. So, K. Chen, Q. Dou, Y. Gao, and P. Abbeel. Any-point trajectory modeling for policy learning, 2023.
[9] X. Zhu, J. Ke, 7. Xu, 7. Sun, B. Bai, J. Lv, Q. Liu, Y. Zeng, Q Ye, C. Lu, M. Tomizuka, and L. Shao. Diff-lfd: Contact-aware model-based learning from visual demonstration for robotic manipulation via differentiable physics-based simulation and rendering. In Conference on Robot Learning (CoRI.), 2023.
[10] J. Lv, Y. Feng, C. Zhang, S. Zhao, L. Shao, and C. Lu. Sam-rl: Sensing aware model-based reinforcement learning via differentiable physics- based simulation and rendering. The Inter- national Journal of Robotics Research, page 027836492412846553 2023.
[11] X. Yu, S. Zhao, S. Luo, G. Yang, and L. Shao. Diffclothai: Differentiable cloth simulation with intersection-free frictional contact and differentiable two-way coupling with articulated rigid bodies. In 2023 IEFF/RS) International Conference on Intelligent Robots and Systems (IROS). IEEE, 2023.
[12] T. Chen, M. Tippur, S. Wu, V. Kumar, E. Adelson, and Agrawal. Visual dexterity: In- hand reorientation of novel and complex object shapes. Science Robotics, 884))eade9244, 2023. doi: 1..11100.c1111 1122/scinobotics.ae99777 10. 1 126 6/ sci robot ics. adc9244.
URL https: httppp:://wwww../ /wwww/wwww//////// www.sce.eeec....cc. www.sscc.iiccecceeeeeeoo abs/
[13] H. Ha and S. Song. Flingbot: The unreasonable effectiveness of dynamic manipulation for cloth unfolding. In Conference Robot Learning, pages 24-33, PMLR, 2022.

---

[14] Y. Wu, W. Yan, T. Kurutach, L. Pinto, and P. Abbeel. Learning to manipulate deformable objects without demonstrations. arXiv preprint urXiv:/9100 1et../990...,,,.. 2019.
[15] Y. Deng, C. Xia, X. Wang, and L. Chen. Deep reinforcement learning based on local gnn for goal-conditioned deformable object rearranging. 2022 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1131-1138. IEEE, 2022.
[16] Y. Wang, Z. Sun, Z. Erickson, and D. Held. One policy to dress them all: Learning dress people with diverse poses and garments. In Robotics: Science and Systems (RSS), 2023.
[17] M. Yan, Y. Zhu, N. Jin, and J. Bohg. Self-supervised learning of state estimation for ma- nipulating deformable linear objects. IEEE robolics and automation letters, 5(2):2372-2379, 2020.
[18] Y. Avigal, Berscheid, Asfour, Kröger, and K. Goldherg. Speedfolding: Learning effi cient bimanual folding garments. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1-8. JEEE, 2022.
[19] D. Seita, N. Jamali, M. Laskey, A. K. Tanwani, R. Berenstein, P. Baskaran, S. Iba, J. Canny, and K. Goldberg. Deep transfer learning of pick points on fabric for robot bed-making, In The International Symposium of Robotics Research, pages 275-290. Springer, 2019.
[20] Y. Deng, K. Mo, C. Xia, and X. Wang. Learning language-conndtionne deformable object manipulation with graph dynamics. arXiv preprint arXiv:2303.01310. 2023.
[21] T. Weng. S. M. Bajracharya, Y. Wang, K. Agrawal, and D. Held. Fabricflownet: Bimanual cloth manipulation with flow-based policy. In Conference Robot Learning, pages 192- 202. PMLR, 2022.
[22] P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, S. Levine, and G. Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1134-1141, IEEE, 2018
[23] Y. J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and A. Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint arXiv:2210.00055 2022.
[24] T. Xiao, I. Radosavovic, T. Darrell, and J. Malik. Masked visual pre-trum pre-training for motor control. arXiv preprint arXiv:2203.06/ 2022.
[25] R. Mendonca, S. Bahl, and D. Pathak. Structured world models from human videos. 2023.
[26] M. Vecerik, C. Doersch, Y. Yang, T. Davchev, Y. Aytar, G. Zhou, R. Hadsell, L. Agapito, and Scholz. Robotap: Tracking arbitrary points for few-shot visual imitation. arXiv preprint DXXi:2330.15973 2023.
[27] S. Bahl, A. Gupta, and D. Pathak. Human-to-robot imitation in the wild. 2022.
[28] H. Bharadhwaj. A. Gupta, S. Tulsiani, and V. Kumar. Zero-shot robot manipulation from passive human videos. arXiv preprint arXiv:2302.02011 2023.
[29] H. Xiong, Q. Li, Y.-C Chen, H. Bharadhwaj, S. Sinha, and A. Garg. Learning by watching: Physical imitation of manipulation skills from human videos. 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7827-7834. IEEE, 2021.
[30] S. Bahl, R. Mendonca, L. Chen, U. Jain, and D. Pathak. Affordances from human videos as the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 13778-13790 2023.
versatile representation for robotics. In Proceedings the
10

---

[31] K. Schmeckpeper, A. Xie, O. Rybkin, S. Tian, K. Daniilidis, S. Levine, and C. Finn. Learning predictive models from observation and interaction. In European Conference on Computer Vision, pages 708-725. Springer, 2020.
[32] A. Escontrela, A. Adeniji, W. Yan, A. Jain, X. B. Peng, K. Goldberg, Y. Lee, D. Hafner, and P. Abbeel. Video prediction models as rewards for reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[33] Y. Du, S. Yang, B. Dai, H. Dai, O. Nachum, Tenenbaum, D. Schuurmans, and P. Abbeel. Learning universal policies via text-guided video generation. Advances Neural Information Processing Systems, 36, 2024.
[34] P.-. Ko, J. Mao, Y. Du, S.-H. Sun, and J. B. Tenenbaum. Learning to act from actionless videos through dense correspondences. arXiv preprint arXiv:2310.08576, 2023.
[35] Y. Zhu, A. Lim, P. Stone, and Y. Zhu. Vision-based manipulation from single human video with open-world object graphs. arXiv preprint ar arXiv:2405. 20:32,,0..0321,, 2024.
[36] C. Chi and S. Song. Garmentnets: Category-level pose estimation for garments via canon- ical space shape completion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3324-3333, 2021.
[37] H. Xue, W. Xu, J. Zhang, T. Tang, Li, W. Du R. Ye, and C. Lu. Garmenttracking: Category- level garment pose tracking. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 21233-21244 2023.
[38] Z. Huang, Lin, and D. Held. Mesh-based dynamics with occlusion reasoning for cloth manipulation. arXiv preprint arXiv:2206. 02881, 2022.
[39] W. Wang, G. Li, M. Zamora, and S. Coros. Trtm: Template- reconstruction and target- oriented manipulation of crumpled cloths. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 12522-12528. IEEE, 2024.
[40] Tang. Y. Fan, H.-., Lin, and M. Tomizuka. State estimation for deformable objects by point registration and dynamic simulation. In 2017 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 2427-2433. IEEE, 2017.
[41] T. Tang, C. Wang, and M. Tomizuka. A framework for manipulating deformable linear objects by coherent point drift. IEEE Robotics and Automation Letters, 3(4):3426-3433, 2018.
[42] C. Chi and D. Berenson. Occlusion-robust deformable object tracking without physics simu- lation. In 20/9 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6443-6450. IEEE, 2019.
[43] Z. Huang, X. Lin, and Held. Self-supervised cloth reconstruction via action-conditioned cloth tracking. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 7111-7118. IEEE, 2023.
[44] J. Yang, M. Gao, Z. Li, S. Gao, Wang, and F. Zheng. Track anything: Segment anything meets vidcos, 2023.
[45] J. Sun, Z. Shen, Y. Wang, H. Bao, and X. Zhou. LoFTR: Detector-free local feature matching with transformers. CVPR, 2021.
[46] C. R. Qi, L. Yi, H. Su, and L. J. Guibas. Pointnet++: Deep hierarchical feature learning on 2017.
point sets metrie space. arXiv preprint
[47] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. http::/pybulli org, 2016-2021.

---

[48] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347 2017.
[49] A. Myronenko and X. Song. Point set registration: Coherent point drift. IEEE Transactions Pattern Analysis and Machine Intelligence, page 2262-2275, Dec 2010. 2010.46. URL http: http::://xxxodooooooooooo orq. 11..101.1000 1109 wte1119tppppaaa tpami. 2010.46.
[50] N. Karaev, I. Rocco, B. Graham, N. Neverova, Vedaldi, and C. Rupprecht. CoTracker: Is better track together. 2023.
[51] N. Tumanyan, A. Singer, S. Bagon, and T. Dekel. Dino-tracker: Taming dino for self- supervised point tracking in a single video, March 2024.
[52] A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann. Stable-baselines: Reliable reinforcement learning implementations, Journal of Machine Learning Research, 22 (268):1-8, 2021. URL http: / wttpoo/pppppp///////ooeeee///... html.
[53] Kinova.
Kinova-movo.
URL
com/
Ainova-movo, http:///eeceocooooooaaaaaaeeeeee
[54] Y. Wang, Praveena, D. Rakita, and M. Gleicher. Rangedik: An optimization- robot motion generation method for ranged-goal tasks. arXiv preprint arXiv:2302. 13935, 2023.
12

---

Appendix
A More Discussion on Real2Sim
A.1 Why not Use Video Tracking?
One may curious about why not use video tracking to extract tie's motion in our work. We test co-tracker [50] and DINO-Tracker [51] on our tie-knotting demonstration videos. For instance, in the towel-folding task, tested DINO-Tracker the demonstration video. From the results shown in Fig. A.I, we can find that even the state-of-the-ar video tracking model cannot provide accurate point-level trajectory.
Start Frame
End Frame
Figure A.1: Track prediction results of the towel-folding task.
A.2 Experiment on Out-of-Distribution Issue of Keypoints Detection
To illustrate the necessity of iteratively updating the detection model, we train neural network with the same structure randomly sampled tie's shape based the initial shape. We trained iterative keypoint detection model on 14 different shapes separately. For each shape, we randomly generate 500 similar shapes for training. For the randomly sampled method, we randomly generate 7000 shapes from the initial tie's shape to train a single keypoint detection model. We compare the result of Ours with random sample RS result in Fig. A.2. Compared to the human annotation result, RS's predictions have larger errors than Ours. RS method encounters an -fitt--iittiittttinn problem. The test shape of the tie cannot be easily sampled, so RS cannot generalize to this test case.
We also test the iterative global keypoint detection and RS method in simulation with ground truth annotation. The quantitative results are listed Tab. A.1.
Position Error(m)
Z-axis Error(")
X-axis Error(")
Ours
0.028
10.68
14.41
RS
0.183
49.00
68.76
Table A.1: Quantitative results of iterative keypoint detection simulation data
A.3
Implementation Details of Keypoints Detection
A.3.1 Keypoint Positions Prediction
1) data generation We first load the mesh model into DiffClothA/ [1]], choose keypoint as control vertices, and apply random perturbations to these vertices to generate different shapes of the tic. Then, we load generated mesh models into PyBullet [47] to render point clouds. We use the same
13

---

حطا صعر
d
RGB image
Point Cloud
Ours
Random Sample
Human A nnotation
Figure A.2: Prediction results of the oriented keypoints on real image and point cloud.
camera intrinsics as demonstrated video and similar camera pose to render point clouds. We generate 500 point clouds as training data.
To annotate training data, we first compute the geodesic distance of all points in the point cloud to keypoints. Then, we convert the geodesic distance to probability using equation 1. d is the geodesic distance, a is hyperparameter, is the probability to evaluate how likely the point is to be keypoint. In our experiments, we use o = 0.15. Thus, a point is close to one of five keypoints, the probability corresponding to that keypoint will be high.
eXp
20
(1)
2) training details Different from the original pointnet+++ semantic segmentation model [46], we change the last layer to sigmoid. The training parameters are listed in Tab. A.2.
parameter name
parameter value
L2(for keypoint positions and offsets prediction) Ll normal and middle line direction prediction) gaussian noise, random scale, random rotation
loss function
data augmentation training epochs batch size learning rate
80
24
le-4
optimizer scheduler
Adam
cosine annealing with 10 epochs warm-up Table A.2: Hyperparameters for training global keypoint prediction
3) inference details Our model takes point cloud as input and outputs a probability matrix P E (0, 1) 1)Nx5, N is the number of points in the point cloud. Each entry P.j represents the probability of point i to be keypoint j. To decode the predicted keypoints positions, we first select points with the top 5% probability as inlier for each column of P. Then, we assign other points' probability to zero and normalize the probability for each column of P. Now we get the normalized probability distribution of each keypoint, denoted P. Finally, compute the average positions of all points N
weighted by normalized probability, 24 >N P. Iz This is the final prediction for keypoint positions.

---

A.3.2 Normal(Z Axis) Prediction
1) data generation We generate data the same way as keypoint positions, For annotation, we first compute the normal direction of each face the mesh. Then, we assign these values to points in the point cloud according to the nearest faces.
2) training details We also remove the log log-soxa layer in pointnet++ [46]. We use LI distance as the loss function. The other training parameters are the same as keypoint position prediction.
3) inference details With predicted keypoints positions and predicted normal directions of all points, we compute the normal each keypoint as the average of neighboring points normal direc- tions.
A.3.3 Middle Line(X Axis) Prediction
1) data generation Same as normal prediction, just change the annotation from normal direction middle line direction.
2) training details
It's the same as normal prediction.
3) inference details
It's the same as normal prediction.
A.4 Results of Local Feature Matching and Keypoints Detection on Real Data
We present some examples local feature matching and keypoints detection on two tie-knotting tasks and a towel-folding task, shown in Fig A.3.
A.5 Ablation Study of HFM on Real Data
We demonstrate the effectiveness of hierarchical matching in cloth state estimation in Fig. A.4. In the first test case, we aim to illustrate the importance of global keypoint detection. Therefore, we chose two images that show differences in positions and orientations. In the third column, Ours method successfully flips the tie and moves forward a little, as shown in the images. Our w/o KP and Ours w/o LF cannot move forward as expected. Because, in this case, local feature matching cannot find correspondences in the tie's left part.
In the second test case, aim to illustrate the importance of local feature matching. We chose two images that contain an operation of lifting side of ring in the air. This action requires detailed information to achieve accurate estimation. The last column shows the result of Ours w/o FM. Our framework cannot accurately estimate the shape only with global keypoint positions and local frames. This global information can only provide general structure guidance instead of detailed shape information.
B More Discussions on Learn @ @Sm
B.1 How to Control Tie in DiffClothAl
Modeling grasping in DiffCloshAl [11] is not simply selecting one vertex on the mesh as the control vertex. Because controlling one vertex is not enough to simulate rotation DiffClothAl, knotting tie requires some rotation actions. Therefore, we select one central vertex and its surrounding vertices as control vertices. shown in Fig.B.5. By controlling small region instead of single vertex, simulate rotation actions in DiffClothAl.
15

---

(a) Local feature matching and keypoints (b) Local feature matching and keypoints detection detection results real-world tie-knotinng sults another real-world tie-knotting demonstration demonstration
Matches: 3512
Matches: 2026
Matches: 1424
Matches: 1156
(c) Local feature matching results on another real-world towel-folding demonstration
Figure A.3: We test local feature matching and keypoints detection on real-world demonstrations. shows that our method works for most tie shapes.
16

---

Image
Image: I+1
Ous
Ours w/o
Ours
Ours w/o FM
(a) Ablation study mainly on keypoint prediction.
top view
L
HP
P
front horizontal view
Imaget
Dure
Ours
Ours FM
(b) Ablation study mainly local feature matching.
Figure A.4: Ablation study on hierarchical feature matching for state estimation.
H
target local frame
simulation tie's local frame before the alignment
after the alignment
Figure B.5: Illustration of control vertices in DiffClothAl.
B.2 How are the grasp and pull actions defined
Grasp Actions For the grasp action, we begin by sampling set vertices from the mesh, which serve as grasping vertex candidates. The grasp action then defined as selecting different combina- tions of these vertices. This selection process is treated as a discrete action space, where policy chooses the optimal vertices for grasping based on the current state of the tie.
Pull Actions The pull action is defined as the target position for the robot's gripper. Once the policy has selected the grasping vertices particular step. the corresponding goal positions are determined by the positions of these vertices on the subgoal mesh. This approach means that the pull action is not learned directly; instead, it is derived automatically based on the selected grasping points and the configuration of the subgoal mesh, leveraging the differentiable cloth simulation.
17

---

B.3
Implementation Details of Teach-Student Training Paradigm
B.3.1 Teacher Policy
We model the grasping point selection as MDP and use model-free RI. to learn the proper grasping point. To simplify the problem, we sample 40 vertices on the middle line of the mesh model as our candidates. vertices directly connected to these candidates in left, right, up, and down defined as their neighbors.
In practice, we evenly sample 40 vertices on the middle line of the mesh model as our candidates. The state 8 is 40 matrix. The action is a 820 x one-hot vector, which contains grasping one vertex(40) and two vertices(40 39/2- 780).
To learn to select grasping points, we use PPO implemented in stable-baseline3 [52]. The hyperpa- rameters are shown in Tab B.3 for all trajectories.
parameter name learning rate batch size gae lamba clip range C1
parameter value
0,0003
64
0.99
0.95
0.2
5
{0.9, 1.5, 2.0, 3.0, 3.0, 1.9)
fitting threshold
(listed in subgoals order)
C2 C3
30
30
Table B.3: Hyperparameters for learning grasping points settings
B.3.2 Student Policy
To learn student policy, first execute the teacher policy multiple times within the DifiClothAT simulation environment. During each run, the teacher policy uses the current mesh and the subgoal mesh to predict the selected grasping vertices on the current mesh, as well as the corresponding placing points. Through these simulations, we generate dataset comprising 3,000 mesh-grasping- placing pairs, which serve as the foundation for training the student policy.
To simulate real-world conditions, load all the generated meshes into PyBullet and use the same camera intrinsics those used in real-worle setup to render point clouds. This ensures that the point clouds accurately reflect the observations robot would receive in practice.
The student policy is then trained using supervised learning, where the input is the rendered point cloud and the output is the predicted grasping points and placing points. The grasping and placing points produced by the teacher policy serve the ground truth labels for this training process. The student policy learns map point cloud observations to the appropriate actions (grasping and placing points), effectively mimicking the decisions made by the teacher policy based the point cloud data alone.
The training details are the same as training keypoints prediction, only changing the number of keypoints from 5 to follow the same training hyperparameters as grasping point prediction for placing point position detection, only changing the pointnet++ semantic segmentation model to the classification model.
More Results on ATM Baseline
We first illustrate example outputs of ATM baseline on two different ties in Fig. B.6. We can that without explicit mesh modeling, ATM will quickly deviate from correct trajectories.
18

---

Normal tie
Larger
Figure B.6: Illustration of ATM rollouts in simulation.
To further examine whether the long-horizon property or points trajectories representations lead to the failure of ATM on tie-knotting tasks, we further conduct experiments on some shorter tasks to see ATM can work. Specifically, we divide the whole tie-knotting task into 6 subtasks, training and testing ATM on each subtask separately. The results are shown in Fig. B.7. We can see that ATM's results look better for the first subtasks. But ATM still cannot complete the last 3 subtasks, which involve complex topology and subtle dynamics. This experiment demonstrates that using points trajectory representation cannot handle such complex tasks even with a shorter horizon.
initial
larget
step1
step2
step3
slep1
slep5
slep6
expl
AP
1
1-0
1
450
4
exp2
cx03
6
P
1
|
1
exp4
6
P
6
6 P
6
P
6
6
6
csp5
exp
Figure B.7: Illustration of ATM results on subtasks.
C More Discussions on Real- Experiment
C.1 Real- World Experiment setup
We set up the real-world experiment with dual-arm robot shown in Fig. C.8. The MOVO robot (53) has two 7 DoF arms and Kinect RGB-D camera overhead. We perform position controls and use RangedIK [54] for solving inverse kinematics. The success state is defined in Fig. C.9.
C.2 Failure Cases and Analysis
Two major failure cases real-world experiments are shown Fig. 10 One is the robot fails to rotate the whole ring structure of the tie, another is the robot fails to insert the little end of the tie into the whole shown in Fig. C.9.
The first case is caused by the subtle dynamics of the tic. To rotate the ring structure, the tie should be a bit harder so that the ring structure will not crumple during the rotation process, while it should not be so hard so that the rotation action won't 't interfere with other parts of the tie. This places high demand on both the tie and the robot. It's hard to solve from the algorithm side.
The second case is caused by partial observation of this task. We use one camera the top of the robot for perception. It cannot perceive the little end of the tie in this case. Thus, the robot has to act blindly, lowering the success rate.
19

---

initial
camera
working space
step1
step2
step3
step
5
step6
step7
step8
Figure C.8: Illustration of real-world experiment settings.
Figure C.9: Mustration of the success state of knotting tie.
20

---

(a) Fail to rotate the ring structure.
(b) Fail to insert little end of the tie.
Figure C.10: Illustration of two major failure cases
D Additional Experiments on Towel-Folding Task
To illustrate our pipeline applies to other cloth manipulation tasks besides the tie-knotting task, conduct experiments on the towel-folding task in the real setting.
D.1 Real-Word World Experiment setup
We set up the real-world experiment with two robot arms as shown in Fig. D.11. The left arm is Flexiv Rizon 4s arm and the right is Flexiv Rizon arm. One Realsense D435 RGB-D camera is mounted on the top of the aluminum tube. The size of the towel is 45em 75cm. This task requires two robot arms to fold the rectangular towel shown in Fig D.12.
21

---

camera
working space
Figure D.11: Illustration of real-world experiment settings for towel-folding task.
first fold action
second action
final state
Figure 1.12: Illustration of the process of the towel-folding task. The blue dash lines are the folding lines
22


Reviews:
[{"review_id": "0ElTZ3yya5", "reviewer": "Reviewer_NNyg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduced a method for tie knotting from video demonstration containing of three phases. The first phase is to estimate mesh sequences from demonstration videos. The second phase use the estimated mesh sequences as subgoals to learn a policy in simulation to act from the point cloud. Finally, the third phase learn a residual policy to aid the simulation policy when deployed in the real.", "summary_of_recommendation": "The paper introduced an interesting method for learning tie knotting from human demonstration videos. The method has certain level of successes and advancement but more experiments should be performed and the paper's clarity should be improved.", "technical_quality": 3, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- The paper stated the student policy is trained using supervised learning; more details should be provided\n- In Table 1, which result comes from simulation and which comes from real hardware?\n- How do the authors connect PyBullet with DiffClothAI to train the student policy?\n- In Table 2, does the result come from averaging all the meshes or just a single one, and what is the unit?\n- It seems that it is pretty easy to experiment with the method for towel folding in the hardware; the authors are encouraged to add such an experiment instead of just showing the performance of HFM. This task can have more baselines to see the performance of the proposed method to state-of-the-art baselines", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pdfVFe8pq0", "reviewer": "Reviewer_XN7q", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work proposes a system for learning to knot a tie by using human video to generate subgoals for supervising policy training in simulation which is then transferred onto real hardware.", "summary_of_recommendation": "I recommend accepting, I think this work has sufficient novelty and evaluation to meet the criteria for acceptance, though it would be helpful to clarify some of the minor concerns and comments.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. Do you need to re-train a policy for each tie? Is the whole pipeline tie specific?\n2. How much randomization is present in real world trials? I found \"We test our real-world policy 10 times. Each time the initial positions of the tie are slightly perturbed about 5cm. The final success rate is 50%\" but could not find a more detailed description in the paper or supplement. \n3. What is the compute required to train the policies, how long does training take? \n4. How are the grasp/pull actions defined, can the methods section be updated to make this much more clear?\n5. 74-75: To the best of our knowledge, this work is the first effort to develop a robot system, integrating perception, modeling, and robot learning. This statement is clearly not true, perhaps the authors would like to amend it to a statement regarding knot tying? \n6. Is the human arm segmented out in the intermediate steps of the video?\n7. Can this method be applied to any other domains?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uVrXnFkuRT", "reviewer": "Reviewer_eicD", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 5, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes Hierarchical Feature Matching, combining correspondence-based local feature matching from RGB and learning to predict global keypoints from pointclouds. This novel state estimation method allows for real2sim transfer of the cloth’s state from a single human demonstration. The resulting sequence of meshes then act as subgoals for policy learning in simulation. To enable sim2real transfer, the authors utilize student-teacher distillation in simulation and residual policy learning in the real world.", "summary_of_recommendation": "The presented method shows impressive tracking ability in a real2sim setting as well as when executed on a real-world robotic platform. Minor improvements like real-world experiments on a different setup (e.g. towel folding instead of tie knotting) and adding ablations for the residual policy learning would further improve the paper.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. The reward function design in Equation 1 mentions variables C_1, C_2, C_3 but the experiment section never provides the concrete values being used. How much does the choice of those matter and how much tuning did you have to do?\n\n2. The sim2real stage includes training a residual policy. Is this policy trained in the real world? Providing additional details, e.g., how many episodes were needed and how performance evolves over time would help better understand the sim2real gap.\n\n3. The success rates in Table 1 show that knotting a tie from a single human demonstration video is still challenging. How do the failure cases look like especially w.r.t. to the different properties of the tie (unseen, softer, harder)?\n\n4. Would collecting more than one demonstration benefit the state estimation and real2sim?\n\n5. Line 296-297: “[The] pipeline still requires manually setting the initial state of the tie at the beginning of the Real2Sim stage.” How does this process look like and can it be automated?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduced a method for tie knotting from video demonstration containing three phases. The first phase is to estimate mesh sequences from demonstration videos. The second phase uses the estimated mesh sequences as subgoals to learn a policy in simulation to act from the point cloud. Finally, the third phase learns a residual policy to aid the simulation policy when deployed in the real.
The paper deals with the very challenging manipulation tie-knotting problem, is well-written, and provides thorough experimentation and analysis especially its tracking ability.

The reviewers had originally important comments and concerns on the specificity of the approach and its components to the tie-knotting problem, the limitations in terms of its performance, and the limited scope of the experiments on generalization for the task and for the object (tie).

While the authors well-addressed the comments of the reviewers, I agree that more experimental evaluation with the real robot should be included in a robotics paper. Therefore I recommend the paper to be accepted for poster presentation.
Decision: Accept