Precise Robotic Needle-Threading with Tactile Perception and Reinforcement Learning
Authors: Zhenjun Yu
Wenqiang Xu
Siqiong Yao
Jieji Ren
Tutian Tang
Yutong Li
Guoying Gu
Cewu Lu
Venue: CoRL 2023 Poster (2023)
Keywords: tactile perception
needle threading

Abstract: This work presents a novel tactile perception-based method, named T-NT, for performing the needle-threading task, an application of deformable linear object (DLO) manipulation. This task is divided into two main stages: \textit{Tail-end Finding} and \textit{Tail-end Insertion}. In the first stage, the agent traces the contour of the thread twice using vision-based tactile sensors mounted on the gripper fingers. The two-run tracing is to locate the tail-end of the thread.
     In the second stage, it employs a tactile-guided reinforcement learning (RL) model to drive the robot to insert the thread into the target needle eyelet.  The RL model is trained in a Unity-based simulated environment. The simulation environment supports tactile rendering which can produce realistic tactile images and thread modeling. During insertion, the position of the poke point and the center of the eyelet are obtained through a pre-trained segmentation model, Grounded-SAM, which predicts the masks for both the needle eye and thread imprints. These positions are then fed into the reinforcement learning model, aiding in a smoother transition to real-world applications.  Extensive experiments on real robots are conducted to demonstrate the efficacy of our method. More experiments and videos can be found in the supplementary materials and on the website: \url{https://sites.google.com/view/tac-needlethreading}.

Precise Robotic Needle-Threading with Tactile Perception and Reinforcement Learning
Zhenjun Yu", Wenqiang Xu", Siqiong Yao, Jieji Ren, Tutian Tang, Yutong Li, Guoying Gu, Cewu Lu Shanghai Jiao Tong University
Abstract: This work presents novel tactile perception based method, named T-NT, for performing the needle-threading task, an application of deformable lin- ear object (DLO) manipulation. This task is divided into two main stages: Tail- end Finding and Tail-end Insertion. In the first stage, the agent traces the con- tour of the thread twice using vision-based tactile sensors mounted on the grip- per fingers. The two-run tracing is locate the tail-end of the thread. In the second stage, it employs tactile-guided reinforcement learning (RL) model drive the robot to insert the thread into the target needle eyelet. 'he RI model is trained in Unity-based simulated environment. The simulation environment supports tactile rendering which can produce realistic tactile images and thread modeling. During insertion, the position of the poke point and the center of the eyelet are obtained through a pre-trained segmentation model, Grounded- SAM, which predicts the masks for both the needle eye and thread imprints. These positions are then fed into the reinforcement learning model, aiding smoother transition real-world applications. Extensive experiments real robots are conducted to demonstrate the efficacy of our method. More experi- ments and videos can be found in the supplementary materials and on the website: breading.
attps://sitess google.
Keywords: tactile perception, needle threading
Introduction
Deformable linear object (DLO) insertion is common task in everyday life. From needle threading particularly interested in the needle-
to suturing in medical surgery scenarios. In this work, threading task as it requires precisely manipulating the highly deformable thread and locating the target eyelet with a small clearance, which is challenging, even for humans.
The task of needle-threading can be subdivided into two stages: Tail-end Finding and Tail-end Insertion. During the Tail-end Finding stage, the primary objective of the agent is locate and secure the terminal end of the thread. Compared to the thread, it is ostensibly less complex to identify the tail-end of other DL.Os such as network and USB cables. The tail-ends of these DL.Os are easily distinguished from the rest of their line part and are often sufficiently large to be detected by a standard RGB-D sensor from its working distance. Conversely, the tail-end of a thread bears a uniform appearance with the remainder of the thread, rendering virtually invisible to a standard camera lens. In the Tail-end Insertion stage, the agent should accurately guide the tail-end of the thread into specified eyelet with minimal clearance. addition, the agent should be able to judge the success or failure of the task exccution.
Based on the observations, we propose Tactile perception-based approach to address the Needle- Threading task and name it T-NT. In comparison with previous solutions which adopt laser scanner [1], high-resolution camera [2], and dual cameras [3] to locate and discern the states of both the
indicates equal contributions.
$ Cewu Lu is the corresponding author.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

T-NT
Spool with Brake
Needle Pose Estimation
Thread
Robot
Eyelet Robot
Tail-end Finding
Tail-end Insertion
Our task setting contains one thread robot with RealSense camera for marker detection and one eyelet robot. There are two stages for our task, Tail-end Finding and Tail-end Insertion. The brake is connected to the scroll to control whether
Figure I:
rotate or not.
thread and eyelet, approach utilizes the tactile sensor which is locally mounted the gripper fingers. We make of MC-Tac [4], GelSlim[5]-like tactile sensor, to provide tactile percep- tion. The tactile imprints produced by the thread and eyelet are distinctly observable when they are pressed on the gel of the tactile sensor.
To find the tail end, the agent is instructed trace the contour of the thread twice. In the first run, the goal is measure the distance from the starting point the extremity of the thread. The endpoint of the thread imprint within the tactile image serves determinant of the gripper reaching the tip. It is crucial to note that, due to the pliable nature of the thread, it can get straightened as the gripper glides over it. Thus, once the robot gripper grasps the starting point, it only needs to follow linear path. The total thread length can be approximated by cumulatively adding the distance traversed by the gripper and the remaining length of the thread imprint visible on the tactile image. For the second run, the agent initiates from the identical starting point, proceeding to the terminal end. This time the gripper stops at a certain distance from the tip. This approach ensures that the tail-end is grasped with minimal residual segment ready for insertion into the eyelet.
The Tail-end Insertion process is facilitated by employing a tactile-guided reinforcement learning (RL) model. The task conceptualized as a goal-conditioned RL task, wherein the objective is for the thread's 's tip to poke the gel at the back of the eyelet and the poke point is within the eyelet's range. To accomplish this, an RL model is trained from simulated environment built on the Unity platform and deployed to the real world. Tactile rendering implemented in the simulator. It can produce tactile images that resemble the images from the real-world sensor. The thread is repre- sented as chaining particles constrained by distance and bend effect, modeled by XPBD (Extended Position-based [y] To obtain the position of the poke point and the eyelet, we adopt pretrained language- grounded segmentation model, Grounded-SAM [7, 8] to predict the masks for both the needle eye and thread imprints. And then we calculate the center mass for the poke point and sample several pixels represent the eyelet area based on the masks. We feed the position information of the poke point and the eyelet inputs train the reinforcement learning model, ensuring smoother transition real-world applications.
language-grounded
We conduct experiments with needles of different sizes and threads of different sizes. We have achieved an average 63.33% success rate in the real world.
We summarize our contribution as follows:
We propose tactile perception- approach for the needle threading task, T-NT, with two stages, Tail-end Finding and Tail-end Insertion.
We conducted real experiments different kinds of threads and needles, with various needle positions and angles, and achieved great success rate in every category.

---

2
Related Works
Our work is directly related to deformable linear object (DLO) manipulation, especially the task of robotic insertion. Besides, since our work also relies on the tactile sensor, we also discuss related works on tactile perception-base -based object manipulation.
Robotic Insertion of Deformable Linear Object. Robotic manipulation of deformable linear ob- jects (DLOs) such as cables, wires, threads, or tubes has been -tandding challenge in the field of robotics. The inherent flexibility of these objects, their interaction with the environment, and the high-dimensional nature of their state space pose unique challenges.
Several manipulation tasks involving DL.Ss have been explored by previous researchers, such as shape control [9, 10, 11], cable routing [12] and knotting/unknotting [13, 14, 15]. A subset of the search has focused specifically the task of robotic insertion of DLOs, including needle-threading [1, 2, 3, 16] and DLO-in-hole assembly tasks [17, 18]. Kim et al. [3] adopted a dual camera setting to capture both peripheral and foveated vision, and leveraged imitation learning to train a policy to execute the needle threading task. Since imitation learning requires expert demonstrations, visual servoing is more direct. However, due to the challenge of locating the thread and needle in the common visual setting. many works adopted enhanced visual settings. Silverio al. [1] adopted laser scanner track the tail-end of the thread. However, the laser scanner is not common choice for general-purpose manipulation settings. Huang et al. [2] employed high-speed camera to perceive the thread. They assumed that rapidly rotating thread can be approximated as rigid object, thereby simplifying the control models. However, this study was based on a specifically de- signed two-degree-or offerreo-f-ffreedm freedom mechanism and did not automate the thread mounting process. Lv et al. [16] propose a model-based RI. method for the theede-t task. They adopt differentiable simulation and rendering techniques to synchronize the thread and needle configuration between the simulation and real-world observations. However, they cannot determine whether the execution is successful unless manually examine the execution.
Tactile perception object manipulation. Tactile perception- based object manipulation is of- ten studied accompanied by the development of tactile sensors. For example, Zanella [18] proposed address the DLO-in-hole assembly tasks with customized low-resolution tactile sensors on the gripper finger. She et al. [19] explored cable manipulation with proposed tactile-reactive gripper. Later, more and more tactile sensors get commercially available or open-source [20, 21, 5], re- searchers can focus on developing algorithms for different tasks, such as object shape reconstruction [22, 23, 24, 25], contour following [12, 26] and dexterous manipulation [27].
3 Background
In this section, we will first describe the task settings and assumptions. Then we describe the prob- lems in the tactile-based theede-threading task.
3.1 Task Setting
The overall task setting is illustrated in Fig. 1. In the experiments, we need two robot arms, one for thread manipulation (denoted as thread robot), and one for eyelet localization (denoted as eyelet robot). To grasp and sense the thread, we mount two tactile sensors on the inner side of the gripper fingers on the thread robot. To touch and sense the eyelet, we mount one tactile sensor on the outer side of one gripper finger on the eyelet robot. In this work, we adopt Franka Emika panda as the thread robot and Kinova Gen2 as the eyelet robot. We adopt MC-Tac sensor [4] as our tactile sensors to provide tactile perception. We mount an Intel RealSense D435 on the thread robot, and conduct eye-in-hand calibration with Easy-HandEye package [28].
The thread is scrolled on spool, and a part of the thread is hung down naturally. The spool is controlled by a brake device. When we trace the thread in Tail-end Finding stage, the spool will not be able to scroll so that the length of the thread dropped can remain the same. While Tail-end Insertion stage, the spool can rotate so that the thread can reach the eyelet.
The needle is inserted into base support because we do not consider picking the needle up. The location of the base support and the angle of the needle can be varied.
3

---

The tactile sensor on the eyelet robot touches the eyelet from the beginning of the experiments, but the exact location of the eyelet is not known.
3.2 Problem Statement
In the Tail-end Finding stage, given a thread, we first find its rest length 4hread by gliding it from the beginning point to the thread tip. The beginning point is defined at the center of 2cm below the spool. Then, we trace the thread from the same beginning point and stop the movement early, so that the tail-end has length of hui- To note, when the gripper holds the tail-end and moves around, gravity might cause the tail-end to bend downwards. Thus, need estimate the tip position in consideration of the gravity force. We solve this problem with neural network f(-):
Pup
fili 0, Pus) Piass
where is the orientation of the tail-end measured from the tactile image, Ptac SO(3) is the position w.r.t thread robot's base, and Pup E 50(3) the position w.r.t thread robot's 's base, as shown in Fig. 2.
In the Tail-end Insertion stage, we drive the tail towards the eyelet area. We first give a rough estimate of the eyelet area with Peyeler Pmarker + To- Pmarker is the relative pose from the marker on eyelet robot to thread robot's base, which is easy to obtain after eye-in-hand calibration. To is the transformation between the marker and the tactile sensor on the eyelet robot, which only needs to measure once for all the experiments. Given the estimated Ptip and Peyelet. we plan trajectory location perpendicular to the gel surface and 1em away. In this way, the thread robot later need only move along the surface to find the eye and perform insertion, the surface is denoted "ue-plane", as shown in Fig. 1.
Then, we read the tactile image stream / IR 121100x1200x3 from the tactile sensor on the eyelet robot and model the insertion task as goal-conditioned reinforcement learning problem. Given an eyelet imprint Leye, segment the hole and the thread tip-gel contact point from Leye, with pretrained segmentation model, Grounded-SAL SAM [7, 8]. We then caleulate the center of mass (COM) of contact point mask C, and we randomly sample N pixel positions from the mask of needle eyelet, and obtain Ceye.t = {meye.t.... .., t. We formulate the problem of Tail-end Insertion as learning a policy æ that sequences move actions ae E A with a robot from tactile observations (ca,e r) E O.
(2) with Tu and T., defined in SE(2), is the displacement of the end-effector of the thread robot in the v directions respectively. We calculate the total number pixels of the poking thread, as well as the number that is within the eyelet mask. If the total number is lower than 500, or the number of steps t that the agent has tried is larger than (based on the average reward of training), we consider it failure. If the number of pixels that in the needle surpasses half of the total pixel number, regard the state successful. One of the steps in the real world is shown in Fig. 3.
x (C,,, Ceyo,t) - a. (T,, To) E
Tail-end Finding
Tail-end Insertion
① Length Estimation
Tip Estimation
Goal-Conditioned Insertion
(cr,2 Ceye.) + reward,
Fixed
CGC
CG
RL Agent xt())
Tactile Sensor
Ptac
Orientation
Movable
Step action (Tu.TT)
tip
prip
needle
Figure 2: The pipeline of T-NT consists of two aforementioned parts: Insertion. We first do the tip pose estimation with a neural network, and adapt our RL agent trained simulation environment to conduct needle threading with the observation from the tactile sensor.
Tail-end Finding and Tail-end

---

Method
4.1
Tactile Image Processing
For all the tactile images read during the whole process, we adopt the Grounded-SAM [7, 8] model with prompt of "line" to locate the thread imprint, and "bump" for the poke imprint. As for the eyelet, we the prompt of "big hole". In our experiments, enough. Thanks to the generalizability of the segmentation model, we don't t have to train a specific segmentation model to get the imprint mask for both thread and eyelet.
find these prompts are robust
4.2
Tip Pose Estimation
As mentioned in Sec. 3.2, once we trace the thread twice, we can obtain Lmn, and then we can estimate the orientation / of the tail-end from the tactile image. huil = dl d2 where dl and d2 the distances the gripper traverses in the times respectively. The orientation of the line is estimated by the incline of the bottom contour, conducted by OpenCV [29], of the line mask in the tactile image, as shown in Fig. 2.
With lail and 0, we need to estimate the tip position. According to Equ. 1, we train a 4-layer MLP f()). To collect the training data, the thread robot grasps the tail-end of the thread and moves to random direction 500 times. Each time, we measure the hail and the transformation between Piip and Prac manually, while network structure and its training can be found in the supplementary materials.
automatically obtain the data of Prac 0. More details about the neural
4.3 Goal-conditioned Insertion
Training Setup in Simulation We adopt Unity-based simulation environment to train the RL model. We set up the same hardware setting in the simulator and the rendering of tactile images has been implemented and adjusted resemble the real images (See supplementary materials on our website). We model the needle as rigid object and attach it to a cube the base support. The clearances of the eyelets replicated from the real items. As for the thread, we model it as constrained particle system with XPBD [6]. It allows for adjustment of the stiffness, thickness, and length of the thread.
We initialize the training by randomly setting the relative positions between the end-effector of the thread robot and the eyelet. Considering the calibration error in the real world, the randomized distance is set most 3cm. The thread robot moves along the uv-plane of the gel surface of the tactile sensor eyelet robot, and tries to poke gel and insert the tip into the eyelet.
As mentioned earlier, to mitigate the sim-to-real gap, instead of utilizing the tactile image as the input to the policy model, we introduce an intermediate representation, i.e., the locations of the eyelet and thread tip calculated by the mask generated on the tactile image from eyelet Loye with the segmentation model. We model the insertion problem the goal-conditioned RI. based this intermediate representation of the observation.
The reward function we use is: reward, = -EY d(cr,1)" /1 When the task is successful, reward, = r and when the task is failed, reward, = =rr With = 100, is the pixel length of the tactile image diagonal, d(-) is the Fuclidean distance. The clear definition of success and failure has been mentioned in Sec. 3.2.
Transfer to Real World After train our RL model in the simulation environment, directly apply it in the real world (See Fig. 3). To note, in the real world, to accomplish the full pipeline of insertion, we need to conduct the Tail-end Finding process and move the tip around the eyclet in near-perpendiculle direction to the gel surface.
Experimental Setup
Tactile Sensors The tactile sensor we adopt is MC-Tac sensor [4], the gel has Young's module of 0.123 MPa. It is important when selecting the thread material since the thread needs to be stiffer to leave the imprint the gel.
5

---

Poking spot
Repeat until success failure
Needle center
a
obs
xle Capoz)
action
obs
eye.
eyye/+1
Step
(Tu TT))
Step t + 1
Figure 3: An example of the real-worid steps for goal-conditioned insertion. The observations
obtained by the tactile sensor, and the model predicts uu- uu-plane transformation for the thread robot. We repeat the step until the agent judges the state success failure.
Saread-Eyellee We select three kinds of threads made of nylon, metal, and glass fiber with Young's module of 8.3 GPa, 50 GPa, and 90 GPa respectively.
Aside from the stiffness, the thickness of the threads and the size of the eyelet clearance are also important show the adaptability of method. We use threads with thicknesses of 0.2mm, 0.5mm,, Imm, and 2mm. The sizes of the eyelets clearance are 0.6 7.5mm², 1.6 x 15mm,, and 2.4 x 9mm? The thread and eyelets we use in the tasks are shown in Fig. 4.
Fumm Sum
2. D.OOnaa
U.Nmmm
1. Lamm
H
5umm
Needle #1
Needle #2
Needle #3
Thread #1
Thread #2
Thread#3
Thread #4
Figure 4: We have selected three kinds of eyelets and four kinds of threads with different sizes, to prove the ability of our method.
As shown in Fig. 5, when we fix the needle onto a base, we vary the angles from 45 to 90", and the base support positions are randomly placed as long as is within the reach of the robot.
50
60
Figure 5: We conduct our experiments on different needle positions and angles, to show the ability of our model to adopt different situations of the needle states.
Training Details In our simulation training, we only train our model with needle #2 and line #2.
The line in the simulation is modeled with Obi [6], an XPBD-based physics engine. For keeping the line straight, we use the damping value 0.9, and the particle resolution for modeling the line 0.7. We train the RL model with Stable baselines3 [30] and RFUniverse [31] to interact with Unity. We train three different models: PPO, SAC, and DDPG with the same learning rate of 3e-4 and total timesteps 1e5. The sample number N of needle pixels for reward function calculation is 500.

---

Results
6.1 Metrics
Success Rate If the imprint of the tail-end is inside the eyelet, we consider a success. We measure the success rate in real-world experiments.
Mean Distance Error We measure the distance between the predicted thread tip and the ground truth, and calculate the average distance for every kind of thread.
6.2 Evaluation on Tip Pose Estimation
As mentioned earlier, we have prepared 500 data samples for each thread. We split 400 of them to train and 100 to test. We report the quantitative results with mean distance error in Table 1. And the qualitative results are shown in Fig. 6. Since use the estimated tip position for planning desired location to insert, thus as long as this location, which is influenced by sum error of tip position estimation, Peyelet estimation, trajectory following for the robot to execute the plan, still in the range of the gel surface, the insertion process can continue. And the gel surface has an area of 15mm x 15mm, thus the tip pose estimation error is significantly small. Additionally, it takes about 1-2 minutes depending on the length of the thread (5 cm-20 cm).
Predicted Tip Position
Needle #2 Thread #1
Needle #1 Thread #2
Ground Truth
Figure 6: Qualitative results for tail-end estimation and insertion. The tip poses estimation and goal- conditioned insertion allow successfully thread needles with different sizes and thicknesses.
Thread
Mean Distance (mm)
Angle
Success Rate
#1 (Fx 0.2mma)
2.2
45
57.89%
#2 (Rx 0.5mm)
1.7
60
63.33%
#3 [ 1mm)
3.5
90
62.89%
#4 ( 2mm)
4.1
Table 2: Average success rate for different
Table Quantitative results for tail-end find-
needle angles.
ing.
6.3 Results of Tail-end Insertion
We conduct experiments all three kinds of needle eyelets and four threads, with 20 random base support locations. The needle is fixed the base support with angle of 60%. According small test (Table 2) the angles, it does not have significant influence on the results.
We have conducted the experiments using a visual servoing method (denoted "VS" Tab. 3) as a baseline to compare. The method is simply calculating the pixel distance between the COMs of the needle mask and the thread imprint, and projecting the real-world distance according the calibrated camera model, and then planning and moving the gripper accordingly with an off-the- shelf planner, Movelt! [32].
The quantitative results of the success rate are shown Tab. 3. Due to needle size, line #3 can't be threaded into needle #1, and line #4 can't be threaded into needle #1 and #2. The results clearly show that, with accurate tail-end finding, the success rates of the Tail-end Insertion significantly high, reaching 63.33%, while visual servoing method only has 47.22%. which proves the great performance of our method. Although only train mediate representation between real- real-world tactile image and mask from our simulation environment
kind of needle and thread, the inter-

---

successfully solves the sim2real gap and thus provides us great generalizability and transferability for different needles and threads. We can also conclude that the difference the size of needles and threads affects the success rate greatly, for the bigger needles give the thread more opportu- nity for trial and error. The difference the reinforcement learning policy hardly has an impact on the results. The insertion process takes an average of 3.41 steps complete, with less than minute. We will elaborate and discuss more factors that influence experiments and results in the supplementary materials.
Thread
Policy
#1 (== 0.2mm)
#2 (== 0.5mm)
#3 ( 1mm)
#4 (+s 2mm)
needle
#1 (0.6 7.5)
35%
35%
*
*
VS
#2 (1.6 x 15)
55%
40%
45%
#3 (2.4 x 9)
65%
50%
50%
50%
#1 (0.6 7.5)
50%
45%
*
*
PPO
#2 (1.6 x 15)
65%
65%
50%
" 60%
#3 (2.4 x 9)
70%
80%
85%
#1 (0.6 X 7.5)
45%
50%
SAC
#2 (1.6 X 15)
65%
65%
45%
* 65%
#3 (2.4 X 9)
80%
85%
75%
1# (0.6 X 7.5)
60%
50%
DDPG
2# (1.6 15)
55%
60%
55%
*
3# (2.4 x 9)
75%
75%
75%
60%
Table 3: Quantitative results for the success rate of four kinds of lines and three different needles, with 3 different RL policy and 20 random base support pose.
6.4 Limitation
We briefly analyze the limitation of our method, more analysis and visual failure cases can be found on our website.
Thread stiffness requirement: The key limitation of our proposed methodology is its inability to work with soft threads such as the sewing threads. due to only the thread which has a stiffness greater than the gel being used can cause the imprint.
Thread fixation to obtain the length: Our method takes two-run process locate the tail-end of the thread. During the Tail-end Finding stage, the thread has to be fixed in end, ensuring that its remaining length does not change. This could complicate the hardware setting.
Tactile Sensor Positioning: The tactile sensor on eyelet robot needs to come in contact with the back side of the eyelet. Thus it requires the eyelet to thin and cannot applied applications such USB insertion where the back side of the hole is not easily accessible. While be used for tasks like screw insertion into nut.
7 Conclusion
In this work, we present T-NT, novel approach to deformable linear object (DLO) insertion tasks, particularly needle threading. Our strategy incorporates tactile perception-bii approach for both Tail-end Finding and Tail-end Insertion stages, utilizing the vision-bar based tactile sensor. The reinforcement learning model for the insertion stage, trained in simulated environment, was shown to be effective and adaptable to real-world scenarios. This tactile-based approach indicates that the use of tactile perception, combined with reinforcement learning, can facilitate research involving highly deformable objects and requires precise manipulation. Further research may benefit from building on this work to extend its application other DLOs and complex manipulative tasks.

---

Acknowledgments
This work was supported by the National Key R&D Program of China (No. 2212D00107044 Shanghai Municipal Science and Technology Major Project (20215HH//72X0002)),, Shanghai Qi Zhi Institute, and Shanghai Science and Technology Commission (21511101200).
References
[1] Silvério, G. Clivaz, and S. Calinon. A laser-based dual-arm system for precise control of collaborative robots. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 9183-9189. IEER, 2021.
[2] S. Huang, Y. Yamakawa, T. Senoo, and M. Ishikawa. Robotic needle threading manipulation based on high-speed motion strategy using high-speed visual feedback. In 2015 IEEE/RS/ International Conference on Intelligent Robols and Systems (IROS), pages 4041-4046. IEEE, 2015.
[3] H. Kim, Y. Ohmura, and Y. Kuniyoshi. Gaze-based dual resolution deep imitation learning for high-precision dexterous robot manipulation. IEEE Robotics and Automation Letters, 6(2): 1630-1637, 2021.
[4] J. Ren, J. Zou, and G. Gu. Mc-Tac: Modular camera-based tactile sensor for robot gripper. In The 16th International Conference on Intelligent Robotics and Applications (ICIRA), 2023.
[5] Taylor, S. Dong, and A. Rodriguez. Gelslim 3.0: High-resolution measurement of shape, force and slip a compact tactile-sensing finger. In 2022 International Conference on Robotics and Automation (ICRA), pages 10781-10787 IEEE, 2022.
[6] V. M. Studio. URL hatt:://oal.iiittttammmtoouuuuu com.
[7] S. Liu, Z. Zeng. T. Ren, F. Li, H. Zhang, J. Yang. C. Li, J. Yang, H. Su, Zhu, and L. Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. 2023.
[8] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick. Segment anything. arXiv:2304.02643 2023.
[9] M. Yu, H. Zhong, and X. Li. Shape control of deformable linear objects with offline and online learning of local linear deformation models. In 2022 International Conference on Robotics and Automation (ICRA), pages 1337-1343. IEEE, 2022.
[10] R. Laezza, R. Gieselmann, F. T. Pokorny, and Y. Karayiannidis. Reform: A robot learning sandbox for deformable linear object manipulation. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 4717-4723. IEEE, 2021.
[11] Y. Yang. J. A. Stork, and Stoyanov. Online model learning for shape control deformable linear objects. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4056 4062. IEEE, 2022.
[12] A. Wilson, H. Jiang W. Lian, and W. Yuan. Cable routing and assembly using tactile-driven motion primitives. arXiv preprint arXiv:2303. 17765 2023.
[13] K. Suzuki, M. Kanamura, Y. Suga, H. Mori, and T. Ogata. In-air knotting of rope using dual- arm robot based on deep learning. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6724-6731 IEEE, 2021.
[14] H. Wakamatsu, E. Arai, and S. Hirai. Knotting/unknotinng manipulation of deformable linear objects. The International Journal of Robotics Research, 25(4):371-395 2006.

---

[15] S. Lin, X. Jiang, and Y. Liu. Cable manipulation with partially occluded vision feedback. In 2022 IEEE International Conference on Robotics and Biomimetics (ROBIO), pages 1245- 1250. IEEE, 2022.
[16] J. Lv, Y. Feng, C. Zhang. S. Zhao, L. Shao, and C. Lu. Sam-rl: Sensing- aware model-based re- inforcement learning via differentiable physics-based simulation and rendering. arXiv preprint 2XXi:2210.15188 2022.
[17] K. Galassi, A. Caporali, and G. Palli. Cable detection and manipulation for dlo-in-hole assem- bly tasks. In 2022 IEEE 5th International Conference on Industrial Cyber-Physica, Systems (ICPS), pages 01-06. IEEE, 2022.
[18] R. Zanella, D. De Gregorio, S. Pirozzi, and G. Palli. Dio-in-hole for assembly tasks with tactile feedback and Istm networks. In 2019 6th International Conference on Control, Decision and Information Technologies (CoDIT), pages 285-290. IEEE, 2019.
[19] Y. She, S. Wang, S. Dong, N. Sunil, A. Rodriguez, and E. Adelson. Cable manipulation with a tactile-reactive gripper. The International Journal of Robotics Research, 40(12-14):1385- 1401, 2021.
[20] M. Lambeta, P.-W. Chou, S. Tian, B. Yang, B. Maloon, V. R. Most, D. Stroud, R. Santos, A. Byagowi, G. Kammerer, et al. Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation. JEEE Robotics and Automation Letters, 5(3):3838-3845, 2020.
[21] W. Yuan, S. Dong, and E. H. Adelson. Gelsight: High-resolution robot tactile sensors for estimating geometry and force. Sensors, 17(12):2762, 2017.
[22] E. Smith, R. Calandra, A. Romero, G. Gkioxari, D. Meger, J. Malik, and M. Drozdzal. 3d shape reconstruction from vision and touch. Advances in Neural Information Processing Systems, 33: 14193-14206, 2020.
[23] E. Smith, R. Calandra, A. Romero, G. Gkioxari, D. Meger, J. Malik, and M. Drozdzal. 3d shape reconstruction from vision and touch. Advances in Neural Information Processing Systems, 33: 14193-14206, 2020.
[24] S. Wang. J. Wu, X. Sun, W. Yuan, W. T. Freeman, J. B. Tenenbaum, and E. H. Adelson. 3d shape perception from monocular vision, touch, and shape priors. In 2018 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems (IROS), pages 1606-1613. IEEF, 2018.
[25] W. Xu, Z. Yu, H. Xue, Ye, S. Yao, and Lu. Visual-tactile sensing for in-hand object reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8812, 2023.
[26] L. Pecyna, S. Dong, and S. Luo. Visual-tactile multimodality for following deformable linear objects using reinforcement learning. 2022 IEEE/RSJ International Conference on Intelli- gent Robots and Systems (IROS), pages 3987-3994. IEEE, 2022.
[27] D. F. Gomes and S. Luo. Geltip tactile sensor for dexterous manipulation in clutter. In Tactile Sensing, Skill Learning, and Robotic Dexterous Manipulation, pages 3-21. Elsevier, 2022.
[28] R. Tsai, K. Lenz, et new technique for fully autonomous and efficient 3 robotics hand/eye calibration. IEEE Transactions on robotics and automation, 5(3):345-358, 1989.
[29] G. Bradski. The opencv library. Dr. Dobb's Journal: Software Tools for the Professional Programmer, 12:0:))1122233 2000.
[30] A. Raffin, A. Hill, M. Ernestus, A. Gleave, A. Kanervisto, and N. Dormann. Stable baselines3. https: / h/ghttttbbb////// com/DLR-RM RM/ stable baselines"s 2019.
10

---

[3] H. Fu, W. Xu, H. Xue, H. Yang, R. Ye, Y. Huang, Z. Xue, Y. Wang, and C. Lu. Rfuniverse: A physics-based action-centric interactive environment for everyday household tasks. arXiv preprint arXiv:2202.00119, 2022.
[32] D. Youakim, P. Ridao, N. Palomeras, F. Spadafora, D. Ribas, and M. Muzzupappa. Moveit!: Autonomous underwater free-floating manipulation. IEEE Robotics & Automation Magazine, 24(3):41-51, 2017.


Reviews:
[{"review_id": "BoCwmCp4GWc", "reviewer": "Reviewer_t9QJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose a robotic system for threading a needling, using image segmentation and deep reinforcement learning.", "summary_of_recommendation": "Impressive systems paper on a complex task. I recommend acceptance. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "None.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I thought this was a very difficult robotic task, so I commend the authors for making progress in this setting. \n\nI do not have a background in this problem setting, so my comments are mainly high level. \n\nI was surprised to see there wasn't an appendix? I would have liked to have seen the RL training curves, to see how hard the task was to learn.\n\nI thought the paper was perhaps too acronym heavy. The T-NT acronym is a bit labored and not really needed as there aren't any baselines to differentiate against. The paper introduces 'Grounded-SAM' a few times, and I wonder if it could just be referred to as a segmentation model. It also appears to be language conditioned based on the passage in 4.1 which isn't mentioned in the text previously. The XPBD acronym is not explained (extended position-based dynamics?)\n\nI would have been interested to see a non-RL baseline, perhaps something hand-engineered like a visual servoing feedback law. It would have been good to see what benefit RL added to the performance over human engineering.  \n\nOpenCV should be cited in Section 4.2.\n\nIt would be good to have a success / failure definition closer to Equation 3. \n\nThe paper formatting is a bit mess, with some clear vspace hacking in various places. There is plenty of space optimization that could be done, such as heading 4.1 being unnecessary, and Equation 3 generating a lot of white space. It's also visually cleaner to have figures and tables either on the top or bottom of a page, rather than floating. It also looks like the ```\\text``` macro is not being used in any of the math, so the math text appears quite wonky (e.g. $reward_t = \\dots $ rather than $\\text{reward}_t = \\dots $).", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "9BgkiDjHvv", "reviewer": "Reviewer_zrs7", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a learning method for tactile-based needle threading tasks with a dual arm setup. The proposed method T-NT will first locate the thread and then employ a tactile-guided reinforcement learning (RL) model to drive the robot to insert the thread. And the paper shows experiments on real and sim with different sizes of needles and threads.\n", "summary_of_recommendation": "The paper shows a solution for tactile-based needle threading. However, more experiments, details, and clarification needed to be added to show the validity of the method. (as described in previous sections)\n\nTherefore, I am leaning toward rejecting this work but am willing to change my mind if the authors provide a convincing rebuttal.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "As described above:\n1) please add more analysis in related works with imitation learning for needle threading (e.g https://www.youtube.com/watch?v=ytpChcFqD5g&ab_channel=UTISILab)\n    more analysis in related work with visual servoing methods\n2) Why using a second arm with tactile sensing is necessary and important \n3) In the second stage, more analysis on why RL method can not reach 100% even when the thread is small and the needle is large\n4) Adding imitation learning baseline and visual servoing baseline\n5) Show the number of inserting trials for each agent\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n\nThe paper addresses an interesting task -- needle threading. The presentation of the method is relatively clear and straightforward. \nThere are sufficient quantitative results and real-world experiments.\n\n\nWeaknesses:\n\nIt seems to me the task setup on attaching a tactile sensor to the eyelet robot is a little bit redundant. In reality, whether the thread could pass the needle could be detected from a front-view camera or the tactile sensor in the thread robot part. Also, given that there exist camera and tactile information and estimated states, it is natural to have a visual servo baseline to compare with. Besides, there should be more analysis of the failure cases of not reaching 100% even when the thread is small (0.2) and the needle is large (2.4x9). \n\nAdditional point:\nFigure2 expresses the algorithm step by step but may need to be approved to make readers understand it easier.\n", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "3xKnQkJx_4X", "reviewer": "Reviewer_uQnm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors in this paper developed a pipeline of using a Gelsight slim on both a Franka and a Kinova arm to first trace the end of one thread by gripping and sliding it while it was held in place by an overhead spool with a brake using the Franka and then using the Kinova to press against the eye of a needle to locate it and then classify whether the end of the thread was correctly inserted into the eye. They used Grounded-SAM to determine the location and orientation of the thread and the tip from the tactile images and also to locate the eyelet. Afterwards, they used reinforcement learning in simulation to learn a policy to insert the tip of the thread into the eyelet once they have both been identified. Their end result is around 60% success rate in the real world on various different thread sizes and eyelet sizes.", "summary_of_recommendation": "I gave these recommendations because it is more of a systems paper. While I do not think there will be as much impact because no extremely \"novel\" techniques were introduced, it was a nice demonstration of how to use gelsight sensors to do a task given the state of the art techniques.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "I don't have any real questions. I just wonder how long one insertion from start to finish would take.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall I think the quality of the paper is good. The clarity is also fine. They have cited previous works that used a laser scanner mounted on the end of another Franka robot to inspect where the eyelet is in order to insert it so it is not an original problem, but they used gelsight slims to solve the problem instead of a more expensive laser scanner. They used Grounded-SAM out of the box to locate the features that they cared about in the tactile images which was nice. They used traditional reinforcement learning techniques of PPO, SAC, and DDPG to train with once they had the reward model. I don't see any real weaknesses in the paper. This seems like a paper that uses tactile sensors and many out of the box components connected together in order to accomplish a pretty difficult task.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "0wqoLbNNXq2", "reviewer": "Reviewer_KsbS", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper provides a method for tail-end finding and insertion in the needle-threading task. Reinforcement learning in simulated environment gives a smooth transition to real-world applications. The authors also conduct extensive experiments on real robots showing that the proposed method is effective.", "summary_of_recommendation": "Great strengths outweighs minor weaknesses. Strong accept.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. Most kinds of threads are soft, and there seems to be no guarantee that it will be detected on the sensors every time. At this point CV seems to provide better recognition.\n2. There seems to be a gap between the images in the simulation environment and those in the real experiments, and how the impact of this gap is overcome?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- The paper expresses the method clearly and presents the results well.\n- Reinforcement learning in simulation environment gives smooth transition to real-world experiements.\n- The real-world experiments show that the method is effective and reliable.\n- Several different kinds of eyelets and threads are selected to test the adaptability of the method.\n\nWeaknesses:\n- Most kinds of threads are soft, and there seems to be no guarantee that it will be detected on the sensors every time. At this point CV seems to provide better recognition.\n- There seems to be a gap between the images in the simulation environment and those in the real experiments, and how the impact of this gap is overcome?", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a system integration for needle threading demonstrated in simulation and real robot experimentation. The method uses Grounded-SAM to determine the location and orientation of the thread and the tip from the tactile images and also to locate the eyelet. Then, reinforcement learning in simulation learns a policy to insert the tip of the thread into the eyelet once they have both been identified. with a 60% success rate in the real world on different thread sizes and eyelet sizes. 

One major concern from the reviews is that the technical contribution is missing since it is mostly a system paper that puts several existing components together. After discussion with reviewers and PCs, we believe that it is a well-written system paper for a very complex task. Its presentation at CoRL is beneficial to the audience and could inspire follow-up works.

Please revise your paper according to the feedback by reviewers, in particular, justify the bi-manual system and (1). why simulating the environment and doing domain randomization is difficult and hard to segment the threads? (2) Why RL agents can not poke the thread's position correctly given the good thread segmentation?
Decision: Accept (Poster)