Proposal of a Method for Estimating Needle Base Position to Decrease
Needle Deflection During Robotic Needle Insertion Using Machine Learning
Axel Yap Shen Yit1, Yuta Fukushima1
1

Teikyo University, Graduate School of Science and Engineering, Utsunomiya, Tochigi, Japan
24FM10022wb@stu.teikyo-u.ac.jp, fukushima@ics.teikyo-u.ac.jp

1. Introduction
1.1

Background

Needle insertion, a fundamental medical technique forpuncturing the human body, is an essential skill that must
be mastered. Needle insertion is performed on varioussites and tissues, including blood vessels such as veins
and arteries, organs such as the liver, intestines, breasts,brain, and prostate, and cavities such as the epidural andpericardial spaces. Additionally, its applications arediverse, ranging from biopsy and treatment to drainageof accumulated fluids, making it a crucial procedure inmany medical interventions. One of the most common
needle insertion procedures is venipuncture for blood
sampling from the arm's veins, which often relies on theoperator's tactile sensation. Consequently, in cases where
the blood vessel's diameter is narrow, the vessel wall has
low elasticity, or the vessel wall is calcified and hardened,
the needle tip may fail to reach the vessel lumen.
Complications of venipuncture include nerve damage,vasovagal
reactions,
infections,subcutaneous
hematomas, and allergies, with fatalities being rare.
However, in procedures that involve needle insertion into
the body's central regions, such as central venous catheter(CVC) placement or pericardiocentesis, despite using
imaging guidance like X-rays or ultrasound, the mortality
rate can be high. Complications associated with CVC
include infections, thrombosis, and mechanicalcomplications. Atypical mechanical complications
include arterial puncture, pneumothorax, hematoma, andhemothorax, with an incidence rate of up to 19% [1].
Pericardiocentesis
complications
encompasspneumothorax, damage to surrounding vital organs,cardiac wall puncture, and death. Before the advent ofechocardiographicguidance,
pericardiocentesis
primarily relied on a blind approach, with a lifethreatening complication rate exceeding 20% and amortality rate of approximately 6% [2]. Currently, withthe widespread use of echocardiographic guidance, thecomplication rate for pericardiocentesis has decreased to5%, and the mortality rate has dropped to 1%. However,understanding ultrasound images requires specialized
knowledge and experience, and it may be challenging tovisualize in obese patients. Additionally, while 18-gauge
needles were traditionally used for pericardiocentesis,
the current trend is to use thinner 21G needles [3]. The

21G needle has probrems that are more prone to bending
and breaking. The causes of complications during needle
insertion can be attributed to various factors, including
the operator's mental state, mechanical factors such as
needle targeting errors (needle misplacement), infections,thrombosis, and others. Targeting errors, a mechanicalfactor, are a crucial factor that significantly impacts
treatment efficacy and diagnostic accuracy. Targeting
errors have been shown to arise from limitations inimaging, image misalignment, target uncertainty, humanerrors, target displacement due to tissue deformation, andneedle deflection [4-7]. To mitigate targeting errors,
extensive research has been conducted on automatingneedle insertion using robotic systems.
Needle insertion robots often employ MRI, CT, or
ultrasound imaging to guide the needle toward the target[8-11]. However, image-guided needle targeting isdifficult in obese patients and in cases with hardened
tissue due to calcification, as halation may obscure thetarget or prevent tracking of the needle tip. During blindneedle insertion, physicians typically hold the base of theneedle and rely on the force feedback perceived by theirfingers to estimate the needle's deflection and performtargeting. Therefore, we propose a method that generates
needle trajectories and controls the needle based on force
feedback, similar to how physicians manipulate needles
during blind insertions. In this proposed method, MRI,
CT, and ultrasound imaging are utilized as
supplementary aids to improve insertion accuracy bytracking the target.
Therefore, given the high incidence of severe
complications and the challenging nature of needle
targeting due to respiratory motion in pericardiocentesis,
our last goal was set to develop an robotic needle
insertion system that used force feedback for needle
control for pericardiocentesis. Our initial objective is toestimate the needle tip position using force feedback andpropose a robust needle control method against external
forces during insertion. Leveraging force feedback forneedle tip position estimation and control can also serve
as a training tool for needle insertion procedures.
Additionally, it offers a wide range of applications, suchas quantitatively evaluating the differences between
experienced and inexperienced operators, integrating
augmented reality (AR) to provide guidance during
human-performed insertions, and supporting otherpotential applications. By utilizing force feedback, ourapproach not only addresses the challenges of needle
targeting but also presents opportunities for training,assessment, and assistive technologies in the field ofneedle-based interventions.

1.2

𝐶̃𝑡 = 𝑡𝑎𝑛ℎ(𝑊𝐶 [ℎ𝑡−1 , 𝑥𝑡 ] + 𝑏𝐶 ) ··· (5)𝐶𝑡 = 𝑓𝑡 ∗ 𝐶𝑡−1 + 𝑖𝑡 ∗ 𝐶̃𝑡 ············ (6)Proposal of Needle Control Method to
Decrease Needle Deflectionℎ𝑡 = 𝑜𝑡 ∗ tanh (𝐶𝑡 ) ················· (7)The proposed method for reducing needle deflection isshown in Figure 1. This research is divided into twosections. The first section aims to estimate the vector of
the needle deflection caused by external forces using
artificial intelligence. Here we explain the needle
deflection vector. When observing from the needle baseto the needle tip, if the needle tip was bent away from theneedle's long axis, as shown in Figure 2, the needle tip
can be represented on a two-dimensional plane with theneedle's axis as the origin [12]. In this study, we refer tothe vector from the origin of this plane to the needle tip
as the needle deflection vector. The needle deflection
vector can be calculated using the following equation,

where 𝑓𝑡 is the forget gate, 𝑖𝑡 is the input gate, 𝑜𝑡 is
the output gate, 𝐶̃𝑡 is the candidate for cell state attimestamp 𝑡, 𝐶𝑡 is the cell state at 𝑡, 𝜎 is the sigmoid
function, 𝑊𝑥 is the weight for the respective gate(𝑥)neurons, ℎ𝑡−1 is the output of the previous lstm block,
𝑥𝑡 is the input at current 𝑡, 𝑏𝐶 is the biases for therespective gates(𝑥), ℎ𝑡 is the output of the current
lstm block.
The input, 𝑥𝑡 for the second artificial intelligence (AI2) includes the forces and torques acting around theneedle base, the insertion depths of the needle tip and theinversion of the needle deflection vector estimated by AI
1. The output, ℎ𝑡 consists of the estimated position of
the needle base to reduce the needle deflection at thecurrent timestamp.
In this report, before proposing a method to estimate
the needle base position that reduces the needle
deflection vector in the second section, we must firstclarify whether it is possible to estimate the needle baseposition that generates the current needle deflection
vector using machine learning, and then report on theaccuracy of the method.

𝑣⃗𝐵 = 𝑣⃗𝑇 − 𝑣⃗𝑅 ······················· (1)Where 𝑣⃗𝐵 is the needle deflection vector, 𝑣⃗ 𝑇 is theposition vector of the needle tip measured using a 3D
position tracking system, and 𝑣⃗𝑅 is the projection vectorof 𝑣⃗ 𝑇 relative to the vector between the needle basestarting position and the target position.
The input for the first artificial intelligence (AI 1)
includes the forces acting around the needle base, theposition of the needle base, and et al. The output consistsof the estimation of the needle deflection vector at thecurrent time. Regarding Section 1, in our previous report,
we elucidated whether it is possible to estimate theneedle deflection vector using artificial intelligence[13].
The second section aims to determine whether a needle
base position can be derived using a artificial intelligence
to reduce the needle deflection. In this research, weemployed a Long Short-Term Memory (LSTM) model.
LSTM is a type of recurrent neural network (RNN)
architecture, designed to overcome the limitations of
traditional RNNs in capturing long-term dependencies insequential data. In an LSTM network, there are special
units, known as memory cells, which are capable ofmaintaining information over long periods of time. The
memory units of LSTM networks resemble the way
humans store and recall memories when learning
behaviors, which led us to hypothesize that they could be
effective for our current objective. An LSTM is made upof 3 gates, Input Gate, Forget Gate, Output Gate. The
equations for an LSTM block are shown as follows:Fig.1 Schematic diagram of needle control method toreduce needle deflection.

𝑓𝑡 = 𝜎(𝑊𝑓 [ℎ𝑡−1 , 𝑥𝑡 ] + 𝑏𝑓 ) ········· (2)𝑖𝑡 = 𝜎(𝑊𝑖 [ℎ𝑡−1 , 𝑥𝑡 ] + 𝑏𝑖 )·········· (3)𝑜𝑡 = 𝜎(𝑊𝑜 [ℎ𝑡−1 , 𝑥𝑡 ] + 𝑏𝑜 ) ········ (4)Fig.2 Needle deflection vector.

2. Experiment
2.1

Experimental Equipment and Conditions

The experimental equipment was shown in Figure 3.
This experimental setup was used get the traning data for
LSTM model. The 6-axis force sensor used to measure
𝐹𝑥,𝑦,𝑧 and 𝜏𝑥,𝑦,𝑧 was set near the base of the needle. Theneedle used was a 21G type needle usually used for
percutaneous transhepatic cholangiography drainage.
The needle tip and needle base positions were measured
using a magnetic type 3D position tracking system(AURORA SYSTEM, NDI). 6-degree-of-freedom(DOF) type and 5DOF type position sensors were used.
In a previous research, the needle insertion phantom wasmade using materials such as gelatin gel, PVA, silicone
rubber, and so on [13]. We decided to use gelatin for thephantom in this experiment because it is inexpensive andit is easy to adjust its hardness. A gelatin phantom with aconcentration of 15% was used for this experiment.
During the needle insertion into the phantom, an
external force was manually applied to the top face of thephantom. The dataset was obtained by inserting theneedle into six points on the phantom surface
perpendicular to the area where the external force wasapplied as shown in Figure 3. The black dots in Figure 3
represents the needle insertion points for training data set,
whereas the red dot represents the needle insertion point
for the test data ( 𝑦 = 40 𝑚𝑚 , 𝑧 = 15 𝑚𝑚 ). The
validation data used during the training of the LSTM
model was collected from the coordinates 𝑦 = 10 𝑚𝑚,𝑧 = 5.0 𝑚𝑚.
The experimental procedure was as follows:
1) The needle was set in front of the insertion surfaceof the phantom.
2) The needle was inserted into the phantom along theneedle axial direction until a certain insertion depth
𝐿1 .
3) The needle base was moved downwards for a⃗⃗.
distance of vector ℎ4) The needle was moved along the needle axial
direction until a certain insertion depth 𝐿2 .
5) The current phantom is then replaced by a new one
and the experiment is repeated again from step 1 for⃗⃗.
all the values of 𝐿1 and ℎ
The deflection vector of the needle tip 𝑣⃗𝐵 wascalculated by measuring the needle tip position and the⃗⃗needle base position. In this experiment, 𝐿1 , 𝐿2 and ℎwere calculated using a motor encoder. Throughout theexperiment, 𝐿2 was kept at a constant value of 50 mm.
⃗⃗ were each divided into three discreet values.
𝐿1 and ℎ⃗⃗
𝐿1 values included 10 mm, 30 mm, 50 mm, while ℎvalues included 5 mm, 10 mm, 15 mm. Given that there
are 6 puncture sites on the gelatin and each puncture site
has 9 conditions each, a total of fifty-four training data
were collected. The root mean squared error (RMSE)
was used for evaluating the accuracy of the estimatedresults.

As explained in section 1.2, the input to the LSTM
model 𝑥𝑡 includes the forces 𝐹𝑥,𝑦,𝑧 (𝑡) and torques
𝜏𝑥,𝑦,𝑧 (𝑡) around the needle base, the insertion depth of
the needle 𝐿1 and the needle tip deflection vector 𝑣⃗𝐵 .
The output of this LSTM model was set as the needle
base vector ℎ(𝑡) . The hyperparameters of the LSTM
model were tuned automatically using Optuna (PreferredNetworks), an open-source automatic hyperparameter
optimization framework.

Fig.3 Experimental equipment

2.2Results

The time-series data results estimating the position ofthe needle base are shown in Figures 4 and 5. Figure 4
shows the Y-axis component, and Figure 5 shows the Zaxis component. The orange line represents the estimatedvalues, the blue line represents the actual measuredvalues, and the green line represents the error values
obtained by subtracting the measured values from theestimated values. From Figure 4, the mean error for theY-axis component was 0.14±0.20 mm. The error value
in the Y-axis component increased when the needle baseis stationary, as seen from the time range 20 s to 30s.
From Figure 5, the mean error for the Z-axis component
was -0.45 ± 1.20 mm. When the needle base wasstationary, the error in the Z-axis component was close tozero, as seen from the time range 20 s to 30 s. The overallroot mean squared error (RMSE) of the results was 0.92mm.

Fig.4 Prediction result (Y axis)

Acknowledgement
This work was supported by JSPS KAKENHI Grant
Number JP21K18087.

Reference
Nguyen CT, Lee E, Luo H, Siegel RJ, “Echocardiographic
guidance for diagnostic and therapeutic percutaneousprocedures”, Cardiovasc Diagn Ther, Vol.1, No.1, pp.1136, 2011.
2) Kumar R, Sinha A, Lin MJ, Uchino R, Butryn T, O'MaraMS, Nanda S, Shirani J, Stawicki SP, “Complications ofpericardiocentesis: A clinical synopsis”, Int J Crit Illn Inj
Sci, Vol.5, No.3, pp.206-212, 2015.
3) Morton Kern, Andrew Doorey, “Conversations inCardiology: Should We Use Micropuncture Technique forPericardiocentesis?”, Cath Lab Digest, Vol.23, No.9, 2015.
4) H. K. Hussain, J.E.Kingston, P. Domizio, A.J.Norton, R. H.
Reznek, “Imaging-guided core biopsy for the diagnosis ofmalignant tumors in pediatric patients”, American journalof roentgenology, Vol.176, No.1, pp.43-47, 2001.
5) P. L. Roberson, V. Narayana, D. L. McShan, R. J. Winfield,P. W. McLaughlin, “Source placement error for permanent
implant of the prostate”, Medical Physics, Vol.24, No.2,
pp251-257, 1997.
6) R. Taschereau, J. Pouliot, D. Tremblay, “Seedmisplacement and stabilizing needles in transperineal
permanent prostate implants”, Radiotherapy andOncology: Journal of the European Society for
Therapeutic Radiology and Oncology, Vol.55, No.1,
pp.59-63, 2000.
7) J. J. Carr, P. F. Hemler, P. W. Halford, R. I. Freimanis , R.
H. Choplin, M. Y. Chen, “Stereotactic localization ofbreast lesions: how it works and methods to improveaccuracy”, Radiographics: A Review Publication of theRadiological Society of North America, Inc, Vol.21, No.2,
pp.463-473, 2001.
8) Samuel Lafreniere, Olivia Lee Sprouse, Ryan Justin
Padilla, Bardia Konh, “DESIGN OF AN MRICOMPATIBLE ROBOT FOR IMAGE-GUIDED
NEEDLE INSERTION PROCEDURES USING ACTIVETENDON-DRIVEN NEEDLES”, 2023 Design of Medical
Devices Conference, 2023.
9) Pedro Moreira, Leanne Kuil, Pedro Dias, Ronald Borra,Sarthak Misra, “Tele-Operated MRI-Guided Needle
Insertion for Prostate Interventions”, Journal of Medical
Robotics Research, Vol.4, No.1, 2019.
10) Hajime Ishii, Tetsushi Kamegawa, Hiroki Kitamura,Takayuki Matsuno, Takao Hiraki, Akio Gofuku,“Development of a prototype of puncturing robot for CTguided intervention”, 2016 IEEE 11th Conference onIndustrial Electronics and Applications (ICIEA), pp10201025, 2016,11) Meaghan Bowthorpe, Mahdi Tavakoli, Harald Becher,
Robert Howe, “Smith predictor based control inteleoperated image-guided beating-heart surgery”, 2013
IEEE International Conference on Robotics andAutomation (ICRA), pp. 5825-5830, 2013.
12) Hiroyuki Kataoka, Toshikatsu Washio, Michel Audette &Kazuyuki Mizuhara, “A Model for Relations Between
Needle Deflection, Force, and Thickness on NeedlePenetration”, Medical Image Computing and ComputerAssisted Intervention – MICCAI 2001, Vol.2208, pp.966974, 2001.
1)

Fig.5 Prediction result (Z axis)

3. Discussion
When the needle base position was moved in the Zaxis direction, the error in the estimation results for theZ-axis component of the needle base position increased.
One possible cause for this is that the movement of theneedle base in the Z-axis direction caused the needle tobend, resulting in non-linear changes in torque at the baseof the needle. In our method, the position of the needle
base needs to be controlled based on the estimation
results of the needle deflection. However, if the error islarge, as observed in the present results, it becomesdifficult to sufficiently reduce the needle deflection, andthe needle tip may not reach the target. Therefore,
improving the estimation accuracy when the needle baseis moved in the Z-axis direction is an important elementin this research. In recent years, ensemble learning has
been extensively studied, but in the present research, webelieve that the data used to train the AI needs to bedivided into smaller tasks. For example, with and without
external forces, the needle base is moved in each axis (X,Y, Z) direction, and different AIs can be trained for eachcondition, which can then be merged to create anintegrated AI. Another reason for the low estimation
accuracy is that feature selection was not fully
implemented in this study. The features provided to theAI are an important factor that affects the estimationaccuracy, so it will be necessary to examine the featuresin the future.

4. Conclusion
The ultimate goal of this research is to develop a robotthat can automate pericardiocentesis, and in this report,
we verified whether it is possible to estimate the positionof the needle base that can reduce needle deflection basedon force information. As a result of the experiments, aproblem arose where the estimation accuracy decreased
when the position of the needle base was moved. In thefuture, we plan to consider ensemble learning and otherestimation methods to improve the estimation accuracy.

13) Axel Yap Shen Yit, Yuta Fukushima. “Proposal of a needle
deflection estimation method during needle insertion using
AI - Examination of estimation accuracy based on
differences in input information to AI -”, JSMBE
Symposium 2023, C-09, 2023.