Three-axis force visualizing system for fiberscopes utilizing highly
elastic fabricTakanobu Iwai, Yoshinori Fujihira, Lina Wakako, Hiroyuki Kagawa, Takeshi Yoneyama, andTetsuyou Watanabe, Member, IEEE
1

Abstract—This paper presents a novel force sensing system
for fiberscopes. The key features of the system are its low cost,high resolution, small size, and ability to measure threedimensional force. A previous study described a novel force
sensing system that could be attached to a very thin fiberscope,based on a force visualization mechanism utilizing panty
stocking fabric—a highly elastic material. However, this systemmeasures force in only one direction. In this paper, the system isextended to measure forces in any of three directions. The system
is targeted for application to neurosurgical examinations. It may
also be useful for other medical and non-medical examinations
that involve the use of fiberscopes.

I. INTRODUCTION
Fiberscopes and endoscopes are widely used to examine
areas where humans cannot see directly. They are commonlyused for medical examinations, although they can also be used
for industrial examinations. However, only visual information
can be obtained with current systems. If tactile information
could be obtained, the examinations would be more valuable.
In particular, tactile/force information is important when thevisible area is limited. A typical example occurs inneurosurgery, where the doctor is required to treat tumors thatare seated deeply within the brain and surrounded by healthy
tissues. Considering this issue, a force sensor system that could
be attached to the tip of an endoscope [1] was previously
developed. However, this system measures the force only in
one direction. This paper presents an extended version of theaforementioned force sensing system that can measure threedimensional forces.
Many force and tactile sensors have been developed for
medical instruments. Puangmali et al. [2] reviewed research onforce and tactile sensors for minimally invasive surgery. The
purpose of force sensing is to provide feedback to the operator
on the sensation of touching an organ (see [3]–[6] for reviews).
Most force sensors that have been developed include electric
devices such as strain gauges. However, electrical devices are
not always suitable for medical devices. Electrical force
sensors require amplifiers, and the size and cost of the overallsystem becomes large. Electrical sensors also require wiring
for signal transfer, although wiring easily causes noise. In
addition, sterilizing or disinfecting such electrical devices isdifficult. One solution to these issues is to build a system
without using electrical parts. Takaki et al. [7] developed aforce sensor based on force visualization by using moiré fringe
patterns. Tadano and Kawashima [8] developed a system to* Research partly supported by JKA.
T. Iwai and Y. Fujihira are with the Department of mechanicalEngineering, College of science and engineering, Kanazawa University,
Ishikawa,Japan(corresponding
authortoprovidee-mail:iwai.t@stu.kanazawa-u.ac.jp).

generate feedback on force sensation without a force sensor by
utilizing a pneumatic servo system. Kawahara et al. [9]developed a system to measure the stiffness of an organ bypushing it with air and capturing its deformation by using acamera. Peirs et al. [10] developed a force sensor that detects
the deformation of a flexible structure by using optical fibers.
Tada et al. [11] developed a force sensor that functions in MRI
environments by detecting, via photo-sensors, changes in thepeak illumination of a point light source attached to the tip ofan elastic frame. These sensors are mainly used forlaparoscopic surgery, in which the part size and forces
measured are larger (in the cm and N ranges, respectively) thanthe corresponding values that would be ideal for endoscopy inneurosurgery (in the mm and mN ranges). Sensors utilizing
force visualization have been developed, although theirintended applications are not medical. Ohka et al. [12] havedeveloped a three-axis force sensor by observing the states ofconical feelers with a camera. Kamiyama et al. [13] havedeveloped a sensor that measures the direction, magnitude, anddistribution of force by observing two layers of sphericalmarkers with a camera. However, applying these concepts toforce sensors in small and thin fiberscopes requires that thesensor size be reduced and a method to construct small
markers be developed. In general, this issue is considered thedisadvantage of sensors utilizing force visualization, as noted
in previous studies [6], [14]. In contrast, a robotic system witha force sensor and feedback systems for neurosurgery wasdeveloped in [15]–[17]. However, because the force sensing
system was based on a strain gauge, the aforementioned issuesof sterilization and MRI compatibility were not resolved.
Considering these issues, a force sensing system based onforce visualization, which can be attached to a fiberscope, wasdeveloped in this study. The key features of the proposed force
sensing system are as follows.
Force visualization mechanism utilizing a highly elastic
panty stocking fabric: To ensure both small size and highresolution, panty stocking fabric (shown in Fig. 1) is used for
detecting force. The fabric is highly elastic and can deform
even if the applied force is extremely small. In addition, thematerial is thin, inexpensive, and lightweight. A force
visualization approach that utilizes this fabric was adopted
because visual information could be obtained throughfiberscopes, and only a force-detecting component is needed

L. Wakako, H. Kagawa, T. Yoneyama, and T. Watanabe are with Schoolof Mechanical Engineering, College of Science and Engineering, Kanazawa
University, Ishikawa, Japan (e-mail: linawakako@se.kanazawa-u.ac.jp;
kagawa@t.kanazawa-u.ac.jp;
yoneyama@t.kanazawa-u.ac.jp;
tewatanabe@ieee.org).

to construct the force sensor. Hence, the force sensing system
can be produced at a very low cost.

Figure 1. Panty stocking fabricHigh resolution and small size: Owing to the highelasticity of the stocking fabric, a resolution of more than
0.015 [N] is realized. The part for detecting force is extremelycompact, and its diameter is less than 3 [mm]. It does notinclude electrical components; therefore, sterilization is simple
and MRI-compatibility easy to achieve. In addition, the part isdisposable and its manufacturing cost is low.

1) Depth direction of endoscope:
The same principle as in the previous study [1] wasconsidered. Fig. 2 shows a schematic diagram of the basic
force sensing concept in the depth direction of endoscope. The
force visualization part is attached to the tip of the fiberscope.
If objects or tissues make contact with the pin and force isapplied along the depth direction of the fiberscope (horizontal
direction in Fig. 2), the pin slides in this direction andapproaches the fiberscope. The pin is in contact with the elastic
fabric shown in the middle area of Fig. 2 (Region A in Fig. 4).
The distance traveled by the pin then depends on themagnitude of the applied/contact force. The elastic fabric plays
the role of a spring. The image of the pin covered with theelastic fabric is captured by the fiberscope. The pin image
becomes larger as the applied/contact force increases. The
magnitude of the applied/contact force is then derived from thesize of the pin image.
Captured imageF

II. FORCE DETECTING PART
Deformation

A. Target situation
In this paper, the situation where a medical doctor
examines a candidate tumor area by touching or nudging itwith the developed sensor to identify the affected area
precisely, while observing the area with other endoscopes is
considered. The examination is normally performed byapplying pressure to tissues, and rarely involves their releasefrom pressure. Therefore, the focus of this paper is on applying
pressure to tissues. In this paper, tangential forces that were
not considered in the previous study [1] will also be considered.
The main purpose of obtaining force information is todetermine the softness of tissue; thus, static force is a focus of
this paper. The measured force value is displayed on themonitor and superimposed on the image obtained by theendoscope. Feedback information is presented at the video
rate; if the computing time is less than the video rate (10–30[Hz]), then (image) processing can be adapted. Therefore, anendoscope and a force visualization technique that does not
require any electrical components are used in this study. The
target area of application is neurosurgery, such as an
examination of tumors of the pituitary gland. For example, in
examinations of tumors in the pituitary gland, the typicalmagnitude of force is around 0.1 [N] [15-17]. A resolution
close to 0.01 [N] is then required [15-17]. To detect these small
magnitudes at high resolution, highly elastic panty stocking
fabric is used in the study.

F

Elastic fabric plays the role of a bearing and a cushion

Figure 3. Principle for sensing force in directions tangential to tip surface

2) Direction tangential to tip surface:Fig. 3 shows a schematic diagram of the basic force
sensing concept in the directions tangential to tip surface(vertical direction in Fig. 3). The elastic part (blue area in Fig.
3 or region B in Fig. 4) was put in to connect the pin and theslider by fitting. The material of the elastic part is a highly
elastic panty stocking fabric. This part serves as a bearing andcushion. If objects or tissues make contact with the pin andforces are applied along directions tangential to the tip surface,
the pin tilts. Depending on the tilt, the pin image slides in theframe captured by the fiberscope. The sliding distance of thecenter position of the pin image corresponds to the distancetraveled by the tip because of the force applied in the tangential
direction. The magnitude of the applied/contact force is thenderived from the sliding distance of the center position of thepin image. Note that the change of shape in the pin image also
corresponds to the applied forces. However, the sliding
distance of the center position was derived because it waseasier.

B. Force sensing principle
Pin

Filled with fabrics

Force

Fabrics

Fiberscope

Slidable

Figure 4. Overview of force visualization partC. Structure of force visualization part

Force

Detect force magnitude from the size of circular pin image

Figure 2. Principle for sensing force in depth direction of endoscope

Fig. 4 shows the designed force visualization part. Its main
components are the pin, fabric, fiberscope, and theirconnecting parts. A 20-denier panty stocking fabric, whichplays the role of a spring, is installed at region A in Fig. 4.

Figure 5. Overview of assembly process for force visualization part

Figure 6. Overview of assembly process for pin (part 2 in Fig. 5)

Figure 7. Photograph of the manufactured force visualization part and a complete view of the developed force sensing system

TABLE I.

SPECIFICATIONS OF CAMERA (LUMENERA LU 135)

Size of image

1392 × 1040

Frame rate

15 [fps]

Exposure time

66 [ms]Whole gain

2.0 [dB]

The same fabric is also installed at region B in Fig. 4. Thisfabric plays the role of a cushion and bearing. Fig. 5 shows an
overview of the assembly process for the force visualization
part. First, the fabric is used to cover part 4, and it is fixed bylinking parts 3 and 4. Second, part 2 (pin) is connected to thelinked part (the fabric and parts 3 and 4). Finally, part 1 isconnected to the linked part (the fabric and parts 2, 3, and 4).
Fig. 6 shows an overview of the assembly process of the pin(part 2). After covering the fabric on part 2-1, part 2-2 isconnected to the linked part (the fabric and part 2-1). Parts 14 were connected by fitting. Fig. 7 shows a photograph of the
manufactured force visualization part and complete view ofthe developed force sensing system. Except for the fabric, thematerial used for the visualization part was duralumin. The
visualization part was 18.5 [mm] long in the maximallyextended state and had a maximum diameter of 3 [mm]. The
fiberscope is a Super Thin Flexible Borescope (MEDIT) witha diameter of 1.9 [mm], full length of 254 [mm], and visual
field of 45[°]. The camera (Lumenera Corporation: Lu 135),
whose specifications (measurement conditions) are listed inTable 1, is connected to the fiberscope. An image captured byusing the Lu 135 can be directly sent to a PC via a USB cable.
Note that the effect of distortion when bending the endoscope
was investigated during the study and no change of intensity
in the captured image was identified.
III. DERIVATION OF FORCE VALUE FROM CAPTURED IMAGE
This section describes how to derive the force value fromthe image captured by a fiberscope. The method for deriving
force in the depth direction of the fiberscope is the same as themethod presented in a previous paper [1]. The method forderiving forces in directions tangential to the tip surface ismainly described.
A. Forces in directions tangential to tip surface
As mentioned above, the force is derived based on thecenter position of the pin image captured by the fiberscope. To
derive forces in directions tangential to the tip surfaces(vertical direction in Fig. 3), the center position of the pin
image is first extracted. The magnitude of the force in atangential direction is then derived from the distance traveledby the center position.
1) Extraction of center position of the pin image
Fig. 8 shows the procedure for extracting the pin area in
the image captured by fiberscope. The original captured imageis shown in Fig. 8 (a). The area surrounded by the relatively
dark region is the area of the pin with the elastic fabric. The
following steps show how to extract the center position of thepin image captured by the fiberscope. The process was
implemented with Halcon (MVTech). The total computing
time was less than 100 [ms].
1.

Smoothing and edge enhancement: Smoothing andedge enhancement are performed to reduce noise andclarify the boundary of the pin area. The obtained
image is shown in Fig. 8 (b).

2.

Converting to a grayscale image: The (color) image
is converted to a grayscale image as a preprocessing
step for binarization. Fig. 8 (c) shows the obtained
image.

3.

Binarizing and selecting area: The grayscale image isbinarized. By searching the region with the maximum
area in the obtained binarized image, the pin area wasdetermined. Fig. 8 (d) shows the obtained image.

4.

Extracting the center position: By calculating thegeometrical center of the obtained pin area, the center
position is extracted.

(a) Original image(b) Smoothing and edge enhancement(c) Grayscale image(d) Binarizing and selecting area

Figure 8. Procedure for extracting the pin area in the captured image

FΔl

Non-linear
spring model

Fulcrum

Figure 9. Simplified model for derivation of forces in directions tangential
to tip surface

2) Extraction of magnitude of force
The simplified model shown in Fig. 9 is considered toderive the value of the force. Because the center position of thepin image is used for the deviation, the pin is described as aline (whose right extreme point corresponds to the center
position of the pin image) in this model. Note that the fabricalso sits at the fulcrum, which corresponds to region B in Fig.
4. However, the effect of the fabric as a spring can be includedin the non-linear spring model shown in Fig. 9. Therefore, theeffect was ignored in this model. As experimentallydemonstrated in a previous paper [1], the relationship between
applied force and the moving distance of pin surface attachedto the fabric (corresponding to ∆ in Fig. 9; see also region A
in Fig. 4) is considered to be nonlinear and is expressed by apolynomial function. In this paper, the relation is derived
experimentally.
Fig. 10 shows the experimental setup for the investigation.
The tip of the force visualization part is placed on the custommade stage shown on the right side of Fig. 10. The custommade stage is placed on the electronic weighting instrument(SHIMADZU TW223N), and the force applied to the stage inthe vertical/downward direction is measured by the instrument.
The force visualization part is attached to the fiberscope.
Another custom-made stage linked to the automatic
positioning stage (SIGMAKOKI SGSP15-10) is attached to
the fiberscope to apply forces, as shown on the left side of Fig.

10. A force in a direction tangential to the tip surface (verticaldirection in Figs. 3, 4, and 10) is applied by downwardly
moving the automatic positioning stage.
Automatic positioning stage

Force visualization part

Custom-made stage
Custom-made stage

Move in verticaldirection bypositioning stage∆(a) Schematic view(b) Photograph
Figure 10. Experimental setup for investigating the relation between theapplied force in direction tangential to the tip surface and the center positionof the pin image

As the fundamental case, the force in the direction
tangential to the tip surface was examined when the force inthe depth direction of the fiberscope (horizontal direction in
Fig. 10) was zero. The initial state is set such that the value
shown by the electronic weighting instrument is 0.000 [g]while the pin of the force visualization part is in contact withthe custom-made stage shown on the right side of Fig. 10.
Subsequently, the automatic positioning stage was moved insteps of 100 [μm] to push the pin attached to the custom-madestage on the electronic weighting instrument until the total
distance traveled reached 3.5 [mm] (The pin was moved 35
times). At each step, the applied force was measured by theelectronic weighting instrument while the image was capturedby the fiberscope. This procedure was repeated five times toconfirm repeatability. Each measurement was conducted at
10-min intervals to minimize dynamic effects.
y = 0.0003x2 + 0.0023x
R² = 0.9919

Applied force F [N]

0.1
0.05
0
5
10
15
20
Moving distance of center position of the pin image from theinitial state (Δl) [pixel](1)

Used (knitted) fabric has a reticular pattern and is
manufactured using multiple (bended and twisted) strings. If
the load is low, deformation of the fabric is mainly caused bystretching the bended strings. The relationship between loadand displacement is then expressed by a nonlinear function[18], [19]. If the load is high, the strings are stretched, and thedeformation is dominated by the stiffness of the strings
themselves. In this case, the relationship between load anddisplacement is expressed by a linear function. In the study,
the maximum applied force was roughly 0.20 [N] and the load
was assumed to be low. In this case, a nonlinear function
should be chosen for regression. Hence, a third-orderpolynomial function is used.
B. Force in the depth direction of fiberscope
The same method as that used in the previous study [1] wasadapted. If the applied force in the depth direction of thefiberscope is large, the size of the pin image is also large. Thatis, the applied force is inversely related to the radius of thecircular image of the pin. The relationship is expressed by thefollowing a second-order polynomial function:(2)is the applied force, is the radius of the circular
whereimage of the pin, and , , and are constants (see [1] forfurther details). This concrete relationship is experimentally
examined in the developed system.

0.2
0.15

0

∆ 

where a and b were 0.30 10 [N/pixel2] and 2.30
10 [N/pixel], respectively. The coefficient of determination
was 0.99. The fitted curve expresses the relationshipconsiderably well. Because (1) is a quadratic polynomial
function, the resolution becomes lowest (i.e., 0.196 [N]) whenthe measured force value is highest. If the force value F isinvestigated so that the lowest Δl that produces 0.196 [N]exceeds the largest Δl that produces F, F turns out to be 0.18
[N]. The resolution is then better than 0.015 [N]. If the rangethat provides a resolution of more than 0.01 [N] is investigated,
the range is 0–0.18 [N]. Therefore, the requirement of aresolution of more than 0.01 [N] is almost achieved.

Electronic weighing instrument

0.25

Fig. 11 shows the experimental results for the relationshipbetween the applied force (F) in a direction tangential to thetip surface (vertical/downward direction in Fig. 10) and the
distance traveled by the center position of the pin image fromthe initial state (Δl). The diamond symbols indicate themeasured points. The center position was derived by themethod described in the previous subsection. A regression
analysis was performed using the following third-orderpolynomial function

25

Figure 11. Relationship between the applied force F) in a direction tangential
to the tip surface (vertical/downward direction in Fig, 10) and distancetraveled by the center position of the pin image from the initial state (Δl).

Fig. 12 shows a schematic view of the experimental setup
for the investigation. Force0 was applied by a force gauge(IMADA ZP-5N, resolution: 0.001 [N]) attached to an
automatic positioning stage (Oriental Motor ELSM2YF030KD-C, movement speed: 100 [mm/s]). A pin attached to thetip of the force gauge was used to push the force visualization
part attached to the fiberscope. The fiberscope was fixed to acustom-made stage so that the direction of pushing could becontrolled. The initial state was set such that the value shownby the force gauge was 0.000 [N] while both pins were inmutual contact. Then, the automatic positioning stage was
moved in steps of 0.02 [N] to push the pin attached to thefiberscope until the applied force reached 0.2 [N]. At each step,
the image was captured using the fiberscope. The procedure
was repeated five times.
Force gauge
Pin

Automatic positioning stage

Figure 12. Schematic view of experimental setup for investigating therelationship between the applied force in depth direction of fiberscope andthe inverse of the radius of the circular pin image
0.25

Force (Fd ) [N]

0.2

0.15

0.1

0.05

0
0.009

0.00950.01

0.01050.011

0.0115ACKNOWLEDGMENT
This work was partly supported by JKA.
REFERENCES[1]

FiberscopeFinvestigated.

0.012

Inverse of theradius of circular image of pin (1/r) [1/pixel]

Figure 13. Relationship between the applied force in the depth direction offiberscope (Fd) and the inverse of the radius of the circular pin image (1/r)

Fig. 13 shows the results. The radius of the circular pin
image was calculated in the same manner as in a previous
study [1]. Note that this is the same method as that describedin section III.A.1 until step 3. At step 4, the radius wascalculated instead of the center position. A regression analysis
was performed using (2), and constant values of
4.63
1.92 10 , and1.63 were obtained. The
10 ,coefficient of determination was 0.99. The fitted curve
expresses the relationship very well, and the resolution was
better than 0.01 [N].
IV. CONCLUSION
A novel 3-axis force sensing system that could be attached
to a fiberscope was presented in this paper. This system is anextension of our previous 1-axis force sensing system [1]. The
main features are its compact size, high resolution, low cost,absence of electrical parts, and disposability. It was
experimentally demonstrated that the system had a resolutionof less than 0.01 [N].
There are several future directions for research. For use in
the real world, a systematic calibration method that includes
camera verification (so that the camera can capture a perfectimage) is needed. Further, forces in the depth direction andtangential directions were separately investigated. In future
work, forces in combined directions will be analyzed. Further,issues such as waterproofing and dynamic forces, which werebeyond the scope of the present study, will also be

T. Watanabe, T. Iwai, Y. Fujihira, L. Wakako, H. Kagawa and T.
Yoneyama, “Force sensor attachable to thin fiberscopes/endoscopes
utilizing high elastic fabric,” Sensors (submitted) vol.14 pp.52075220.
[2] P. Puangmali, K. Althoefer, D. Seneviratne, D. Murphy, and P.
Dasgupta, “State-of-the-art in force and tactile sensing for minimallyinvasive surgery,” Sensors Journal, Vol. 8, No. 4, pp. 371–381, 2008.
[3] E. W. Van der Putten, R. Goossens, J. Jakimowicz, and J. Dankelman,“Haptics in minimally invasive surgery-a review,” Minimally Invasive
Therapy & Allied Technologies, Vol. 17, No. 1, pp. 3–16, 2008.
[4] A. Okamura, C. Verner, and M. Mahvash, “Haptics for robotassisted
minimally invasive surgery,” Robotics Research, 2011, pp. 361–372.
[5] E. V. Poorten, E. Demeester, and P. Lammertse, “Haptic feedback for
medical applications, a survey,” Proceedings Actuator, 2012, pp.18–20.
[6] M. I. Tiwana, S. J. Redmond, and N. H. Lovell, “A review of tactile
sensing technologies with applications in biomedical engineering,”
Sensors and Actuators A, Vol. 179, pp. 17-31, 2012.
[7] T. Takaki, Y. Omasa, I. Ishii, T. Kawahara, and M. Okajima, “Forcevisualization mechanism using a moire fringe applied to endoscopic
surgical instruments,” Proc. of IEEE Int. Conf. on Robotics andAutomation, 2010, pp. 3648–3653.
[8] K. Tadano and K. Kawashima, “Development of 4-dofs forceps withforce sensing using pneumatic servo system,” Proc. of IEEE Int. Conf.
on Robotics and Automation, 2006, pp. 2250–2255.
[9] T. Kawahara, S. Tanaka, and M. Kaneko, “Non-contact stiffness
imager,” Int. J. of Robotics Research, Vol. 25, No. 5-6, pp.537–549,
2006.
[10] J. Peirs, J. Clijnen, D. Reynaerts, H. V. Brussel, P. Herijgers, B.
Corteville, and S. Boone, “A micro optical force sensor for force
feedback during minimally invasive robotic surgery,” Sensors andActuators A, Vol. 115, pp. 447-455, 2004.
[11] M. Tada, S. Sasaki, and T. Ogasawara, “Development of an optical 2axis force sensor usable in MRI environments,” IEEE Sensors, Vol. 2,
pp. 984-989, 2002.
[12] M. Ohka, Y. Mitsuya, Y. Matsunaga, and S. Takeuchi, “Sensing
characteristics of an optical three-axis tactile sensor under combined
loading,” Robotica, Vol. 22, No. 2, pp. 213-221, 2004.
[13] K. Kamiyama, K. Vlack, T. Mizota, H. Kajimoto, N. Kawakami, and S.
Tachi, “Vision-based sensor for real-time measuring of surface traction
fields,” IEEE Computer Graphics and Applications, Vol. 25, Issue 1,
pp. 68-75, 2005.
[14] R. S. Dahiya, G. Metta, M. Valle, and G. Sandini, “Tactile Sensing From Humans to Humanoids,” IEEE Trans. on Robotics. Vol. 6, Issue
1, pp. 1-20, 2010.
[15] T. Yoneyama, T. Watanabe, H. Kagawa, J. Hamada, Y. Hayashi, and
M. Nakada, “Force-detecting gripper and force feedback system forneurosurgery applications,” Int. J. for Computer Assisted Radiology
and Surgery, 2013.
[16] Y. Kanada, T. Yoneyama, T. Watanabe, H. Kagawa, N. Sugiyama, K.
Tanaka, and T. Hanyu, “Force feedback manipulating system forneurosurgery,” Procedia CIRP, 2013. Vol. 5, pp. 133–136.
[17] Y. Fujihira, T. Hanyu, Y. Kanada, Y. Yoneyama, T. Watanabe, and H.
Kagawa, “Gripping Force Feedback System for Neurosurgery,” Int. J.
of Automation Technology, Vol. 8, No. 1, pp. 83-94, 2014.
[18] S. Kawabata, “Linearizing Method” for Solving Non-linear Biaxial
Tensile Property of fabric,” J. Text. Mach. Soc. Japan, Vol. 39, No. 11,
T169-T173, 1968.
[19] S. Kawabata, “A Method of Theoretical Calculation of Tensile
Properties of Knitted Fabric from its Structure and Yarn Properties,” J.
Text. Mach. Soc. Japan, Vol. 23, No. 1/2, T30-T34, 1970.