Visualization method based stiffness sensing system for endoscopes
Takanobu Iwai, Toshio Koyama, Hiroyuki Kagawa, Takeshi Yoneyama, and Tetsuyou Watanabe,
Member, IEEE

Abstract— This research developed novel stiffness sensing
system attachable to endoscope. The system is an extension of
our previous force sensing systems utilizing force visualization
mechanism. The sensing part is attached to endoscopes. The
force is visualized at the sensing part, and can be measured as
visual information via endoscopes. The sensing part also has astructure of limiting the pressing amount. By measuring force atthe limitation, the stiffness can be measured. The developed
sensing part has the features of no electrical components,
disposable, simple, easy sterilization, MRI-compatibility, andlow-cost. The validation of the system was experimentally shown.

I. INTRODUCTION
Endoscope is a powerful medical instrument. Via
endoscope, medical doctors can minimal-invasively examine
deep area where doctors cannot see directly. It is used for not
only medical checks but also surgeries. However endoscope
provides only visual information. If doctors can have not only
visual but also haptic information, more accurate diagnosis canbe expected. For example, palpation is a useful method for
diagnosis at small hospitals and clinics. The tactile informationin abdomen surgeries is very important to detect the difference
between normal tissues and tumors. With this in mind, at ourprevious papers [1, 2], we developed force sensing system
attachable to fiberscopes/endoscopes. By applying force
visualization mechanism utilizing a highly elastic material(panty stocking fabric), the force sensing system has the very
good features of high resolution, small size, low cost, noelectrical part, and disposable of sensing part. However, thereally required functions to detect affected area are to measure
not only force but also stiffness or softness of tissues. With thisin mind, this paper presented a novel stiffness sensing systemthat can be attached to endoscope. The main features are as
follows:
Visualization mechanism based sensing system
attachable to endoscope: The developed stiffness sensing
system is an extension of our force sensing system [1, 2]. The
sensing part is attached to endoscopes. The force ordisplacement are visualized at the sensing part, and can bemeasured as visual information. The sensing part has thefeatures of no electrical component, disposable, simple, easysterilization, MRI-compatibility, and low-cost.

There are many kinds of force and tactile sensors for
medical usage [1-7]. Most force and tactile sensors require
electrical components, and are not easy to sterilize or disinfect.
Additionally, amplifiers are required and the size and cost of
the overall system is then large. Sensors without electricalcomponents are suitable in the medical field. Concerning this,several force sensors without electric components have beendeveloped, utilizing force visualization [8], pneumatic actuator[9], and optical fibers [10]. However, stiffness was not thetarget and cannot be measured. Only Kawahara et al. [11]developed a system to measure the stiffness of an organutilizing air jet and camera. Drawbacks are not simple(requiring multiple endoscope-like systems) and largeness ofoverall system.

Data

Sensing part

Endoscope

PC

Figure 1. Ovewview of stiffness sensing system
4mm

14mm

18mm

Figure 2. Schematic views of sensing part and its cross-selction

II. STRUCTURE AND MECHANISM
Fig. 1 shows the overview of the developed stiffness
sensing system. If the sensing part makes a contact with anobject or an organ, the image change associated withforce/load and displacement is captured by camera equippedwith the endoscope, and analyzed by PC. The sensing part has
a structure of limiting the pressing amount, and by sensing
force at the limitation, the stiffness information of the object
or organ can be obtained.

Stiffness sensing: The sensing part has a structure oflimiting the pressing amount. By measuring force at thelimitation, the stiffness can be measured.

A. Structure of sensing partThe schematic views of sensing part and its cross-section
are shown in Fig. 2. The sensing part is constructed by head,spring, and base parts, shown in Fig. 3 (left). These three parts
are connected by fitting. Spring part is made of silicone
produced by Shin-etsu Silicone Company (silicone: KE-1308,
hardener: CAT1300L-3, hardener volume: 6 %, see [12, 13]

T. Iwai and T Koyama are with the Department of mechanicalEngineering, College of science and engineering, Kanazawa University,
Ishikawa,Japan(corresponding author
toprovidee-mail:iwai.t@stu.kanazawa-u.ac.jp).

H. Kagawa, T. Yoneyama, and T. Watanabe is with the Institute of
Science and Engineering, Kanazawa University, Kakuma-machi, Kanazawa,
9201192Japan(e-mail:kagawa@t.kanazawa-u.ac.jp;
yoneyama@t.kanazawa-u.ac.jp; te-watanabe@ieee.org).

for stiffness information), and Head and base parts (ABS plus:
Young's Modulus: 2.2 [GPa]) were made by 3D printer(Stratasys Uprint SE). Fig. 3 (right) shows the manufactured
one. If the head part makes a contact with an object or organ,
the spring part deforms and the pin displaces according to thedeformation. By capturing the displacement of the pin via
camera, force or displacement can be estimated.
Head part

Spring part

Pin

Base part

Figure 3. Components of sensing part (left) and Manufactured one(right)
Model

Analyzed image

BeforeObtained image

C. Principle for sensing stiffness
We defined that stiffness is 𝐹/𝑥 [N/mm] when the load
𝐹[N] is applied to the head part, and the target object deforms
with the displacement (𝑥 [mm]) of the head part due to the load,
as seen in Fig. 5. We assumed that the surface of target object
around contact area is flat. Now, suppose the case of pressing
a target object with the sensing part. If the surface of target
object reaches the top surface of spring part, the increasing rateof the pressing distance/amount changes. We call the pointchange point 𝑃c . By detecting the change point 𝑃c and sensing
load 𝐹 at the point, load 𝐹 when pressing the target object byexactly 4 [mm] can be obtained. Note that 4 [mm] is the heightof the head part from the top surface of the spring part.
Therefore, the stiffness can be derived from the relation
between the 𝐹 [N] and 4 [mm]. Note that because 4 [mm] does
not always equal to the displacement of the head part, thedisplacement ( 𝑥 [mm]) has to be derived from the pin
displacement 𝐿 (= 4 − 𝑥) . 𝐿 can be derived from 𝐴 − 𝐴0 ,similarly to the deviation of 𝐹.

A0

CameraAfter

F

Force gauge Sensing partA

CameraL

Figure 6. Schematic view of experimental setup for evaluating propertiesof sensing parts.

Figure 4. Principle for sensing force in the longitudinal direction of pin

III. EXPERIMENTAL EVALUATION

B. Principle for sensing force
Before describing the principle for stiffness sensing, theprinciple for force sensing is described. The method to detectthe displacement of pin is the same as the method presented in
our previous papers [1, 2]. The difference is the structures fordisplacing pin according to load and for limiting thedisplacement for sensing stiffness. Three-axis force can bemeasured in principle. Because the main target is stiffness
sensing in this paper, we focus on the measurement of force inthe longitudinal direction of the pin. Fig. 4 shows the principle.
If load 𝐹 is applied to the head part, the pin displaces
according to the deformation of the spring part. Then, thecaptured area for the pin changes from 𝐴0 (initial area withoutload) to 𝐴, as seen in Fig. 4. Therefore, the load 𝐹 can bederived from 𝐴 − 𝐴0 if we know the relation between 𝐹 and
𝐴 − 𝐴0 .

A. Properties of sensing parts
Fig. 6 shows the schematic view of experimental setup forevaluating the properties of sensing part. The sensing part was
attached to the camera (Elecom UCAM-DLT30HSV:
resolution: 640×480 [pixel]), and fixed. Note that here thecamera instead of endoscope was used for easy setup. Forcegauge (IMADA DS2-500N) was attached to the automatic
positioning stage (IMADA MX2-500N). The flat plate was
attached to the tip of the force gauge. By moving the force
gauge, we pressed the head part of the developed sensing partwith the flat plate. Initially it is set that the palate was in contact
with the sensing part with the force value of 0.00 [N]. Then,the automatic positioning stage was controlled to increase themagnitude of the load by 0.25 [N], until the load reached 5.00
[N]. At every step, the value of force gauges and the moving
distance of automatic positioning stage were recorded, andphoto was taken by the camera to obtain the area for the pin 𝐴.

The target object

6F

F = 1.41(A-A0) - 6.80
R² = 0.99

5Load (F) [N]

4mm

x

4

F = 0.397(A-A0) + 0.0815
R² = 0.99

3

2
1
0

L=4-x

0

2
4
6
8
10
The difference of the area of captured pin
3
image from the initial one (A-A0) [10 pixel]

Figure 5. Principle for sensing stiffness
Figure 7. Load F [N] v.s. the difference of the area of captured pin image
from the initial one (𝐴 − 𝐴0 )

Moving distance of pin fromthe initial state (L) [mm]

5
4
3

2
1

L = -0.0330(A-A0)2 + 0.839(A-A0) - 0.0716
R² = 0.99

0
0

2

4

6

8

10

The difference of the area of captured pin image fromthe initial one (A-A0) [103 pixel]

object. We used two target objects made of silicone andmelamine foam. The shape for these objects was cylinder withthe height of 11 [mm] and the radius of 11 [mm]. The stiffness
properties for the soft objects are experimentally obtained
when pushing the head part only against the soft objects. Fig.
10 and Fig. 11 shows the properties. The procedure was thesame as the previous experiment where increasing pushingforce by 0.25 [N] until the pushing force reached 5 [N]. At
every step, pushing distance and force value were recorded andphoto (pin image) was taken by camera. The overall procedure
was repeated 5 times for every object.
Sensing part Camera
Force gauge
Soft object

Figure 8. Moving distance of pin from the initial state L [mm] v.s. thedifference of the area of captured pin image from the initial one (𝐴 − 𝐴0 )Figure 9. Schematic view of experimental setup for validating thedeveloped stiffness sensing sytem
4
3.5

2
1.5
1

0.5
0

0

F = 0.0318d2 + 0.106d - 0.0115
R² = 0.99

1.2
1Load F [N]

0.8
0.6
0.4
0.2

0
0

1

2

3

4

5

6

Pushing distance d [mm]Figure 11. Load versus pushing distance when pushing the head part only
against the object made of melamine foam
6

Step 4: Derive the stiffness 𝐹/𝑥 [N/mm].

Fig. 9 shows the schematic view of experimental setup to
validate the developed stiffness sensing system. The
experimental system is the same as the one shown in Fig. 6,
except that a soft object is attached to the plate as a target

5

1.4

Step 2: The load 𝐹 and the moving distance of pin from theinitial state L [mm] at the point 𝑃c were derived,
utilizing 𝐴 − 𝐴0 and the regression curves shownin Fig. 7 and Fig.8.

5regression curve

Load F [N]

If increasing pushing force against soft objects with thestep of a certain value (for example, 0.25 [N]), the increasingrate of the pushing distance/amount or 𝐴 − 𝐴0 suddenly
changes due to the contact between the object and the top
surface of spring part. 𝑃c at step 1 is the point at the change,
and can be detected by detecting the change (Note thatpractical method for the detection will be described later).

1
2
3
4
Ppushing distance d [mm]Figure 10. Load versus pushing distance when pushing the head part only
against the object made of slicone

Step1: Detect the point 𝑷𝐜 where an object makes acontact with the top surface of the spring part.
𝐴 − 𝐴0 at the point 𝑃c is derived.

Step 3: Derive the displacement of the head part from theinitial stage 𝑥 [mm] at the point 𝑃c , by 𝐿 = 4 − 𝑥.

+ 0.165d -0.0438

2.5

Fig. 8 shows the relation between L and 𝐴 − 𝐴0 . The
quadratic function was used for regression. It is seen that aquadratic function can represent the relation very well.
B. Sensing stiffness of soft objects
By utilizing the relations obtained at the previous
subsection (see Fig. 7 and Fig. 8), we tried to sense the stiffnessof soft objects. The procedure of sensing stiffness is as follows:

F = 0.15
R2 = 0.99

3Force F [N]Fig. 7 shows the obtained relation between load 𝐹 [N] andthe difference of the area of captured pin image from the initialone (𝐴 − 𝐴0 ). It is seen that load 𝐹 linearly increases with theincrease of 𝐴 − 𝐴0 . However, around 𝐴 − 𝐴0 = 7 × 103[pixel] the increasing rate of load 𝐹 suddenly changed. Thischange is due to the occurrence of contact between the plateand (the top surface of) the spring part. Two regression curves
were obtained by dividing the data taking the change asboundary. Each regression curve corresponds to the data for
𝐿 = 0~4 [mm] and for 𝐿 = 4~ [mm]. The obtained
regression curves are shown in Fig. 7 with the coefficient ofdetermination. It is seen that a linear function can represent therelation very well.

4

The measured result

3
2

1
0
0

1

2

3

4

5

6

7

The difference of the area of captured pin image from the initialone (A-A0) [103 pixel]

Figure 12. Load versus the difference of the area of captured pin image
from the initial one (𝐴 − 𝐴0 ) with regression curve when pushing silicone

IV. CONCLUSION5
4.5

The regression curve

4

The mesured result

Load F [N]

3.5
3

2.5
2
1.5
1
0.5

0
0

2

4

6

8

The difference of the area of captured pin image from theinitial one (A-A0) [103 pixel]

Figure 13. Load versus the difference of the area of captured pin image
from the initial one (𝐴 − 𝐴0 ) with regression curve when pushing melamine
foamFig. 12 and Fig.13 shows the relationship between applied
load and mean 𝐴 − 𝐴0 with standard deviation. 𝑃c is the point
where changes happened, and can be easily detected in Fig. 12
and Fig.13. But we have to know 𝑃c only from 𝐴 − 𝐴0 . It isseen that initial several points are on the regression curve, andthe force value can be derived from 𝐴 − 𝐴0 before reaching
the point 𝑃c . Namely the investigation of increasing pushingforce with a step of a certain value without the force gauge(another force sensors) is available. But after reached the point
𝑃c the data points dropped off the regression curve. It is also
seen that after reached 𝑃c , the increasing rate of 𝐴 − 𝐴0
decreased; each interval between data points became small.
Then, 𝑃c can be detected by letting 𝑃c be the point where theincreasing rate of 𝐴 − 𝐴0 changes (when increasing pushingforce with a step of a certain value).

Stifness [N/mm]

According to the procedure for sensing stiffness, thestiffness was calculated. Fig. 14 and Fig. 15 shows the meanderived and actual stiffness with standard deviation. Note that
actual stiffness also can deviate due to the change of pushingdistance (see Fig.10 and Fig.11), although it is very small. It isseen that in both cases, the stiffness was estimated with small
error. But, in case of melamine-foam object, the error was
relatively larger. Because the melamine-foam object has small
stiffness, the investigation with the step of small force value
might be required for getting more accurate stiffness.
0.5
0.4

0.3
0.2
0.1

0
Derived one

Actual one

Figure 14. Derived stiffness and actual stifness of silicone object

Stifness [N/mm]

0.25
0.2
0.15
0.1
0.05

0
Derived one

Actual one

Figure 15. Derived stiffness and actual stifness of melamine-foam object

This research presented novel stiffness sensing system thatworks by attaching the developed sensing part to theendoscope or camera. If an object or organ touches the sensing
part, the pin of the sensing part displaces according to themagnitude of force. By detecting the displacement as visual
information, force value can be derived. Additionally thedisplacement of pin is limited by the structure of spring part.
The limitation of the pushing distance can be detected bymonitoring the sudden change of increasing rate of pushingdistance when pushing object or organ with a step of a certain
value of pushing force. The stiffness of object or organ can bederived by deriving the force at the sudden change
corresponding to the known limitation of pushing distance.
Experimental evaluation showed the validity of the developed
system. Remark that experimental results indicated that if
investigating the limitation of the pushing distance roughly,
the accuracy of the derived stiffness decreased. How to keep
accuracy in any case might be our future work.
REFERENCES[1] T. Watanabe, T. Iwai, Y. Fujihira, L. Wakako, H. Kagawa and T.
Yoneyama, “Force sensor attachable to thin fiberscopes/endoscopes
utilizing high elastic fabric,” Sensors, vol.14, No.3, pp.5207-5220,
2014.
[2] T. Iwai, Y. Fujihira, L. Wakako, H. Kagawa, T. Yoneyama, and T.
Watanabe, “Three-axis force visualizing system for fiberscopes
utilizing highly elastic fabric,” Proc. of the IEEE/ASME Int. Conf. on
Advanced Intelligent Mechatronics (AIM), pp.1110-1115, 2014.
[3] P. Puangmali, K. Althoefer, D. Seneviratne, D. Murphy, and P.
Dasgupta, “State-of-the-art in force and tactile sensing for minimallyinvasive surgery,” Sensors Journal, Vol. 8, No. 4, pp. 371–381, 2008.
[4] E. W. Van der Putten, R. Goossens, J. Jakimowicz, and J. Dankelman,“Haptics in minimally invasive surgery-a review,” Minimally Invasive
Therapy & Allied Technologies, Vol. 17, No. 1, pp. 3–16, 2008.
[5] A. Okamura, C. Verner, and M. Mahvash, “Haptics for robotassisted
minimally invasive surgery,” Robotics Research, 2011, pp. 361–372.
[6] E. V. Poorten, E. Demeester, and P. Lammertse, “Haptic feedback for
medical applications, a survey,” Proceedings Actuator, 2012, pp.18–20.
[7] M. I. Tiwana, S. J. Redmond, and N. H. Lovell, “A review of tactile
sensing technologies with applications in biomedical engineering,”
Sensors and Actuators A, Vol. 179, pp. 17-31, 2012.
[8] T. Takaki, Y. Omasa, I. Ishii, T. Kawahara, and M. Okajima, “Forcevisualization mechanism using a moire fringe applied to endoscopic
surgical instruments,” Proc. of IEEE Int. Conf. on Robotics andAutomation, 2010, pp. 3648–3653.
[9] K. Tadano and K. Kawashima, “Development of 4-dofs forceps withforce sensing using pneumatic servo system,” Proc. of IEEE Int. Conf.
on Robotics and Automation, 2006, pp. 2250–2255.
[10] J. Peirs, J. Clijnen, D. Reynaerts, H. V. Brussel, P. Herijgers, B.
Corteville, and S. Boone, “A micro optical force sensor for force
feedback during minimally invasive robotic surgery,” Sensors andActuators A, Vol. 115, pp. 447-455, 2004.
[11] T. Kawahara, S. Tanaka, and M. Kaneko, “Non-contact stiffness
imager,” Int. J. of Robotics Research, Vol. 25, No. 5-6, pp.537–549,
2006.
[12] T. Watanabe and Y. Fujihira, “Experimental investigation of effect offingertip stiffness on friction while grasping an object,” Proc. of IEEE
Int. Conf. on Robotics and Automation, 2014, pp. 889-8943.
[13] Y. Fujihira, K. Harada, T. Tsuji, and T. Watanabe, “Experimental
Investigation of Effect of Fingertip Stiffness on Resistible Force in
Grasping,” Proc. of IEEE Int. Conf. on Robotics and Automation, 2014,
pp. 4334-4340.