This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at
http://dx.doi.org/10.1109/JSEN.2016.2574322> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <1

Stiffness measurement system using endoscopeswith a visualization method
Tetsuyou Watanabe, Member, IEEE, Takanobu Iwai, Toshio Koyama, and Takeshi Yoneyama
Abstract—A novel stiffness-sensing system was developed thatworks by attaching the proposed sensing part to endoscopes orcameras. The system provides a method to investigate the stiffnessof tissues or objects in deep areas that can only be observed withendoscopes in order to detect abnormalities. The system is anextension of our previous force sensing system that utilized a force
visualization mechanism. The force is visualized at the sensing part,and can be measured as visual information via endoscopes orcameras. The sensing part also has a limiting structure used as a
threshold for the applied force. By measuring the force at thelimitation, the stiffness can be measured. The limitation point is
detected by the brightness changes of the captured images. The
developed sensing part has the advantages of having no electronic
components, being disposable, simple, easy to sterilize, MRIcompatible, and low-cost. Image processing methods for realizing
the mechanism are also proposed. The system was experimentally
validated.
Index Terms—Force sensors; Endoscopes; Stiffness sensors;
Force visualization method

E

I. INTRODUCTIONNDOSCOPES are powerful instruments that enablevisualization of areas that people cannot enter or touchdirectly. In the medical field, endoscopes are widely used forinvasive examinations. They are used not only for medical
checks, but also in surgeries. Endoscopes are also useful for
detecting faults inside machines. However, they typicallyprovide only visual information. If operators and doctors could
access not only visual, but also haptic information, then moreaccurate diagnoses could be made. For example, palpation is afrequently used diagnostic method in small hospitals and clinics.
The tactile information during abdominal surgery can be used
to detect the difference between normal and malignant tissues.
Tightening of screws could be achieved if tactile information
was available. With this in mind, in our previous papers [1], [2],we developed a force sensing system that can be attached tofiberscopes or endoscopes. Its main applications fell within thefield of medicine. Utilizing a highly elastic material (panty
stocking fabric) to create a force visualization mechanism
results in a force sensing system with high resolution, small size,
An earlier version of this paper was presented at the IEEE EMBC
Conferenceandwaspublishedinits
Proceedings
http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=7319869. This work
was supported by JSPS KAKENHI Grant Number 16H04298.
T. Iwai and T Koyama are with the Department of mechanical Engineering,
College of science and engineering, Kanazawa University, Ishikawa, Japan.

low cost, no electrical components, and disposable parts.
However, the functions required to detect afflicted tissue areas
include not only force measurements, but also stiffness or
softness of the tissue. A similar analogy can be observed inother fields such as factories and plants. Therefore, this paper
presents a novel stiffness-sensing system that works byattaching a force sensing component to an endoscope to detect
abnormalities. The main features are as follows: Visualization mechanism-based sensing system: The
developed stiffness-sensing system is an extension of
our force sensing system [1], [2]. The sensing system
was completed by appending the sensing part to the endof endoscopes or cameras. The force or displacement of
the sensing part is visualized using the scope, andconverted into force and stiffness measurements. The
sensing part has the advantages of having no electricalcomponents, being disposable, a simple design, easysterilization, MRI-compatibility, and low-cost.
 Mechanical structure for limiting the applied force:
The sensing part has a structure used to limit theencountered force. By measuring the force at thislimitation, the stiffness can be determined.
 Strategy for detecting the force limitation: Thechange in brightness of the image is used to detect theforce limitation. The light source is located outside of
the sensing part. The limiting structure also works toblock the light. Therefore, if the brightness substantiallydecreases, it indicates the applied force has met thestructural limitations of the system. Through detecting
this change, the point for measuring stiffness is obtained.
This paper also proposes new image processing
methodologies to realize mechanisms that include thesefeatures.
Our previous conference paper [3] provides a summarized
description of the concept used in this sensing system. However,the strategy for detecting the force limitation was unclear.
Therefore, this paper presents the strategy to detect force
limitations used to determine stiffness, in addition to thestructural modifications to the sensing part required for
implementation of the strategy.
After introducing related works, this paper is organized asT. Yoneyama and T. Watanabe are with the Institute of Science andEngineering, Kanazawa University, Kakuma-machi, Kanazawa, 9201192,
Japan (e-mail: yoneyama@t.kanazawa-u.ac.jp; te-watanabe@ieee.org).
(corresponding author to provide e-mail: te-watanabe@ieee.org)

Copyright (c) 2016 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at
http://dx.doi.org/10.1109/JSEN.2016.2574322> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <follows. First, the main structure and principle are described.
Second, the strategy for detecting the force limitation ispresented. After describing how to derive the force anddisplacement information from the visual information, theexperimental validations are shown. Lastly, the conclusion ispresented.
A. Related works
Many force and tactile sensors have been developed. Dahiya
et al. presented a detailed review for tactile sensors [4]. Tiwana
et al. reviewed tactile sensors in the biomedical field [5]. The
presented sensing system in this paper works with endoscopes.
Force and tactile sensors have wide applications in the medical
field. Primarily, the sensors are used to provide tactile feedback
to a medical doctor or an operator. Puangmali et al. [6],Westebring et al. [7], Okamura [7], and Poorten et al. [8]presented reviews for this type of haptic technology.
Most of the haptic feedback systems used for minimallyinvasive robotic surgery require electronic components such as
strain gauges [9]–[11]. However, electrical devices are notalways preferable in the medical field. The electrical force
gauge and tactile sensors often require other devices, such as
amplifiers, to function, resulting in increased total costs andlarge-sized systems. Difficulties in sterilizing or disinfecting
are also common problems. With this in mind, several sensor
systems without electronic components have been developed.
Takaki et al. developed a force sensor embedded in forceps byutilizing moiré fringe patterns [12]. The group of Kawashima
developed a pneumatic force sensing system embedded inpneumatically driven forceps [13], [14]. Peirs et al. developed
a force sensor utilizing optical fibers to visualize thedeformation of a flexible structure embedded in the sensor [15].
Tada et al. developed a force sensor that detects peakillumination changes of a point light source attached to a
flexible structure [16]. Ohka et al. developed a three-axis force
sensor that detects the position and orientation of columnar andconical markers by cameras, which changes depending on theapplied force [17]. Kamiyama et al. developed a sensor that canestimate the magnitude, distribution, and direction by detecting
the motion of two layered markers with cameras [18]. Chorley
et al. developed a biologically inspired vision-based tactile
sensor (TACTIP) [19].
With regarding to stiffness sensors, Kawahara et al.
developed a stiffness measuring system that supplies air
streams to a target and captures the deformation caused by the
air pressure by using endoscopes or cameras [20]. The
drawback of this system is its complexity and large size. Fukuda
et al. developed a sensing system that detects the softness oftissue via acoustic reflection [21]. However, only relative
comparison is available, and quantitative values of stiffness
cannot be obtained. Although there are some types of stiffness
sensing systems, their development is limited, and still have
many unresolved problems. Therefore, this paper aims todevelop a low-cost and simple stiffness-sensing system.
II. STRUCTURE AND PRINCIPLE
Fig. 1 shows the overview of the developed stiffness-sensing

2

system. The system consists of a sensing part, an endoscope orcamera, a light source, and a PC. If the sensing part makes
contact with an object, the image change associated with theforce/load and subsequent displacement is captured by thecamera (contained within the endoscope), and analyzed by a PC.
The sensing part has a structure that acts as a force limit, and by
sensing the force at this limit, the stiffness information of theobject can be obtained.

Fig. 1. Schematic view of the proposed stiffness-sensing system.

Here, we define stiffness. Let [N] be the force/load applied
to the object by the sensing part, and [mm] be the deformationof the object, as shown in Fig. 2. The stiffness is defined asfollows

stiffness

N/mm

(1)

Note that the stiffness given in (1) was used rather than theelastic modulus in order to take future applications to haptic
feedback systems into account.

Fig. 2. Schematic of stiffness.

A. Structure of sensing partFig. 3 shows the (a) overview, (b) photo, (c) side view, and(d) cross-sectional view of the proposed sensing part. The
sensing part is composed of a pin, a limiting component, asilicon part, and main body part, as shown in Fig. 4. These fourparts are connected with a fitting. The silicone part played a keyrole acting as a spring, and was made of silicone: KE-1308 andhardener (volume: 6 %): CAT1300L-3 (Shin-etsu Silicone, see[22], [23] for details on the stiffness information; Fig. 17 (a)shows the material information for the relationship between theload and displacement). Note that the viscosity of this material(silicone) has negligible difference compared to other polymer
materials (e.g., [24]). This is one reason why this material waschosen. The silicone part was transparent so that the light source
could transmit through the sensing part. The pin, limiting
component, and main body part (ABS plus: Young's Modulus:
2.2 [GPa]) were manufactured by a 3D printer (Stratasys Uprint
SE). The pin had a spherical-shaped head and a cylindricalextension. The pin head was a sphere having a diameter of 8[mm]. It was designed such that the distance between the top of

Copyright (c) 2016 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at
http://dx.doi.org/10.1109/JSEN.2016.2574322> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <the pin and the limiting component ( ) was 7 [mm] when thepin reached the limitation of displacement, as shown Fig. 3 (d).
is defined as the point at which the pin reaches the limit. It
should be noted that the distance between the top of the pin andthe limiting component ( ) was set as 7 [mm]; however,
7 wasbecause of a manufacturing and processing error,
not always convenient. Therefore, was used to represent thisdistance in the later discussion.
Fig. 5 shows a schematic of the pins making initial contact
with an object (left side) and reaching their limitation point(right side). Fig. 5 also shows the corresponding images
captured at these stages. When the pin head makes contact withan object or organ, the silicone part deforms, and the pin
displaces based on the deformation. By capturing thedisplacement of the pin via camera, force or displacement canbe estimated.

(a)

(b)

(c)

(d)

3

the relationship between the displacement of the pin [mm] andthe deformation of the object [mm] is represented by(2)
Therefore, if andat the limitation point
can beobtained, then the stiffness given in (1) can be calculated
utilizing (2). In sum, the necessary information for stiffness
estimation is as follows:
1. Detection of the limit
2. Estimation of the displacement and the load at
1) Detection of the limitation point
A light source is assumed to be located outside the sensing
part. When the images at the initial state (where the pin head
initially contacts the object) and limitation point (see Fig. 5)
are compared, it can be seen that the brightness clearly differs.
This is due to the difference in the transmissivity of light fromthe light source. The yellow arrows in Fig. 5 describe a modelof this principle. At the initial state, there is space for lighttransmission. Because of the transparency of the silicone, thelight can easily transmit inside the sensing part, and an image
having high brightness can be obtained. On the other hand, atthe limitation point , the object covers the limitation structure
and there is little space for light transmission. The amount oflight that can transmit through the sensing part then decreases
significantly. Therefore, we get an image with low brightness.
By detecting the change in the brightness, the limitation point
can be obtained.

Fig. 3. The proposed sensing part (a) overview; (b) photo; (c) side view; (d)cross-sectional viewFig. 5 Schematic of the pins making initial contact with an object (left side) andreaching their limitation point (right side), and the corresponding images
captured at these stages.

Fig. 4. The components of the sensing partB. Principle for stiffness estimation
Let be the displacement of the pin as shown in Fig. 5. Here,consider the case when the pin reaches its limit . In this case,2) Estimation of the displacement and the load at thelimitation point
Load [N] and the displacement [mm] can be estimated by
the same strategy as presented in our previous works [1]–[3],although different image processing methods are required torealize the strategy because of the different brightness
conditions. If a load is applied to the pin head, then the pin
displaces based on the deformation of the silicone part. The
captured area for the pin changes from(initial area withoutload) to , as seen in Fig. 5. Therefore, the load can be
if we know the relation between andderived from. Note that the relation includes the nonlinear elastic

Copyright (c) 2016 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at
http://dx.doi.org/10.1109/JSEN.2016.2574322> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <(a) Color image(b) Greyscale image
Fig. 6 Evaluation area for detecting the limitation point ; (a) the captured
color image, (b) converted greyscale image highlighting the evaluation area

be clear. The centers for both semicircles coincided with thecenter of the pin. The procedure for the evaluation was asfollows:
1. The captured image was converted to a grayscale image.
2. The mean of the pixel values in the evaluation area ( )was calculated.
The obtained
value was the target for evaluating thebrightness. The threshold for detecting the limitation point
was experimentally derived.

Pixel value for evaluation area [-]property of the silicone part at the sensing part. By a similar
method, can also be estimated. Note that the three-axis force
can theoretically be measured as proposed in a previous study[2]. The main purpose in this paper pertains to the estimation of
stiffness, specifically in the longitudinal direction of the pin.

4

80
70
60
50
40
30
20
6

6.1

6.2

6.3

6.4

6.5

6.6

Displacement of the force gauge
from the initial state ( ) [mm]Fig. 8. Relationship between the pixel value for the evaluation area and thedisplacement of the force gauge from the initial state ( ).

Fig. 7 Schematic view of the experimental setup

III. PROCEDURE FOR DETECTING THE LIMITATION POINT
When the brightness of the captured image is lower than thethreshold, it signifies that the pin head has reached thelimitation point . The key factors involved in applying thismethod include the area for brightness evaluation and thethreshold. The brightness of the captured image is not uniform
because of the position of light source. As shown in Fig. 5, thelight source was located such that the light came from the rightand transmitted towards the left. Therefore, a relatively largechange in brightness could be obtained in the left half of theresulting image. If evaluating both left and right areas of animage, the change in brightness could be unclear. Additionally,
there was little change in the brightness around pin. Therefore,
we defined the evaluation area to be that shown in Fig. 6. The
evaluation area was constructed by subtracting a smaller leftsemi-circle area (radius: 80 [pixel]) from a larger semi-circle
area (radius: 160 [pixel]) so that the change in brightness could

A. Experiment for determining the threshold
Fig. 7 shows a schematic view of the experimental setup. The
sensing part was attached to the camera (Elecom UCAMDLT30HSV: resolution: 640 480 [pixel]). Note that here thecamera was used instead of the endoscope to simplify the setup.
A force gauge (IMADA DS2-500N) was attached to theautomatic positioning stage 1 (SIGMAKOKI TAMM40-10C).
The flat plate was attached to the tip of the force gauge. The
automatic positioning stage 1 was also attached to the automatic
positioning stage 2 (IMADA MX2-500N). The role ofautomatic positioning stage 1 was precise motion, while the roleof automatic positioning stage 2 was large motion. A black-out
curtain was used to block the ambient light in order to isolate
the light from the light source (MITSUTOYO Megalight100)
located to the right of the sensing part. The force gauge wasadjusted until the pin head of the developed sensing part waspressed up to the flat plate. Initially, the force gauge was set toa value of 0.00 [N] when the plate just barely contacted thesensing part. Then, the automatic positioning stages were
controlled to increase the displacement by increments of 1.00[mm], until the total displacement reached 6.00 [mm]. After
that, the displacement increased by increments of 0.10 [mm]until the total displacement reached 6.60 [mm]. It should be
noted that when the total displacement reached 6.60 [mm], asudden and substantial increase in load was observed. At thispoint, we stopped increasing the displacement. This indicatedthat the contact between the limiting structure and the plate
occurred when the total displacement changed from 6.50 to 6.60

Copyright (c) 2016 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at
http://dx.doi.org/10.1109/JSEN.2016.2574322> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <[mm]. At each step, the value of the force gauge and the moving
distance of the automatic positioning stage ( ) were recorded,
and a photo was taken with the camera. These experimental
results were used not only to determine the deviation of thethreshold for detecting the structural limitation but also for
estimating the load and the displacement from the image ofpin.
Let be the pixel value for the evaluation area at thelimitation point . Here, we tried to derive not only thethreshold but also (the distance between the top of the pin
and the limiting part), taking the manufacturing and processing
errors into account. For this, we used the data from when the
total displacement went from 6.00 to 6.60 [mm]. Fig. 8 shows
the obtained relationship between the pixel value for theevaluation area and the displacement of the force gauge fromthe initial state ( ). It was confirmed that the contact between
the limiting structure and the plate occurred when the total
displacement went from 6.50 to 6.60 [mm]. The corresponding
pixel values for the evaluation area were
32.8 at 6.50[mm] and31.2 at 6.60 [mm]. We set
32 to be thethreshold value for detecting the limitation point.
6.50 [mm] in (1). Note that this value
Additionally we set
was different from the designed value 7 [mm] (see Fig. 3 (d)).
This was due to manufacturing and processing errors.

5

in Fig. 5, the target image (at the limitation point ) was dark
and is not easy to derive the area of the pin. A method that
employs another light source located inside the sensing part,
could potentially be used. However, the ability to derive thearea of the pin from the dark image, would enable constructionof a simpler system. Therefore, we challenged the problem byusing the following algorithm (see Fig. 9):
Step 1. Convert to greyscale image and cut the right half areaof pin.
Step 2. Adjust brightness.
Step 3. Extract the area including left lower area for pin.
Step 4. Adjust contrast
Step 5. Binarize
Step 6. Invert black and white areas
Step 7. Make the boundary clear and extract left lower areaof pinFig. 10. The evaluation areas for the brightness adjustment.

Fig. 9 Procedure for deriving the left lower area for pin ().

IV. PROCEDURE FOR ESTIMATING THE LOAD

AND THEDISPLACEMENT

Because the difference in the area of the pin from its initialsate,, corresponds to a load and a displacement , weand ( ).
considered deriving the relationship between
However, as mentioned the above, the image around pin is not
uniform because of the position of light source. Therefore, we
derived the left lower area for pin
instead of , because theleft lower area for pin was relatively stable and uniform. Weand ( ), letting
then used the relationship between
berepresent the initial state. The left lower area for thepin is a quarter circle, so(/4. As shown

At Step 1, as a preprocessing step for extracting the left lower
area for pin, the color image was converted to a greyscale image.
The right half of the image was cut, but the left upper area was
not cut because it was used for brightness adjustment.
In order to extract a quarter circle area for the pin from a dark
image, we conducted two-step adjustments. In the first step, we
adjusted the brightness. In Step 2, we defined the reference
pixel value for the brightness adjustment. The purpose of thisadjustment was to increase the total brightness of the image andto remove the effects of shadows. was set to be the mean
pixel value of the evaluation area where the shadow effect wasstrong, as shown in Fig. 10 (left upper arced area with a radiusfrom 80 to 160 [pixel]). Letandbe the at the initialstate of the nominal case shown in Fig. 7 and the limitation
point . In order to letbe the constant stable value, weused at the initial state of the nominal case (when contacting
flat plate) shown in Fig. 7. Let , be the image pixel value atof the image after Step 1, and , be the image
the point ,pixel value after the brightness adjustment. Then,,min 255, min

,

,

(3)

where
is the specified value used to prevent the brightness
from being too high. We set
9 when at the initialstate was over 210 and
8 when it was under 210.
After cutting the left upper area of the pin in Step 3, we

Copyright (c) 2016 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at
http://dx.doi.org/10.1109/JSEN.2016.2574322> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <adjusted the contrast in Step 4, in order to make the boundaryof area for pin clear. Letting , be the image pixel value afterStep 3, and , be the image pixel value after the contrast
adjustment, we used the following equation:255
min 255,,,(4)

max 0,,,

6

displacement , the following regression curve was used:

(6)
where
0.663 ,
4.15 , and b
0.265 . The
coefficient of determination was 0.99. It can be seen that the
regression curve represented the relationship very well.
Note that (5) and (6) include the nonlinear elastic property ofthe silicone part at the sensing part.
7

min 3x,y
x,y

Here,
and
(
) represent the range foradjustment, corresponding to 0 and 255 in the adjusted
240 in this study.
image, respectively. We set
After the contrast adjustment, binarization was conducted in
Step 5, utilizing a modified Otsu method [25]. We took into
account only pixel values under 230 in the Otsu method [25].
In Step 6, the image was inverted to get the information of thearea of the pin. Finally, in Step 7, we extracted the left lower
area for pin after making the boundary clear by computing theconvex hull of the area.
8

Displacement ( l ) [mm]

where

6
5
4
3
2
1
0
0

0.5

1

1.5

2

2.5

3

The difference of the left lower circular area of the[10 pixel]
pin from the initial stateFig. 12. Relationship between the difference of the left lower circular area ofthe pin from the initial state
and the displacement .

Load () [N]76

V. EXPERIMENTAL VALIDATION OF STIFFNESS ESTIMATION

5
4
3
2
1
0
0

0.5

1

1.5

2

2.5

3

The difference of the left lower circular area of thepin from the initial state[10 pixel]
Fig. 11. Relationship between the difference of the left lower circular area ofthe pin from the initial state
and the load .

A. Relationship between the area for pin and the load (thedisplacement )We used the data obtained from the experiments described in
the previous section to get the relationship between the load(the displacement ), and the difference of the left lower circular.
wasarea of the pin from the initial state,
derived by the method described above. Fig. 11 shows theand the load , while Fig. 12
relationship between
shows the relationship between
and thedisplacement .
and the load , theFor the relationship between
following regression curve was used:

(5)
where
0.726 ,
4.69 , and
0.151 . The
coefficient of determination was 0.99. It can be seen that the
regression curve represented the relationship very well.
For the relationship between
and theIn order to see the validity of the developed system, we
conducted experiments estimating the stiffnesses of objects
made of silicone (silicone: KE-1308 and hardener (volume:6 %): CAT1300L-3, produced by Shin-etsu Silicone) andmelamine. The experiment not only checks whether or not the
stiffness value can be obtained but also sees whether or not the
proposed sensing system can detect the difference between hard(silicone) and soft (melamine) objects, supposing abnormal andnormal objects. As shown in Fig. 13, the target objects wereprismatic columns with a height of 11 [mm]. The base prism
was inscribed in a circle with a radius of 11 [mm]. The shape
was set so that the objects could match the outside of the sensing
part (see Fig. 3). Note that this setting is not a discrepancy withJIS K 6254 corresponding to ISO 7743 (Rubber, vulcanized or
thermoplastic—Determination of stress-strain properties). Fig.
14 shows the schematic view of the experiment, which is thesame as that shown in Fig. 7, except for the existence of a soft
object between the metal plate and the sensing part. Initially,
once the soft object (silicone or melamine) had just contacted
the sensing part, the force value was set to 0.00 [N]. Then, theautomatic positioning stages were controlled to increase thedisplacement by 1.00 [mm], until the total displacement
reached 6.00 [mm]. After that, the automatic positioning stages
were controlled to increase the displacement by 0.10 [mm],until the total displacement reached 6.60 [mm]. At each step,
the value of the force gauges and the moving distance of theautomatic positioning stage ( ) were recorded, and a photo wastaken by the camera. The experiment was repeated 5 times foreach sample.

Copyright (c) 2016 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at
http://dx.doi.org/10.1109/JSEN.2016.2574322> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <

7

(a)
(b)
Fig. 13 Photos of the target objects in the experiment; (a) silicone and (b)melamine form

Fig. 16 Schematic view of the experimental setup for obtaining actual stiffness.
The relationship between the load and the displacement was obtained.
10

Load ( ) [N]

8
6
4
2
0
0

2

Displacement (46) [mm](a) Target object was silicone
2.5

Load ( ) [N]2

Fig. 14 Schematic view of the experimental setup for estimating the stiffnessesof objects made of silicone and melamine form.

0.5

0

2

Displacement (

1.2

4

6

) [mm](b) Target object was melamine form
Fig. 17 The relationship between the load ( ) and the displacement (silicone and (b) melamine form.

1); (a)

0.8

0.4
0.2
0

Estimated one

Actual one(a) Target object was silicone
1.4
1.2
1

/

) [N/mm]0.6

Stiffness (Stiffness [N/mm]

1

0

1.4

Stiffness [N/mm]

1.5

14
12
10
8
6
4
2
0
0
0.5
1
1.5
2
2.5
The difference of the left lower circular area of thepin from the initial state[10 pixel]

0.8
0.6

Fig. 18 The relationship between the difference of the left lower circular area
and the stiffness / , with theof the pin from the initial stateexperimental values; It indicates the resolution of the developed sensing
system.

0.4
0.2
0Estimated one

Actual one(b) Target object was melamine form
Fig. 15 Results of stiffness estimation and experimentally obtained actual
stiffness values; (a) silicone and (b) melamine form

Using the previously described methods, we derived thedifference of the left lower circular area of the pin from the, the load(see (5)), and theinitial state,
displacement (see (6)). Utilizing (1) and (2), we derived the

Copyright (c) 2016 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at
http://dx.doi.org/10.1109/JSEN.2016.2574322> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <stiffness of the soft objects. The obtained results are shown in
Fig. 15. For comparison, actual stiffness values are also shown.
Actual stiffness values were obtained experimentally. Fig. 16
shows the schematic view of the setup for the experiment. A
sphere whose size and material were the same as the pin head
was fixed to a table, instead of the sensing system. This part wasdifferent from the setup shown in Fig. 14. The other settings
were the same. The procedure was also the same. Initially, theforce was set to 0.00 [N] when the soft object (silicone ormelamine form) just barely contacted the sphere. Then, theautomatic positioning stages were controlled to increase thedisplacement until the total displacement reached 6.00 [mm].
The value of force gauges and the moving distance of automatic
positioning stage ( ) were recorded at an interval of 0.0083[mm]. Fig. 17 shows the obtained relationship between the load
and the displacement . Because the stiffness varied with thedisplacement , the relationship could be obtained. In theexperiment of stiffness estimation, the actual displacement ( )information was obtained. By letting be , we derived theactual and corresponding stiffness from the relationship shownin Fig. 17.
From Fig. 15, it can be seen that there was little difference
between the estimated and the actual stiffness values. Although
the variation for the estimated stiffness values was larger thanthat of the actual stiffness values, it was still small. In addition,
the difference in stiffness could be detected. These results provethe validity of the developed sensing system.
From (1), (2), (5), and (6), we can obtain the relationshipbetween the stiffness and the difference of the left lower circular:area of the pin from the initial sate

[N/mm](7)

Fig. 18 shows the relationship with the experimental values.
This relationship indicates the resolution of the developed
sensing system. It can be seen that as the stiffness of the target
increases, the resolution becomes lower. If the stiffness exceeds
12 [N/mm], it is not easy to estimate the stiffness with enoughresolution. We then show the relationship for the stiffness tohave a range of 0 to 15 [N/mm]. In other words, the range of thedeveloped system is 0 to 12 [N/mm]. This means that the
stiffness from 0 to 2 [N/mm] is expected to be estimated withhigh accuracy. This is consistent with the results shown in Fig.
15.

estimated with image processing. The displacement of the pin
was structurally limited by the limiting component. By
estimating the load at the limiting point, the stiffness of thetarget object could be estimated. The limiting point was
detected by a brightness change of the captured image. The lightsource was located outside the sensing system. Because of thetransparency of the silicone, images having high brightness
could be captured at the initial state of examination, and images
having reduced brightness could be captured at the limiting
point. By detecting the change in brightness, the limiting point
was detected. An image processing methodology was alsopresented to realize the mechanisms. The experimental results
discriminating silicone and melamine objects showed thevalidity of the developed system. We effectively usedtransparent silicone as the spring. The range and resolution
depend on the material properties of the silicone. By changing
the material or hardness, we would have another range orresolution for the sensing system. The sensor configuration
should be compatible for attachment depending on the size ofendoscopes or cameras used. Identifying various ranges andresolutions of the system, in addition to optimally sizing sensors
will be part of our future works.
REFERENCES
[1]

[2]

[3]

[4]
[5]
[6]

[7]

VI. CONCLUSION
This paper presented a novel stiffness-sensing system thatworked by attaching a sensing part to endoscopes or cameras.
The system provides a method for investigating the stiffness oftissues or objects in deep areas that can only be observed withendoscopes in order to detect abnormalities. The sensing partconsisted of a pin, a limiting component, a transparent silicon,
and body parts. When pressing a target object with the pin, thepin displaces based on the magnitude of load because ofsilicone’s mechanical spring property. The area of the pin in thecaptured image corresponded to the load and the displacement.
Therefore, the values of load and displacement could be

8

[8]
[9]

[10]

T. Watanabe, T. Iwai, Y. Fujihira, L. Wakako, H. Kagawa, and T.
Yoneyama, “Force Sensor Attachable to Thin
Fiberscopes/Endoscopes Utilizing High Elasticity Fabric,” Sensors,
vol. 14, no. 3, pp. 5207–5220, Mar. 2014.
T. Iwai, Y. Fujihira, L. Wakako, H. Kagawa, T. Yoneyama, and T.
Watanabe, “Three-axis force visualizing system for fiberscopes
utilizing highly elastic fabric,” in 2014 IEEE/ASME International
Conference on Advanced Intelligent Mechatronics, 2014, pp. 1110–
1115.
T. Iwai, T. Koyama, H. Kagawa, T. Yoneyama, and T. Watanabe,“Visualization method based stiffness sensing system forendoscopes,” in 2015 37th Annual International Conference of theIEEE Engineering in Medicine and Biology Society (EMBC), 2015,
pp. 6449–6452.
R. S. Dahiya, G. Metta, M. Valle, and G. Sandini, “Tactile
Sensing&#x2014;From Humans to Humanoids,” IEEE Trans.
Robot., vol. 26, no. 1, pp. 1–20, Feb. 2010.
M. I. Tiwana, S. J. Redmond, and N. H. Lovell, “A review of tactile
sensing technologies with applications in biomedical engineering,”
Sensors Actuators A Phys., vol. 179, pp. 17–31, 2012.
P. Puangmali, K. Althoefer, L. D. Seneviratne, D. Murphy, and P.
Dasgupta, “State-of-the-art in force and tactile sensing forminimally invasive surgery,” Sensors Journal, IEEE, vol. 8, no. 4,
pp. 371–381, 2008.
E. P. Westebring – van der Putten, R. H. M. Goossens, J. J.
Jakimowicz, J. Dankelman, E. P. der Putten, R. H. M. Goossens, J.
J. Jakimowicz, J. Dankelman, E. P. Westebring – van der Putten, R.
H. M. Goossens, J. J. Jakimowicz, and J. Dankelman, “Haptics inminimally invasive surgery-a review,” Minim. Invasive Ther. Allied
Technol., vol. 17, no. 1, pp. 3–16, Jan. 2008.
E. Vander Poorten, E. Demeester, P. Lammertse, E. V. B. Vander
Poorten, E. Demeester, and P. Lammertse, “Haptic feedback for
medical applications, a survey,” Proc. Actuator, pp. 18–20, 2012.
T. Yoneyama, T. Watanabe, H. Kagawa, J. Hamada, Y. Hayashi,
and M. Nakada, “Force-detecting gripper and force feedback system
for neurosurgery applications,” Int. J. Comput. Assist. Radiol. Surg.,
vol. 8, no. 5, pp. 819–829, Sep. 2013.
Y. Kanada, T. Yoneyama, T. Watanabe, H. Kagawa, N. Sugiyama,
K. Tanaka, and T. Hanyu, “Force Feedback Manipulating System
for Neurosurgery,” in Procedia CIRP, 2013, vol. 5, no. 1, pp. 133–
136.

Copyright (c) 2016 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at
http://dx.doi.org/10.1109/JSEN.2016.2574322> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
[11]
[12]

[13]

[14]

[15]

[16]
[17]
[18]

[19]

[20]
[21]

[22]

[23]

[24]
[25]

Y. Fujihira, T. Hanyu, Y. Kanada, T. Yoneyama, T. Watanabe, and
H. Kagawa, “Gripping Force Feedback System for Neurosurgery,”Int. J. Autom. Technol., vol. 8, no. 1, pp. 83–94, 2014.
T. Takaki, Y. Omasa, I. Ishii, T. Kawahara, and M. Okajima, “Forcevisualization mechanism using a Moir&#x00E9; fringe applied to
endoscopic surgical instruments,” in 2010 IEEE International
Conference on Robotics and Automation, 2010, pp. 3648–3653.
K. Tadano and K. Kawashima, “Development of 4-DOFs forceps
with force sensing using pneumatic servo system,” in Robotics andAutomation, 2006. ICRA 2006. Proceedings 2006 IEEE
International Conference on, 2006, pp. 2250–2255.
D. Haraguchi, K. Tadano, and K. Kawashima, “A prototype of
pneumatically-driven forceps manipulator with force sensing
capability using a simple flexible joint,” IEEE Int. Conf. Intell.
Robot. Syst., pp. 931–936, 2011.
J. Peirs, J. Clijnen, D. Reynaerts, H. Van Brussel, P. Herijgers, B.
Corteville, and S. Boone, “A micro optical force sensor for force
feedback during minimally invasive robotic surgery,” Sensors
Actuators A Phys., vol. 115, no. 2–3, pp. 447–455, 2004.
M. Tada, S. Sasaki, and T. Ogasawara, “Development of an optical
2-axis force sensor usable in {MRI} environments,” Proceedings ofIEEE Sensors, vol. 2. pp. 984–989, 2002.
M. Ohka, Y. Mitsuya, Y. Matsunaga, and S. Takeuchi, “Sensing
characteristics of an optical three-axis tactile sensor under combined
loading,” Robotica, vol. 22, no. 2, pp. 213–221, Mar. 2004.
K. Kamiyama, K. Vlack, T. Mizota, H. Kajimoto, K. Kawakami,
and S. Tachi, “Vision-based sensor for real-time measuring of
surface traction fields,” IEEE Comput. Graph. Appl., vol. 25, no. 1,
pp. 68–75, Jan. 2005.
C. Chorley, C. Melhuish, T. Pipe, and J. Rossiter, “Development ofa tactile sensor based on biologically inspired edge encoding,” inAdvanced Robotics, 2009. ICAR 2009. International Conference on,
2009, pp. 1–6.
T. Kawahara, S. Tanaka, and M. Kaneko, “Non-contact stiffness
imager,” Int. J. Rob. Res., vol. 25, no. 5–6, pp. 537–549, May 2006.
T. Fukuda, Y. Tanaka, M. Fujiwara, and A. Sano, “Softness
measurement by forceps-type tactile sensor using acoustic
reflection,” in 2015 IEEE/RSJ International Conference onIntelligent Robots and Systems (IROS), 2015, pp. 3791–3796.
T. Watanabe and Y. Fujihira, “Experimental investigation of effectof fingertip stiffness on friction while grasping an object,” in 2014
IEEE International Conference on Robotics and Automation(ICRA), 2014, pp. 889–894.
Y. Fujihira, K. Harada, T. Tsuji, and T. Watanabe, “Experimental
investigation of effect of fingertip stiffness on resistible force ingrasping,” in 2015 IEEE International Conference on Robotics andAutomation (ICRA), 2015, pp. 4334–4340.
T. K. Kim, J. K. Kim, and O. C. Jeong, “Measurement of nonlinear
mechanical properties of PDMS elastomer,” Microelectron. Eng.,
vol. 88, no. 8, pp. 1982–1985, Aug. 2011.
N. Otsu, “A Threshold Selection Method from Gray-Level
Histograms,” IEEE Trans. Syst. Man. Cybern., vol. 9, no. 1, pp. 62–
66, 1979.

Tetsuyou Watanabe (M’02) received the
B.S., M.S., and Dr.Eng. degrees inmechanical engineering from Kyoto
University, Kyoto, Japan, in 1997, 1999,
and 2003, respectively. From 2003 to 2007,
he was a Research Associate with the
Department of mechanical Engineering,
Yamaguchi University, Japan. From 2007
to 2011, he was an assistant professor with
Division of Human and Mechanical Science and Engineering,
Kanazawa University. Since 2011, he has been an associate
professor with Faculty of Mechanical Engineering, Institute of
Science and Engineering, Kanazawa University. From 2008 to
2009, he was a visiting researcher at Munich University ofTechnology. His research interests include mechanics and9control of robotic systems, medical sensors, surgical systems,
grasping, object manipulation, and micro-manipulations.
Dr. Watanabe has received several awards, including the2005 MAGDA Conference Best Paper Award from the Japan
Society of Applied Electromagnetics and Mechanics, the 2008
Young Investigation Excellence Award from Robotics Societyof Japan, the 2010 JSME Young Engineers Award from theJapan Society of Mechanical Engineers, and the 2012 SICE SI
division conference excellent presentation Award.
Dr. Watanabe is a member of the IEEE Robotics andAutomation Society and IEEE Engineering in Medicine &Biology Society. He is also a member of the Japan Society of
Mechanical Engineers, the Robotics Society of Japan, theJapanese Society of Medical and Biological Engineering and
the Society of Instrument and Control Engineers.
Takanobu Iwai received the B.S. and M.S.
degrees in mechanical engineering from
Kanazawa University, Kanazawa, Japan, in2014 and 2016. His research interests
include sensors, medical sensors, andsurgical systems.

Toshio Koyama received the B.S. degree
in mechanical engineering from Kanazawa
University, Kanazawa, Japan, in 2015. His
research interests include sensors, medical
sensors, and surgical systems.

Takeshi Yoneyama received the B.S.,
M.S. and Dr. Eng. degrees in mechanicalengineering from The University of Tokyo,
in 1979, 1981 and 1984 respectively.
From 1984 to 1991, he was a Research
Associate in the Department of MechanicalEngineering of Kanazawa University.
From 1991 to 2001, he was an Associate
Professor in the same department. From2001 he has been a Professor in the school of MechanicalEngineering, Faculty of Mechanical Engineering, Institute of
Science and Engineering, Kanazawa University. His researchinterests include sensors, machine design, surgical systems,
sports engineering, production engineering for plastic
deformation process, injection molding and composite material
processes.
Prof. Yoneyama received several awards, including 2004
Education Award from the Japan Society of MechanicalEngineers, 2006 Best Paper Award from the Japan Society ofTechnology of Plasticity.
Prof. Yoneyama is a member of the Japan Society of
Mechanical Engineers, the Japan Society of Technology of
Plasticity, the Japan Society for Precision Engineering,
Japanese Society for Medical and Biological Engineering

Copyright (c) 2016 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.