Vision-based Cutaneous Sensor to Measure Both Tactile and Thermal
Information for Telexistence
112

Katsunari Sato , Hiroyuki Shinoda , and Susumu Tachi
The University of Tokyo, 2Keio University, JAPAN

1ABSTRACTWe have proposed a vision-based cutaneous sensor fortelexistence that can simulate the physical interaction between ahuman fingertip and an object. The proposed sensor comprises afinger-shaped GelForce and a thermo-sensitive paint. The fingershaped GelForce enables us to measure tactile information interms of the distribution of forces that are calculated from thedisplacements of markers inside the sensor body. The thermosensitive paint is employed to measure thermal information on thebasis of its color, which changes according to its temperature. In
this study, we have described the design of the proposed
cutaneous sensor, constructed its prototype, and discussed its
efficiency for telexistence.
KEYWORDS:Telexistence.

Vision-based

sensor,Force,Temperature,

INDEX TERMS: H.5.2 [Information System]: User Interfaces—
Haptic I/O; I.2.9 [Computing Methodologies]: Robotics—Sensors
1INTRODUCTION

The transmission technology of haptic information is essential fortelexistence [1] (or telepresence) systems. When a humaninteracts with an object, he/she perceives the properties of theobject such as stiffness, weight, shape, texture, and temperature
through his/her haptic sense. Without haptic sense, a telexistencesystem user cannot perceive any property of an object that he/shetouches, thereby eliminating the reality of the telexistenceexperience. Haptic sensors and displays are required to realizetransmission of haptic information. The purpose of this study is todevelop a haptic sensor for telexistence systems that involve
robotic fingers.
Haptic sense can be divided into the following two types basedon the position of the receptors that acquire sensory information:kinesthetic and cutaneous senses. Conventional haptic sensors fortelexistence can also be classified into kinesthetic and cutaneous
sensors based on the measured information. A number of threeaxis force or torque sensors have been developed for application
as kinesthetic sensors for robotic fingers. Moreover, to date, it has
been difficult to develop cutaneous sensors primary because of
Katsunari Sato is with the Graduate School of Information Scienceand Technology, The University of Tokyo, Tokyo, Japan (phone: +813-5841-6927; e-mail: Katsunari_Sato@ipc.i.u-tokyo.ac.jp).
Hiroyuki Shinoda is with the Graduate School of Information
Science and Technology, The University of Tokyo, Tokyo, Japan (email: Shino@alab.t.u-tokyo.ac.jp).
Susumu Tachi is with the Graduate School of Media Design, Keio
University, Kanagawa, Japan (e-mail: Tachi@tachilab.org).

two problems. One of the problems is that cutaneous sensors arerequired to measure two types of physical information, i.e., forceand temperature. The cutaneous sense of a human consists oftactile and thermal senses that are required to perceive themechanical deformation and temperature of the skin surface,respectively. Therefore, cutaneous sensors are expected tomeasure tactile and thermal information, i.e., they should be able
to sense the applied forces and temperature changes on the sensor
surface simultaneously. The other problem is associated with theinteractivity of the haptic sensing. When a human actually touches
an object, deformation and thermal changes occur on both his/herskin and the surface of the object. Therefore, the sensor should
simulate the interaction caused by the contact of an object withthe human skin so as to measure cutaneous information correctly.
In order to appropriately simulate the interaction between thefingertip and an object, both the mechanical and thermal
properties of the sensor should be the same as those of thefingertip. In particular, it is important that the shape, size,compliance, temperature, and thermal contact coefficient [2] ofthe sensor are the same as those of the fingertip.
Conventional cutaneous sensors for telexistence systems
measure either tactile or thermal information. For example,
Maeno et al. developed a tactile sensor for the transmission oftextures of objects [3]. Sato et al. have also developed a force
sensor and constructed a transmission system for the spatial
distribution of force [4]. These sensors measure only tactile
information and not thermal information. On the other hand,
Guiatni et al. [5] proposed a thermal transmission system using aPeltier sensor and a display for transmitting realistic thermalinformation. However, this system could not sense and display
tactile information. Some previous studies have attempted tomeasure both the tactile and the thermal information [6][7][8].
These studies integrated a thermal sensing element, such as aPeltier element, with a tactile sensing unit. These sensors weredeveloped for an autonomously controlled robotic hand and not tosimulate a physical interaction between a fingertip and an object.
As a cutaneous sensor for telexistence, we propose a visionbased cutaneous sensor that can measure both the tactile and thethermal information. The proposed sensor can simulate theinteraction caused by the contact of an object with the human skinIn this paper, we introduce the design of the vision-based
cutaneous sensor and implement the proposed sensor.
2VISION-BASED CUTANEOUS SENSOR FOR TELEXISTENCE

2.1
Vision-based sensing method
We have proposed a vision-based cutaneous sensor comprising afinger-shaped GelForce [9] and a thermo-sensitive paint [10].
2.1.1
Finger-shaped GelForce
We use the finger-shaped GelForce to measure tactile information,in terms of the distribution of three-dimensional forces (or asurface traction field). This sensor consists of a transparent elastic
body, colored marker matrixes, and a color charge-coupled device(CCD) camera that captures the displacements of the markers.

When force is applied to the surface of the sensor, the markers aredisplaced in response to the magnitude and direction of theapplied force (Figure 1). The displacements of the markers arecaptured by the camera. We determine the center of the markers
before and after the application of the force in order to calculate
their degree of displacement. When we assume that thedeformation of the elastic body is linear, we find that therelationship between the two-dimensional displacements of mmarkers, u, and the three dimensional n-force vectors, f, isexpressed in terms of the conversion matrix H.

f  Hu(1)

In this equation, the 2m × 3n conversion matrix H is determinedfrom observational method [9]. First, we applied forces at eachforce sampling point and recorded the applied force. Then, wemeasured the movement of the markers. We estimated H from therecorded forces and the measured movement of the markers.
The finger-shaped GelForce does not require any electronic
components such as resistors or capacitors. Therefore, it is easy toadapt the shape and size of the GelForce to those of a humanfingertip. In addition, new developments in camera technology
may help us to improve the spatial and temporal resolutions of thesensor. Furthermore, the finger-shaped GelForce can easily
implement a thermal measurement method described in below
subsection.
2.1.2
Thermo-sensitive paintWe use the thermo-sensitive paint to measure thermal information.
The color of this paint changes on the basis of a change in its
temperature. This paint is used to make thermal information
visible and, hence, it is applied for the temperature managementof products such as food or industrial machines. Thermo-sensitive
paint is also used for analysis of fluids. In this study, we haveused thermo-sensitive paint to measure the change in thetemperature of the surface of the haptic sensor, for telexistence.
The thermo-sensitive paint is applied on the inner side of thesensor surface so that its color changes according to the change intemperature of the sensor surface.
In order to convert the data from a color to a temperature format,
we use the hue of the captured image. In a previous study [10], it
has been indicated that the change in the hue corresponds to thechange in temperature of the thermo-sensitive paint. Therefore,
we consider that the temperature of the sensor surface can becalculated from an equation developed for converting hue h totemperature T.

Figure 1. Displacements of markers in the elastic sensor bodycaused by force applied to the sensor surface [9].

T  g (h)(2)

In this equation, g(·) denotes the conversion function from the hueof the captured thermo-sensitive paint to its temperature.
As in the case of the finger-shaped GelForce, the thermal
measurement method involving the use of the thermo-sensitive
paint does not require any electronic components. Therefore, thismethod does not influence the physical properties of the fingershaped GelForce.
2.2
Design of vision-based cutaneous sensor
Figure 2 shows the configuration of the proposed vision-based
cutaneous sensor comprising the finger-shaped GelForce and thethermo-sensitive paint. This sensor consists of an elastic sheetcoated with the thermo-sensitive paint, colored makers, atransparent elastic body, a CCD camera, a heat source, and a lightsource. The camera detects the displacement of the markers andthe color of the thermo-sensitive paint; using this information we
can determine the tactile and thermal information of the sensor
surface.
We can control the shape, size, compliance (Young’s modulus),
temperature, and thermal contact coefficient of the elastic body tobe the same as that of a human fingertip. A heat source is used toensure that the temperature of the sensor is the same as that of thehuman fingertip. Therefore, the proposed sensor can simulate theinteraction caused by the contact of an object with the human skin.
3EVALUATION

We implemented a prototype of the proposed vision-based
cutaneous sensor to ensure that it can simultaneously measure thedistribution of the applied force and temperature.
3.1

Implementation of prototype

3.1.1
Components
Figure 3(a) shows the constructed prototype. The shape of thetransparent elastic body is partly spherical and partly cylindricalresembling a human fingertip. The dimensions of the elastic bodyare 18 mm × 9 mm × 19 mm. The transparent elastic body iscomposed of urethane gel (Human Skin Gel, Exseal Corp.) owing
to the fact that its Young’s modulus (5.0 × 104 Pa) is
approximately the same that of a human fingertip (1.36 × 105 Pa
for the epidermis, 8.0 × 104 Pa for the dermis, and 3.4 × 104 Pa forthe subcutaneous fat [11]).
However, the thermal contact
coefficient of this material (urethane rubber; 400–600 J/m2s1/2K)
differs from that of the human skin (1,181 J/m2s1/2K [12]). In
future studies, we must appropriately select the material of theFigure 2. Configuration of proposed vision-based cutaneoussensor.

elastic body by considering its thermal properties. The frame rateand resolution of the CCD camera (ViewPlus, Inc.) are 15Hz and640 pixel × 480 pixel (VGA), respectively. Figure 3(b) shows animage captured by the CCD camera. 16 red and 8 green circularmarkers are arranged at different depths of the transparent elasticbody. The diameter of the markers is 1.0 mm and the centers ofthe markers are separated by 3.0 mm. We have used a thermosensitive liquid sheet (c-task, Inc., JP) for thermo-sensitive paint.
The 3.0 mm × 3.0 mm liquid sheet is placed at the center of theelastic sheet. The thickness of the elastic sheet is 0.5 mm. The
light source comprises 8 LEDs. The temperature of the sensor ismaintained at approximately 35°C by the heat generated from theLEDs when the surrounding air temperature is 26°C.
3.1.2
Force calculation
The proposed sensor calculates the applied forces from thedisplacement of the markers. The positions of the markers arecalculated as the centroid of each R and G element of the captured
RGB format image. Employing the observational method
described in [9], we allocated 12 (3 × 4) force sampling points,
which are 3.0 mm apart.
We evaluated the accuracy of the proposed sensor in measuring
the applied force using the constructed prototype sensor. We fixed
the prototype sensor to an xyz stage (VSQ-601XYV, Chuo
Precision Industrial Co. Ltd.) and applied a force of 0–500 gf
vertically to the sensor surface by increasing its magnitude insteps of 50 gf. The force was applied by a six-axis force sensor(BL NANO, BL Autotech Ltd.) with a cylindrical probe ofdiameter 10 mm. We recorded the measured force using the sixaxis force sensor (applied force) and the total force at every
sampling point (calculated force). The measurement procedure
was repeated five times. Figure 4 shows the evaluation results.
The graph shown in this figure indicates a liner relationshipbetween the applied force and the calculated force, confirming theforce measurement accuracy of the prototype sensor.

have cropped a 10 pixel × 10 pixel block from the captured imageand calculated the average hue within the block. The center of thecropped image is calculated from the positions of the markers at
the four corner of the thermo-sensitive liquid sheet. The camera
image is captured in an RGB format; it is then converted it to an
HSV format using OpenCV (Open Source Computer Vision)
library.
To develop a function to convert the hue of the captured image
to the temperature of the sensor surface (g(·) in equation (2)), weplaced the center of the sensor surface on a cool plate (NCP-2215,Nissinrika Corp.). The temperature of the cool plate was increased
from 31 to 36°C at intervals of 0.5°C. After the temperature of thecool plate was fixed at the target value, the contact between thesensor and the plate was maintained for more than 120 sec. The
hue values of the thermo-sensitive liquid sheet were recorded 10times for each temperature. Figure 5 shows the result of thecalibration. From the graph shown in figure, we can confirm thatthe function of hue and temperature monotonically decreases inthe temperature range of 31.5–35°C.
3.2
Measurement of cutaneous information
We measured the applied force and temperature simultaneously
using the constructed prototype sensor. We fixed the prototype
sensor to an xyz stage, and pressed the surface of the sensor
against to an aluminum plate suspended in air at 26°C. Force wasapplied to the sensor at two different locations, by pressing thealuminum plate vertically to the sensor surface, producing
displacements of approximately 1.0 mm and 2.0 mm. Then, werecorded the temporal changes in the applied force andtemperature measured by the prototype sensor at intervals of 0.1 s
when the temperature was 32–35°C.

3.1.3
Thermal calculation
In order to calculate the temperature of the sensor surface, weFigure 4. Relationship between applied force and calculated
force by prototype sensor.

Figure 3. (a) Implemented prototype of the vision-based
cutaneous sensor and (b) captured image.

Figure 5. Relationship between temperature of the sensor
surface and hue of thermo-sensitive liquid sheet.

Figures 6 and 7 show the recorded results. In Figure 6, theapplied forces and temperatures are represented by arrows andtheir colors, respectively. The color continuously changes fromred at 36°C, to green at 34°C, to blue at 32°C. The measured
forces do not appear to change even when the temperaturechanges. In Figure 7, the solid and dashed lines represent the total
force and temperature, respectively. The magnitude of force isconstant irrespective of the change in temperature. The decreasein temperature was larger when the aluminum plate caused a
displacement of 1 mm than the decrease when it caused a
displacement of 2 mm. When the aluminum plate was pressedagainst sensor, the density of the sensor increased. From thethermal conductivity theory, we infer that the decrease intemperature is more pronounced with an increase in the density.
However, the result of our measurement does not agree with thistheory. We hypothesize that excess heat from the heat source
could cause this unexpected disparity. Therefore, in the future, wewill study this effect and make required changes to the design ofthe sensor.
From these measurements, we can confirm that the prototype
sensor can measure tactile and thermal information
simultaneously and independently.
4CONCLUSION

In this paper, we proposed a vision-based cutaneous sensor tomeasure tactile and thermal information simultaneously. The
proposed sensor comprises a finger-shaped GelForce and thermosensitive paint, enabling it to simulate the physical interactions
between a human fingertip and an object so as to correctly
measure cutaneous information. We implemented a prototype of
the proposed sensor and confirmed that it could measurecutaneous information. However, the thermal measurement
method was not adequately implemented and evaluated. We willfurther develop the thermal measurement method to realize acutaneous sensor for telexistence. Then, we will also develop acutaneous display that will be able to present both tactile andthermal senses.
ACKNOWLEDGMENT
This work was partly supported by Grant-in-Aid for CREST andJSPS Fellows (20•10009).
REFERENCES
[1]

[2]

[3]

[4]

[5]

[6]

Figure 6. Visualized results of force and temperature
measurement.

Tachi, S., Kawakami, N., Nii, H., Watanabe, K. and Minamizawa, K,“TELEsarPHONE:MutualTelexistenceMasterSlave
Communication System based on Retroreflective Projection
Technology,” SICE Journal of Control, Measurement, and System
Integration, Vol.1, No.5, pp.335–344, 2008.
J. A. Businger and K.J.K. Buettner, “Thermal Contact Coefficient (ATerm Proposed for Use in Heat Transfer),” Journal of AtmosphericSciences, vol. 18, No. 3, pp.422–422, 1961.
T. Yamauchi, S. Okamoto, M. Konyo, Y. Hidaka, T. Maeno and S.
Tadokoro, “Real-Time Remote Transmission of Multiple Tactile
Properties through Master-Slave Robot System,” Proc. of the 2010
IEEE Int’l Conf. on Robotics and Automation (ICRA'10), pp. 1753–1760, 2010.
K. Sato and S. Tachi, “Evaluation of Transmission System for
Spatially Distributed Tactile Information,” Proc. of EuroHaptics
2010, Part I, pp. 279–284, 2010.
M. Guiatni, A. Benallegue, and A. Kheddar, “Thermal Display for
Telepresence Based on Neural Identification and Heat Flux Control,”Presence, Vol. 18, No. 2, 2009.
D. Siegel, I. Garabieta, and J. M.Hollerbach. “An integrated tactileand thermal sensor,” Proc. of Int’l Conf. on Robotics andAutomation (ICRA ’86), pp. 1286–1291, 1986.

Figure 7. Recorded force and temperature measured by theprototype sensor.
[7]G. J. Monkman and P. M. Taylor, “Thermal Tactile Sensing”, IEEETrans. on Robotics and Automation, Vol. 9, No. 3, pp. 313–318,
1993.
[8] D. Taddeucciand C.Laschi, R.Magni, P.Dario, and A.Starita, “Anapproach to integrate tactile perception,” Proc. of IEEE Int’l Conf.
on Robotics and Automation (ICRA’96), pp. 3215–3221, 1996.
[9] K. Sato, K. Kamiyama, N. Kawakami, and S. Tachi, “Finger-Shaped
GelForce: Sensor for Measuring Surface Traction Fields for RoboticHand,” IEEE Trans. on Haptics, Vol. 3, No. 1, pp.37–47, 2010.
[10] M. Ozawa, U. Miller, L.Kimura, and T. Takamori, “Flow andTemperature Measurement of Natural Convection in a Hele-Shaw
Cell Using a Thermo-Sensitive Liquid Crystal,” Experiments inFluids, Vol. 12, pp. 213–222, 1992.
[11] T. Maeno, K. Kobayashi, and N. Yamazaki, “Relationship between
the structure of human finger tissue and the location of tactile
receptors,” JSME Int’l Journal, Vol. 41, No. 1, pp. 94–100, 1998.
[12] L. A. Jones and H.-N. Ho, “Warm or Cool, Large or Small? The
Challenge of Thermal Displays,” IEEE Trans. on Haptics, Vol. 1,No. 1, 2008.