International Conference on Artificial Reality and TelexistenceEurographics Symposium on Virtual Environments (2018)
G. Bruder, S. Cobb, and S. Yoshimoto (Editors)

Soft Finger-tip Sensing Probe Based on Haptic Primary Colors
Fumihiro Kato†1, Yasuyuki Inoue‡1

and Susumu Tachi§1

1 Institute of Gerontology, The University of Tokyo, Japan

Figure 1: (a) acquisition signal from force, acceleration, temperature sensor, (b) construction design of tactile probe,(c) developed probe
Abstract
This paper describes a novel tactile sensing probe based on haptic primary colors (HPCs) and a tactile classifying system.
We developed a finger-type soft tactile probe incorporating a sensor to measure three physical quantities: force, vibration, andtemperature. We also constructed a tactile probe sliding system on the surface of the material repeatedly. The tactile fluctuation
obtained from the tactile probe was recorded, and a frequency analyzed image was generated. In the evaluation experiments,
the tactile images were generated by sliding the tactile probe on seven materials (ray fish skin, aluminum plate, rusting hemp
fabric, MDF board, tatami mat fabric, acrylic board and rubber sheet). A convolutional neural network (CNN) was constructed
and its classification performance was evaluated. In addition, we used tactile images to clarify the classification performancethrough TLAlexnet (transfer learned Alexnet). Pre-trained TLAlexnet was generated by domain adaptation using the tactile
images. The results of TLAlexnet showed the great performance to be 85.0%, 91.7%, and 85.7% with respect to single primarycolors of force, vibration, and temperature, respectively, and it improved to 96.4% when using three HPCs. In addition, theclassification performance of the proposed seven-layered another CNN that was trained with the obtained tactile images was
98.2% of the CNN constructed using common filtering parameters. Thus, highly accurate classification was realized by using
three HPCs elements.
CCS Concepts• Human-centered computing → Virtual reality; Haptic devices; • Hardware → Haptic devices; • Computing methodologies → Neural networks;

1. Introduction† fumihiro.kato@tachilab.org‡ y-inoue@tachilab.org§ tachi@tachilab.org

Many applications for virtual reality (VR) technology have beendeveloped rapidly in recent years. High-fidelity transmission of
sensory information and somatic sensation such as tactile sense,
in addition to audiovisual sensation, will provide work flexibility
and efficiency. The presentation of tactile information is considered necessary for remote working systems, because of its effec-

c 2018 The Author(s)Eurographics Proceedings c 2018 The Eurographics Association.

DOI: 10.2312/egve.20181322

https://www.eg.org

https://diglib.eg.org

108

Fumihiro Kato , Yasuyuki Inoue & Susumu Tachi / Soft Finger-tip Sensing Probe Based on Haptic Primary Colors

tiveness for the efficient VR operation [FTB∗ 00, KH13]. Our goal
is to develop a telexistence [Tac15] robot system with soft finger
hands. The robot transmits material information through material
sensing and classification results with tactile finger-tip probe. To
achieve the material classification, we developed a proposed system to classify the tactile information, to extract features with certain tactile characteristics based on haptic primary colors (HPCs)
theory [TMFF15], and also to present tactile features classification
results by using machine-learning methods. In order to acquire therequired amount of tactile infromation for the machine learning, asoft tactile sensing probe and a recording system were constructed.
1.1. Haptic Primary Colors
Tactile sensation is based on information obtained from human tactile sensors and is a type of human somatosensory sensation. The
HPCs is a theory that tactile sensing is composed by elements oftactile stimulus. It is the same as the three elements of light. In
the HPCs theory, vibration, force, and temperature correspond totactile sensation receptors and thermal sensation receptors. Tactile
receptors that accept mechanical stimulation include Meissner corpuscles, Pacinian corpuscles, Merkel’s cell, and Ruffini ending. Experiments have been conducted to measure the vibration detection
area by sinusoidal vibration stimulation ranging from about 1-500
Hz for the measurement of the skin-vibration detection area. As tactile information, it is necessary to measure mechanical vibrations ofabout 1 kHz stimulus. The force is sensed on muscle, tendon, andjoints as a deep sensation. The force is sensed on muscle, tendon,and joints as a deep sensation. The vibration is like the alternatingcurrent component, the force is like the direct current component,
thus both vibration and force sensor is needed. The frequent neuralfiring of temperature receptors at 40 - 45◦ C in warm sensation andat about 30◦ C in cold sensation. In this case, temperature sensing
is required in the temperature range between 0-50◦ C.
1.2. Related Works
The tactile signal acquisition methods with finger or hand shapeprobe has been proposed by several reports. In our previous study[KFIT17], we determined the vibration component of a tactile sensation when sliding an object of materials by using fingertip-like acceleration probe. There are many reports [AS95, BHH∗ 17, CLK14,
MPC16, PR03] using a rigid metal-tip stylus. Some reports used
gel state tactile sensor [HNNH12, VMK∗ 05, YZO∗ 17], while theother reports using thermal sensor hand [BWK15]. To the contrary we aim to realize soft finger-tip tactile sensing probe like
Biotac [WSJL08]. A study [GHKD16] reported the use of for two
fingers mounted on a robot hand; the authors classified tactile feelthrough hand gestures such as squeezing and holding. The Biotac
has functions for sensing pressure, vibration through fluid, and coretemperature; however, its sampling rate is not high enough to reproduce human sensation; the sampling rate of vibration and thermal
capacity are 100 Hz, except for fluid vibration (2.2 kHz). The authors also reported that the classification performance of results byusing only haptic is not very high. Moreover, the sensor’s sensitivity of Biotac might not be high because the pressure vibration
sensor is separated by fluids, and the stimulus transmitting throughfluids might have high impedance. Several studies have conductedclassified tactile signals with machine learning method like convolutional neural networks (CNNs) constructed from one or multiple
physical quantities and multimodal sensations. A few of these studies [HMN16, KFIT17, SSS15] classified various material surfaces
through tactile signals acquired from acceleration probe by constructing neural networks/CNN, while the others developed CNNs
based on acceleration and by using visual images [ZFJ∗ 16]. In addition, force sensor and microphone devices are included on a report [SSIS17], IR sensor metal detector is on a report [SBS17]. In
the former study, its scanning system extracts acceleration, hardness, roughness, friction, sound and image features. The latter one
defines tactile features of reflectance from IR led and metal detection, in addition. Both of them conducted the evaluation using
some supervised machine learning approaches which are not the
deep neural network. Another study [SBS17] proposed a classification method which focused on the velocity of normal force changes
while the probe slides the material. On the other hand, in the thisstudy, we aimed to classify signals about three axes of both force
and vibration, material surface temperature, directly. A classifying
system using CNN classifier was also exploited. In our system,
the feature is not extracted explicitly because human finger justsenses the force, vibration, temperature. Proposed sensing formulais based on haptic primary colors theory [TMFF15].
A tactile information recording system with a finger slides linearly on a material surface is also proposed [MPC16]. It scans friction coefficients, both lateral and normal force using an external
force sensor. In our proposed system, all sensors are embedded ina fingertip to measure the very surface vibration and force well.
Thus, in the current study, we propose the tactile probe thatsenses force, vibration, and temperature similar to a human finger.
It enables high-frequency sampling rate similar to that of the humantactile receptor. The tactile sensors of our probe are distributed on
the probe surface for high sensitivity. We also developed a tactile
signal recording system that slides the tactile probe linearly and automatically on the surface of materials. This type of automatic system enables us to easily gather a large amount of material textures.
After recording a signal, our system produces a tactile spectrogram
through a fast Fourier transform (FFT), which is frequently usedin audio analysis research [SK03], in which acquired time seriesaudio signal are produced to process frequencies.

1.3. Contribution
We proposed the tactile probe that acquires tactile information of
three HPCs (force, vibration, and temperature). In addition, we developed a tactile recording system to conduct the tactile probe sliding on materials. The acquired tactile information was classified
by using CNN. The result of cross-validation by using the proposed system classifies seven specific materials with an accuracyof 98.2%. The result also confirmed that combining force, acceleration, and temperature image improves the classification accuracy.
Our proposed system enables distinguish between different typesof materials according to their surface texture.
c 2018 The Author(s)Eurographics Proceedings c 2018 The Eurographics Association.

Fumihiro Kato , Yasuyuki Inoue & Susumu Tachi / Soft Finger-tip Sensing Probe Based on Haptic Primary Colors

2. Tactile Probe, Tactile Feeling Recording System, andClassification System Construction
In this paper, we describe the recording system and classification
system of tactile sensation that satisfies the above requirements. In
order to obtain tactile information about tactile sensations and temperature sensations, we developed a system that is based on the
HPCs to record the force, vibration, and temperature. The force islike the direct current component of the tactile sense, and the vibration is the alternating current component. We propose the tactile
probe (Figure 1(b) and (c)) that acquires a tactile signal using three
sensors of force, acceleration, and temperature. The probe imitates
the fingertip of a human, and it is assumed that tactile signals are
obtained by sliding the surface of material samples.

2.1. Tactile probe with built-in three HPCs sensors
We acquired the three-axis of force, which consists of one verticalforce axis against a material and a horizontal biaxial moment force
using µDynPick MAF-3 piezoresistive force sensor, (1 kHz, threeaxes, manufactured by Wako Tech). To obtain the vibration, we
employed LIS3DH acceleration sensor (1.6 kHz, tri-axial, manufactured by ST Microelectronics). We employed an infrared thermopile TMP-007 (manufactured by Texas Instruments, Inc.) for
non-contact temperature sensors that are connected by an FPC connection cable. To acquire the contact temperature, we employed a
56 A 1002 - C 3 thermistor (manufactured by Alpha Technics Co.,
Ltd.), and both are connected to the embedded controller ArduinoDue (32 bit, equipped with Arm Cortex - M 3, 84 MHz). We established a connection from the embedded controller and force sensor
by serial USB communication to the PC to record tactile sensing
signals (Figure 1(a)) and preserve time-series fluctuations.
The configuration of the tactile probe is shown in Figure 1(b).
The tactile probe shape was designed with 3D CAD (Autodesk Inventor), and it was a three-dimensional design modeled by a stereoscopic image printer (Figure 1(c)). The finger pad of the tactile
probe was made hollow with a thickness of 1 mm to facilitate vibrating, and it was molded using a flexible but hard material (FLX9995). We assumed that it is mounted on the robot’s fingertips or
extremities for remote work. The ease of grasping of objects is important owing to their softness, but the vibration and force that canbe acquired become dull in the finger pulp, which is too soft. Thebase of the force sensor (Figure 1(b)) is made of hard plastic (ABSresin). This is because a base with hardness is better for obtaining the force. The arm of the force sensor is on the opposite sidefrom the base, and touches the back and outside of the finger pulp
tube (Figure 1(b)). In Figure 1, the force sensor can measure thevertical force and the horizontal biaxial moment on the bottom of
the finger pulp tube. In order to obtain the force from the finger
pulp surface to the force sensor, the support column (Figure 1(b))
was extended from the finger pulp bottom surface to the arm of
the force sensor. In addition, it is thought that the vibration is moreeasily measured at the part closer to the contact surface, with theobject on the ventral side of the finger pad, where vibrations are
generated. Therefore, the acceleration sensor is disposed inside theventral side of the finger pulp tube (Figure 1(c); the acceleration
sensor is transparent).
c 2018 The Author(s)Eurographics Proceedings c 2018 The Eurographics Association.

109

2.2. Temperature Sensor and Peltier Heater
The contact temperature sensor is attached to the front/outside of
the ventral side of the finger pulp tube. The non-contact temperature sensor is mounted in a direction such that it can measure thetemperature of the surface of the substance to be contacted fromthe measurement hole, which is opened on the ventral side of thefinger pulp tube (Figure 1(b)). For the contact temperature sensor,
we used a thermistor that has a small measurement volume andgood reactivity. The non-contact temperature sensor can measure
the temperature of the cone-shaped bottom surface region with thesensor surface as the apex, and the finger can measure the temperature immediately before contact with the object, but the effect of
the thermal emissivity of the object to be measured and the environment. It is difficult to measure the temperature accurately, easily,and stable. Therefore, we used only thermistors as contact temperature sensors to measure the temperature.
On the ventral surface of the finger pulp tube, we placed a Peltierelement to maintain the temperature of the finger pad (Figures 1(b),(c)). We aim to identify materials from temperature fluctuations of
the contact object. In order to identify an object’s material, the human acquires the temperature fluctuation of the surface of the finger at the time of contact with the object. This is because the thermos receptor resides in the skin surface. In order to discriminate
an object on the temperature axis, if the tactile probe has a bodytemperature, as is the case with humans, heat exchange is causedby the temperature difference with an object, which is usually atroom temperature. Therefore, it is easier to measure the characteristics of object temperature change. Figure 2 shows the temperaturechange example of our proposed tactile probe was pressed against
both an aluminum plate and a ray fish skin plate. The tactile probe
was pressed against the surface of the sample using a tapping operation, after which it was heated, and the temperature change wasrecorded. The Peltier element with the built-in tactile probe wasmaintained at 42◦ C, which is the upper limit of human skin, whichis controlled using a voltage of 5V and a current of 0.5A. Two thermistors are located on finger pulp. One thermistor (T1 ) measures thePeltier device temperature for thermal control. The other thermistor(T2 ) is located 5 mm forward from Peltier device on the finger pad.
Figure 2 shows when the T2 became 28.9 ◦ C (higher than room
temperature), the fingertip pulp was brought into contact with thematerial for 5 min. In the aluminum sample, the temperature rose
after the temperature decreased by 1.5 ◦ C toward the room temperature. After the fingertip pulp contacts the ray fish skin, whereas
the fingertip pulp temperature nearly monotonically increased. Because aluminum has a higher thermal conductivity than ray fishskin. Because the thermal conductivity of the ray fish skin is low
and the heat conduction to the surroundings of the fingertip pulp
is slow, it is believed that the magnitude of the temperature decrease was small. From the above, by touching the object with theheated tactile probe on the human skin, it is possible to determine
the tendency of the thermal conductivity of the material. Because
the temperature fluctuation is a variable that is unique to each substance that is neither concave nor convex on the surface of the specimen, it can be used independently of the force and vibration as anindicator to determine the material using the temperature for theclassification of tactile sensation. The reports [BWK15] also proposed material classification using a heating element and a surface

110

Fumihiro Kato , Yasuyuki Inoue & Susumu Tachi / Soft Finger-tip Sensing Probe Based on Haptic Primary Colors

thermistor. It sensing temperature decreasing with 200Hz. In our
method, the heating element is as one roll of HPCs sensors whichslides material surface.

2.3. Tactile Feeling Recording System
Using the proposed tactile probe, we constructed a recording system (Figure 3) to obtain the texture of the surface material of
the object. We built a system for sliding a tactile probe by coupling a linear actuator and a servomotor. The linear actuator,
which uses a high-precision ball screw SG2605-A300P-A3CS-NNPSR(Kuroda Seiko Co., Ltd.) and the servomotor MSMF5AZL1A2(both of which are manufactured by Panasonic Corporation) areconnected by a coupling SCIW-19-5-8 (Misumi Corporation). They
are operated by sending command values from the servo amplifier
MADLT01SF. A specimen material used to acquire the tactile sensation to be recorded is attached to the work of the actuator, andit moves iteratively. We do not move the tactile probe using thelinear actuator. This is because there is no acceleration due to sliding. Two pillars that are located on the sides of the linear actuator
support the tactile probe above the linear actuator. Then, the tactile
probe has extra freedom along the vertical axis force (blue text in
Figure 3). The tactile probe weighed 0.1 kg, and it was pressed witha force of 1 N. The cylinder position was constantly measured by
the optical encoder of the motor. In the recording system used in
the report [CLK14], the vibration was excited by the reciprocating
movement. However, in the proposed recording system, the noisy
vibration can be reduced by the cylinder using the high-precisionball screw.

Figure 2: Example of temperature change when tactile probe
pressed against both an aluminum plate and a ray fish skin plate.

2.4. Imaging of Tactile Information
In the recording system, waveforms of force, acceleration and temperature are obtained as tactile information. The obtained tactile
information undergoes the Fast Fourier Transform (FFT) within aunit time delimited in the time direction to generate an image inwhich the intensity of force, acceleration or temperature frequency
varies over time. Image acquisition parameter shows on the Table 1.
In the imaging after the FFT, the intensity distribution of frequencies included in the signal is generated.

Figure 3: Recording system of surface material using tactile probe.

size and size of the max-pooling layer. The tactile image is generated using single HPCs or a combination of two or three HPCs.

2.5. Tactile Information Classification System
For the classification of imaged tactile information, we employed amachine-learning method using a CNN, which is well used in thefield of computer vision. Alexnet [MPC16] is also frequently usedas the benchmark. In this study, we developed a transfer-learning
CNN by using Alexnet (which we call TLAlexNet) with tactile images. Alexnet is a pre-trained network through one million generalimages of thousand categories database [KSG12, Ima12]. But it is
not specialized in tactile images, therefore another seven-layered
CNN was also constructed for evaluation. The tactile image enters
the input layer, and the convolution layer is the second layer. Next,
the ReLU layer is configured to become the entire binding layer,then the soft-max layer and the output layer are used for the classification via the two-dimensional max-pooling layer. It will be determined at the section of experiments; convolution filter size, stride

3. Experiments
Using the tactile probe and tactile information recording system,
we acquired HPCs tactile information from the sliding operation ofseven material surfaces (Figure 5 and 6).
3.1. Acquisition of Force, Acceleration, and Temperature
Information
The sampling frequency of the force and acceleration of the tactile
information are 1 kHz for the force sensor and 1.6 kHz for the acceleration sensor. Tactile information was acquired by repeatedly
sliding a tactile probe, which is pressed 1 N on a material sample, and it is operated at a stroke distance of 20 cm (Table 1). A
time for one-way slide takes in 2 - 3 seconds. Figures 5 and 6 showc 2018 The Author(s)Eurographics Proceedings c 2018 The Eurographics Association.

Fumihiro Kato , Yasuyuki Inoue & Susumu Tachi / Soft Finger-tip Sensing Probe Based on Haptic Primary Colors

111

Table 1: HPCs Images Acquisition Parameter(left), Parameters of generating images, Images Amount Used for TLAlexnet/Proposed CNN
and Classification Accuracy of TLAlexnet/Proposed CNN(right)

3.2. Classification using TLAlexnet

Figure 4: Combination of HPCs images. Example of ray fish skin.

the waveforms of the recorded force, acceleration, and temperature
as well as the frequency analysis results. For the material sample,
we used a ray fish skin, which is rugged but has little friction, atatami mat fabric, which is smooth, a rusting hemp fabric, an MDF
boards, which are dry and rustic, an aluminum plate and an acrylic
plate, which presents cool tactile and smoothness, and rubber sheet
which has also smoothness. It is shown that the frequency component included in the forces recorded in our proposed system isconcentrated mainly in the low-frequency range of 50 Hz or less.
The frequency components included in the acceleration, which are
set to 800 Hz, especially in the sliding direction. By focusing on a
specific frequency, it is possible to confirm the band-like features.
Because the frequency components that can be acquired differ for
the force and acceleration, it is considered that indicators regarding
the discrimination of materials can be independently obtained by a
combination of obtained frequency components. We obtained thetemperature change during sliding in the same way as the acceleration and force. We measured the temperature of the Peltier device(T1 ) and the surface temperature (T2 ) of the finger pulp tube 5 [mm]from the Peltier element while controlling the ON/OFF switchingof the power supply; as a result, the Peltier device of the tactile
probe became 42◦ C. The temperature sampling rate was about 516
Hz. The temperature waveform (Figure 6, right column) was imaged as well as the force and acceleration.
c 2018 The Author(s)Eurographics Proceedings c 2018 The Eurographics Association.

Pre-trained Alexnet randomly selected 70% of the images generated from the tactile signals of force, acceleration, and temperature
as shown in Table 1, and the domain adaptation was performed
through transfer learning. The classification performance was thenevaluated with the remaining 30% of the images. Transfer learning
was conducted on each contact image of force, acceleration, andtemperature (15 × 501 pixels), a combination of two tactile images(15 × 501 pixels) for the f orce + acceleration(F + A), the f orce +temperature(F + T ), theacceleration + temperature(A + T ), anda combination of three tactile images (15 × 501 pixels) for
the f orce+acceleration+temperature(F +A+T ). For combining
several HPCs images, we compound a temperature image next toa force image for the f orce + temperature(F + T ). We constructeda learning system using MATLAB. The size of the mini batch was[28,28]. Figure 4 shows the combination images of HPCs.

3.3. Classification using Proposed CNN
Similar to the classification using Alexnet, CNN was constructedusing each touch image and evaluated. In the construction of CNN,
the filter size of the convolution layer was set to a width and heightof 8 and 33, respectively. In the pooling layer, the pool size was set
to 2 × 2, and the stride was set at 2 × 2. In addition, the output of thetotal coupling layer was configured to be 5. The CNN that achieves
the best performance by changing the classification performancethrough parameters for each touch was evaluated.

4. Classification Results and Discussion
Compared both situations, 1) The performance classified by independently imaging the tactile information of force, acceleration,and temperature, and 2) The performance in the case of simultaneously categorizing the two and three sets of tactile information
into one image, with respect to TLALexnet, the proposed CNN.
Table 1 shows the evaluation result by using TLAlexnet / proposed
CNN respectively. We used just y-axis of accelerometer because
the accuracy was not high when using 3 axes.

112

Fumihiro Kato , Yasuyuki Inoue & Susumu Tachi / Soft Finger-tip Sensing Probe Based on Haptic Primary Colors

Figure 5: Waveform of force and image after frequency analysis. Waveform (left) and frequency analysis (∼500 Hz) results (right) obtainedin the vertical direction and sliding direction of the force when sliding by reciprocating the skin of ray fish, tatami mat, hemp cloth, and MDF
plate.

4.1. Classification Performance using TLAlexnet
By comparing the classification accuracy of the TLAlexnet, the accuracy was found to be good in the order of temperature(T ) 'f orce(F) < acceleration(A) for single HPCs images (Table 1(right)). In combination, the order was F + T < F + A < A + T ,which is higher than for a single HPCs. Even with the combinations of tactile images with low classification performance,
higher classification performance can be realized. The combinations of the f orce + acceleration, the f orce + temperature,and the acceleration + temperature were complemented, and arethought to be similar to the robust joint Haar-like feature quantity [MKSH08] with a combination of weak classifiers used in therecognition of facial images. This kind of features are called weakclassifier, it is known that these combinations of weak classifiers

become robust strong classifiers. High classification performance
was obtained even with the combination of temperature images
with low classification accuracy. Moreover, by combining three tactile combinations, it was possible to obtain maximum classification
performance. It contains images of accelerations that can be classified with high accuracy and these seem to have become strong
feature extraction entities.

4.2. Classification Performance using Proposed CNN
Table 1 shows the results by using proposed CNN, which is produced by using only tactile images, the classification accuracy of
all of HPCs image set increased (it shows right edge column ofTable 1). Middle column at Table 1 (right) shows the accuracy ofc 2018 The Author(s)Eurographics Proceedings c 2018 The Eurographics Association.

Fumihiro Kato , Yasuyuki Inoue & Susumu Tachi / Soft Finger-tip Sensing Probe Based on Haptic Primary Colors

113

Figure 6: Left table (Acceleration) shows waveform of acceleration and image after frequency analysis. Waveform (left) and frequency
analysis (∼500 Hz) result (right) in the vertical direction and sliding direction of the acceleration. Right table (Temperature) showsT2
waveform (left) and frequency analysis (∼258 Hz) result (right) when sliding. Both tables are results by reciprocating the skin of ray fishskin, tatami mat, hemp fabric, MDF plate, aluminum plate, acrylic board and rubber sheet.

proposed CNN of which is made from both filter size [13,13] and35 filters by using HPCs images, F, A, F + A, F + T , A + T and
F + A + T , respectively. The accuracy by using single T image set
is almost the same but has a little decreased. It is because that temperature images amount is small as well as low sampling rate, and
it is also considered that thermal conduction phenomena changes
slowly than the other HPCs, frequency analyze images of temperature has small features. However it seems by combining different HPCs images could increase the classification accuracy evenwith temperature, temperature image could cover the pair HPCs
images. Classification accuracy by using three HPCs images scored98.2% by using a common filter parameter. From the above, proposed CNN also shows the highest classification accuracy by using
three combinations of HPCs images.

4.3. Limitations
Tapping and grasping operation is out of scope in this study, though
the robot hand is required such operation. Tactile sensing patterndepends on pressure. Adapting to the pressure changes during suchoperation is a future work. Our proposed system will be usefulwhen the robot hand touches a object, with normalized velocity.
The velocity of robot finger will be acquired because a robot handshould be equipped motion sensing/calculating system to control
of it. In this study seven materials are featured, in order to achievevarious tactile transmits, more materials need to be analyzed formore tactile information.

5. Conclusion
In this study, we proposed a tactile probe that acquires tactile information of three haptic primary colors. We also devel-c 2018 The Author(s)Eurographics Proceedings c 2018 The Eurographics Association.

114

Fumihiro Kato , Yasuyuki Inoue & Susumu Tachi / Soft Finger-tip Sensing Probe Based on Haptic Primary Colors

oped the recording system which slides material repeatedly. Tactile information was acquired automatically by the tactile probe
pressed against the material surface. The sampling rate of theforce/acceleration/temperature is 1 k / 1.6 k / 500 Hz, respectively,
and it is sufficiently fast to be sensed by the skin receptors. The
evaluation was conducted by using both pre-trained TLAlexnet andanother trained CNN with seven sets of single HPC image or the
combinations of two or three HPC images. The high accuracy ofclassification was guaranteed by using three combinations of HPC
images, and even the combination of low accuracy HPCs provided
a sufficient accuracy of classification.

Acknowledgment
The authors wish to thank all the ACCEL project member. This
work was supported in part by a grant from JST ACCEL “Embodied Media Technology based on Haptic Primary Colors”.

References[AS95] A KUTAGAWA T., S AITO G.: Multiplex Proton-Transfer andElectron-Transfer Natures Based on the 2,2’-Bi-1H-imidazole System.
I. Acid Dissociation Constants and Redox Properties in Solution., 1995.
doi:10.1246/bcsj.68.1753. 2[BHH∗ 17] B URKA A., H U S., H ELGESON S., K RISHNAN S., G AO Y.,
H ENDRICKS L. A., DARRELL T., K UCHENBECKER K. J.: Proton: A
visuo-haptic data acquisition system for robotic learning of surface properties. IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (2017), 58–65. doi:10.1109/MFI.
2016.7849467. 2[BWK15] B HATTACHARJEE T., WADE J., K EMP C.: Material Recognition from Heat Transfer given Varying Initial Conditions and ShortDuration Contact. Robotics: Science and Systems XI (2015). doi:
10.15607/RSS.2015.XI.019. 2, 3
[CLK14] C ULBERTSON H., L ÓPEZ D ELGADO J. J., K UCHENBECKER
K. J.: One hundred data-driven haptic texture models and open-source
methods for rendering on 3D objects. IEEE Haptics Symposium, HAPTICS (2014), 319–325. doi:10.1109/HAPTICS.2014.6775475.
2, 4[FTB∗ 00] F ROHLICH B., T RAMBEREND H., B EERS A., AGRAWALA
M., BARAFF D.: Physically-based manipulation on the Responsive
Workbench. Proceedings of IEEE Virtual Reality (Cat. No.00CB37048),
figure 1 (2000), 5–11. doi:10.1109/VR.2000.840357. 2[GHKD16] G AO Y., H ENDRICKS L. A., K UCHENBECKER K. J., DAR RELL T.: Deep learning for tactile understanding from visual and haptic data. Proceedings - IEEE International Conference on Robotics
and Automation (IROS) June (2016), 536–543. arXiv:1511.06065,
doi:10.1109/ICRA.2016.7487176. 2[HMN16] H ANAMITSU N., M ORITA M., NAKATANI M.: Use of recurrence plots for haptic surface recognition in convolutional neural networks. In Euro Haptics Work in Progress (2016), pp. 1–4. 2[HNNH12] H O V. A., NAGATANI T., N ODA A., H IRAI S.: What can beinferred from a tactile arrayed sensor in autonomous in-hand manipulation? IEEE International Conference on Automation Science and Engineering (2012), 461–468. doi:10.1109/CoASE.2012.6386384.
2[Ima12]

Imagenet http://www.image-net.org, 2012. 4[KFIT17] K ATO F., F ERNANDO C. L., I NOUE Y., TACHI S.: Classification method of tactile feeling using stacked autoencoder based on haptic
primary colors. Proceedings - IEEE Virtual Reality (2017), 391–392.
doi:10.1109/VR.2017.7892341. 2[KH13] K ATO F., H ASEGAWA S.: Interactive Cooking Simulator: Showing food ingredients appearance changes in frying pan cooking. Proceedings of the 5th international workshop on Multimedia for cooking
and eating activities(CEA) (2013), 33–38. doi:10.1145/2506023.
2506030. 2[KSG12] K RIZHEVSKY A., S UTSKEVER I., G EOFFREY E. H.: ImageNet Classification with Deep Convolutional Neural Networks. Advances in Neural Information Processing Systems(NIPS) (2012), 1–9.
arXiv:1102.0183, doi:10.1109/5.726791. 4[MKSH08] M ITA T., K ANEKO T., S TENGER B., H ORI O.: Discriminative feature co-occurrence selection for object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 7 (2008), 1257–
1269. doi:10.1109/TPAMI.2007.70767. 6[MPC16] M EYER D. J., P ESHKIN M. A., C OLGATE J. E.: Tactile
Paintbrush: A procedural method for generating spatial haptic texture.
IEEE Haptics Symposium, HAPTICS April (2016), 259–264. doi:
10.1109/HAPTICS.2016.7463187. 2, 4
[PR03] PAI D. K., R IZUN P.: The WHaT: A Wireless Haptic Textue
Sensor. Proceedings of the 11th International Symposium on Haptic
Interfaces for Virtual Environment and Teleoperator Systems (HAPTICS)(2003), 3–9. 2[SBS17] S TRESE M., B OECK Y., S TEINBACH E.: Content-based surface
material retrieval. IEEE World Haptics Conference (WHC), June (2017),
352–357. doi:10.1109/WHC.2017.7989927. 2[SK03] S OON I. Y., KOH S. N.: Speech enhancement using 2-D Fouriertransform. IEEE Transactions on Speech and Audio Processing 11, 6(2003), 717–724. doi:10.1109/TSA.2003.816063. 2[SSIS17] S TRESE M., S CHUWERK C., I EPURE A., S TEINBACH E.:
Multimodal Feature-Based Surface Material Classification. IEEE Transactions on Haptics 10, 2 (2017), 226–239. doi:10.1109/TOH.
2016.2625787. 2[SSS15] S TRESE M., S CHUWERK C., S TEINBACH E.: Surface classification using acceleration signals recorded during human freehand
movement. IEEE World Haptics Conference (WHC) (2015), 214–219.
doi:10.1109/WHC.2015.7177716. 2[Tac15] TACHI S.: Telexistence, 2nd ed. WORLD SCIENTIFIC, 2015.
doi:10.1142/9248. 2[TMFF15] TACHI S., M INAMIZAWA K., F URUKAWA M., F ERNANDO
C. L.: Haptic media: Construction and utilization of human-harmonized"tangible" information environment. Proceedings of the 23rd International Conference on Artificial Reality and Telexistence (ICAT) (2015),
145–150. 2[VMK∗ 05] V LACK K., M IZOTA T., K AWAKAMI N., K AMIYAMA K.,
K AJIMOTO H., TACHI S.: GelForce: a vision-based traction field computer interface. Proceedings of ACM CHI 2005 Conference on Human
Factors in Computing Systems 2 (2005), 1154–1155. doi:10.1145/
1056808.1056859. 2[WSJL08] W ETTELS N., S ANTOS V. J., J OHANSSON R. S., L OEB
G. E.: Biomimetic Tactile Sensor Array. Advanced Robotics - journal of the Robotics Society of Japan 22, 8 (2008), 829–849. doi:
10.1163/156855308X314533. 2[YZO∗ 17] Y UAN W., Z HU C., OWENS A., S RINIVASAN M. A., A DEL SON E. H.: Shape-independent hardness estimation using deep learningand a GelSight tactile sensor. Proceedings - IEEE International Conference on Robotics and Automation (IROS) (2017), 951–958. arXiv:
1704.03955, doi:10.1109/ICRA.2017.7989116. 2[ZFJ∗ 16]Z HENG H., FANG L., J I M., S TRESE M., O ZER Y., S TEIN BACH E.: Deep Learning for Surface Material Classification Using Hap-

tic and Visual Information. IEEE Transactions on Multimedia 18, 12(2016), 2407–2416. arXiv:1512.06658, doi:10.1109/TMM.
2016.2598140. 2

c 2018 The Author(s)Eurographics Proceedings c 2018 The Eurographics Association.