Twech: A Mobile Platform to Search and Share Visuo-tactile Experiences
Nobuhisa HANAMITSU*1, Haruki NAKAMURA*2, Masashi NAKATANI*1 and Kouta MINAMIZAWA*1*1

Graduate School of Media Design, Keio University*2
Tokyo National College of Technology
aFigure 1.1: Example of recorded experience data (a scratching cloth)1.

Introduction

Using social networks in our daily life, users sometimes upload
their experience; for example, foods, magnificent views, agathering scene with friends etc. These events are taken by acamera, recorded by a microphone, or archived by a video camera.
These data provide us with visual, sound or integrated audiovisual experiences through the Internet. However, up until now,the sharing of corresponding haptic experiences has not beenpossible. If this haptic experience can be shared, this sensory
feedback will be compelling enough and is easy to understandwhat people are experiencing by a first-person perspective.
The TECHTILE toolkit [Minamizawa et al. 2012] is an easy-touse device that can record and playback realistic haptic
experiences. This device captures solid borne wave produced by
friction-evoked sound using a microphone. As a haptic feedback
actuator, they use a vibrator and directly input amplified audio
signal that is captured by a microphone. Since recorded audio
signal is highly associated with haptic event, we can enjoy highfidelity haptic experiences using this device.
In this works, we propose a mobile platform, called Twech,
which enables users to collect and share visuo-tactile experiences.
This platform also provides a function that enables to findmaterials that can provide similar haptic feeling. User can recordand share visuo-tactile experiences by using a visuo-tactile
recording and displaying attachment for smartphones. This systemallows users to instantly share their experiences with social media,and re-experience shared data such as visuo-motor coupling.
Further, Twech’s search engine finds similar experiences, whichwere collected by scratching material surfaces, communicatedwith animals or other experiences. Tactile data is analyzed using
deep learning algorithm that ware expanded for recognizing
tactile materials. When this engine was included in web shopping
store, user may find similar clothes based on analysis of recorded
signals. In other words, user can search tactile data for texturesinstead of using explicit verbal expressions.

2.

bc

Figure 1.2: (a)Record textile (b)Search textile(c)Present and Suggest

System Description

Twech is consisting of two different components: one part ismeant to capture video and audio signals that are associated withtactile experience. Another part is meant to provide haptic
experience that is similar to recorded signals. Below, we will
discuss how these captured images are analyzed in the developed
signals.

2.1. Attachment for Record and Display
We proposed a smartphone extension for sharing visual and tactile
experiences [Nakamura et al. 2015]. As shown in Figure 2 (left),the device consists of a tactile amplifier (AMP15W-8006), atactile microphone (PRIMO MX-M4758) that is able to capture
frequencies as low as approximately 10Hz, a voltage divider and ahaptic actuator (Tactile Lab Haptuator Mark-II). This actuator isstrong enough to provide haptic feedback with entire smartphonedevice. This configuration is also effective to make participants beaware to touching it. The microphone, which was embedded in
the contact probe, helps users to pay their attentions to the contact
point between the probe and materials. The captured data wasrecorded by audio signal. This audio signal is converted into
tactile signals by presented with the Haptuator. Previous study
showed that high fidelity tactile feedback was achieved with thismethod [Minamizawa et al. 2012]. To re-experience recorded byusers, this system uses visuo-motor coupling situation andaccordingly playbacks the recorded tactile data. In addition, thismethod is our previous research provides sufficient information to
re-experience the scene by body motion using badminton racket[Mizushina et al. 2013]. These give us an alternative method toprovide embodied experiences through shared experience, whichwere recorded by other person.	 

*e-mail: hanamitsu@kmd.keio.ac.jp
Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor commercial advantage and that copies bear this notice and the full citation on thefirst page. Copyrights for third-party components of this work must be honored. For allother uses, contact the Owner/Author.
Copyright is held by the owner/author(s).
SA’15 Emerging Technologies, November 02 – 06, 2015, Kobe, Japan.
ACM 978-1-4503-3925-4/15/11.
http://dx.doi.org/10.1145/2818466.2818495

Figure 2: Attachment overview (left) and
Visuo-tactile experience (right)

2.2.

Haptic search engine

To achieve what to search for tactile data, we develop haptic
search algorithm, which finds similar tactile data. We employ aconvolutional neural network algorithm for tactile recognition[Krizhevsky et al. 2012]. This architecture is capable of learning
scale and translation invariances. Applying this learning algorithm
in tactile search system, scale invariance of tactile data isequivalent to the amplitude of audio signal. Translation invarianceis associated with frequency shift that is caused by the differenceof touching speed. Those invariances successfully analyze ourcollected dataset; for example, the system can classify how users
touch an object with slow or fast speed. The input data was also
analyzed using spectrogram, and the data was divided into low
frequency (0Hz-1000Hz) and high frequency (1000Hz-20kHz)
components. Spatial configuration of contact surface is alsoclassified based on the time structure of measured signal,suggesting whether the surface texture is rough or smooth (seeFigure 3). Final output of analyzed data is shown in Figure 4,
suggesting that collected signals were successfully classified into
small number of categories. This network recognizes search
results, which based on vibro-tactile map, as shown in Figure 5.
This figure shown that X-axis is low frequency vector and Y-axis
is high frequency vector. When network gets new material, ourdeveloped recognition system is capable of calculating thesimilarity of new input with known materials. As you can see inthis figure, left bottom area’s materials has roughness texture, forexample washboard, shell and LEGO. On the other hand, right top
area’s materials have smooth textures and almost of them aretextiles.

3.

Possible User Experience

Twech is a mobile platform that enables users to share visuotactile experience and search other experiences for tactile data.
User can record and share subjective experiences using visuotactile information recorded and presented with developed
attachment for smartphone. This setup allows the user to instantly
such as tweet, and re-experience shared data.
Further, Twech’s search engine finds similar visuo-tactile
experiences, which cam be the feelings of scratched material
surfaces, communicated with animals or other experiences, fromcollected tactile data. This similarity is automatically calculated
by using search engine is based on deep learning, Twech provides
sharing and finding haptic experiences and users re-experience
uploaded visual-tactile data from cloud server, as shown in Figure
1.2.
In addition to, this system also provide platform to shareinteractive experience with animals. The recorded data will beutilized recorded enhanced interacting with animals along visuoaudio-haptic feedback. Users who can’t even touch them, may beable to have interactive experience with animals.

Figure 6: Record play with a cat

4.

Acknowledgements

This work is supported by JST-ACCEL Embodied Media projectand JSPS KAKENHI Grant #26700018.
Figure 3: Washboard’s photo and spectrum

References
K. MINAMIZAWA, Y. KAKEHI, M. NAKATANI, S. MIHARA,
AND S. TACHI. 2012. TECHTILE toolkit: a prototyping tool for
design and education of haptic media. In Proceedings of the2012 Virtual Reality International Conference, Article 26, 2p.
Figure 4: Machine Learning of tactile recognition

H. NAKAMURA, N. HANAMITSU AND K. MINAMIZAWA, 2015.
A(touch)ment: a smartphone extension for instantly sharing
visual and tactile experience. In Proceedings of the 6th
Augmented Human International Conference, 223-224,1p.
Y. MIZUSHINA, W. FUJIMURA, T. SUDOU, C. L. FERNANDO,K.MINAMIZAWA AND S.TACHI. 2015. Interactive instant replay:sharing sports experience using 360-degrees spherical images
and haptic sensation based on the coupled body motion. In
Proceedings of the 6th Augmented Human International
Conference, pp. 227-228, 2p
A. KRIZHEVSKY, I. SUTSKEVER AND G. HINTON. 2012. Imagenet
classification with deep convolutional neural networks,
In Advances in Neural Information Processing Systems, vol. 25,
pp. 1106-1114, 8p.
K. J. KUCHENBECKER, J. ROMANO AND W. MCMAHAN, 2011.
Haptography: Capturing and recreating the rich feel of realsurfaces. In Robotics Research, pp. 245-260, 5p.

Figure 5: Tactile map: Blue) shell and LEGO, Red) textiles