Twech: A Mobile Platform to Search and Share Visuo-tactile Experiences
Nobuhisa HANAMITSU*1, Haruki NAKAMURA*2, Masashi NAKATANI*1 and Kouta MINAMIZAWA*1*1

Graduate School of Media Design, Keio University*2
Tokyo National College of Technology(b) Captured video image: the probe data is seen at the bottom of the image(a) Recording contacted textures using an embodied probe
Figure 1: Twech is at once an attachment for capturing tactile experiences and a device for playing back haptic experience. As shown in(a), the device attaches to a smartphone. To record a visual and tactile experience, the user attaches the device through the smartphone’saudio jack. The user can then record a visual and tactile experience by simultaneously capturing the haptic data and using the phone torecord a video, as shown in (b). The headphone jack is used both as a microphone and an earphone jack. Therefore, in order to reexperience the haptic recording, the user uses the same configuration. The haptic actuator in the back of the device responds to the hapticvibration data transferred via the headphone jack.

1.

Introduction

When using social networking, users often upload digital media
to capture their experiences, and this includes video or photos ofmeals, landscapes, gatherings of friends, etc. These events arecaptured with a camera, recorded by a microphone, or archived
using a video camera. Such media provides us with visual, audible,
or integrated audio-visual experiences. However, up until now,
sharing one’s corresponding haptic experiences has not beenpossible. If this haptic experience can be shared, the sensory
feedback will be sufficiently compelling and easy to understand inorder for a more complete experience from a first-person
perspective.
The TECHTILE toolkit [Minamizawa et al. 2012] is an easy-touse device that can record and playback realistic haptic
experiences. This device captures solid-borne sound waves
resulting from friction-evoked sounds using a microphone. As ahaptic feedback actuator. Because recorded audio signals areassociated with haptic events, users can enjoy high-fidelity haptic
experiences using this device.
In this work, we propose a mobile platform, called Twech, whichenables users to collect and share visuo-tactile experiences. Thisplatform can also discover materials that can provide a similar
haptic feeling. Users can record and share visuo-tactile
experiences by using a visuo-tactile recording and displaying it
through an attachment to a smartphone. The proposed systemallows users to share their experiences instantly over social media,
and to re-experience shared data such embodiment visualization.
*E-mail: hanamitsu@kmd.keio.ac.jp
Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made ordistributed for profit or commercial advantage and that copies bear this notice andthe full citation on the first page. Copyrights for third-party components of thiswork must be honored. For all other uses, contact the Owner/Author.
Copyright is held by the owner/author(s).
SA'15 Symposium on MGIA , November 02-06, 2015, Kobe, Japan
ACM 978-1-4503-3928-5/15/11.
http://dx.doi.org/10.1145/2818427.2818461

Further, Twech’s search engine queries similar experiences—thetexture of a scratched surface or an animal’s fur, for example—
and these can be shared with others. Tactile data is analyzed using
a deep-learning algorithm that is expanded in order to recognize
tactile materials. Thus, were this engine included in an online
shopping store, for instance, users could find clothes with aparticular feel to the fabric based on an analysis of the recorded
signals, as shown Figure 2. In other words, users could search fortactile data through textures, rather than explicit verbalexpressions.

Figure 2: Example scene of online shopping2.

Related WorkFor both digital information and the real world, EyeRing[Nanayakkara et al. 2013]. EyeRing is a device attached to theuser’s finger for capturing color, words, and views of real objects
uses an actual index finger to the embodiment position.
FingerReader [Shilkrot, et al. 2015] is a similar device that is able
to recognize where a user points in real-time. FingerReader
facilitates expanded embodiment with increased target contacts.
TouchCast [Takeuchi et al. 2012] was proposed as a vibro-tactile
search engine based on voice recognition using the Earth Mover’sDistance and Mel-Frequency Cepstrum Coefficients. This method
is common in the field of image recognition for solving
invariances. The first convolutional neural network proposed wascalled LeNet [LeCun et al. 1989]. Subsequently, AlexNet[Krizhevsky et al. 2012] was proposed. AlexNet was developed inILSVRC2012, and has since become one of the most popularconvolutional neural networks. Unlike LeNet, AlexNet offers highperformance for recognizing images. In this paper, we adopt
AlexNet to develop a search engine for vibro-tactile signals.

3.

System Description

We propose a mobile platform that is able to provide fourfeatures of a haptic experience. The proposed system uses visualand tactile feedback for displaying a recorded experience.
 Visual feedback from recorded video.
 Tactile feedback from recorded tactile signals.
 Enhanced visual feedback using tactile signals.
Enhanced tactile feedback using pseudo-haptic feedback.
Twech consists of two components: one component captures
video and audio signals that are associated with tactile
experiences; the other provides a haptic experience that is similar
to the recorded signals. We discuss how these captured images areanalyzed with our system.

is capable of learning scale and translational invariances. For
example, the scale invariance of tactile data is equivalent to theamplitude of an audio signal. Translational invariance isassociated with a frequency shift that is caused by the difference
in touching speed. These invariances successfully analyze ourcollected dataset. For example, the system can classify how users
touch an object, whether at slow or fast speeds. The input data isalso analyzed using a spectrogram, and the data is divided into
low-frequency (0 Hz – 1000 Hz) and high-frequency (1000 Hz –20 kHz) components. The final output of the analyzed data isshown in Figure 5, showing how collected signals are successfullyclassified into a limited number of categories. This network
recognizes search results that are based on the vibro-tactile map.
When network acquires new material, our developed recognition
system is capable of calculating the similarity of new inputs toknown materials. The materials in the area have a rough texture(e.g., a washboard, shell, or a LEGO block). By contrast, thematerials in area have a smooth texture (e.g., textiles).

3.1. Attachment for Record and Display
We developed a smartphone attachment for sharing visual andtactile experiences [Nakamura et al. 2015]. As shown in Figure 3,
the device consists of a tactile amplifier (AMP15W-8006), amicrophone (PRIMO MX-M4758) that is able to capture
frequencies as low as approximately 10 Hz, a voltage divider, anda haptic actuator (Tactile Lab Haptuator Mark-II). The actuator isstrong enough to provide haptic feedback with the entire
smartphone device. This configuration is also effective at makingusers aware that they are in contact with an object. The
microphone, which is embedded in the contact probe, helps users
to focus on the contact point between the probe and the materials.
The captured data is recorded as an audio signal. This audio signal
is converted into tactile signals with Haptuator. To re-experience
the sensations recorded by users, the haptuator uses visuo-motor
coupling and accordingly plays back the recorded tactile data. Our
previous research showed that this haptic feedback systemprovided sufficient information to re-experience a scene with themotion of swinging a badminton racket [Mizushina et al. 2013].
The proposed device is intended to offer an alternative method forsharing embodied experiences recorded by another person.

Figure 4: Machine learning for dual-layered tactile recognition:
convolution layer and a subsampling layer for the neural network[Krizhevsky et al. 2012].

Figure 3: Attachment overview: microphone includes anembodied probe; captured signals convey audio through a jackattached to the smartphone; the vibro-tactile wave is displayedusing a tactile actuator though a tactile amplifier from thesmartphone.

3.2.

Haptic search engine

To build the haptic search system, we developed an algorithm
that finds tactile data that is similar to the target data. We
employed a convolutional neural-network algorithm for tactile
recognition [Krizhevsky et al. 2012](Figure 4). This architecture

Figure 5: Tactile map showing 46 material thumbnails. The
tactile wave uses a dataset comprising 460 items for capturing
items touched at slow or fast speeds. In this map, materials areseparated into four clusters: soft-thin (e.g., cloth); soft–thick (e.g.,a soft toy), fine-textured (e.g., a shell); hard-rough (e.g., a LEGOblock).

4.

Possible User Experience

Twech is a mobile platform that enables users to share visuotactile experiences and search other experiences for tactile data.
User can record and share subjective experiences using the visuotactile information recorded and presented with the proposed
smartphone attachment. This setup allows the user to share and reexperience shared data over social media.

Further, Twech’s search engine locates similar visuo-tactile
experiences—for instance, the texture from a scratched surface or
the fur of an animal—from collected tactile data. The similarity isautomatically calculated by using the proposed search engine,which is based on deep learning. Twech allows users to share andfind haptic experiences from visual-tactile data uploaded to a
cloud server, as shown in Figure 6. In addition, the proposedsystem provides a platform for sharing the experience ofinteracting with pets. The recorded data is utilized for recording
an enhanced interaction with a pet to facilitate visuo-audio-haptic
feedback. Users who are not in a position to touch the animalshave an interactive experience with them, as depicted in Figure 7.

References
K. MINAMIZAWA, Y. KAKEHI, M. NAKATANI, S. MIHARAAND S. TACHI. 2012. TECHTILE toolkit: a prototyping tool fordesign and education of haptic media. In Proceedings of the2012 Virtual Reality International Conference, Article 26, 2p.
A. HETTIARACHCHI, S. C. NANAYAKKARA, K. P. YEO, R.
SHILKROT AND P. MAES. FingerDraw: More than a Digital
Paintbrush, ACM SIGCHI Augmented Human, 4p.
Y. TAKEUCHI, H. KATAKURA, S. KAMURO, K. MINAMIZAWA
AND S. TACHI. TouchCast: an on-line platform for creation andsharing of tactile content based on tactile copy & paste. InAdjunct proceedings of the 25th annual ACM symposium onUser interface software and technology, 13-14, 2p.
Y. LECUN, L. BOTTOU, Y. BENGIO, and P. HAFFNER.
Gradient-based learning applied to document recognition.
Proceedings of the IEEE 1998, 2278-2324, 46p
A. KRIZHEVSKY, I. SUTSKEVER AND G. HINTON. 2012. Imagenet
classification with deep convolutional neural networks,
In Advances in Neural Information Processing Systems, vol. 25,pp. 1106-1114, 8p.
S. NANAYAKKARA, R. SHILKROT, K. PEEN YEO AND P. MAES,EyeRing: A Finger Worn Input Device for Seamless
Interactions with our Surroundings, In the proceedings of the4th International Conference on AugmentedHuman, 7p.

Figure 6: Proposed architecture for displaying recorded data andrecognizing vibro-tactile waves. In this case, we used aconvolutional neural network.

H. NAKAMURA, N. HANAMITSU AND K. MINAMIZAWA. 2015.
A(touch)ment: a smartphone extension for instantly sharing
visual and tactile experience. In Proceedings of the 6th
Augmented Human International Conference, 223-224, 1p.
Y. MIZUSHINA, W. FUJIMURA, T. SUDOU, C. L. FERNANDO,K. MINAMIZAWA AND S. TACHI. 2015. Interactive instantreplay: sharing sports experience using 360-degrees sphericalimages and haptic sensation based on the coupled body motion.
In Proceedings of the 6th Augmented Human International
Conference, pp. 227-228, 2p

Figure 7: Case experience: recording interactions with a cat.

5.

Future direction

In this paper, we proposed a mobile platform, called Twech,
which enables users to collect and share visuo-tactile experiences.
The proposed system includes an attachment. The attachment
enables users to re-experience a recorded experience. The haptic
search engine was based on a convolutional neural network—apopular deep-learning algorithm for recognizing images. We
applied this algorithm for recognizing tactile signals.
In future research, we design an embodiment visualization for
capturing tactile data and for displaying the recorded data to users
who do not have the proposed device. In addition, we shallpropose a tactile data format for the search engine,

6.

Acknowledgements

This work was funded by JSPS KAKENHI Grant #26700018.

K. J. KUCHENBECKER, J. ROMANO AND W. MCMAHAN. 2011.
Haptography: Capturing and recreating the rich feel of realsurfaces. In Robotics Research, pp. 245-260, 5p.