情報処理学会 インタラクション 2024
IPSJ Interaction 2024

1P-96
2024/2/28“Touching Our Heritage”:
Exploration of Crossmodal Senses
in Experiencing Heritage
Indonesian “Batik” Textiles
Maximillian Sutrisno†1

Masashi Nakatani †2

Abstract: Culture binds groups of humans together, and as a byproduct, culture produces tangible artifacts that people canexperience. Cultural heritage is preserved via these artifacts, creating a historic, distinct identity in which people can group under.
The actors within a culture that contribute directly to its preservation, such as craftsmen, historians, and so forth, are the minority
within the general population. Since culture is an identity adopted by a general population, extending the experience of directlyinteracting and forming connections with our culture beyond those who directly contribute to its preservation is imperative. This
paper utilizes Indonesian batik textile as an example of such cultural artifacts, as it is a hand-made form of art that only skilledcraftsmen can create. Utilizing audiovisual synesthetic sciences and software, it is possible to create novel immersive experiencesthat allow viewers to form a new, direct connection with culture. This paper is an exploration of culture utilizing TouchDesigner,
experimenting with the connection between audiovisual sensory perception and haptics in the context of Indonesian batik textiles.

1. Introduction

automation claiming that true batik is limited to a specific handdrawn technique. Despite their polarizing stances, both sides of

1.1 INDONESIA: BATIK

the argument are in favor of preserving batik as IndonesianIndonesia, the largest archipelagic nation in the world, harboring

cultural heritage. Thus, it is imperative to view cultural

over 17,000 islands, is rich in cultural heritage. Though thepreservation based neither on what is currently defined or how it

country has historically been a melting pot of identities due to its

was defined, but to think on what is possible in the future.

nature as a seaport and trading hub, it still possesses a uniquecultural identity independent, or minimally influenced by outsidegroups. One such tradition, the Batik textiles, continue to persist,and will do so as long as the Indonesian people continue topreserve it as a cultural symbol.
Batik, derived from “mbat”, an archaic Indonesian word meaning“dotting”, is a type of Indonesian cultural artifact that is createdby hand-stenciling traditional motifs with wax on a plain sheetof fabric that is then dyed to produce a patterned cloth [2]. These
cloths have a variety of uses, most of which includes clothing or
ornamentation, particularly in Indonesian traditional uniforms such as with the Balinese Paya Agung or paired with the widely-

1.2. TRANSCENDING AS AN OBSERVER

The field of interactive art is blooming, and with it, comes totallynovel ways in which to experience artistic intent. This paper isan exploration into the conveyance of the batik creation process
through adapting batik as a form of interactive art in which
viewers can narrow the observer-artist gap through a mode ofcross modal sensory experiences, which include vision, hearing,and haptics in order to facilitate an intimate connection with thecultural artifact beyond viewing a final product.

2. Synesthetic Science

used Kebaya, among others [3]. Batik is not restricted to

2.1. CROSS-MODAL SENSES

traditional use, as it is also commonly used as a material in

The way our senses interact with each other in order to form

creating clothing such as western-style suits and shirts.

perception show that each sense does not run parallel with eachThough batik is commonly associated with a lengthy process ofother, instead they act as supplements that cross over in order tohand-drawn patterns, technological advances have allowed batik

create a much larger whole. For instance, human olfactory

craftsmen to increase efficiency using automation - which has

systems work with gustatory (taste) systems and act as

caused controversy amongst communities and their definition ofchemoreceptors that paint a perception of food or drink. Ocularbatik [4][5][6]. The general controversy surrounding automatedreceptors also allow the visual cues of said food item to bebatik-making processes is multi-faceted, with those pro-

understood, and haptic perception allows us to feel the texture ofautomation citing its use as a way to efficiently spreadthe food item. Each sensory system works in tandem with eachIndonesian culture to a wider audience and those anti-

other to provide a full sensory experience [7].

†1 Keio University Shonan-Fujisawa Campus†2 Keio University Shonan-Fujisawa Campus© 2024 Information Processing Society of Japan612

情報処理学会 インタラクション 2024
IPSJ Interaction 2024

1P-96
2024/2/28

2.2. SEEING IS FEELING: HAPTICS AND VIDEO GAMESVideo games are a unique form of media, like interactive art, inwhich the observer can manipulate information provided to them(quite literally, with a controller). One of the first breakthroughsin video game haptics was the 1997 “Rumble Pak” developed to
be used alongside the Nintendo N64 console, a haptic accessory
that was to be attached to the regular N64 controller allowing for
haptic force feedback when playing certain games like Super
Mario 64 and Star Fox 64 [12].
In Sony’s Spider-Man 2 for the PlayStation 5, sequences inwhich the titular hero is required to lift heavy objects arecommon. As Spider-Man is seen lifting heavy boulders anddebris within the game, players are prompted to push the left andright trigger buttons, of which DualSense haptic technology isfitted into. Players are then required to delicately balance the left
and right trigger buttons as if they were Spider-Man’s own arms
[13]. Similarly, in Infinity Ward’s 2022 video game, Call of Duty:
Black Ops: Cold War [14], this haptic resistance technology isused to imitate the feeling of using various different firearmFig. 1: Summary of final visualizer

Synesthesia is an exaggerated example of this sensory crossover.
It is described as the overlap between senses, or rather thetriggers, of which there are many in the game. This application of
haptic technologies allows for immersive qualities that older
video games were unable to reach.

overlapped activation of the senses [8]. A synesthete could
ascribe colors to numbers, or “see” sounds as if they were part ofthe color spectrum. Grapheme-colour synesthesia, a type ofsynesthesia that allows synesthete to associate graphemes, or thesmallest meaningful building block of a writing system (i.eletters, numbers, etc) with specific colors, is caused by amutation that allows for stronger cross-activation between the V4(the visual cortex) and the number area in the brain [9].
Ramachandran & Hubbard’s 2001 paper entitled “Synaesthesia A Window Into Perception, Thought, and Language”, reviews

Fig. 2: Rudimentary TouchDesigner node functions

synesthesia and other cross-modal sensory perception (such as

With the advent of new applications and technologies allowing

the well-known and well-reported “Bouba-Kiki effect” [10]),for downscaling and a wider range of applications, haptic

explaining that human synesthetic capabilities may have had antechnologies can be used effectively in creating novel multimodal

impact in the creation of language, particularly metaphor andexperiences.

other non-literal methods of communication. Regardless of its
evolutionary origin, it is clear that fundamentally the brain relies

2.3. FEELING CULTURE

on cross-talk between its various regions in order to perceive the

Batik is a hand-drawn textile - how can we convey this to an

world around us.

average observer? This is where the potential for haptic

Interactive art allows us to tap into this cross-modal sensory

technologies lies. In creating an audiovisual experience that also

perception system on a more direct manner, immersing ourselves

utilizes haptic technology for the purpose of sensory immersion,as observers able to shape the piece in front of us. Haptic

representing the hand-drawn nature of each Batik stroke istechnologies allows us to feel what we see, and in turn can be apossible to a degree, of which the objective of this paper is so:valuable asset in creating novel interactive experiences.

Exploring the applications of synesthetic science and haptic
technologies to create an immersive Batik exhibition highlighting© 2024 Information Processing Society of Japan613

情報処理学会 インタラクション 2024
IPSJ Interaction 2024

1P-96
2024/2/28

its hand-drawn nature.
This is to be achieved via software capable of acting as a medium

3.2 FINAL VISUALIZER: EXPLAINED

between audiovisual output and haptic output, whereby generated
Batik visuals can induce haptic response.

3. Cultural Visualization
3.1. VISUALIZING BATIK VIA TOUCHDESIGNER

TouchDesigner, described by its developer, Toronto-based
Derivative, as “visual development platform” is a node-based
programming language created for the development of realtime
multimedia works. At its most basic, it allows for most inputs, be
it MIDI data, camera data, microphone data, and so forth, to be

Fig. 4. Visual and Audio
description of visualizer

manipulated, creating any output intended by the artist. It
functions parametrically, meaning that variables in every node

The final visualizer is an infinitely looping moving image of acan be tied to realtime data. As an example, via TouchDesigner, it

Batik pattern that perpetually zooms in and out, while individual

would be possible to take camera input, convert said input into

Batik strokes undulate in and out. As it does so, audio oscillators

color data, and use that color data as variables of, for instance, aequally undulate, associating the movement of the pattern withlens distortion effect node. The lens distortion effect node would

audio output. (Please refer to the appendix for a video link to thethen alter its output based on the color data converted from thevisualizer itself)

camera input, meaning that whenever the camera detects a

3.2.1 Step 1: Imagery

change in color, it results in a parametric alteration of the lensIn order to replicate Batik patterns in TouchDesigner, it isdistortion node’s values.

necessary to break down the pattern into its simplest shapes, as

TouchDesigner is also capable of generating its own inputs that

TouchDesigner is capable of creating curves that can be

can be manipulated into output visuals. This includes audio input

parametrically adjusted. By generating each curve, movement(via TouchDesigner’s provided oscillators) or visual input (via

can then be introduced, leading us into the second step,

3D primitives and more). Thus, to create an immersive Batik

processing.

exhibition utilizing haptic technologies, it would be necessary to

As there are an innumerable amount of Batik patterns local touse Batik imagery as an input that is then manipulated, of which

various Indonesian regions, the Parang pattern was chosen for its

the manipulation triggers haptic transducers to induce vibrations.

simplicity and commonality in Indonesia’s most populous region,

In this case, as we are exploring the replication of Batik strokes,
we will be utilizing TouchDesigner in creating moving Batik
patterns via its in-built 3D functionality. By replicating these
Batik patterns within TouchDesigner, we can then manipulate theJava. Following instructional videos regarding the drawing of thepattern, it was possible to break down the pattern into at least
three simple curves. They were then manually adjusted to fit into
the Batik Parang pattern.

stroke length. Utilizing the stroke length data, we can then link

3.2.2. Step 2: Processingthese values parametrically to an oscillator, which will generate

In this step, the Batik pattern will have movement induced via

sound that is then converted into haptic feedback via a haptic

TouchDesigner’s inbuilt low-frequency oscillator (LFO) nodes,

transducer. What results is a visualizer that depicts moving

allowing a pulsating range of values to be parametrically inputtedstrokes of Batik that generate audio and haptic feedback based on

into the length of each curve. This creates a rhythmicallyits movements, creating an immersive multimodal experience.

pulsating movement. This movement value will then be

The Batik strokes expand and contract, generating audio output

associated with the frequency parameters of TouchDesigner’s

listened through headphones, while that same audio output isaudio oscillator nodes. This results in audio frequencies to beconverted into haptic vibrations felt through the viewer’s hands

generated as each Batik curve moves.

via small haptic transducers.
3.2.3. Step 3: Output

Using the length value of the Batik curves, audio is generated via
the audio oscillator node, as explained step 2. This audio© 2024 Information Processing Society of Japan614

情報処理学会 インタラクション 2024
IPSJ Interaction 2024

1P-96
2024/2/28

oscillation is then run through a Foster AP15d, amplifying theCurved wagara patterns also exist, like the seigaiha and shipposignal at a maximum of +15 W. This signal is then fed through[16], and an exploration into how the angularity of a pattern

the amplifier into two Acouve VP2 haptic transducers which

affects the entire synesthetic experience could be a direction thatfurther amplify low-frequency signals that are best felt by

future projects can take. Mimicking the well-known Bouba-kikimechanoreceptors in human skin. The original audio signal iseffect, patterns that move, sound, and are colored in the same

simultaneously played through a set of headphones. By wearing

ways but possess different angularities may be perceived

the headphones and holding the haptic transducers in the palms,

differently. Similarly, if any of these factors were to be changed,

while also viewing the visualizer simultaneously, an audiovisualan exploration into how the displayed traditional patterns areexperience becomes an audio-visual-haptic holistic experience.

perceived would also be possible.

3.3 EXHIBITION

The full audio-visual-haptic experience was exhibited at the Keio
Shonan-Fujisawa Open Research Forum (Keio SFC ORF) on the
23rd and 24th of November, 2023. Please refer to Appendix 2 forimages pertaining to the ORF, intended to showcase theexhibition. Please refer to Appendix 3 for hardware
specifications.
Via informal interviews with visitors that experienced theexhibition, it was clear that the haptic qualities of the exhibition
were novel and thus, most memorable in terms of the exhibition
as a whole. Visitors were able to associate the haptic feedback of
the transducers with the visualizations presented to them, andnoted that the audio was “soothing” and immersive. With theaddition of the soft, curved patterning of the batik, a wave-likemotion was achieved, which could have had an impact on theperception of the movement of the pattern.

4. Discussion and Conclusion
4.1. FURTHER CULTURAL EXPLORATION

Limiting the scope of the project to Indonesian Batik patterns,
and more specifically, Batik Parang, patterns, breaking down thepattern into simple, parametrically editable shapes was a direct
and precise task. This can extend to other cultural patterns anddesigns, specifically other patterns that possess the simple-toreproduce shapes similar to Batik Parang. As long as clear,patterned shapes are distinctly identifiable, it is possible to adapt
the same exhibition utilizing the same audio-haptic-visualframework for use with other traditional patterns.

4.2 Expanding Cross-Modal Sensory Experiences
It is important to consider the current project’s scale and scope,and areas in which it lacked immersive qualities. Sonically,noise-canceling headphones were utilized, creating a pocket ofsound distinct away from environmental noise. This creates animmersive soundscape, and ensures that no outside interference,sonically, was experienced. Visually, there is still an observabledistance between a viewer and the artwork, and this distanceallows for objects in the periphery of one’s vision to interferewith the perception of the visualizer being displayed. A solutionthat could be considered is virtual reality technology, particularly
VR headsets. VR headsets are praised for their immersive
qualities, as much like noise-canceling headphones, they allow
the user to completely shut off any outside stimuli that may
interfere with their immersion of the content they are consuming.
Haptically, the feedback experienced by exhibit viewers was
localized to the palm of their hands. Human skin possesses haptic
mechanoreceptors on virtually every portion of its surface, withvarying sensitivities based on the area. For instance, the feet are
generally more sensitive to haptic feedback, and portions of thehuman body where skin-to-bone contact is more direct may
resonate differently when encountering haptic feedback [17][18].
A solution to this would be to utilize full-body haptic suits, or, or
considering an economical scale, wearable haptic technology thatcan be fitted onto any portion of the body. It would then be
possible to manipulate with portions of the body receive haptic
feedback, and how this relates to the perception of traditional
patterns can be explored.

Many traditional Japanese patterns fit the description ofgeometrically simple, repeating shapes. Specifically, the traditionof wagara and its many sub-patterns evoke a similar repeating,patterned structure like Indonesian Batik. Unlike Batik, however,common wagara patterns like asanoha, yagasuri, and kikko, aremuch more angular in shape, featuring sharp corners and distinct
edges [15]. Thus, the perception of movement that the patterns
can invoke within exhibition visitors can be drastically different,
contrasting the “warm” and “soothing” undulations of batik.

© 2024 Information Processing Society of Japan

Fig. 5: Software, hardware, andsensory flow of exhibition

615

情報処理学会 インタラクション 2024
IPSJ Interaction 2024

4.3 Conclusion

Culture is a form of human expression that is transferable acrossgenerations, encompassing a long timescale. The way individuals
perceive culture is dependent on this developed, generational
perception, creating unique identities for human beings to rally
under. As this process takes decades, if not centuries, there areclear disconnects between different cultural groups, and theperception of cultural artifacts. Utilizing technology, forming
novel connections between individuals and culture is not only
possible, but is actively being undertaken to ensure culturalpreservation.
Utilizing haptic technology specifically allows for a direct skinto-culture contact that cannot be achieved through other sensory
means. As experienced through the final exhibition, individuals
were able to “feel” a culture they had never experienced, via
novel means. Through this, there is a precedent not only for
haptic and cultural research, but research regarding human perception
and memorability. Similarly, beyond an artistic exploration, there is apotential for cognitive/psychological/neurological sciences to be involved
within the research itself.
In conclusion, by exploring culture through novel sensory andtechnological means, we hold the potential to connect individuals ofdifferent cultures, allowing us to communicate our cultural values to eachother in a global manner.

fi

fi

Szalontai, Á., Ünal-Logacev, Ö., ... Winter, B. (2022). The bouba/kiki
effect is robust across cultures and writing systems. PhilosophicalTransactions of the Royal Society B: Biological Sciences, 377(1841),
Article 20200390.
(https://doi.org/10.1098/rstb.2020.0390)[11]
The sensorama: One of the first functioning efforts in virtual
reality : history of information. (n.d.). Retrieved December 21, 2023,(https://www.historyofinformation.com/detail.php?id=2785)[12] Buchanan, L. (2008, April 3). Happy birthday, Rumble Pak. IGN.
(https://www.ign.com/articles/2008/04/03/happy-birthday-rumblepak)[13] Buchanan, L. (2008, April 3). Happy birthday, Rumble Pak. IGN.
(https://www.ign.com/articles/2008/04/03/happy-birthday-rumblepak)[14] Ramnath, R. (2020). PS5 Adaptive trigger for different GUNS in
Call of Duty BO: Cold War! [Video]. In YouTube.
(https://www.youtube.com/watch?v=Bzr7a1gnzgw)[15] What are “Wagara (traditional Japanese patterns)”? Meaning andPrayers Accompanying the Main Japanese Patterns. (2022, July 15).
Living in JAPAN | Living in Japan Is a Web Magazine Dedicated toForeign Nationals. We Provide a Lot of Useful Information Such as
Japanese Culture and Manners to Navigate Life in Japan. We Also
Provide Job Hunting and VISA Support.
(https://livinginjapan.net/2022/07/00681/)[16] What are “Wagara (traditional Japanese patterns)”? Meaning andPrayers Accompanying the Main Japanese Patterns. (2022, July 15).
Living in JAPAN | Living in Japan Is a Web Magazine Dedicated toForeign Nationals. We Provide a Lot of Useful Information Such as
Japanese Culture and Manners to Navigate Life in Japan. We Also
Provide Job Hunting and VISA Support.
(https://livinginjapan.net/2022/07/00681/)[17] Purves D, Augustine GJ, Fitzpatrick D, et al., editors. Neuroscience.
2nd edition. Sunderland (MA): Sinauer Associates; 2001.
Mechanoreceptors Specialized to Receive Tactile Information. Available(https://www.ncbi.nlm.nih.gov/books/NBK10895/)[18] O’Donoghue, Patrick et al. “Investigation of bone resonance during
femoral reaming in hip replacement surgery.” 2013 Seventh International
Conference on Sensing Technology (ICST) (2013): 5
Appendix:

References:

Appendix 1: Visualizer video link

[1] What is Culture? (n.d.). Retrieved December 21, 2023
(https://sphweb.bumc.bu.edu/otlt/mph-modules/PH/
CulturalAwareness/CulturalAwareness2.html)[2] Anderson, A. E. (2023, July 24). Batik 101: Where it comes from,
how it’s made, and how to use it in an interior. Architectural Digest.
(https://www.architecturaldigest.com/story/batik-explained)[3] Serafica Gischa. (2021, May 4). Payas agung, pakaian adat bali.
Kompas.Com.
(https://www.kompas.com/skola/read/2021/05/04/130847569/
payas-agung-pakaian-adat-bali)[4] The Jakarta Post. (n.d.). Differences between printed batik andauthentic batik 'tulis’. The Jakarta Post. Retrieved December 21,
2023,(https://www.thejakartapost.com/life/2018/02/23/differencesbetween-printed-batik-and-authentic-batik-tulis.html)[5] Sarosa, A. P. (2017, November 15). Tips Memilih Kain batik Agar
Tidak Tertipu. TEMPO.CO.
(https://cantik.tempo.co/read/1034163/tips-memilih-kain-batikagar-tidak-tertipu)[6] The Jakarta Post. (n.d.-b). Preserving traditional values throughstamped batik. The Jakarta Post. Retrieved December 21, 2023, fromhttps://www.thejakartapost.com/life/2018/10/07/preserving-traditionalvalues-through-stamped-batik.html[7] Molnar, C., &#38; Gair, J. (2015, May 14). 17.3 Taste and smell.
BCcampus.
(https://opentextbc.ca/biology/chapter/17-3-taste-and-smell/)[8] professional, C. C. medical. (n.d.). Synesthesia. Cleveland Clinic.
Retrieved December 21, 2023,(https://my.clevelandclinic.org/health/symptoms/24995-synesthesia)[9] Ramachandran, V. S., & Hubbard, E. M. (2001). Synaesthesia--awindow into perception, thought and language. Journal of Consciousness
Studies, 8(12), 3–34.
[10] Ćwiek, A., Fuchs, S., Draxler, C., Asu, E. L., Dediu, D., Hiovain, K.,
Kawahara, S., Koutalidis, S., Krifka, M., Lippus, P., Lupyan, G., Oh, G.
E., Paul, J., Petrone, C., Ridouane, R., Reiter, S., Schümchen, N.,

https://www.youtube.com/shorts/5G3oGeqvPrM© 2024 Information Processing Society of Japan

fi

1P-96
2024/2/28

Appendix 2: Exhibition images

Appendix 3: Hardware Specifications

Table 1: Full Experience Hardware Specifications
HardwareDimensions

Foster AP15d
Ampli er

110mm (W) x30mm (H) x75mm (D)Acouve VP2
HapticTransducers

43mm
diameter x
15mm(t)OutputType
Digitalsignalout
HapticfeedbackoutSonyWH-1000xm5
Headphones

-

Audio
signalout

Monitor1280*1280p

Vi s u a lfeedbackout

Function
Ampli es audio
signals, feeds
signal intotransducers
Ampli es lower
frequencies,
allowing haptic
feedback to beexperienced
Noise-cancelingheadphones
allow for fullaudio immersion
Viewing thevisualizer

616