3URFHHGLQJVRIWKH60&WK6RXQGDQG0XVLF&RPSXWLQJ&RQIHUHQFH-XO\3DGRYD,WDO\C. ELEGANS MEETS DATA SONIFICATION: CAN
WE HEAR ITS ELEGANT MOVEMENT?
Hiroko Terasawa1, Yuta Takahashi1, Keiko Hirota1, Takayuki Hamano2,
Takeshi Yamada1, Akiyoshi Fukamizu1, Shoji Makino1
Life Science Center of TARA, University of Tsukuba, Ibaraki, Japan.1
JST, ERATO, Okanoya Emotional Information Project, Tokyo, Japan. 2
terasawa@tara.tsukuba.ac.jp, y-takahashi@tara.tsukuba.ac.jp,
hirota@tara.tsukuba.ac.jp, hamano@japan.com, takeshi@cs.tsukuba.ac.jp,
akif@tara.tsukuba.ac.jp, maki@tara.tsukuba.ac.jp

ABSTRACT

Molecular biologists are currently experiencing majoradvances in their research methods. The development ofhigh-resolution video recording and the usage of fluorescent protein tags have greatly expanded the possibilities
for in-situ observation (i.e. non-destructive, real-timeobservation of living organisms), leading to high expectations for new discoveries by observing dynamic motions.
Although such dynamic data are currently observed by“eyeballing” video data, we expect that sonifying temporal elements from video data could be equally advantageous to visual display, offering another modality of data
observation in molecular biology.

We introduce our video-data sonification of Caenorhabditis elegans (C. elegans), a small nematode worm thathas been extensively used as a model organism inmolecular biology. C. elegans exhibits various kinds of
movements, which may be altered by genetic manipulations. In pursuit of potential applications of data sonification in molecular biology, we converted video data of thisworm into sounds, aiming to distinguish the movements
by hearing. The video data of C. elegans wild type andtransgenic types were sonified using a simple motiondetection algorithm and granular synthesis. The movement of the worm in the video was transformed into thesound cluster of very-short sine-tone wavelets. In theevaluation test, the group of ten participants (from both
molecular biology and audio engineering) were able todistinguish sonifications of the different worm types withan almost 100% correct response rate. In the postexperiment interview, the participants reported more detailed and accurate comprehension on the timing of theworm’s motion in sonification than in video.

Since sonification is still a new and unconventional
approach for many people outside the auditory display
community, we suspect suddenly switching to an abstract
auditory display might seem initially implausible to biologists. In order to gain acceptance of sonification as aconvincing research method, we need simple andstraightforward sonification examples, where the causality between the original data and the resulting sounds canbe easily grasped. Therefore, we employed video recording, which is often assumed to be the most concrete visual data, for our preliminary investigation in sonification.
Our goal in this study is to examine whether we could“hear the movements we see” in a transparent manner,and thus to motivate further research directions.

1. INTRODUCTION
1.1 Background and Goal
One of the most promising directions in data sonificationis the sonification of time-series data, because auditory
perception is very sensitive to changes in time [1, 2].
EEG data sonification and seismic data sonification offer
successful examples that intuitively display transitions
over time, inspiring sonification of other kinds of dynamic, time-series data [3, 4]. In disciplines such as biology, researchers observe organisms by visualization orquantitative measurements, and sonification has seldom
been applied as a data-observation technique. However,biological research investigates temporal change of life
and organisms, and we expect that sonification may
provide a means to comprehend biological phenomena
from a new angle.

1.2 Dynamic Movements of Model Organisms
Caenorhabditis elegans (C. elegans) is a small nematode,
only about 1 mm long and formed from precisely 959
somatic cells. In the 1960s, Sydney Brenner began using
the tiny worm to study the genetics of development, andit has since been used extensively as a model organism[5]. Brenner, John Sulston, and Robert Horvitz shared the2002 Nobel prize in physiology or medicine for their discoveries in C. elegans concerning genetic regulation ofdevelopment and programmed cell death.
C. elegans has been a popular model organism becauseof its favorable characteristics for biological experiments.
Various genetic and biological techniques have been invented, enabling experiments on development, the nervous system, and aging/longevity.

Copyright: © 2011 Terasawa et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License 3.0
Unported, which permits unrestricted use, distribution, and reproduction
in any medium, provided the original author and source are credited.

3URFHHGLQJVRIWKH60&WK6RXQGDQG0XVLF&RPSXWLQJ&RQIHUHQFH-XO\3DGRYD,WDO\

1.3 Sonification of Image and Motion

Wild type: Bristol N2 wild type was provided by the
Caenorhabditis Genetics Center [14].
Red-fluorescent type: We prepared transgenic worms
expressing red fluorescent protein in their pharynxes (theworm’s throat) by injecting DNA (promoter myo2::DsRed plasmid) into wild-type worms [15].
Rolling type: To generate transgenic worms displaying
the rolling movement phenotype, the plasmid containing
the rol-6 (su1006) gene was injected into wild-type
worms [16].

“What we see” in video data is an image in motion, andthe sonification of a moving image poses its own uniquechallenge, in contrast to the sonification of a static image
or of motion. One common approach in the sonificationof a static image is to unfold the image data into a rastersequence of pixel data and to interpret it as a waveform
representation of sound [6, 7]. However, this method
does not translate the visual sensation of up/down andleft/right into audio in an intuitive way, and information
about the object position is lost. Meanwhile, sonificationof motion is often approached using acceleration sensor
data or motion-capture data, at pre-selected measuring
points [8, 9]. Substituting sensors and motion capture
with video data has also been proposed, such as in humangait sonification, for which Boyd et al. extracted a phase
configuration that describes the timing pattern of motionsin the gait and sonified that data [10].

C. elegans was grown on E. coli lawn (as food), on
agar plates. Video recording was done using a Leica
MZFL III microscope and Leica DFC500 digital camera
with a resolution of 1168x878 pixels.
Out of these recordings, we prepared four videos (A, B,
C, and D) of 20-second duration each. Table 1 shows thelist of worm types and video data, and Fig. 1 shows thescreenshots of the videos.

As that example indicates, video data represent raw
image data, and so it is necessary to computationally extract meaningful and intuitive information about visualobjects (such as size, speed, position, etc.) and eliminate
what is unnecessary background image. Pelletier discussed the matching between visual object and sound object
and proposed a framework to sonify the vector representation of corner displacement, which is a perceptuallysalient feature in vision [11]. This system could employ
any kind of synthesis method, but the use of granularsynthesis [12], which can represent the addition of largenumber simple components, is suggested as one of thenatural choices.

Type of Worm

Video Data

Wild

A, C

Red-fluorescent(transgenic)

B

Rolling (transgenic)

D

Table 1. The List of C. elegans Types and Video Data

1.4 Framework of the StudyIn this work, we sonified video data of C. elegans wild
type and transgenic types. Worm movement in the video
was transformed into sound cluster, using a simple motion-detection algorithm and granular synthesis. We examined the resulting sounds with an evaluation test, inwhich both biologists and audio engineers participated.
The effect of sonification was measured with an identification task, in which the participants judge which video
the presented sound was generated from. In the next sections, we describe the genetic manipulation, sonification
method, and evaluation test, followed by discussion andideas for further research.
2. GENETIC MANIPULATION OF C.

ELEGANSIn this study, we investigated three kinds of C. elegans,the wild type, the red-fluorescent type, and the rolling
type. The latter two types were transgenic strains, andthey were generated using standard microinjection methods [13], in which DNA solution was injected into worm
gonads.

Figure 1. The snapshots of the video data (A: top-left, B:
top-right, C: bottom-left, D: bottom-right).

3URFHHGLQJVRIWKH60&WK6RXQGDQG0XVLF&RPSXWLQJ&RQIHUHQFH-XO\3DGRYD,WDO\

3. SONIFICATION DESIGN AND SOUND

the pixels for the entire worm is heard as the sum of all
the corresponding wavelets.

SYNTHESISThe mapping for the granular synthesis was decided
upon by trying several configurations. We designed thevertical axis to correspond with the pitch, the horizontal
axis to correspond with the attack-time and duration, andthe pixel value to correspond with the intensity of the
wavelet. All of these acoustical characteristics are designed to vary exponentially based on the perceptual scaling [18, 19].

3.1 System Overview
The algorithm for sonification was implemented using
Max/MSP/Jitter [17] as shown in Fig. 2. We apply a simple motion-detection algorithm on the video data to extract the moving worm from the background, then theimage resolution is rescaled to 80 x 60 pixels. The downsized video shows a rough figure of the worm with acluster of pixels. Granular synthesis translates the clusterof pixels into a sound cluster.

4. EVALUATION TEST

3.2 Motion Detection

4.1 Procedure

The worm is filmed on an agar plate and exhibits somebackground objects such as traces of the earlier movements. In order to extract the moving worm, the system
reads the video frame every 40 milliseconds. The absolute difference between the current read-out frame andthe prior read-out frame is calculated, enhanced by raising the value to the fourth power. Then the extracted motion is smoothed out using envelope-following. The resolution is rescaled to 80 x 60 pixels to minimize the dataflow, and thus the computational power required for granular synthesis.

We conducted an evaluation test to judge the effect of thesonification using an identification task. We designed thetask to resemble the expected use, in which the users arewell informed about the sonification concept and the algorithm.
Four sounds (A, B, C, D) were created from the four
video-data sources (A, B, C, D, respectively). Then four
video clips with matching sounds (i.e., video A and soundA) were produced. These videos are available on our
website:http://www.tara.tsukuba.ac.jp/~terasawa/Worms/SMC2011.htm

The participants first received an explanation of thesonification algorithm by watching the Max/MSP/Jitter
patch, and then they proceeded to the practice session, in
which they watched the four video clips together withsound a few times to familiarize themselves with thesound. In the test session, the participants then listened tothe sounds only and were asked to identify the video fromwhich the sound was generated. During the test session,
we provided a card showing the snapshots of the video
with the names (A, B, C, D), so that the participants did
not need to memorize the video names. The participants
were allowed to listen to the stimuli several times if they
wished.
Each stimulus was presented four times, resulting in 16
stimulus presentations. The stimuli were presented in arandomized order, but ensuring that any particular stimulus would not appear twice in succession. The chancelevel for making a correct response at each trial is 25 %.

Figure 2. Max/MSP/Jitter implementation of the sonification algorithm.

4.2 Participants and Environment
Five molecular biologists (three graduate students andtwo faculty members) and five audio engineers (twograduate students and three faculty members) participatedin the experiment.

3.3 Granular Synthesis
The extracted worm image is sonified using the granularsynthesis technique, since the granular synthesis is suitable to capturing pixel representation of the image andreflects the complexity of the image into sound. For eachpixel of the worm, a small particle of sine wave (wavelet
or sound grain) was generated using the parameters ofhorizontal and vertical positions in the frame (x and yaxes) and the intensity of the pixel value. The cluster of

The test was conducted in a normal laboratory officespace (i.e., not particularly quiet) to resemble realistic
user conditions. The stimuli were presented with built-in
audio of a laptop computer (Apple MacBook Air) with aclosed-type stereo headphone (Sony MDR-7506). The
participants adjusted the volume to a comfortable level.

3URFHHGLQJVRIWKH60&WK6RXQGDQG0XVLF&RPSXWLQJ&RQIHUHQFH-XO\3DGRYD,WDO\

4.3 Questionnaire and Interview

With questions 2 and 3, both molecular biologists andaudio engineers provided the same type of responses.
Most of them recognized the change in position, but they
were not confident that they heard the “rhythm” of the
movement. We asked question 3 expecting that the participants could identify some patterns of the movement.
Perhaps the use of the word “rhythm” was not appropriate because it implies very periodic patterns, while theworm shows only pseudo-periodic patterns with its
movement.

After the test session, we asked four yes/no/maybe questions. The questions were:(1) Is it easy to associate the sound and video?
(2) Can you hear the change of the worm’s position?
(3) Can you hear the rhythm of the worm’s movement?
(4) Do you think you will improve your hearing withmore practice?

With question 4, all of the participants answered thatthey could improve the hearing by practice. The participants showed almost 100% accuracy in the identification
task. Improving the hearing would lead to the easiercomprehension of the sounds, if the accuracy is alreadyaccomplished.

After the participant answered these questions, theexperimenter had a free-form interview with the participant to gather useful comments and suggestions.
5. RESULTS
5.1 Evaluation Test

Question#

The mean correct response rate from the identification
task is shown in Table 2, which was calculated by averaging the percentage of correct responses across the participants.
All the molecular biologists performed the identification
task perfectly. Audio engineers performed slightly lesswell, but still above 95% correct. Overall, the participants
performed the identification task very accurately.

Group

Mean Correct
Response Rate

All

98.2%

Molecular Biologists

100%

Audio Engineers

96.4%

Molecular
Biologists

Audio
Engineers

1

40%

90%

2

100%

90%

3

60%

60%

4

100%

100%

Table 3. The “Yes” Answer Percentage of the Questionnaire

6. DISCUSSION
During the interview, many of the participants stated thatthe main identification cues were the density of sound(i.e., the amount of wavelets) and the pattern of pitchchange (i.e., vertical displacement).
The density of sound becomes very low with the redfluorescent type. Because the number of visible cells in
the worm’s body is very few, only a small number ofwavelets exist in the sound. With the other types, the density of sound is low when the worm is stopping or showing only tiny movements. The participants used the timing of sparse sounding and silence as a cue to identify the
movement patterns.

Table 2. The Percentage of Correct Responses in the
Identification Task

5.2 Questionnaire
The percentage of “yes” answers for the questionnaire isshown in Table 3. The percentage was calculated by taking the sum of responses by counting “yes” as 1, “no” as
0, and “maybe” as 0.5, divided by the number of participants.

The biologists also reported that they were able to focus on the worm movements more accurately with soundthan with video. There are some moments when theworms briefly stop their motion. With sounds, such moments are easily detected. But with video, the participants
tend not to notice such moments, and have the perceptionthat the worms are moving smoothly without any interruption.

With question 1, molecular biologists and audio engineers showed different attitudes with their confidence inhearing. This difference may be because audio engineers
are more used to working with sounds, or that there weremore musicians among the audio engineers.

3URFHHGLQJVRIWKH60&WK6RXQGDQG0XVLF&RPSXWLQJ&RQIHUHQFH-XO\3DGRYD,WDO\International Community for Auditory Display Editorial Committee and Co-Authors, 1999.

Both engineers and biologists reported they used thedensity and pitch cues concurrently. However, the parameters that corresponded to the horizontal position (attack time and duration) did not seem to affect the perceived quality of sound. With the use of sound cluster,
the wavelets overlap each other, and such overlaps may
preclude accurate perception of attack time and duration.

7. CONCLUSION AND FUTURE WORKIn this study, we investigated the potential of data sonification in molecular biology. The movements of wild-type
and transgenic C. elegans were sonified using motion
detection and granular synthesis, so that the sound clusterof wavelets represents the visual cluster of pixels in aperceptually matching manner. The evaluation test showed that both molecular scientists (C. elegans specialists)
and audio engineers (non-specialists) could accurately
comprehend the motion of worms through hearing, demonstrating that data sonification may have a strong potential for applications in molecular biology.
From this collaboration of biologists and audio engineers, various ideas for future directions are emerging.
Technical ideas for improving sonification include theuse of spatial audio, different kinds of mapping in thesynthesis, and using acceleration as a parameter insteadof position. However, beyond the technical ideas, wecame to realize the potential of sonification in discovering new knowledge in molecular biology. Investigating
the rhythmic aspects in the dynamic motion of worms
would be of interest, such as the pumping gesture observed at the throat of a worm. Sonification may also be
useful for observing the behaviors of model organisms.

[2]

S. Barrass and G. Kramer, “Using sonification,” inMultimedia Systems No. 7, 1999, pp. 23-31.

[3]

T. Hermann et al., “Vocal Sonification of
Pathologic EEG Features,” in Proceedings of the
12th International Conference on Auditory Display,
London, 2006, pp. 158-163.

[4]

F. Dombois, “Auditory Seismology on Free Oscillations, Focal Mechanisms, Explosions and Synthetic Seismograms” in Proceedings of the 2002 International Conference on Auditory Display, Kyoto,
2002, pp. 1-4.

[5]

S. Brenner, “The genetics of Caenorhabditis elegans,” in Genetics No. 77, 1974, 71-94.

[6]

W. S. Yeo and J. Berger, “Raster Scanning: A New
Approach to Image Sonification,” in Proceedings ofthe International Computer Music Conference,
2006.

[7]

K. Jo and N. Nagano, “Monalisa: See the Sound,Hear the Image” in Proceedings of International
Conference on New Interfaces for Musical Expression, 2008.

[8]

S. Barrass, N. Schaffert, and T. Barrass, “Probing
Preferences between Six Designs of Interactive
Sonifications for Recreational Sports, Health andFitness,” in Proceedings of ISon 2010, 3rd Interactive Sonification Workshop, Stockholm, 2010.

[9]J. M. Pelletier, “Sonified Motion Flow Fields as a
Means of Musical Expression” in Proceedings ofInternational Conference on New Interfaces forMusical Expression, 2008.

In this project, we sonified the already-visible aspects
and discovered that sounds can convey some informationthat we tend to dismiss with vision. However, the sonification of non-visual aspects in biology is a further promising direction. “Listening to the phenomena we cannot
see at all” may lead to the most fascinating new
discoveries, and seeking such model examples is a desirable next-stage goal.

[10] J. E. Boyd and A. Sadikali, “Rhythmic Gait Signatures from Video without Motion Capture,” inProceedings of the16th International Conference on
Auditory Display, Washington, D.C, 2010, pp 187191.

Acknowledgments[12] C. Roads, Microsound, MIT press, 2002.

We would like to thank Peter Wang for his generous support in preparing this manuscript. This work was supported by the Kawai Foundation for Sound Technology andMusic, Japan.

[13] C. C. Mello, et al., “Efficient gene transfer inC.elegans: extrachromosomal maintenance and integration of transforming sequences,” in EMBO J.
No. 10, 1991, pp. 3959-3970.

[11] J. M. Pelletier, “Perceptually Motivated Sonification of Moving Images,” in Proceedings of the International Computer Music Conference, 2009.

[14] Caenorhabditis Genetics Center, University of
Minnesota. URL: http://www.cbs.umn.edu/CGC/

8. REFERENCES[1][15] P. G. Okkema, et al., “Sequence requirements formyosin gene expression and regulation in
Caenorhabditis elegans,” in Genetics No. 135, 1993,
pp. 385-404.

G. Kramer et al., “The Sonification Report: Statusof the Field and Research Agenda,” Prepared for
the National Science Foundation by members of the3URFHHGLQJVRIWKH60&WK6RXQGDQG0XVLF&RPSXWLQJ&RQIHUHQFH-XO\3DGRYD,WDO\[16] C. A. Peixoto, et al., “Ultrastructural analyses of
the Caenorhabditis elegans rol-6 (su1006) mutant,which produces abnormal cuticle collagen,” in J.
Parasitol No. 84, 1998, pp. 45-49.
[17] Cycling 74, Max 5. URL: http://cycling74.com/[18] E. Zwicker and H.Fastl, Psychoacoustics: Facts andModels, Springer, 1999.
[19] S. McAdams et al., “Perceptual scaling of synthesized musical timbres: Common dimensions,specificities, and latent subject classes,” in Psychological Research 58, 1995, pp. 177–192.

