Results from this study actually show that context-sensitive feature can help to better analyze gesture expressivity. The same expressive features are differently weighted depending on their spatial and temporal rarity. It put in evidence salient human motion either at the level of a single individual (rarity of behavior over time) or at the collective level (relative rarity of one member's behavior with respect to the others).

We plan to further investigate the potentialities of the rarity index in two directions: (i) by applying it to a more sophisticated set of expressive features (.e.g contraction, expansion, fluidity, impulsiveness) (ii) by analyzing how the visual feedback computed on the rarity index affect the subject behavior (e.g. whether it fosters expressive behavior).

## ACKNOWLEDGMENTS

This work has been achieved in the framework of the eNTERFACE 2008 Workshop at the LIMSI Lab of the Orsay University (France). It was also included in the Numédiart excellence center (www.numediart.org) project 3.1 funded by the Walloon Region, Belgium. The authors thank to Johan DECHRISTOPHORIS who helped in IR sensor hardware components set-up. Finally, this work has been partially supported by the Walloon Region with projects BIRADAR, ECLIPSE, and DREAMS, and by EU-IST Project SAME (Sound And Music for Everyone Everyday Everywhere Every way).

[13] Mancas, M., "Computational Attention: Towards Attentive Computers", Similar edition, CIACO University Distributors, ISBN: 978-2-87463-099-6, 2007

[14] Mancas M., Gosselin B., Macq B., "A Three-Level Computational Attention Model", Proc. of ICVS Workshop on Computational Attention & Applications, 2007, Germany.

[15] Hubel, D.H., "Eye, brain and vision", New York: Scientific American Library, N°22, 1989

[16] Mancas, M., "Image perception: Relative influence of bottom-up and top-down attention", Proc. of the WAPCV workshop of the ICVS conference, Santorini, Greece, 2008

[17] Velastin, S., Boghossian, B., Lo, B., Sun, J., Vicencio-Silva, M.: Prismatica: toward ambient intelligence in public transport environments. IEEE Trans. Syst.Man Cybern. Part A 35(1), 164–182, 2005

[18] Camurri, A., De Poli, G., Leman, M., and Volpe, G., 2005. "Toward Communicating Expressiveness and Affect in Multimodal Interactive Systems for Performing Art and Cultural Applications", IEEE Multimedia, 12,1, 43-53, 2005

[19] Glowinski, D., Camurri, A., Coletta, P., Bracco, F., Chiorri, C., Atkinson, A. An investigation of the minimal visual cues required to recognize emotions from human upper-body movements, (in press) Proceedings of the ACM 2008 International Conference on Multimodal Interfaces (ICMI), Workshop on Affective Interaction in Natural Environments (AFFINE) Crete, 2008

## REFERENCES

[1] Hongeng, S. and Nevatia, R., "Multi-agent event recognition" in ICCV, 2001, pp. II: 84-91

[2] eNTERFACE 2008 Workshop website: http://enterface08.limsi.fr/

[3] A. Camurri, A., Coletta, P., Varni, G., & Ghisio, S. "Developing multimodal interactive systems with EyesWeb XMI", Proceedings of the 2007 conference on new interfaces for musical expression (NIME07) (pp. 305-308). New York, USA, 2007

[4] OpenCV static wiki: http://opency library.sourceforge.net/wiki-static

[5] Dhavale, N., and Itti, L., "Saliency-based multi-foveated mpeg compression", IEEE Seventh International Symposium on Signal Processing and its Applications, 2003

[6] Yee, H., and Pattanaik, S., "Spatiotemporal sensitivity and visual attention for efficient rendering of dynamic environments", IACM, 2001

[7] Parkhurst, D.J., Niebur, E., "Texture contrast attracts overt visual attention in natural scenes", European Journal of Neuroscience, 19:783–789, 2004

[8] Itti, L., Baldi, P., "Bayesian Surprise Attracts Human Attention", Advances in Neural Information Processing Systems, Vol. 19 (NIPS 2005), pp. 1-8, Cambridge, MA:MIT Press, 2006

[9] Le Meur, O., Le Callet, P., Barba, D., and Thoreau D. "A Coherent Computational Approach to Model Bottom-Up Visual Attention", PAMI(28), No. 5, pp. 802-817, 2006

[10] Liu, F., and Gleicher, M., "Video Retargeting: Automating Pan-and-Scan", ACM Multimedia, 2006

[11] Zhang, V., and Stentiford, F.W.M., "Motion detection using a model of visual attention", IEEE ICIP, 2007

[12] Boiman, O., Irani, M., "Detecting Irregularities in Images and in Video", International Conference on Computer Vision (ICCV), 2005

## Matei Mancas.

<figure><img src="image_122.png" /></figure>

Matei Mancas was born in Bucarest in 1978. He holds an ESIGETEL (Ecole Supérieure d'Ingénieurs en informatique et TELcommunications, France) Audiovisual Systems and Networks engineering degree, and a Orsay University (France) MSc. degree in Information Processing. He also holds a PhD in applied sciences from the FPMs (Engineering Faculty of Mons, Belgium) on computational attention since 2007.

His past research interest is in signal and, in particular, image processing. After a study on nonstationary shock signals in industrial tests at MBDA (EADS group), he worked on medical image segmentation. He is now a Senior Researcher within the Information Processing research center of the Engineering Faculty of Mons, Belgium. His major research field concerns computational attention and its applications.

## Donald Glowinski.

<figure><img src="image_123.png" /></figure>

Donald Glowinski, Paris, 27-02-1977. He is doing a PhD in computing engineering at InfoMus Lab - Casa Paganini, in Genoa, Italy. (dir: Prof. Antonio Camurri). His background covers scientific and humanistic academic studies as well as high-level musical training.- EHESS (Ecole des Hautes Etudes en Sciences Sociales) MSc, in Cognitive Science, CNSMDP (Conservatoire National Supérieur de Musique et de Danse de Paris) MSc in Music and Acoustics, Sorbonne-Paris IV MSc. in Philosophy.

He was chairman of the Club NIME 2008 (New Interfaces for Musical Expression), Genoa, 2008. His research interests include multimodal and affective human-machine interactions. He works in particular on the modeling of automatic gesture-based recognition of emotions.
### Pierre Bretéché.

<figure><img src="image_124.png" /></figure>

Pierre Bretéché, Ivry sur Seine, 1981, received a MSc. degree in Computer Science in 2006 at University of Rouen (France). He is doing a PhD in Information and Communication Sciences at Laseldi Lab - University of Franche-Comté in Montbéliard (France). He previously worked in AI with a using massive multi-agent system to build semantic picturing of an environment.

He is now part of Organica research project. His research interest has moved to studying and designing new technology applications for public, cultural and artistic purpose.

### Jonathan Demeyer.

<figure><img src="image_125.png" /></figure>

Jonathan Demeyer received a MSc. in electrical engineering, Université Catholique de Louvain, Louvain-la-Neuve, Belgium in 2005 and a MSc. in applied sciences from the FPMs (Faculté Polytechnique de Mons, Belgium) in 2008.

He previously worked in the TCTS lab (Faculté Polytechnique de Mons, Belgium) developing a mobile reading assistant for visually impaired people.

He is now working on a project on automatic processing of high speed videoglottography for physicians. His main interests are medical image processing and computer vision.

### Thierry Ravet.

<figure><img src="image_126.png" /></figure>

Thierry Ravet was born in Brussels, Belgium on 31st Augustus 1976. He received an Electrical Engineering degree in 1999 in ULB (Université Libre de Bruxelles).

Afterwards, he worked as researcher in the Electronic - Microelectronic - Telecommunication department of ULB for 4 years with medical instrumentation projects.

Since February 2008, he has joined the TCTS Lab (Faculté Polytechnique de Mons). His main experiences are in non-invasive instrumentation, microprocessor system development, artefacts filtering and data fusion in the field of cardiorespiratory monitoring and polysomnographic research.

### Gualtiero Volpe.

<figure><img src="image_127.png" /></figure>

Gualtiero Volpe, Genova, 24-03-1974, PhD, computer engineer. He is assistant professor at University of Genova. His research interests include intelligent and affective human-machine interaction, modeling and real-time analysis and synthesis of expressive content in music and dance, multimodal interactive systems.

He was Chairman of V Intl Gesture Workshop and Guest Editor of a special issue of Journal of New Music Research on "Expressive Gesture in Performing Arts and New Media" in 2005. He was co-chair of NIME 2008 (New Interfaces for Musical Expression), Genova, 2008. Dr. Volpe is member of the Board of Directors of AIMI (Italian Association for Musical Informatics).

### Antonio Camurri.

<figure><img src="image_128.png" /></figure>

Antonio Camurri (born in Genova, 1959; '84 Master Degree in Electric Engineering; 1991 PhD in Computer Engineering) is Associate Professor at DIST-University of Genova (Faculty of Engineering), where he teaches "Software engineering" and "Multimedia Systems". His research interests include multimodal intelligent interfaces, non-verbal emotional and expressive communication, kansei information processing, multimedia interactive systems, musical informatics.

He was President of AIMI (Italian Association for Musical Informatics), is member of the Executive Committee (ExCom) of the IEEE CS Technical Committee on Computer Generated Music, Associate Editor of the international "Journal of New Music Research", a main contributor to the EU Roadmap on "Sound and Music Computing" (2007). He is responsible of EU

IST Projects at DIST InfoMus Lab of University of Genova. He is author of more than 80 international scientific publications. He is founder and scientific director of the InfoMus Lab at DIST-University of Genova (www.infomus.org). Since 2005 he is scientific director of the Casa Paganini centre of excellence at University of Genoa on research in ICT integrating artistic research, music, performing arts, and museal productions (www.casapaganini.org).

### Paolo Coletta.

<figure><img src="image_129.png" /></figure>

Paolo Coletta born in Savona, Italy, in 1972. He received the "Laurea" degree in computer science engineering in 1997 and the Ph.D. degree in electronic engineering and computer science in 2001, both from the University of Genoa, Genoa, Italy.

From January 2002 to December 2002, he was a Research Associate at the Naval Automation Institute, CNR-IAN (now ISSIA) National Research Council, in Genoa.

In 2004 he was adjunct Professor of "Software Engineering and Programming Languages" at DIST (Dipartimento di Informatica, Sistemistica e Telematica), University of Genoa. Since 2004 he is a collaborator at DIST, University of Genoa.
# Expressive Violin Synthesis
## A Study Case in Realtime Manipulation of Complex Data Structures

Nicolas d'Alessandro, Alexis Moinet, Thomas Dubuisson, *PhD students, Faculty of Engineering of Mons*<fnref n="1" />,
Jehan-Julien Filatriau, Wenyang Chu, *PhD students, Catholic University of Louvain*<fnref n="2" />

*Abstract*—This report gives the results of the project #10 of eNTERFACE 2008 summer workshop on multimodal interfaces. This research highlights the problems of content-oriented instrumental sound synthesis, and an example for the violin is developed. The project has been planned as an extensive sound analysis processus, where multiple aspects of a given database have been studied. Then different resynthesis strategies have been implemented and some relevant choices in this field are discussed.

*Index Terms*—violin, spectral analysis, granulation, viterbi

## I. INTRODUCTION

From the early days of electrical (even acoustical, through organs) modeling of instrumental sounds, a lot of work have been achieved. Today end-user solutions such as Garage-Band [1] give an incomparable access to high-quality musical sounds, allowing professionnal and non-professionnal musicians to compose nice soundtracks. With the same kind of temptation than in speech processing (thanks to high-quality unit selection synthesizers [2]) we could say that this topic reached a production level, where it is now just a matter of funding, in order to propose best collections of sounds.

However this is not really the point of view of performers. From their side, the possibility of producing high-quality musical sounds remains slightly schizophrenic. On the one hand, sensors and controllers have never been so user-friendly and exciting (e.g. WiiRemote), thanks to softwares like Max/MSP [3] and OSC protocol [4]. On the other hand, synthesis techniques often have to be re-designed from scratch (e.g. inside Max/MSP itself), as MIDI protocol is strangling the access to high-quality sample-based softwares. This is not just transmission conventions to be updated (e.g. OSC replacing MIDI), but also the architecture of these algorithms which is not compatible with more expressive setups.

Ircam's CataRT granulator [5] and its related softwares represents a significant break-through in this context. It brings creative ways of interacting with databases in the hands of the end-user. However we think that some important aspect of this "database-to-synthesis" pipeline could be re-discussed and more optimal solutions could be found. First because

instrument sound synthesis is a complex and particular pitch-synchronous case of granulation where usual texture synthesis benchmarking [6] is no longer efficient. Then because libraries like FTM/Gabor/Mnm propose an "all-in-the-patch" view of the computer music language which makes some simple things really unobvious to implement.

In this paper we propose to focus our investigation on a particular case study: the realtime synthesis of the violin. In section II we give an overview of what is the violin timbre from a perceptual point of view, thus allowing us to know what is the structure and dimensionality of the sound we want to control. In this project we use a monophonic violin database from FreeSound Project [7]. In section III we describe analysis algorithms that have been implemented offline in order to convert wavefiles into a structured cloud of frames. In section IV we finally explain some tracks that we followed in the field of data clustering and the possibility of generating a synthesis signal by browsing the cloud and selecting frames for OLA.

## II. BACKGROUND IN VIOLIN TIMBRE PERCEPTION

Timbre space is a feature space which is constructed by perceptual attributes selected based on human perceptual properties. The selection of perceptual attributes is one of the decisive factors of the success of granular synthesis of musical instruments. It decides not only the effectiveness of the segmentation of original sound tracks in the analysis phase but also that of unit selection from the granular database in the synthesis phase. The best selected attributes can construct a timbre space which allows to distinguish and classify all grains in the database.

To construct a timbre space which can be efficiently browsed and provides efficient mapping between feature trajectory and gesture trajectory, except pitch information, the signal processing timbral attributes of the violin has been seen the important clues to define the timbre space. However, only little research has been carried on the violins timbre space in a signal processing point of view. In the following survey, we try to summarize those important papers and make a list of the possible attributes.

### A. *Selection of the parameters*

1) *Categories of the attributes and their correlation:* In the paper of James McDermott et al. [8], a set of forty timbral, perceptual, and statistical sound attributes are defined, studied,

<footnote n="1">¹ Information Technology Research Center, Faculty of Engineering of Mons, Belgium. Address: Boulevard Dolez, 31 - B7000 Mons (Belgium). Email: firstname.lastname@fpms.ac.be.</footnote>

<footnote n="2">² Telecommunications and Teledetection Laboratory, Catholic University of Louvain, Belgium. Address: Place du Levant, 2 - B1348 Louvain-la-Neuve (Belgium). Email: firstname.lastname@uclouvain.be.</footnote>
and categorized into the following domains: (1) Time-domain, (2) Fourier-transform, (3) Partial-domain, (4) Trajectory, (5) Periodic, and (6) Statistical, where the high correlation between attributes within each subset has been observed [8]. To avoid having high correlation among selected attributes and to have the minimum number of the attributes, we will not choose more than one attribute within each set in the early analysis stage.

Since the most well known perceptual properties are pitch, loudness, brightness, and attack time, it is reasonable to first find the corresponding statistical attributes which can represent those perceptual sound properties better. Then we further eliminate the less dominant attributes in each subset. First, we obtain the following attributes (description in the quotients represents the corresponding perceptual sound attributes): fundamental frequency (pitch), RMS energy (loudness), spectral-centreofgravity(CoG) (brightness) [8]. The change of them over time, which is perceptual, shows another information – trajectories inside the timbre space. Spectral flatness measure (SFM) can show the distinction between harmonic and non-harmonic sounds [8]. We avoidtoppickupattributesdependin-gonVibratomeasures because it isisdifculttobedened mea-sured.

2) *Attributes especially for the violin:* The violin timbre and its perceptual attributes have been shown its dominant influence. Some interesting conclusion can be also drawn from the work of Jane Charles, etc [9]. Except the attributes mentioned in the last paragraph, SFM and the spectral contrast measure (SCM) are considered important because of their suitability as violin timbre features [9].

3) *Other perceptual study:* According to the research of Jan Stepanek [10] based on the human auditory experiment, levelsincriticalbands(barks), CoGand?overallspectrumlevelandlevelsofindividual play important roles in the perception. The experiment results show that the CoG and the first harmonic level have a dominant influence on the perception of violin tones [10].

### III. ANALYSIS OF EXPRESSIVE VIOLIN TIMBRE

This section aims at describing the features that have been computed for the five pieces of the database. A strategy for the mapping of a source piece to a target piece is also presented.

#### A. Computation of the Fundamental Frequency

The fundamental frequency $F_0$ (and its temporal correspon-dance, the fundamental period $T_0$) is surely one of the most obvious parameters for the description of musical sound. In a tonal music context, it is indeed related to the perceived note, played by the performe, inside a given scale.

In speech and music processing domains, many pitch de-tection algorithms have been proposed. All these algorithms have to deal with strong difficulties of estimation when the fundamental frequency is high (which may be the case if sources like violin or soprano voice is considered). Among the proposed algorithms, the YIN algorithm [11] emerged for the last years and proved its ability to deal with high fundamental frequencies. For the purpose of this project,

the fundamental frequency has been estimated in MAX/MSP environment, using the pitch detection tool included in the FTM/GABOR library. The values of fundamental frequency and the voiced/unvoiced decision have then been stored in files loadable in the MATLAB environment.

#### B. Pitch Contour Correction

Although YIN algorithm is known as efficient for musical sounds, it has been observed that the detected fundamental period suffers of several mistakes, essentially octave jumps. These jumps may be isolated or grouped into bursts. These artefacts have to be removed because there is a need of a reliable fundamental period, this value being the base of an efficient pitch-synchronous analysis. In order to correct the pitch contour, an algorithm consisting in two steps is proposed in this study.

First median filtering. This step aims at removing the isolated peaks in the pitch contour. It consists on computing, for every pitch occurence, the median of three values: the current pitch value and the two ones located on its left and right. Once the median filtered pitch contour is available, the difference between this contour and the original one is computed. For a given occurence, if the difference is higher than a given threshold (let us name it $Th1$), the pitch value is the one obtained after median filtering, otherwise the pitch value is the original one. The tests on different parts of the database showed that 100 seems to be a correct value for $Th1$.

<figure><img src="image_130.png" /><figcaption>Fig. 1. Pitch contour correction</figcaption></figure>

Then an anti-rebounce procedure. Although the previous step is able to deal with isolated peaks, it is not adapted to burst of peaks. In order to avoid these bursts, an anti-rebound procedure has been implemented. First the derivative of the corrected pitch contour from the first step is computed. It allows to highlight strong variations in the contour. Then, for each occurence of the derivative, if it is higher than a given threshold (let us name it $Th2$), a temporal interval is considered, beginning at the current pitch occurence and ending after a certain number of pitch occurences (let us name it $Th3$). If in this interval there is at least one value of derivative higher than $Th2$, the entire interval is tagged as a burst, which has to be removed. In the tagged interval, the corrected pitch values are obtained by computing the linear interpolation between the pitch values located at the boundaries of the interval. The tests on different parts of the
database showed that 50 and 5 seem to be a correct values for respectively *Th2* and *Th3*. Fig. 1 shows the two steps of correction for an piece of pitch contour computed for the first piece of the database. For each occurrence that has been corrected, its index is stored in order to avoid to use the corresponding frame in the synthesis operation.

### C. Computation of Pitch Marks

As said before, the aim of this study is to perform a pitch-synchronous analysis, meaning that the descriptors are computed for frames centered on particular instants called pitch marks. At this point, an analogy with speech has been made. Indeed, particular instants during the phonation correspond to Glottal Closure Instant (GCI). These GCIs occur every fundamental period and are particularly suited to sensitive analysis methods such as *ZZT* decomposition [12]. Besides these instants may be considered as particular pitch marks. For these reasons, it has been decided to compute analog instants for violin sounds, meaning that the same techniques than in speech are used. From now on, the computed pitch marks for violin sounds are called GCIs. Among the techniques proposed in litteratures, an interesting one use the temporal center of gravity (COG) [13] to locate GCIs:

$$Temporal\ COG = \frac{\sum_{i=1}^{N} n \times x(n)^2}{\sum_{i=1}^{N} x(n)^2} \quad (1)$$

where $x(n)$ stands for a particular violin frame. The COG is computed for frames whose hopsize is one sample. A first estimation of GCIs is obtained by finding the samples at which the COG crosses the zero-axis from positive to negative values. Fig. 2 shows the evolution of COG for successive periods of violin and Fig. 3 shows the detected GCIs for a sequence of COG.

<figure><img src="image_131.png" /><figcaption>Fig. 2. Temporal Center of Gravity</figcaption></figure>

Some GCIs may be missed during this analysis step, which is problematic for the following analysis. As explained above, the GCIs are spaced by the current fundamental period. This fact is exploited in order to detect the missing GCIs. For this purpose, the temporal interval between successive GCIs and the difference between the pitch contour and the evolution of this interval are computed. For each pair of successive GCIs, if this difference is higher in absolute value than 80 percents of the corresponding value, it is decided that a GCI has been

missed between the two successive GCIs. A new GCI is then inserted in the middle of the interval defined by the two existing GCIs. Fig. 4 shows an example of pitch marks insertion.

<figure><img src="image_132.png" /><figcaption>Fig. 3. Detected Pitch Marks from COG</figcaption></figure>

<figure><img src="image_133.png" /><figcaption>Fig. 4. Insertion of missing Pitch Marks</figcaption></figure>

### D. Pitch-Synchronous Analysis

Once the pitch marks are available, descriptors are computed for successive short time periods. In the case of pitch-synchronous analysis, these periods are frames centered on each pitch mark and whose length is twice the fundamental period associated to the considered pitch mark. In order to avoid strong discontinuities at the boundaries of the frame, each one is weighed by a Hanning window (very common in speech processing).

Many descriptors are proposed in signal processing literature, some of them being particularly dedicated to speech or music analysis. For this study, it has been chosen to consider some descriptors implemented in the framework of CUIDADO project [14] aiming at developing a new chain of applications through the use of audio/music content descriptors. The chosen descriptors are presented in the following, where $x(n)$ stands for a frame and $X(f)$ for its magnitude spectrum.

1) Energy: The temporal energy of the frame $x(n)$ is defined as below (expressed in dB).

$$Temporal\ Energy = 10 \times log_{10} \sum_{i=1}^{N} x(n)^2 \quad (2)$$
2) Total loudness: The loudness is defined as the energy in a band of the Bark scale [15]. This non linear scale has been defined so as two sinusoids of same amplitude and located in the same band are perceived in the same way while, if they are located in different bands, their perceived intensity is different. The Bark scale is used to mimic the behaviour of human ear in spectral domain. The loudness in a given frequency band is defined as below. The total loudness is the sum of all the loudness along the Bark scale. In this study, it has been chosen to consider 24 frequency bands.

$$ \text{Loudness} = \sum_{\text{Band}} X(f)^{0.23} \qquad (3) $$

3) Fundamental frequency (Hz): See Section III-A.

4) Spectral decrease: This descriptor aims at quantifying the amount of decrease of the amplitude spectrum. Coming from perceptual studies, it is supposed to be more correlated with human perception. It is computed as:

$$ \text{Decrease} = \frac{\sum_{f=2}^{\frac{f_e}{2}} \frac{X(f) - X(1)}{f-1}}{\sum_{f=2}^{\frac{f_e}{2}} X(f)} \qquad (4) $$

5) Spectral Center of Gravity: The spectral center of gravity (also known as spectral centroid) is a very common feature in MIR domain. Perceptually connected to the perception of brightness, it indicates where the "center of mass" of the spectrum is. The spectral center of gravity of the amplitude spectrum is known as an economic spectral descriptor giving an estimation of the major location of spectral energy. It is computed as:

$$ \text{COG} = \frac{\sum_{f=1}^{\frac{f_e}{2}} f \times X(f)}{\sum_{f=1}^{\frac{f_e}{2}} X(f)} \qquad (5) $$

6) Spectral Flatness Measure: The spectral flatness measure is defined as the ratio between the geometrical and arithmeric mean of the spectrum.

7) Tristimulus 1: According to the Bark scale, the first tristimulus is defined as a particular energy ratio:

$$ \text{Tristimulus}_1 = \frac{\sum_{f_{\text{Band}[1]}} X(f)}{\sum_{f_{\text{Band}[1...24]}} X(f)} \qquad (6) $$

where $f_{\text{Band}[1]}$ stands for the frequency bins included in the first Bark band and $f_{\text{Band}[1...24]}$ for the frequency bins included in the frequency band covered by Bark band from 1 to 24.

8) Tristimulus 2: According to the Bark scale, the second tristimulus is defined as a particular energy ratio:

$$ \text{Tristimulus}_2 = \frac{\sum_{f_{\text{Band}[2,3,4]}} X(f)}{\sum_{f_{\text{Band}[1...24]}} X(f)} \qquad (7) $$

9) Tristimulus 3: According to the Bark scale, the third tristimulus is defined as a particular energy ratio:

$$ \text{Tristimulus}_3 = \frac{\sum_{f_{\text{Band}[5...24]}} X(f)}{\sum_{f_{\text{Band}[1...24]}} X(f)} \qquad (8) $$

Energy and fundamental frequency are base dimensions of control in a synthesis system from a performer point of view. For our database, these two features are distributed as shown in Fig. 5.

<figure><img src="image_134.png" /><figcaption>Fig. 5. Energy vs Notes for the five pieces of the database.</figcaption></figure>

The total loudness has not been considered in the final synthesis system but it was computed for all the database. Fig. 6 shows the distribution of loudness vs notes for the five pieces of the database. It is interesting to note that it is highly correlated with the human perception (loudness curve).

<figure><img src="image_135.png" /><figcaption>Fig. 6. Total Loudness vs. Notes for the five pieces of the database.</figcaption></figure>

Considering the energy and the pitch in a synthesis system only allows to control the melodic part of the sound. Thus it would be interesting to include more dimensions in order
to reach other regions of expressivity (see [16] where more
than 2 dimensions are used to synthesize singing voice). As
3 dimensions remains manageable and viewable by human,
it has been chosen to select one spectral feature among the
ones defined above to be the third dimension of our synthesis
engine.

It has been observed that the COG and the
*SpectralFlatnessMeasure* are highly correlated.
Conserving these two descriptors is useless since they
bring similar information. It has been thus chosen to conserve
the COG because it is widely used in the MIR domain.
Concerning the three tristimuli, only the first is kept since
the two others are highly correlated with the first one. Fig.
7, 8 and 9 show respectively the distribution of COG,
*SpectralDecrease* and *Tristimulus*₁ for the first piece of
the database.

<figure><img src="image_136.png" /><figcaption>Fig. 7. Spectral CoG vs. Energy/Notes for the first piece of the database.</figcaption></figure>

<figure><img src="image_137.png" /><figcaption>Fig. 8. Spectral decrease vs. Energy/Notes for the first piece of the database.</figcaption></figure>

In order to select one feature among the three ones, the
3D space is projected on a 2D space by representing the
distribution of each of these features according to the fun-
damental frequency. Fig. 10, 11 and 12 show respectively the
distribution of COG, *SpectralDecrease* and *Tristimulus*₁
for the five pieces of the database according to the notes.

Among the three features, one has to find the one which
shows the highest variation for each zone of note and energy,
but with a sufficient density of frames in order to have large
possibilities of exploration. When looking at Fig. 10, 11 and
12, one can see that COG is the feature that meets the best
these conditions.

<figure><img src="image_138.png" /><figcaption>Fig. 9. First tristimulus vs. Energy/Notes for the first piece of the database.</figcaption></figure>

<figure><img src="image_139.png" /><figcaption>Fig. 10. Spectral CoG vs. Notes for the five pieces of the database.</figcaption></figure>

<figure><img src="image_140.png" /><figcaption>Fig. 11. Spectral decrease vs. Notes for the five pieces of the database.</figcaption></figure>

<figure><img src="image_141.png" /><figcaption>Fig. 12. First tristimulus vs. Notes for the five pieces of the database.</figcaption></figure>

Three features are thus considered in our system to car-
acterize violin frames: the energy, the fundamental frequency
and the spectral center of gravity. In order to test strategies of synthesis, the database is split in two parts: the frame belonging to the first piece and the ones belonging to the four other pieces. The distribution of these two parts is shown in Fig. 13.

<figure><img src="image_142.png" /><figcaption>Fig. 13. The two parts of the database in the 3D space.</figcaption></figure>

## IV. BROWSING THE CLOUD FOR SYNTHESIS

Let us note by $x_{(i,j)}$, the $j^{th}$ frame of the $i^{th}$ violin piece of the database and by $X_{(i,j)}$ its corresponding vector of features. We also name $x_t$ the frame $x_{(i,j)}$ selected at time $t$ and $X_t$ its vector of features.

We need to find a sequence of T database frames ($x_1, ..., x_T$) whose features ($X_1, ..., X_T$) best match a given sequence of features ($Y_1, ..., Y_T$), whether these are extracted from an actual violin piece or produced by some gesture-based heuristic. The sequence of $X_t$ needs to be consistent in many ways (realistic and continuous feature trajectories, overall acoustic and musical quality, ...). Moreover the selection process has to be fast enough to be used in a real-time environment such as Max or Pd.

### A. Clustering and Pruning

First a standard k-means clustering is performed on the $f_0$-Energy-CoG normalized space of features so that its 150000 frames are spread around 128 centroids $C_k$ ($k = 1, ..., 128$). This k-means clustering is achieved thanks to Spider<fnref n="1" />, a Matlab library of objects for machine learning purpose, released under the terms of the GPL. The result of the clustering in the $f_0$-Energy space is shown in Figure 14

Note that this clustering is an offline process that has to be done only once, when the database is created. Obviously a clustering for every $X_t$ sequence generation would be everything but a real-time process.

Then for every vector $Y_t$ that has to be matched by a vector $X_t$ from the database we find the k-means centroid $C_k$ that minimizes $M(Y_t, C_k)$, the Mahalanobis distance,

$$M(Y_t, C_k) = \sqrt{(Y_t - C_k)^T \Sigma^{-1} (Y_t - C_k)}, \quad (9)$$

<figure><img src="image_143.png" /><figcaption>Fig. 14. Result of the k-mean clustering. Horizontal axis is $f_0$, vertical axis is energy. Red dots marks the position of the cluster centroids.</figcaption></figure>

where $\Sigma$ is the covariance matrix of all the $X_{(i,j)}$ in the database.

$X_t$ will be selected amid the subset of vectors $X_{(i,j)}$ attached to that centroid. Let us denote by $X^{k_t}$ that subset.

In order to reduce the algorithm duration we can downsize that subset by discarding all the elements of $X^{k_t}$ whose Mahalanobis distance to the target vector $Y_t$ is beyond a certain threshold. Another approach is to limit the number of frames in the subset to the N closest to $Y_t$. We selected the latter in our implementation and fixed $N = 10$.

Finally the last step, detailed in section IV-B, is to add some continuity constraint when selecting one of the N remaining vectors in $X^{k_t}$ to match $Y_t$.

### B. Selection

The continuity constraint is added through a Viterbi-based method. Indeed the $X_t$ sequence has to minimize a global cost which is a cumulative sum of target and concatenation costs.

The target costs at time $t$ are computed as the Mahalanobis distance between each one of the N $X^{k_t}$ and the target $Y_t$. These distances have already been computed during the pruning step.

As for the concatenation cost, it corresponds to “the price to pay” to go from one of the N vectors of $X^{k_{t-1}}$ at time $t-1$ to one of the N vectors of $X^{k_t}$ at time $t$. A zero cost should correspond to a transition between two vectors that are in successive order in the original data used in the database.

For instance, if one of the $X^{k_{t-1}}$ is $X_{(2,64)}$ and one of the $X^{k_t}$ is $X_{(2,65)}$ (i.e. respectively the 64$^{th}$ and 65$^{th}$ frames of the second violin piece), the concatenation cost between the two should be null since they can be concatenated seamlessly. Another way to phrase this is : “if the frame at time $t-1$ is $X_{(i,j)}$ then the optimal frame at time $t$ would be frame $X_{(i,j+1)}$”.

Therefore we measure the concatenation cost between a feature vector $X_{(i,j)}$ from $X^{k_{t-1}}$ and $X_{(i',j')}$ from $X^{k_t}$ as the Mahalanobis distance between $X_{(i,j+1)}$ and $X_{(i',j')}$.

Finally the sequence $X_1, ..., X_T$ is obtained by using the Viterbi algorithm to minimize the overall cost $\delta_T$.

$$\delta_t = \min(\delta_{t-1} + M(X_{(i,j+1)}, X_{(i',j')}) + M(X^{k_t}, Y_t), \quad (10)$$

where each $\delta_t$ is a N-dimension vector and

<footnote n="1">¹http://www.kyb.tuebingen.mpg.de/bs/people/spider</footnote>
$$ \delta_1 = \mathcal{M}(X^{k_1}, Y_1) \quad (11) $$

Then $X_T$ is selected as

$$ X_T = \operatorname{argmin}(\delta_T), \quad (12) $$

and then the backtracking step of the algorithm selects every $X_t$, starting with $t = T - 1$, and so on.

$$ X_t = \operatorname{argmin}(\delta_t + \mathcal{M}(X_{(i,j+1)}, X_{t+1})), \quad (13) $$

that is to say $X_t$ will be the $X_{(i,j)}$ whose following frame $X_{(i,j+1)}$ minimizes the overall cost to go to $X_{t+1}$.

## V. CONCLUSION

In this project we had the opportunity of extensively study a sound space which was not in the track of what we did before. It gave us the possibility to develop our analysis tools (pitch analysis, pitch marking, spectral feature computation) in new directions. The problem of content-oriented sound synthesis with extra-small timing resolution has also been developed. In this context we can conclude that violin sound is particularly difficult to process by overlap-add and that new tracks have to be discussed in this field.

## ACKNOWLEDGMENT

The authors would like to thank their respective PhD supervisors for the possibility of participating to a long workshop in another lab and country. We also thank the LIMSI/CNRS lab for the organisation and support. Finally we would like to thank the RW/Numediat project because of this opportunity of combining both regional and international human resources.

## REFERENCES

[1] "Garage band," http://www.apple.com/ilife/garageband.

[2] "Loquendo," http://www.loquendo.com/en/technology/tts.htm.

[3] "Max/msp," http://www.cycling74.com/products/max.

[4] M. Wright, A. Feed, and A. Momeni, "Opensound control: State of the art 2003," in *Proceedings of NIME'03*, 2003, pp. 153-159.

[5] N. Schnell and D. Schwarz, "Gabor: Multi-representation real-time analysis/synthesis," in *Proceedings of DAFx'05*, 2005.

[6] M. Cardle and S. Brooks, "Directed sound synthesis with natural grains," in *Proceedings of the Cambridge Music Processing*, 2003.

[7] "Freesound project," http://www.freesound.org.

[8] J. McDermott, N. Griffith, and M. O'Neill, "Timbral, perceptual, and statistical attributes for synthesized sound," in *Proceedings of the International Computer Music Conference*, 2006.

[9] J. Charles, D. Fitzgerald, and E. Coyle, "Quantifying violin timbre," in *DMRN-06: DMRN Doctoral Research Conference*, 2006.

[10] J. Stepanek, "Spectral sources of basic perceptual dimension of violintimbre," in *CFA/DAGA*, 2004.

[11] A. de Cheveigné and H. Kawahara, "Yin, a fundamental frequency estimator for speech and music," *Journal of the Acoustical Society of America*, vol. 111, pp. 1917-1930, 2002.

[12] B. Bozkurt, L. Couvreur, and T. Dutoit, "Chirp group delay analysis of speech signals," *Speech Communication*, vol. 49, pp. 159-176, 2007.

[13] H. Kawahara, Y. Atake, and P. Zolfaghari, "Auditory event detection based on a time domain fixed point analysis," in *ICSLP 2000, Proceedings of the International Conference on Spoken Language Processing*, ISCA, Ed., vol. 4, 2000, pp. 664-667.

[14] G. Peeters, "A large set of audio features for sound description (similarity and classification) in the cuidado project," *IRCAM, Tech. Rep.*, 2004.

[15] E. Zwicker, "Subdivision of the audible frequency range into critical bands," *Journal of the Acoustical Society of America*, 1961.

[16] N. d'Alessandro and T. Dutoit, "Handsketch bi-manual controller," in *Proceedings of the NIME'07*, 2007, pp. 78-81.

<figure><img src="image_144.png" /></figure>

Nicolas d'Alessandro holds an Electrical Engineering degree from the FPMs since 2004. He did his master's thesis in the Faculty of Music of the University of Montréal. That work gathered the development of an application based on perceptual analogies between guitar sounds and voice sounds, and a study of mapping possibilities between gestures and speech production models. He just started a PhD. thesis in the TCTS Lab of the FPMs related to the real-time control of unit-based synthetizers.

<figure><img src="image_145.png" /></figure>

Alexis Moinet holds an Electrical Engineering degree from the FPMs (2005). He did his master thesis at the T.J. Watson research Center of IBM (2005). He is currently working on the IRMA project in the Signal Processing Lab of the FPMs. He is particularly interested in source/iter decomposition of speech and music, phase vocoder, voice activity detection, voice conversion and HMM synthesis.

<figure><img src="image_146.png" /></figure>

Thomas Dubuisson was born in Tournai, Belgium, in 1983. He received the Electrical Engineering Degree from the Faculty of Engineering, Mons (FPMs) in 2006. He is currently PhD Student in the Circuit Theory and Signal Processing (TCTS) Lab, FPMs, where he is working on assessment of speech pathologies and source-tract separation of speech, in the framework of the WALEO II Research Project ECLIPSE (Walloon Region, Belgium).

<figure><img src="image_147.png" /></figure>

Jehan-Julien Filatriau received the Engineering degree from Telecom Lille I in June 2002, and graduated a MSc in Acoustics at University of Marseille in June 2004. Since October 2004, he is research assistant at the Communications and Remote Sensing Lab of Université Catholique de Louvain. His research interests include sound processing, computer music and gestural control of audio systems; he started in 2005 a Ph.D. Thesis at UCL, dealing with the synthesis and control of sonic textures.

<figure><img src="image_148.png" /></figure>

Wenyang Chu received the B.S. degree in electronic engineering from Fu Jen Catholic University, Taipei, Taiwan in 2004, and the M.S. degrees in information and communication technologies from Universitat Politecnica de Catalunya (Spain) and Universite Catholique de Louvain (Belgium) in 2008. He currently is pursuing the Ph.D degree in Laboratoire de Telecommunications et Tele detection, Ecole Polytechnique de Louvain (UCL). His research interests include multimodal signal processing and fusion, and human-computer interaction.
# A Database for Stylistic Human Gait Modeling and Synthesis

Joëlle Tilmanne, Raphaël Sebbe, and Thierry Dutoit

***Abstract***—This project aimed at recording a database which can be used as training data for the development of three-dimensional statistical models of various human gait styles. The three-dimensional motion data was captured using the IGS-190 motion capture suit from Animazoo, an integrated wireless motion tracking system equipped with inertial sensors. Fourty participants were recorded, walking at several speeds and performing several direction changes.

***Index Terms***—motion capture, gait modeling, database.

## I. INTRODUCTION

Synthesis of realistic human motion is one of the greatest challenges in computer graphics, as classical animation methods are very time-consuming. A new trend in motion synthesis consists in using Machine Learning techniques to model and reproduce humanlike motion data. When using this kind of techniques, there is a need for representative data. This project is inscribed in this context, as its main goal is to record a database of three-dimensional human gait data, which could later be used for the training of statistical models of human gait and provide mechanisms to generate stylistic variations of these models.

In the next section, we give a brief introduction to the context of walk synthesis and character animation. In section 3, we introduce our database, and on what purpose it was built. In Section 4, we will give some information about the IGS-190 motion capture suit which is used for the recordings. Section 5 will explain the recording setup and instructions, and thus explain the content of the database. Section 6 will explain the format under which the data is represented. In the last section, we will make some remarks on the recorded data, and give some perspectives about the use of the database.

## II. ANIMATION OF VIRTUAL CHARACTERS

As of today, the only application where motion synthesis is truly used and is not only studied for research purposes is the animation field. Animation consists of the rapid display of a sequence of images, which creates an optical illusion of movement due to the human phenomenon of retinal persistence. The animation process is applied to various domains, ranging from the coarse motions seen in video games to the precise humanlike motions of 3D movies, and including fields like virtual reality or character animations for human-computer interactions. There are currently three main kinds of techniques which are used for motion synthesis in the animation field:

* keyframe animations (hand-driven),

* model-driven techniques in which a software based on a set of expert-based rules (i.e. the model) helps the animator for motion production,

* motion capture (data-driven technique).

Not that long ago, those methods were the only ones existing. Unfortunately, all of them have strong drawbacks. The solutions that exist for human character animation or synthesis of human motion in general are either time and labor consuming, or they are unrealistic and of poor quality.

For a few years now, a new trend has appeared. It consists of using statistical learning techniques to automatically extract the underlying rules of human motion, without any prior knowledge, directly from training on 3D motion capture data. Starting from the statistical models trained that way, new motion sequences can be automatically generated, using only some high-level commands from the user. Two movements generated by the same command (for example executing two consecutive steps in a walk sequence) will never be exactly identical. The result presents indeed a random aspect as can be found in the human execution of each movement, and becomes potentially more realistic than the repetition of the same capture sequence over and over. The movements produced that way are thus visually different, but are all stochastically similar to the training movements.

Bipedal locomotion is by nature a periodic time sequence of movements, which involve all body parts, with an underlying sequence of repeated phases. Following [1], human gait can be viewed as a dynamical process with a simple temporal structure (i.e., repetitions of basic stance and swing phases) but with several stylistic variations (i.e., walking, running, trudging, ...). Besides, the gait observations depict high variability that can be explained by the incapacity of a walker to produce perfectly twice the same movements and by the varying nature of its environment. All these elements create the naturalness that is so difficult to achieve in walking avatar animation. The modelling of such a process can be addressed by using probabilistic graphical models [2], and more especially Dynamic Bayesian Networks (DBN) [3] and Hidden Markov Models (HMM) [4], which provide mathematically tractable and computationally efficient solutions for their estimation.

## III. NEED FOR A DATABASE

In all Machine Learning techniques, the first major issue is to obtain enough representative training data. As only motion capture can give realistic 3D human motion data, it is the only way to obtain representative training data for statistical modeling of human motion. The first option is of course to
use an existing database of captured motions. Several motion capture databases are available for free on the Internet, like the widely known Carnegie Mellon University (CMU) Graphics Lab Motion Capture Database [5], or motion provided for free by a society called Eyes, from Japan, referred to as “motion-data.com” [6], but in both cases the motions are only roughly described with a single definition word like “walk”, “jump”, etc. Some other sites sell the motions, like “Truebones” [7] but again, the description of those movements is very brief as they are aimed at a direct use in an animation sequence. There is thus almost no documentation on the recorded sequences of motion, on the people who performed them, or on the motion capture area. This makes it hard to compare walk sequences from different people as they are not all recorded in the same conditions, and the information about each recording is very sparse. The HDM05 database from Bonn University [8], [9] gives a little bit more information, as they have the same sequence played by five different subjects. The sequence is a series of different motions corresponding to instructions like walk 5 steps, turn right, walk 5 steps backwards, etc. But there are no speed variations nor extended study of direction changes, and the subdivision of the whole sequence into each separate class of motion is not made for style variations classes of walk.

For model training, we needed better documented data. Moreover, we are not interested in having as many different sequences as possible, but in acquiring enough examples of the same motion to train a model. If possible, we also want to take into account more precise parameters in the training, like the style of the motion, which is, along with other factors, linked to the subject performing it. This project gave the opportunity to massively collect human gait data with an accurate motion capture system. For each of the fourty subjects of the database, the recording conditions and instructions were the same, which enables us to truly compare the inter-people variations. One could then benefit from this valuable database to estimate complex statistical models. The data are collected for various gait conditions, with an inertial motion tracking system, the Inertial Gyroscopic System (IGS 190) from Animazoo [10].

## IV. MOTION CAPTURE EQUIPMENT: THE IGS 190

In most research, the three-dimensional nature of human gait is overlooked [11]. Observations are often reduced to measurements from the sagittal plane (e.g., video recordings from the side plane) and symmetry between left and right sides is assumed. From this limited perspective, much valuable information is lost. Observations can be improved by including measurements from other planes such as coronal (e.g., front plane) and transverse (e.g., top plane). However, human gait is ideally observed via three-dimensional measurements. To do so, various motion capture equipments can be proposed. All so called “motion capture” systems have the same goal, which is to record a real motion by transcribing it under mathematical form usable by a computer. This is achieved by tracking a number of key points through space across time, and combining them to obtain a tridimensional unified representation of the performance [12]. The subject’s body

is then most of the time represented under the form of a kinematic chain, whose root is situated at the hip level, and whose various segments represent a kind of simplified representation of the skeleton.

Several techniques can be used for motion capture. First, multiple camera views can be triangulated to estimate body marker positions. Such system requires fine calibration and robust computer vision algorithms to yield accurate data [13]. Marker-less optical solutions are more comfortable yet more complex [14]. Besides, mechanical solutions allow directly measuring joint angles but require intrusive sensors. In this project, we adopt the Inertial Gyroscopic System (IGS) developed by Animazoo [10]. The main advantage of this system is to deliver remotely accurate data in a unified manner with no extra solving and over large capture areas.

The IGS 190 motion capture suit contains 19 inertial sensors (InertiaCube3 from Intersense) consisting each of a 3 axis accelerometer, a 3 axis gyroscope and a 3 axis magnetometer. The data from those three sources are integrated and combined directly in the inertial sensor box, and angle data is thus provided straight from the sensor. Those angles which give the configuration of the inertial sensor are then postprocessed realtime trough the software provided with the suit. The output of the system is the set of angles between the body segments, at a rate of 30, 60 or 120 frames per second. In this database recording, the frame rate is chosen to be 60 fps (frames per second).

Although each of the sensors is wired to the main processing unit which is attached to the suit, the system is wireless as this main processing unit acts as emitter and communicates trough radio frequency with the receiver linked to the laptop trough USB connection. The wireless range of the suit is around 100 meter indoors and up to several hundred meters outdoors.

The AutoCal software enables the user to build a skeleton fitting any new actor wearing the suit with just two photographs taken in a given stance and within a 3D cube used for setting the scale of the picture. The actor will need to stand in the same stance, facing north, in the beginning of each motion capture session, in order to calibrate the system (which contains magnetometers).

There is no 3D position tracking system in the IGS suit, and the position of the actor is calculated by the software given a known initial position, using the length of the skeleton segments from the feet to the hip, the angles recorded between those segments for each frame, and always considering that the lower point of the skeleton is in contact with the ground.

Because of that, drifts can appear, but from our experiences with the equipment, if the calibration step is properly done, as well for the skeleton calibration as for the initial position of the actor at the beginning of each motion capture session, we obtain in most capture sequences very good quality of data.

## V. RECORDING SETUP

This database contains 17 walk sequences for 40 subjects. Those 17 sequences correspond, for each subject, to rectilign walks at different speeds, and to segments with direction changes of various amplitudes in the trajectory. The subjects
were participants of other projects of the workshop who agreed to volunteer for the motion capture recording.

The motion capture area was a 6m by 4m space, in which subjects were asked to walk given some instructions. The first step was to take the calibration pictures and to calibrate a skeleton for each participant. Once this was done, each subject came for approximately one hour of recording. They were first asked to remove their shoes and put on the motion capture suit. The walk in the database is thus always barefoot, on a perfectly even floor.

The first part of the motion capture session was the recording of the trajectory changes. Several crosses were drawn on the floor, as shown in figure 1. For each one of the five direction change sequences, the performer began on the red cross on the left upper side of the figure, facing right. The instruction was to walk straight until they reached the red line (which was also drawn on the floor), and then they were free to change direction the way they wanted to, they just had to reach one of the other crosses drawn on the floor, with different angles from the initial straight trajectory. For the first sequence, the cross to reach was just straight ahead (0 degree), and for the following ones, the angles of the direction change were respectively 45, 90, 135 and 180 degrees.

<figure><img src="image_149.png" /><figcaption>Fig. 1. Sketch of the motion capture area for the five direction change recordings. Only the red crosses and line were drawn on the floor.</figcaption></figure>

For the second part of the motion capture, the aim was to study how walk changes when we change our velocity. This time, the performer was told to walk straight from one corner of the capture area to the opposite corner (approximately 7 meters). He was given four different instructions, and the sequence corresponding to each of these instructions had to be repeated three times. Those four instructions were the following:

* Walk (normally).

* Imagine you are in a park, the weather is nice, you take your time.

* You are going to work, but you are not late.

* You have to catch a train, and you are very late, but you are not allowed to run.

These instructions let to the performer the freedom to choose his speed for each scenario, even if the gradation from slower to faster was the same for everybody, with the first "walk normally" instruction as a reference for a standard self selected speed.

## VI. DATA REPRESENTATION

The output of our motion capture system are angles between body segments, and the calculated 3D position of the root (hips) which is evaluated from the angles and the segment lengths of the legs, as explained previously (section IV). That data is recorded under the form of a Biovision file format (.bvh), which is a "de facto" standard format for 3D motion data. A bvh file contains two parts. The first part is the hierarchical description of the skeleton, starting from the root (hip) to the extremities of each limb, and the geometric position of each joints, still standing. The second part of the file contains one line of data for each motion frame. The skeleton from the Animazoo software contains 20 body segments in the skeleton (see figure 2), and each data line contains thus 3 (3D position of the root) + 3 × 21 (3D angles for each segment and for the whole body) = 66 values. Three of the segments are in fact only added to make the skeleton look closer to the real skeleton but have no degree of freedom in the motion capture, and their value does thus not vary during motion capture. There are thus finally 57 values to analyze and model in our data.

<figure><img src="image_150.png" /><figcaption>Fig. 2. Skeleton calibrated and displayed in Animazoo's software during motion capture.</figcaption></figure>

As we have said in section IV, drifts can appear when using such an inertial motion capture system and it is the case for a few motions of the database, where the 3D position is not properly calculated. The appearance of the walk is then still good when looked at without 3D displacement, but the whole body looks like if instantly drifted backwards. Another small disturbance that can sometimes be observed is when one of the sensors moves because of the suit and not because of the subject's motion.

## VII. DATA VISUALIZATION AND SEGMENTATION

The project has a need for a tool to manually segment the motion recordings to keep only those parts needed by the training process. In this context, Motion Editor.app was
developed, which consists of a 3D view of the skeleton and an interactive timeline. Moreover, this application allows the labeling of time segments that can be used for specific training purposes.

The software parses the Biovision file (.bvh) and builds a memory representation of its content. This includes both the skeletal representation, i.e. the relative offsets of the joints, and the time sampled 3-axis rotations of each joint <fnref n="1" />. Then, it displays the skeleton at any point in time as an Open Scene Graph <fnref n="2" /> 3D scene view.

A timeline is also provided, which enables time navigation with immediate display of the skeleton. Additionally, the timeline permits to select temporal segments, and associated motion frames can be deleted. Alternatively, the time segment can also be given a label for identification (and further processing). The edited, in-memory frame representation can then be saved as a new .bvh file, and the labels, if any, can be saved to a separate labels file.

The graphical user interface of Motion Editor is shown of
figure 3.

<figure><img src="image_151.png" /><figcaption>Fig. 3. Sketch of the motion capture area for the five direction change recordings. Only the red crosses and line were drawn on the floor.</figcaption></figure>

VIII. CONCLUSION

The flexibility of the IGS 190 equipment enabled the recording of a large database of human walk and its variations. A preliminary visual analysis of the database already allowed us to observe the obvious style differences between walk from a woman and a man, the way that a person walks at different speeds, as well as the differences in walk parameters for two subjects with comparable morphology. From now on, this new database will allow us to study inter-subject variability in walk, walk variations linked to speed, and the way humans manage direction changes when they walk on an even ground without obstacles, and to train statistical models to model and to synthesize new motion.

ACKNOWLEDGMENT

The authors would like to thank all eNTERFACE’08 atten-
dants who participated in the creation of the database.

This project was partly funded by the Ministry of Région
Wallonne under the Numediart research program (grant
N°0716631). J.Tilmanne receives a PhD grant of the Fonds
de la Recherche pour l'Industrie et l'Agriculture (F.R.I.A.),
Belgium.

REFERENCES

[1] M. Brand and A. Hertzmann, “Style machines,” in Proc of SIGGRAPH 2000, 2000, pp. 183–192. [Online]. Available: http://www.merl.com/reports/docs/TR2000-14.pdf

[2] S. L. Lauritzen, “Graphical models,” 1996.

[3] Z. Ghahramani, “Learning dynamic bayesian networks,” in Adaptive Processing of Sequences and Data Structures. Springer-Verlag, 1998, pp. 168–197.

[4] L. Rabiner, "A tutorial on hidden markov models and selected applications in speech recognition," in Proc. of IEEE, vol. 77, no. 2, February 1989, pp. 257–286.

[5] "The Carnegie Mellon University Graphics Lab Motion Capture Database," http://mocap.cs.cmu.edu.

[6] “The “mocapdata” database,” http://www.mocapdata.com.

[7] “Truebones website,” http://www.truebones.com.

[8] "The HDM05 motion capture database," http://www.mpi-inf.mpg.de/resources/HDM05.

[9] M. Müller, T. Röder, M. Clausen, B. Eberhardt, B. Krüger, and A. Weber, "Documentation mocap database HDM05," Universität Bonn, Tech. Rep. No. CG-2007-2, June 2007. [Online]. Available: http://www.mpi-inf.mpg.de/resources/HDM05/07_MuRoCIEbKrWe_HDM05.pdf

[10] "Animazoo inertial motion capture system IGS-190," http://www.animazoo.com.

[11] C. L. Vaughan, B. L. Davis, and J. C. OConnor, *Dynamics of Human Gait*, 2nd ed. Kiboho Publishers, 1999.

[12] A. Menache, *Understanding motion Capture for Computer Animation and Video Games*. San Francisco, CA, USA: Morgan Kauffman Publishers Inc., 1999.

[13] B. Rosenhahn, T. Brox, and H.-P. Seidel, “Scaled motion dynamics for markerless motion capture,” in *Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, June 2007.

[14] A. Colahi, M. Hoviatalab, T. Rezaeian, M. Alizadeh, and M. Bostan, “Implementation of optical tracker system for marker-based human motion tracking,” in *Proceedings of the IASTED International Conference on Applied Simulation and Modelling (ASM)*, June 2006.

Joëlle Tilmanne holds an Electrical Engineering degree from the Faculté Polytechnique de Mons (Belgium) since June 2006. She did her master thesis in the field of sleep signals analysis, at Lehigh University (USA). She is pursuing a PhD thesis in the TCTS (Circuit Theory and Signal Porcessing) Lab of FPMs since september 2006, in the field of HMM based motion synthesis.

Raphaël Sebbe holds a PhD in the field of image processing as well as electrical engineering degrees from both FPMs and Supélec. Raphaël has worked on speech and image-related technologies and software development. He is now with TCTS Lab of FPMs, pursuing research on Numédiart project.

Thierry Dutoit graduated as an electrical engineer and Ph.D. in 1988 and 1993 from the Faculté Polytechnique de Mons, Belgium, where he is now a full professor. He spent 16 months as a consultant for AT&T Labs Research in Murray Hill and Florham Park, NJ, from July, 1996 to September, 1998. He is the author of two books on speech processing and text-to-speech synthesis, and the coordinator of the MBROLA project for free multilingual speech synthesis. T. Dutoit was an Associate Editor of the IEEE Transactions on Speech and Audio Processing (2004-2006) and a member of the INTERSPEECH07 organization committee. He was the initiator of eNTERFACE workshops and the organizer of eNTERFACE05.

<footnote n="1">¹this 66 DOF record is called a frame</footnote>

<footnote n="2">²Open Scene Graph is an open source library based on OpenGL use to display 3D content</footnote>
eNTERFACE'08 Participants

* Bénédicte Adessi, France, benedicteadessi@hotmail.com

* Rami Ajaj, LIMSI-CNRS, Orsay, France rami.ajaj@limsi.fr

* Samer Al Moubayed, Royal Institute of Technology KTH - Speech Music and Hearing Dept., Sweden, sameram@kth.se

* Gopal Ananthakrishnan, Royal Institute of Technology KTH - Speech Music and Hearing Dept., Sweden, agopal@kth.se

* Oya Aran, Bogazici University - Computer Engineering, Turkey, aranoya@gmail.com

* Carole Arrat, France, acarole17@hotmail.com

* Malek Baklouti, Thales - D3S, France, malek.baklouti@thalesgroup.com

* Jean Bernard, Université Paris VIII, France, j.bernard@iut.univ-paris8.fr

* Tifanie Bouchara, LIMSI-CNRS, Orsay, France, tifanie.bouchara@hotmail.fr

* Pierre Breteche, LASELDI, Université de Franche-Comté, France, pierre.breteche@univ-fcomte.fr

* Pavel Campr, University of West Bohemia - Department of Cybernetics, Czech Republic, campr@kky.zcu.cz

* Antonio Camuri, University of Genova, Italia, antonio.camurri@unige.it

* Aleksandra Cerekovic, Croatia, aleksandra.cerekovic@fer.hr

* Mohamed Chetouani, Université Pierre et Marie Curie, France, mohamed.chetouani@upmc.fr

* Hee-Seung Choi, choimode@gmail.com

* Matthieu Courgeon, CNRS-LIMSI, Orsay, France, courgeon@limsi.fr

* Laurent Couvreur, Faculté Polytechnique de Mons - Théorie des Circuits et Traitement du Signal, Belgium, laurent.couvreur@fpms.ac.be

* Nicolas d'Alessandro, Polytech.Mons - Signal Processing Lab, Belgique, nicolas@dalessandro.be

* Christophe d'Alessandro, LIMSI-CNRS, Orsay, France, cda@limsi.fr

* Nicolas Déflache, Independant, France, contact@nicolasdeflache.fr

* Jonathan Demeyer, Faculté Polytechnique de Mons - TCTS Lab, Belgium, Jonathan.Demeyer@fpms.ac.be

* Hamdi Dibeklioglu, Bogazici University - Computer Engineering Department, Turkey, hamdimail@gmail.com

* Thomas Dubuisson, Faculté Polytechnique de Mons - TCTS Lab, Belgium, thomas.dubuisson@fpms.ac.be

* Thierry Dutoit, Faculté Polytechnique de Mons - TCTS Lab, Belgium thierry.dutoit@fpms.ac.be

* Nesli Erdogmus, Turkey, neslibozkurt@hotmail.com

* Jean-Julien Filatriau, Université catholique de Louvain (UCL) - Communication Lab (TELE), Belgium, jehan-julien.filatriau@uclouvain.be

* Takuya Furukawa, Kyoto Univ, Japan, furukawa@ii.ist.i.kyoto-u.ac.jp

* Simone Ghisio, INFOMUS, Genova, Italia, ghisio@infomus.org

* Donald Glowinski, University of Genova, Italia, donald.glowinski@unige.it

* David Antonio Gómez Jáuregui, Télécom SudParis - EPH, France, david.gomez@it-sudparis.eu
* Fabienne Gotusso, France, fabiennegot@orange.fr

* Catherine Guastavino, University McGill, Montreal, Quebec, Canada, catherine.guastavino@mcgill.ca

* Gökhan Himmetoglu, Turkey, hhimmetoglu@ku.edu.tr

* Marek Hruz, University of West Bohemia - Department of Cybernetics, Czech Republic, mhruz@kky.zcu.cz

* Hung-Hsuan Huang, Kyoto University - Graduate School of Informatics, Japan, huang@ii.ist.i.kyoto-u.ac.jp

* Christian Jacquemin, LIMSI-CNRS, Orsay, France, jacquemin@limsi.fr

* Alexey Karpov, SPIRAS, Russia, karpov@iias.spb.su

* Loïc Kessous, LIMSI-CNRS, Orsay, France, loic.kessous@limsi.fr

* Byung Jun Kwon, byungjun@gmail.com

* Ju-Hwan Lee, University of Oxford - Crossmodal Research Laboratory, UK (but Korean), juhwan.lee@psy.ox.ac.uk

* Martin Lojka, martin.lojka@tuke.sk

* Adolfo Lopez, UPC - Image & Video Processing Group, Spain, alopez@gps.tsc.upc.edu

* Benoît Macq, Université catholique de Louvain (UCL) - Communication Lab (TELE), Belgium, Benoit.Macq@uclouvain.be

* Ammar Mahdhaoui, ISIR - Percetion and Movement P&M, France, Ammar.Mahdhaoui@robot.jussieu.fr

* Matei Mancas, Faculté Polytechnique de Mons - TCTS Lab, Belgium Matei.Mancas@fpms.ac.be

* Jean-Claude Martin, LIMSI-CNRS, Orsay, France, martin@limsi.fr

* Hildeberto Mendonca, Université catholique de Louvain - Electric Engineering, Belgium, hildeberto.mendonca@uclouvain.be

* Alexis Moinet, Université catholique de Louvain (UCL) - Communication Lab (TELE), Belgium, alexis.moinet@fpms.ac.be

* Camille Moussette, Umea Institute of Design, Sweden, camille.moussette@dh.umu.se

* Emma Murphy, University McGill, Montreal, Quebec, Canada, emma.murphy@mail.mcgill

* Yukiko Nakano, y_nakano@agate.plala.or.jp

* Daniel Neiberg, KTH - TMH, Sweden, daniel.neiberg@gmail.com

* Toyoaki Nishida, Kyoto University - Graduate School of Informatics, Japan, nishida@i.kyoto-u.ac.jp

* Stanislav Ondas, Faculty of Electrical Engineering and Informatics, Technical University of Košice, Slovakia - Department of Electronics and Multimedia Communications, Slovakia, stanislav.ondas@gmail.com

* Michal Osowski, michalosowski@yahoo.com

* Thomas Pachoud, www didascalie net, France, thomas.pachoud@gmail.com

* Igor Pandzic, igor.pandzic@fer.hr

* Catherine Pélachaud, catherine.pelachaud@inria.fr

* Serafeim Perdikis, ITI/CERTH, Greece, perdik@iti.gr

* Laurent Pointal, CNRS - LIMSI, France, laurent.pointal@limsi.fr

* Thierry Ravet, Thierry.Ravet@fpms.ac.be

* Albert Rilliard, CNRS - LIMSI, albert.rilliard@limsi.fr

* Renaud Rubiano, didascalie.net, France, renaud@renaudrubiano.com

* Usman Saeed, Eurecom - Multimedia Dept, France, usman.saeed@eurecom.fr
* Albert Ali Salah, CWI - Signals and Images, The Netherlands, a.a.salah@cwi.nl

* Pinar Santemiz, Boğaziçi University - Computer Engineering, Turkey, psantemiz@gmail.com

* Ao Shen, University of Birmingham, UK, shenany@gmail.com

* Yannis Stylianou, yanni@csd.uoc.gr

* Shinya Takeda, Japan, shinya_ddr@yahoo.co.jp

* Joëlle Tilmanne, Faculté Polytechnique de Mons, Belgium, joelle.tilmanne@fpms.ac.be

* Jana Trojanova, University of West Bohemia - Department of Cybernetics, Czech Republic, trojana@kky.zcu.cz

* Dimitrios Tzovaras, Dimitrios.Tzovaras@iti.gr

* Jérôme Urbain, Faculty of Engineering, Mons - Circuit Theory and Signal Processing Laboratory, Belgium, jerome.urbain@fpms.ac.be

* Charles Verron, charles.verron@orange-ftgroup.com

* Athanasios Vogiannou, CERTH/ITI, Greece, tvog@iti.gr

* Gualtiero Volpe, DIST - University of Genova - Casa Paganini - InfoMus Lab, Italy, gualtiero.volpe@unige.it

* Olga Vybornova, olga.vybornova@uclouvain.be

* Chu Wen-Yang, Taiwan, wenyang.chu@gmail.com

* Yuji Yamaoka, Japan, 50007646208@st.tuat.ac.jp

* Mehmet Mustafa Yilmaz, Turkey, mehyilmaz@ku.edu.tr

* Milos Zelezny, University of West Bohemia - Department of Cybernetics, Czech Republic, zelezny@kky.zcu.cz
Table of Contents

Editorial, Christophe d'Alessandro ......................................................................................... Page 3

Project 1: Implementing a Multiparty Support in a Tour Guide system with an Embodied Conversational Agent, Aleksandra Čereković, Hsuan-Hung Huang, Takuya Furukawa, Yuji Yamaoka, Igor S. Pandžić, Toyoaki Nishida, Yukiko Nakano. ........................................ Page 5

Project 2: Multimodal High Level Data Integration, Olga Vybornova, Hildeberto Mendonça, Daniel Neiberg, David Antonio Gomez Jauregui, Ao Shen ........................................ Page 14

Project 3: Sign-language-enabled information kiosk, Pavel Campr, Marek Hrùz, Alexey Karpov, Pınar Santemiz, Miloš Železný, and Oya Aran ........................................................................ Page 24

Project 4: Design and Evaluation of an Audio-Haptic Interface, Emma Murphy, Camille Moussette, Charles Verron, Catherine Guastavino................................................................................ Page 34

Project 5: Capture and machine learning of physiological signals, Benedicte Adessi, Rami Ajaj, Carole Arrat, Ivan Chabanaud, Matthieu Courgeon, Nicolas Déflache, Nesli Erdogmus, Fabienne Gotusso, Hikmet Gökhan Himmetoğlu, Christian Jacquemin, Loïc Kessous, Jean-Claude Martin, Michal Osowski, Thomas Pachoud, Jana Trojanova. page 39

Project 7: Multimodal Feedback from Robots and Agents in a Storytelling Experiment, S. Al Moubayed, M. Baklouti, M. Chetouani, T. Dutoit, A. Mahdhaoui, J.-C. Martin, S. Ondas, C. Pelachaud, J. Urbain, M. Yilmas ................................................................................................................ Page 43

Project 8: Activity-related Biometric Authentication G. Ananthakrishnan, H. Dibeklioglu, M. Lojka, A. Lopez, S. Perdikis, U. Saeed, A.A. Salah, D. Tzovaras, A. Vogiannou ...... Page 56

Project 9: Tracking-dependent and interactive video projection, Matei Mancas, Donald Glowinski, Pierre Bretéché, Jonathan Demeyer, Thierry Ravet, Gualtiero Volpe, Antonio Camurri, Paolo Coletta................................................................................................................................................ Page 73

Project 10: Expressive Violin Synthesis A Study Case in Real-time Manipulation of Complex Data Structures, Nicolas d'Alessandro, Alexis Moinet, Thomas Dubuisson, Jehan-Julien Filatriau, Wenyang Chu................................................................................................................ Page 84

Project 2: A Database for Stylistic Human Gait Modeling and Synthesis, Joëlle Tilmanne, Raphël Sebbe, and Thierry Dutoit ................................................................................................................ Page 91

List of participants ................................................................................................................................................ Page 95