Four recording sessions were done by an Arabic listener and an Arabic speaker.

Five recording sessions were done by a speaker and a listener who do not speak or understand each other's languages; these recordings can be used to study the isolated effect of prosody on the engagement in a storytelling context. The languages used in these sessions were Arabic, Slovak, Turkish, and French.

## Annotation Schema

A part of the MUMIN [26] multimodal coding scheme was used for annotating the database. MUMIN was originally created to experiment with annotation of multimodal communication in video clips of interviews taken from Swedish, Finnish and Danish television broadcasting and in short clips from movies. However, the coding scheme is also intended to be a general instrument for the study of gestures and facial displays in interpersonal communication, in particular the role played by multimodal expressions for feedback, turn management and sequencing.

The videos were annotated (with at least two annotators per session) for describing simple communicative signals of both speaker and listener: smile, head nod, head, shake, eye brow and acoustic prominence. These annotations were done using the ANVIL [27] annotation program (Fig. 2) and are summarized in Table 1.

<figure><img src="image_50.png" /><figcaption>Fig. 2 Anvil, as used for our database annotation</figcaption></figure>

<table><caption>Table 1: Coding scheme used for eNTERFACE_STEAD annotations.</caption><thead><tr><th rowspan="2">Facial feature</th><th rowspan="2">display</th><th colspan="2">Form of expression/Movement values</th></tr><tr><th>Value</th><th>Short tag</th></tr></thead><tbody><tr><td>General face</td><td></td><td>smile</td><td>smile</td></tr><tr><td rowspan="2">Mouth(opening)</td><td></td><td>Open mouth</td><td>Open-M</td></tr><tr><td></td><td>Closed mouth</td><td>Close-M</td></tr><tr><td rowspan="2">Head</td><td></td><td>Nod</td><td>Head-Nod</td></tr><tr><td></td><td>Shake</td><td>Head-shake</td></tr><tr><td rowspan="2">Eyebrows</td><td></td><td>Frowning</td><td>Frown</td></tr><tr><td></td><td>Raising</td><td>Raise</td></tr><tr><td rowspan="2">Acoustic</td><td></td><td>Prominence</td><td>Prominence</td></tr><tr><td></td><td>laughter</td><td>laughter</td></tr></tbody></table>

Annotators had instructions to annotate prominence in the audio recording of the speaker by only listening to the audio signal without looking to the video recording; some annotators spoke French, others did not.

## Annotation evaluation

Manual annotations of videos were evaluated by computing agreements using corrected kappa [28] computed in the Anvil tool [27], shown in Fig. 2:

$$ \kappa = (P0 - 1/z) / (1 - 1/z) $$

where z is the number of categories and P0 is like in Cohen's kappa.

Table 2 presents the agreements among annotators for each track. We can see that the best agreement is obtained for the Listener.Acoustic track which is expected since the listener is not assumed to speak and when he/she does simple sounds are produced (filled pauses). Other tracks have a lower agreement such as Speaker.Acoustic. The speaker always speaks during the session and prominent events are less identifiable. However, the agreements measures are high enough to allow us to assume that selected communicative signals might be reliably detected.

<table><caption>Table 2 Agreement between annotators of our database (eNTERFACE_STEAD)</caption><thead><tr><th>Track Name</th><th>Cohen's Kappa</th><th>Corrected Kappa</th><th>Agreement(%)</th></tr></thead><tbody><tr><td>Speaker.Face</td><td>0.473</td><td>0.786</td><td>89.306</td></tr><tr><td>Speaker.Acoustic</td><td>0.099</td><td>0.786</td><td>84.500</td></tr><tr><td>Listener.Face</td><td>0.436</td><td>0.559</td><td>77.960</td></tr><tr><td>Listener HeadNod</td><td>0.464</td><td>0.694</td><td>84.622</td></tr><tr><td>Listener.Acoustic</td><td>0.408</td><td>0.929</td><td>95.972</td></tr></tbody></table>

## The eNTERFACE_STEAD License

The eNTERFACE_STEAD contents, and all the annotations are released under an MIT-like free software license and is available from the eNTERFACE'08 website (www.interface.net/enterprise08).

## IV. SPEECH ANALYSIS

The main goal of the speech analysis component is to extract features from the speech signal that have been previously identified as key moments for triggering feedbacks (cf. section VII). In this study, we do not use any linguistic information to analyze the meaning of the utterances being told by the speaker, but we focus on the prosodic cross-language features which may participate in the generation of the feedback by the listener.

Previous studies have shown that pitch movements, especially at the end of the utterances, play an important
role in turn taking and backchannelling during human dialogue [29]. In this work, we propose in this work to use the following features extracted from the speaker's speech signal: Utterance beginning, Utterance end, Raising pitch, Falling pitch, Connection pitch, and Pitch prominence.

To extract these important features from the speech stream, we decided to work in Pure Data [30], a graphical programming environment for real-time audio processing. The patch we developed for speech feature extraction is shown in figure 3; it provides the following features: Utterance beginning, Utterance end, Raising pitch, Falling pitch, Stable pitch, and Acoustic prominence.

<figure><img src="image_51.png" /><figcaption>Fig. 3. Pure Data patch for Speech Feature Extraction</figcaption></figure>

Audio acquisition is performed by the `adc~` object. It provides 64-samples blocks (at a sampling frequency set to 16kHz) to objects `vad_framing~` and `pitch_framing~`, which are responsible for voice activity detection and pitch estimation for prominence estimation. These algorithms are written in C, using audio processing functions from a C library developed by the Center of Speech Technology at KTH, Sweden. They were compiled as externals for Pure Data, so that they can be used as ordinary Pure Data objects.

Since we wanted to compute some features on overlapping audio segments longer than 64 samples, we developed a specific framing routine. The objects `vad_framing~` and `pitch_framing~` take two arguments `X` and `Y`, which impose blocks of `X` samples with a shift of `Y` samples between successive blocks: a buffer of `X` samples is filled with the input blocks of 64 samples; when this buffer is full, it is sent to the analysis algorithm, the samples in the buffer are then shifted by `Y` samples, and the buffer is filled again, etc.

For every input speech frame, `vad_framing~` and `pitch_framing~` output an integer, which is set to 0 if there is no event (feature) is detected in the speech, and to a number (index) indicating the id of the detected feature

otherwise. These indices are sent to the "route" PureData object, which triggers a string depending on its input; this string is later sent through a tcp/ip connection to the Multimodal Fusion module.

The `vad_framing~` object is a Voice Activity Detection object, which contains an adaptation of the SPHINX Vader functionality [31]. This object sends "1" if there is a detected change in the audio stream from Silence to Speech, and "2" when there is a detected change from Speech to Silence, otherwise the output of this object is always "0".

The `pitch_framing~` object is used to extract the rest of the speech features. This object contains an implementation of the realtime fundamental frequency tracking algorithm YIN [32]. For cleaning the output of the YIN algorithm, a median filter of size "5" (60 msec) is applied on the extracted F0 to compensate for outliers and octave jumps. This object sends "1" when a Raising Pitch is detected, "2" for Falling pitch, "3" for Stable Pitch, and "4" for Acoustic prominence.

### The TILT model:

The TILT model [33] is used to extract Raising pitch, Falling pitch, Connection pitch. We tried in this implementation to not compensate for the unvoiced segments by using any type of interpolation; nevertheless, the movements of the pitch are detected only at the end of the voiced segments, and no movements are detected when the voiced segment duration is shorter than 125 msec.

### Audio prominence estimation

In the literature, several definitions of acoustical prominent events can be found showing the diversity of this notion [34, 35]. Terken [35] defines prominence as words or syllables that are perceived as standing out from their environment. Most of the proposed definitions are based on linguistic and/or phonetic units.

We propose in this project another approach using statistical models for the detection of prominence. The key idea is to assume that a prominent sound stands out from the previous message. For instance, during our storytelling experiment, speakers emphasize words or syllables when they want to focus the attention of the listener on important information. These emphasized segments are assumed to stand out from the other ones, which makes them become salient.

Prominence detectors are usually based acoustic parameters (fundamental frequency, energy, duration, spectral intensity) and machine learning techniques (Gaussian Mixture Models, Conditional Random Fields) [36, 37]. Unsupervised methods have been also investigated such as the use of Kullback-Leibler (KL) divergence as a measure of discrimination between prominent and non-prominent classes [38]. These statistical methods provide an unsupervised framework adapted to our task. The KL divergence needs the estimation of two covariance matrices (Gaussian assumption):
$$KL_{ij} = \frac{1}{2} \left[ \log \frac{\Sigma_j}{\Sigma_i} + \text{tr}(\Sigma_i \Sigma_j^{-1}) + (\mu_i - \mu_j)^T \Sigma_j^{-1} (\mu_i - \mu_j) - d \right]$$

where $\mu_i$, $\mu_j$ and $\Sigma_i$, $\Sigma_j$ denote the means and the covariance matrices of i-th (past) and j-th (new event) speech segments respectively. d is the dimension of the speech feature vector. An event j is defined as prominent if the distance from the past segments (represented by the segment i) is larger than a pre-defined threshold.

One major drawback of the KL divergence approach is that since the new event is usually shorter than the past events, the estimation of their covariance matrices is less reliable. In addition, it is well-known that duration is an important perceptual effect for the discrimination between sounds. Taking these points into account, we propose to use another statistical test namely the T² Hotteling distance defined by:

$$H_{ij} = \frac{L_i L_j}{L_i + L_j} \left[ (\mu_i - \mu_j)^T \Sigma_{i \cup j}^{-1} (\mu_i - \mu_j) \right]$$

where $i \cup j$ is the union of i-th (past) and j-th (new event) segments. $L_i$ and $L_j$ denote the length of the segments. The T² Hotteling divergence is closely related to the Mahalanobis distance.

In this work only the fundamental frequency (F0) is used as a feature to calculate the Hotteling distance between two successive voiced segments. In this sense, a prominence is detected when the Hotelling distance between the current and the preceding Gaussian distributions of F0 is higher than a threshold. We have used a decaying distance threshold over time, where the initial value of this threshold is the highest distance during the first utterance of the speaker; whenever this threshold is reached by a following segment, a Pitch Prominence event is triggered, and the new distance becomes the distance threshold. Since we estimate a Gaussian distribution of the pitch for a voiced segment, we only estimate it when there are enough pitch samples during the voiced segment, (we set this duration threshold to 175 msec).

## V. FACE ANALYSIS

The main goal of the face analysis component (Fig. 4) is to provide the feedback system with some knowledge of communicative signals conveyed by the head of the speaker. More specifically, detecting if the speaker is shaking the head, smiling or showing neutral expression are the main activity features we are interested in. The components of this module (Fig. 5) are responsible for face detection, head shake and nod detection, mouth extraction, and head activity analysis. They are detailed below.

<figure><img src="image_52.png" /><figcaption>Fig. 4. Screenshot of the face analysis component, which runs on a PC and shows the analysis results through a MATLAB-based user interface It sends its results to the multimodal fusion module via TCP-IP.</figcaption></figure>

<figure><img src="image_53.png" /><figcaption>Fig. 5. Overview of the face analysis module</figcaption></figure>

### Face detection

The face detection algorithm that we used exploits Haar-like features that have been initially proposed by Viola & Jones [39]. It is based on a cascade of boosted classifiers working with Haar-like features and trained with a few hundreds of sample views of faces. We used the trained classifier available in OpenCV.

The face detection module outputs the coordinates of existing faces in the incoming images.

### Smile detection

Smile detection is performed in two steps: mouth extraction followed by smile detection. We use a colorimetric approach for mouth extraction. A thresholding technique is used after a colour space conversion to the YIQ space (Fig. 6).

<figure><img src="image_54.png" /><figcaption>Fig. 6. Mouth extraction. (a) original images (b) color conversion and thresholding (c) elimination of small regions</figcaption></figure>

Once the mouth is extracted, we examine the ratio between the two characteristic mouth dimensions, P1P3 and P2P4 (see Fig. 7), for smile detection. We assume that when smiling, this ratio increases. The decision is obtained by thresholding.
<figure><img src="image_55.png" /><figcaption>Fig. 7. smile detection</figcaption></figure>

Head shake and nod detection

The purpose of this component is to detect if the person is shaking his/her head or doing a nod. The idea is to analyze the motion of some feature points extracted from the face along the vertical and horizontal axes.

<figure><img src="image_56.png" /><figcaption>Fig. 8. Feature points extraction</figcaption></figure>

Once the face has been detected in the image, we extract 100 feature points using a combined corner and edge detector defined by Harris [40]. Feature points are extracted in the central area of the face rectangle using offsets (Fig 8).

These points are then tracked by calculating the optical flow between a set of corresponding points in two successive frames. We make use of the Lucas Kanade [41] algorithm implementation available from in the OpenCV library (http://sourceforge.net/projects/ opencvlibrary/).

Let n be the number of feature points and $Pt_i(x_i, y_i)$ the ith feature point defined by its 2D screen coordinates $(x_i, y_i)$. We then define the overall velocity of the head as:

$$V = \begin{cases} V_x = \frac{1}{n} \sum_{i=1}^{n} (x_i - x_{i-1}) \\ V_y = \frac{1}{n} \sum_{i=1}^{n} (y_i - y_{i-1}) \end{cases}$$

<figure><img src="image_57.png" /><figcaption>Fig. 9. Feature point velocity analysis</figcaption></figure>

Fig. 9 shows the velocity curves along the vertical and horizontal axes. The sequence of movements represented is composed by one nod and two head shakes. We notice that the velocity curves are the sum of two signals: (1) a noise movement which is a low frequency signal representing the global head motion and (2) a high frequency signal representing the head nods and head shakes.

The idea is then to use wavelet decomposition to remove the low frequency signals. More precisely, we decomposed the signal using symlet-6 wavelet. Fig. 10 shows the reconstruction of the detail at the first level of the signal shown in Fig. 8. The head nod and shake events can be reliably identified by this process.

<figure><img src="image_58.png" /><figcaption>Fig. 10 Signal denoising via wavelets.</figcaption></figure>

Head activity analysis

Analysis of recordings of the storytelling experience has shown a correlation between the head activity of both speaker and listener. To characterize the head activity, we use the velocity of the feature points defined in (1), to quantify the overall activity A:

$$A = \sum_{t \in timewindow} V_{x,t}^2 + V_{y,t}^2 \quad (2)$$

where timewindow is set to 60 frames (30 frames/s).
This measure provides information about the head activity levels. In order to quantize head activity into levels (high, medium or low), we analyzed the head activity of all the speakers of the eNTERFACE08_STEAD corpus. Assuming that the activity of one given speaker is Gaussian, we set up various thresholds defined in Table 3. By using these thresholds, the algorithm becomes more sensitive to any head movement of a stationary speaker, while it raises the thresholds for an active speaker, thus resulting in a flexible adaptive modeling.

<table><caption>**Table 3.** Segments are categorized according to the amplitude of their maxima. Mean and standard deviation statistics are related to head activity.</caption><thead><tr><td>Amplitude</td><td>Interpretation</td></tr></thead><tbody><tr><td>&lt; mean</td><td>LOW ACTIVITY</td></tr><tr><td>&lt; mean + standard deviation</td><td>MEDIUM ACTIVITY</td></tr><tr><td>Otherwise</td><td>HIGH ACTIVITY</td></tr></tbody></table>

## VI. AGENT/ROBOT STATE CONTROL

In the feedback model proposed in [4], the state of the agent/robot is characterized by the following features: disagreement, agreement, acceptance, refusal, belief, disbelief, liking, disliking, interest, no_interest, understanding, no_understanding, and mimicry. For this project, we have reduced this set to: interest, understanding, and liking. Our goal was then the design means of modifying these features through some analysis of the audio and video streams from the speaker.

To achieve this goal, we have used an English speaking ASR system based on keyword spotting, which makes it possible to modify the agent state according to the recognized words. The ASR system is thus integrated into an Agent State Manager (ASM) module (Fig 11), which consists in three main parts: the ASR engine, the State Planner and the Message Generator. These components are detailed in the next paragraphs.

### ASR engine

The speech engine we have used is based on ATK/HTK [42] and is available as a dynamic-link library (dll), with a simple API.

It uses freely available British English triphone acoustic models, which are part of the ATK distribution, and were trained on the WSJCAM0 speech corpus [43] recorded at the Cambridge University and composed of readings of the Wall Street Journal.

As a language model we have used a speech grammar which enables the recognition of keywords in phrases. Other words are modeled as “filler words”, and not recognized. The keyword spotting grammar actually puts all keywords in parallel (with no specific syntactic constraints), together with a filler model. It is written in BNF (Backus Naur Form) format and then it is translated to HTK-compatible SLF format.

<figure><img src="image_59.png" /><figcaption>**Fig. 11.** Architecture of the Agent State Manager</figcaption></figure>

For the purpose of storytelling we needed to define appropriate keywords. The best way to obtain this information was to look at our storytelling recordings, available in the from eNTERFACE08_STEAD database (see Section III). We used two recordings by native English speakers, as well as recordings in Slovak, and transcribed them into English words. The transcriptions were then analyzed in terms of word counts via a simple Python script. After eliminating the articles (*the*, *a*), conjunctions (*and*) and pronouns (*she*, *he*, *it*), which were naturally the most frequent words, we identified a group of keywords for our storytelling experiment, containing: hotel, reception, door, room, stairs, luggage, baggage, bags, birdcage, umbrella, clerk, tomcat, cat, silvester, woman, lady, tweety, bird, knock, carry, call, run, trick, discover, hit, hide, ring, uncover, pick up, cartoon, story, treatment, outside.

Notice that the ASR engine also uses a simple pronunciation dictionary to specify the expected pronunciation of keywords. We modified the default pronunciation of words in the dictionary by removing the expected short pauses (sp) after each word. (As a matter of fact, we use continuous speech in this project, in which there are no short pauses between words.)

### State Planner

The State Planner is the main part of the Agent state Manager. Its task is to modify the state of the agent according to spoken input (keywords). For this purpose it uses a rule-based approach. This component is initialized with the initial agent state as well as with a set of rules loaded from an initialization file. It takes spoken keywords as input, and changes the value of the *interest*, *understanding*, and *liking* as output.

Each rule consists of the following fileds: feature, keyword, step, max_value and opposite feature (which is optional). When the speech recognizer recognizes a keyword, State Planner looks for an appropriate rule. If it finds it, it increases the value of the related feature by a given step value, while max_value is not reached. If some opposite feature is defined in the rule, it is decreased by the same step.

### Message Generator

Every three words which have triggered the application of rules, the Message Generator prepares an XML file containing the feature values, which represent the new agent
state. After converting special characters into hexadecimal, it sends the XML file through a TCP socket to the ECA (Greta).

## VII. MULTIMODAL FUSION AND FEEDBACK BEHAVIORS FOR AIBO AND GRETA

### Extracting rules from data

Based on the selected communicative signals, we have defined some rules to trigger feedbacks. The rules are based on [44, 45], which involved mainly only mono-modal signals. The structure of such rules is as follows:

> "If some *signal* (eg. head-nod, pause, pitch accent) is received, then the listener sends some *feedback signal* with probability X."

We have extended these rules by analyzing the data annotated from our eNTERFACE08_STEAD storytelling database. We looked at the correlation of occurrence between each speaker mono-modal and multi-modal signal and each listener feedbacks (where we understand *multi-modal signal* as any set of overlapping signals that are emitted by the speaker within a time window, defined as the time interval of any speaker signal plus 2 seconds). This gave us a correlation matrix between speaker and listener signals, whose elements give, for each speaker signal, the probability that the listener would send a given feedback signal. In our system we use this matrix to select listener's feedback signals. When a speaker's signal is detected, we choose from the correlation matrix, the signal (ie feedback) with the higher probability.

From this process, we identified a set of rules (which can be found in the repository of the project) such as:

* Mono-modal signal $⇒$ mono-modal feedback: *head_nod* is received, then the listener sends *head_nod_medium*.

* Mono-modal signal $⇒$ multi-modal feedback: *smile* is received, then the listener sends *head_nod and smile*.

* Multi-modal signal $⇒$ mono-modal feedback: *head_activity_high* and *pitch_prominence* are received, then the listener sends *head_nod_fast*.

* Multi-modal signal $⇒$ multi-modal feedback: *pitch_prominence* and *smile* are received, then the listener sends *head_nod* and *smile*.

Rules can be made probabilistic via associated probabilities: in case there is more than one rule with the same input, every rule will have a probability of execution.

### Multi-modal fusion

The multi-modal fusion module is responsible for activating the rules mentioned above, when input signals are detected, and will eventually trigger feedbacks from the agent/robot.

For realtime consideration, the rule contains a response time variable, which defines when the output of the rule should be executed after the reception of the last input signal; the last variable is rule duration, rule duration defines how long this rule can be active, so in case not all

the input signals are received, the rule will be deactivated after this specified period.

### Reactive behaviors

In our architecture, we aim to drive different types of virtual and/or physical agents: the GRETA ECA (Fig 12), and The AIBO ERS-7 Robot (Fig 13). To ensure high flexibility we are using the same control language to drive all the agents, the Behavior Markup Language BML [5]. BML encodes multimodal behaviors independently from the animation parameters of the agents.

Through a mapping we transform BML tags into MPEG-4 parameters for the GRETA agent and into mechanical movements for the AIBO robot. Various feedbacks are already available for GRETA such as acceptance (*head_nod*), non-acceptance (*head_shake*) or smile. Concerning AIBO, we developed similar feedbacks conveying the same meaning but in a different way. To develop the reactive behavior of AIBO, we used the URBI (Real-Time Behavior Interface) library [2] allowing a high-level control of the robot.

<figure><img src="image_60.png" /><figcaption>Fig. 12 The GRETA ECA</figcaption></figure>

<figure><img src="image_61.png" /><figcaption>Fig. 13 The ERS-7 Sony Aibo Robot. (from http://www.sony.net/Products/aibo/)</figcaption></figure>
## VIII. ASSESSMENT AND DISCUSSION

Evaluation research is still underway for virtual characters [46, 47, 48] and for human-robot interaction [49, 50].

Since the goal of the project was to compare feedback provided by two types of embodiments (a virtual character and a robot) rather than to evaluate the multimodal feedback rules implemented in each of these systems, we decided to have users tell a story to both Greta and Aibo at the same time. The feedback system tested was the one described in the previous Sections, with the exception of the ASR system, which was not used here.

An instruction form was provided to the subject before the session. Then users watched the cartoon sequence, and were asked to tell the story to both Aibo and Greta (Fig 14). Finally, users had to answer a questionnaire. The questionnaire was designed to compare both systems with respect to the realization of feedback (general comparison between the two listeners, evaluation of feedback quality, perception of feedback signals and general comments). The evaluation form is provided in appendix. Sessions were videotaped using a Canon XM1 3CCD digital camcorder.

<figure><img src="image_62.png" /><figcaption>Fig 14. The assessment set-up</figcaption></figure>

<figure><img src="image_63.png" /><figcaption>**Table 4** Comparing the feedback provided by the virtual character and the robot</figcaption></figure>

As illustrated by table 4, 8 out of 10 users estimated that GRETA understood better the story than AIBO. Yet, 8 out of 10 users felt that AIBO looked more interested and liked the story more than GRETA did.

Further evaluations could be investigated with such a system. Another possibility would be to have the speaker tell two different stories one to Greta, and then another one to Aibo. The order of the listeners should be counterbalanced across subjects. This would avoid having the speaker to switch his attention between Aibo and Greta. Perceptive tests on videos combining speakers and Aibo/Greta listeners could also be designed to have subjects 1) compare random feedback with feedback generated by analyzing user's behavior, or 2) rate if the listener has been designed to listen to this speaker or not.

## IX. CONCLUSION AND FURTHER DEVELOPMENTS

We presented a multi-modal framework to extract and identify Human communicative signals for the generation robot/agent feedbacks during storytelling. We exploited face-to-face interaction analysis by highlighting communicative rules. A real-time feature extraction module has been presented allowing the characterization of communicative events. These events are then interpreted by a fusion process for the generation of backchannel messages for both AIBO and GRETA. A simple evaluation was established, and results show that there is an obvious difference in the interpretation and realization of the communicative behavior between humans and agents/robots.

Our future works are devoted to the characterization of other communicative signals using the same modalities (speech and head). Prominence detection can be improved by the use of syllable-based analysis, which can be computed without linguistic information. Another important issue is to deal with the direction of gaze. This communicative signal conveys useful information during interaction and automatic analysis (human) and generation (robot/agent) should be investigated.

## ACKNOWLEDGMENTS

We are grateful to Elisabetta Bevacqua for her advice in the organization of our work and her help on interfacing our software with GRETA.

We also want to acknowledge Yannis Stylianou for the feedback he gave during discussions on our project.

Some members of this were partly funded by Région Wallonne, in the framework of the NUMEDIART research programme.

FP6 IP CALLAS is also gratefully acknowledged for having funded participants of this project.
REFERENCES

[1] K. Dautenhahn (2007) Methodology and Themes of Human-Robot Interaction: A Growing Research Field. International Journal of Advanced Robotic Systems 4(1) pp. 103-108.

[2] J.C. Baillie (2006) URBI tutorial [online] http://www.gostai.com/doc/en/urbi-tutorial/

[3] Cassell, J., Sullivan, J., Prevost, S., and Churchill, E. (eds). Embodied Conversational Agents. MIT Press, 2000.

[4] E. Bevacqua, M. Mancini, and C. Pelachaud, A listening agent exhibiting variable behaviour, Intelligent Virtual Agents, IVA'08, Tokyo, September 2008.

[5] H. Vilhjalmsson, N. Cantelmo, J. Cassell, N. E. Chafai, M. Kipp, S. Kopp, M. Mancini, S. Marsella, A. N. Marshall, C. Pelachaud, Z. Ruttkay, K. R. Thorisson, H. van Welbergen, R. van der Werf, The Behavior Markup Language: Recent Developments and Challenges, Intelligent Virtual Agents, IVA'07, Paris, September 2007.

[6] T. Fong, I. Nourbakhsh and K. Dautenhahn (2003): A Survey of Socially Interactive Robots, Robotics and Autonomous Systems 42(3-4), 143-166.

[7] C. Breazeal (2004). "Social Interactions in HRI: The Robot View," R. Murphy and E. Rogers (eds.), in IEEE SMC Transactions, Part C.

[8] Oudeyer P-Y, Kaplan, F. and Hafner, V. (2007) Intrinsic Motivation Systems for Autonomous Mental Development, IEEE Transactions on Evolutionary Computation, 11(2), pp. 265--286.

[9] Sidner, C.L.; Lee, C.; Kidd, C.D.; Lesh, N.; Rich, C., "Explorations in Engagement for Humans and Robots", Artificial Intelligence, May 2005

[10] Cassell, J., Bickmore, T., Billinghurst, M., Campbell, L., Chang, K., Vilhjalmsson, H. and Yan, H. (1999). "Embodiment in Conversational Interfaces: Rea." Proceedings of the CHI'99 Conference, pp. 520-527. Pittsburgh, PA.

[11] J. Rickel and W.L. Johnson, Animated Agents for Procedural Training in Virtual Reality: Perception, Cognition, and Motor Control, Applied Artificial Intelligence, 13, pp. 343-382, 1999.

[12] S. Kopp and I. Wachsmuth, Synthesizing Multimodal Utterances for Conversational Agents, The Journal Computer Animation and Virtual Worlds, 15(1), PP 39-52, 2004.

[13] J. Cassell and K. Thórisson, The Power of a Nod and a Glance: Envelope vs. Emotional Feedback in Animated Conversational Agents, Applied Artificial Intelligence, 13(3), 1999.

[14] S. Marsella, Interactive Pedagogical Drama: Carmen's Bright IDEAS Assessed. Intelligent Virtual Agents, pp 1-4, 2003.

[15] E. André, T. Rist, S. van Mulken, M. Klesen and S. Baldes, The automated design of believable dialogues for animated presentation teams, in J. Cassell, J. Sullivan, S. Prevost and E. Churchill (Eds) Embodied

Conversational Characters, MITpress Cambridge, MA, 2000.

[16] W.L. Johnson, H. Vilhjalmsson, S. Marsella, Serious Games for Language Learning: How Much Game, How Much AI?, 12th International Conference on Artificial Intelligence in Education, Amsterdam, The Netherlands, July 18-22, 2005.

[17] A.L. Baylor, S. Kim, C. Son and M. Lee, Designing effective nonverbal communication for pedagogical agents, Proceedings of AI-ED (Artificial Intelligence in Education), Amsterdam, July 2005.

[18] J. Gustafson, N. Lindberg, M. Lundeberg, The August soplex dialog system, Proceedings of Eurospeech'99, Budapest, Hungary, 1999.

[19] Martin, J.C., Buisine, S., Pitel, G., and Bernsen, N. O.: Fusion of Children's Speech and 2D Gestures when Conversing with 3D Characters. In T. Dutoit, L. Nigay and M. Schnaider (Eds.): Multimodal Human-Computer Interfaces. Special Issue of Signal Processing (Elsevier) Vol. 86, Issue 12, December 2006

[20] T. Bickmore, J. Cassell, Social Dialogue with Embodied Conversational Agents", in J. van Kuppevelt, L. Dybkjaer, N. Bernsen (Eds) Advances in Natural, Multimodal Dialogue Systems, Kluwer Academic, New-York, 2005.

[21] L. Hall, M. Vala, M. Hall, M. Webster, S. Woods, A. Gordon and R. Aylett, FearNot's appearance: Reflecting Children's Expectations and Perspectives, Intelligent Virtual Agents IVA, Marina del Rey, USA, pp. 407-419, 2006.

[22] M. Lohse, K. J. Rohlfing, B. Wrede; G. Sagerer, "Try something else!" - When users change their discursive behavior in human-robot interaction, IEEE Conference on Robotics and Automation, Pasadena, CA, USA, 3481-3486, 2008.

[23] J. Allwood, J. Nivre, and E. Ahlsen. On the semantics and pragmatics of linguistic feedback. Semantics, 9(1), 1993.

[24] I. Poggi. Backchannel: from humans to embodied agents. In AISB. University of Hertfordshire, Hatfield, UK, 2005.

[25] D. McNeil, Hand and mind: What gestures reveal about thought, Chicago IL, The University, 1992.

[26] Jens Allwood, Loredana Cerrato, Kristiina Jokinen, Costanza Navarretta and Patrizia Paggio. The MUMIN coding scheme for the annotation of feedback, turn management and sequencing phenomena. Journal of Language Resources and Evaluation, Springer. Volume 41, 2007. pages 273-287

[27] Michael Kipp (2001) *Anvil - A Generic Annotation Tool for Multimodal Dialogue*. *Proceedings of the 7th European Conference on Speech Communication and Technology (Eurospeech)*, pp. 1367-1370.

[28] R. L. Brennan, D. J. Prediger: Coefficient κ: Some uses, misuses, and alternatives. In: Educational and Psychological Measurement. 41, 1981, 687-699.
[29] R. M. Maatman, Jonathan Gratch, Stacy Marsella: Natural Behavior of a Listening Agent. IVA 2005: 25-36.

[30] Pure Data: http://www.puredata.org

[31] The CMU Sphinx open source speech recognizer http://cmusphinx.sourceforge.net

[32] de Cheveigne, A., Kawahara, H.: YIN, a fundamental frequency estimator for speech and music. The Journal of the Acoustic Society of the America 111. 2002.

[33] Paul Taylor. The Tilt Intonation model, ICSLP 98, Sydney, Australia. 1982.

[34] B.M. Streefkerk, L. C. W. Pols, L. ten Bosch, Acoustical features as predictors for prominence in read aloud Dutch sentences used in ANNs, Proc. Eurospeech'99, Vol. 1, Budapest, 551-554, 1999.

[35] J.M.B. Terken, Fundamental frequency and perceived prominence of accented syllables. Journal of the Acoustical Society of America, 95(6), 3662-3665, 1994.

[36] N. Obin, X. Rodet, A. Lacheret-Dujour, "French prominence: a probabilistic framework", in International Conference on Acoustics, Speech, and Signal Processing (ICASSP'08), Las Vegas, U.S.A, 2008.

[37] V. K. R. Sridhar, A. Nenkova, S. Narayanan, D. Jurafsky, Detecting prominence in conversational speech: pitch accent, givenness and focus. In Proceedings of Speech Prosody, Campinas, Brazil. 380-388, 2008.

[38] D. Wang, S. Narayanan, An Acoustic Measure for Word Prominence in Spontaneous Speech. IEEE Transactions on Audio, Speech, and Language Processing, Volume 15, Issue 2, 690-701, 2007.

[39] P.Viola and M.J.Jones, "Robust Real-Time Face Detection", International Journal of Computer Vision, 2004, pp 137-154.

[40] C.G. Harris and M.J. Stephens, "A combined corner and edge detector", Proc. Fourth Alvey Vision Conf., Manchester, pp 147-151, 1988

[41] Lucas, B., and Kanade, T. An Iterative Image Registration Technique with an Application to Stereo Vision, Proc. of 7th International Joint Conference on Artificial Intelligence (IJCAI), pp. 674-679.

[42] S. Young: "ATK: An application Toolkit for HTK, version 1.4, Cambridge University, May 2004

[43] T. Robinson, J. Fransen, D. Pye, J. Foote, and S. Renals. WSJCAM0: A British English speech corpus for large vocabulary continuous speech recognition. In Proc IEEE ICASSP, pages 81-84, Detroit, 1995.

[44] R. M. Maatman, Jonathan Gratch, Stacy Marsella, Natural Behavior of a Listening Agent. Intelligent Virtual Agents, IVA'05, 25-36, 2005.

[45] N. Ward, W. Tsukahara, Prosodic features which cue back-channel responses in English and Japanese. Journal of Pragmatics, 23, 1177-1207, 2000.

[46] Dehn, D. M. and van Mulken, S. (2000). "The impact of animated interface agents: a review of empirical

research." International Journal of Human-Computer Studies(52): 1-22.

[47] Ruttkay, Z. and Pelachaud, C. (2004). From Brows to Trust - Evaluating Embodied Conversational Agents, Kluwer.
http://wwwhome.cs.utwente.nl/~zsofi/KluwerBook.htm

[48] Buisine, S., Martin, J.-C. (2007) The effects of speech-gesture co-operation in animated agents' behaviour in multimedia presentations. International Journal "Interacting with Computers: The interdisciplinary journal of Human-Computer Interaction". Elsevier, 19: 484-493.

[49] Dan R. Olsen, Michael A. Goodrich (2003) Metrics for Evaluating Human-Robot Interactions. Performance Metrics for Intelligent Systems Workshop held in Gaithersburg, MD on September 16-18, 2003.
http://www.isd.mel.nist.gov/research areas/research en gineering/Performance_Metrics/PerMIS_2003/Proceed ings/Olsen.pdf

[50] Mohan Rajesh Elara, Carlos A. Acosta Calderon, Changjiu Zhou, Pik Kong Yue, Lingyun Hu, Using Heuristic Evaluation for Human-Humanoid Robot Interaction in the Soccer Robotics Domain. Second Workshop on Humanoid Soccer Robots @ 2007 IEEE-RAS International Conference on Humanoid Robots Pittsburgh (USA), November 29, 2007
http://www.informatik.uni-freiburg.de/~rc06hl/ws07/papers/HSR-2-110.pdf

## X. APPENDIX

### Instructions

You are going to watch a short cartoon sequence.

Then you will have to tell this story.

Your story will be listened to and watched by the Greta virtual agent displayed on a screen and an Aibo robot at the same time.

Both will react to the story that you are telling.

Please tell the story as you would tell a story to two people listening to you at the same time.

### Evaluation form

Subject number :

Date of session :

### INFORMATION ABOUT SUBJECT

Last name :

First name :

Age :

Male / Female

e-mail :

### GENERAL COMPARISON BETWEEN THE TWO LISTENERS

Did you like telling your story to Greta?

I liked very much / I liked / I did not like

Did you like telling your story to Aibo?
I like very much / I liked / I did not like

Which one did you prefer?

Greta / Aibo

Why?

Which listener was mostly interested in your story?

Greta / Aibo

Which listener understood better your story?

Greta / Aibo

Which listener liked better your story?

Greta / Aibo

What did you like in Greta that was not present in Aibo?

What did you like in Aibo that was not present in Greta?

## EVALUATION OF FEEDBACK QUALITY

Who most clearly displayed when it was interested in your story?

Aibo / Greta

Who most clearly displayed when it understood what you said?

Greta / Aibo

Who most clearly displayed when it liked what you said?

Aibo / Greta

How would you qualify the behavior displayed by Greta when you told your story ?

How would you qualify the behavior displayed by Aibo when you told your story ?

## PERCEPTION OF FEEDBACK SIGNALS

How did Greta displayed that she was interested in your story?

How did Greta displayed that she was understanding in your story?

How did Greta displayed that she liked in your story?

How did Aibo displayed that it was interested in your story?

How did Aibo displayed that it was understanding in your story?

How did Aibo displayed that it liked in your story?

## GENERAL COMMENTS

How did you evaluate the fact to interact with both at the same time?

Please feel free to provide any additional comments:

<figure><img src="image_64.png" /></figure>

Sàmer Al Moubayed is a PhD student since 2008 at the Center of Speech Technology CTT, Royal Institute of Technology KTH, Sweden, and at the Graduate School of Language Technology GSLT, Gothenburg, Sweden. He obtained his MSc degree in Artificial Intelligence and Speech and Language Technology from KULeuven, Belgium.

His main research is carried out in the field of speech communication, and is doing his thesis in Multimodal Speech Synthesis, and is involved in many National and European projects. Samer main research interest is in Talking Agents and behavior prediction, machine learning and pattern recognition in speech, and psycholinguistics

<figure><img src="image_65.png" /></figure>

Malek Baklouti graduated as engineer in from the Tunisian Polytechnic School and received the M.S. degree in Applied Mathematics in 2006. She is currently a PhD student in Robotics and Signal processing at Thalès Security System and Services, France and University of Versailles.

She will be in PAMI lab at the University of Waterloo (Canada) as a Visiting Scholar.

<figure><img src="image_66.png" /></figure>

Mohamed Chetouani received the M.S. degree in Robotics and Intelligent Systems from the University Pierre and Marie Curie (UPMC), Paris, 2001. He received the PhD degree in Speech Signal Processing from the same university in 2004. In 2005, he was an invited Visiting Research Fellow at the Department of Computer Science and Mathematics of the University of Stirling (UK). Dr. Chetouani was also an invited researcher at the Signal Processing Group of Escola Universitaria Politecnica de Mataro,

Barcelona (Spain).

He is currently an Associate Professor in Signal Processing and Pattern Recognition at the University Pierre et Marie Curie. His research activities, carried out at the Institute of Intelligent Systems and Robotics, cover the areas of non-linear speech processing, feature extraction and pattern classification for speech, speaker and language recognition.

He is a member of different scientific societies (ISCA, AFCP, ISIS). He has also served as chairman, reviewer and member of scientific committees of several journals, conferences and workshops.

Thierry Dutoit graduated as an electrical engineer and Ph.D. in 1988 and 1993 from the Faculté Polytechnique de Mons, Belgium, where he is now a full professor.

He spent 16 months as a consultant for AT&T Labs Research in Murray Hill and Florham Park, NJ, from July, 1996 to September, 1998. He is the author of two books on speech processing and text-to-speech synthesis, and the coordinator of the MBROLA project for free multilingual speech synthesis.

T. Dutoit was an Associate Editor of the IEEE Transactions on Speech and Audio Processing (2004-2006) and a member of the INTERSPEECH'07 organization committee. He was the initiator of eNTERFACE workshops and the organizer of eNTERFACE'05.

<figure><img src="image_67.png" /></figure>

<figure><img src="image_68.png" /></figure>

Ammar Mahdhaoui was born in Tunisia, on january 31, 1983. He received the M.S. degree in Engineering of Person-System communication from the university of Grenoble, Grenoble, 2007. Since October 2007, he is PHD Student in signal processing and pattern recognition at the University Pierre and Marie Curie Paris-6, his research activities carried out at the Institute of Intelligent Systems and Robotics,
<figure><img src="image_69.png" /></figure>

**Jean-Claude Martin** is Associate Professor at CNRS-LIMSI, France.

His research topic is multimodal communication, both in human-human and human-computer contexts ; the study of individual differences, the multimodal expression and perception of social behaviors and the evaluation of user's multimodal interaction with embodied conversational agents.

He received the PhD degree in Computer Science in 1995 from the Ecole Nationale Supérieure des Télécommunications (ENST, Paris).

He passed his habilitation to direct research in 2006 on Multimodal Human-Computer Interfaces and Individual Differences.

Annotation, perception, representation and generation of situated multimodal behaviors. He is the head of the Conversational Agents topic of research within the Architecture and Models for Interaction Group (AMI). He co-organised a series of three international workshops on multimodal corpora at LREC 2002, 2004, 2006 and he is a guest editor of a special issue of the international Journal on Language Resources and Evaluation to appear in 2007 on multimodal corpora.

<figure><img src="image_70.png" /></figure>

**Stanislav Ondas** graduated from the Technical University of Košice in 1999. In 2007 he finished your PhD. study at the Department of Electronics and Multimedia Communications at the same university and he is currently an assistent at mentioned department.

His research interests in spoken dialogue systems, dialogue management, voice service designing and natural language processing. He has been involved in several national projects related to spoken and multimodal dialogue systems (ISCI, MOBILTEL).

From October 2008, **Catherine Pelachaud** will be a director of research at CNRS in LTCI, TELECOM ParisTech. She received her PhD in Computer Graphics at the University of Pennsylvania, Philadelphia, USA in 1991. Her research interest includes representation language for agent, embodied conversational agent, nonverbal communication (face, gaze, and gesture), expressive behaviors and multimodal interfaces. She has been involved in several European and national projects related to multimodal communication (EAGLES, IST-ISLE), emotion (FP5 NoE Humaine, FP6 IP CALLAS, FP7 STREP SEMAINE), to believable embodied conversational agents (IST-MagiCster, FP5 PF-STAR, RIAM ACE, ANR My-Blog-3D), and to handicap (CNRS-Robea HuGeX, RIAM LABIAO).

<figure><img src="image_71.png" /></figure>

<figure><img src="image_72.png" /></figure>

**Jérôme Urbain** graduated as an electrical engineer from the Faculté Polytechnique de Mons (FPMs), Belgium, in 2006. He is currently PhD student at the Signal Processing and Circuit Theory (TCTS) Lab of the same University, working on speech processing in the framework of FP6 IP CALLAS.

He is focusing on laughter modeling, synthesis and recognition.

**Mehmet Yilmaz** is a senior undergraduate student at Koç University, Electrical and Electronics Department. His research interests are visual tracking, and interactive multimodal systems.

<figure><img src="image_73.png" /></figure>
# Activity-related Biometric Authentication

G. Ananthakrishnan, H. Dibeklioğlu, M. Lojka, A. Lopez, S. Perdikis, U. Saeed, A.A. Salah, D. Tzovaras, A. Vogiannou

*Abstract*—This project aims at developing a biometric authentication system exploiting new features extracted by analysing the dynamic nature of various modalities, including motion analysis during ordinary tasks performed in front of a computer, analysis of speech, continuous face and facial movement analysis, and even patterns for grasping objects. We test the potential and contribution of each of these modalities for biometric authentication in the face of natural, uncontrolled environments, as well as their fusion.

*Index Terms*—Biometric authentication, activity recognition, face recognition, motion analysis, speaker recognition, audio based event recognition

## I. INTRODUCTION

This project attempts to address the limitations of unimodal biometrics by deploying activity-related multimodal biometric systems that integrate the evidence presented by multiple sources of information. Therefore, the combination of a number of independent modalities is explored to overcome the possible restrictions set by each modality. With a simple sensor setup, we aim at more robust biometric identification through the fusion of physiological, behavioral and soft biometric modalities, keeping also in mind the unobtrusiveness and comfort of the subject.

The term behavioral biometrics refers to Person Recognition using shape based activity signals (gestures, gait, full body and limb motion) or face dynamics. Activity-specific signals [6], [23] provide the potential of continuous authentication, but state-of-the-art solutions show inferior performance compared to static biometrics (fingerprints, iris). This drawback could hopefully be eliminated by the inferential integration of different modalities.

Behavioral information from face videos for person recognition may also be investigated in order to exploit the underlying temporal information in comparison to image-based recognition [39]. Methods for person recognition from face dynamics can be classified into holistic methods (head displacements and pose evolution [30]), feature-based methods (exploitation of individual facial features [12]) and hybrid methods [14]. Various probabilistic frameworks have been proposed in recent works, usually employing a Bayesian Network (Hidden Markov Models, Coupled and Adaptive HMMs, etc.) as the mathematical model for recognition [35].

Soft biometrics (gender, height, age, weight etc.) are believed to be able to significantly improve the performance of a biometric system in conjunction with conventional static biometrics [22], yet their exploitation remains an open issue. Microphones for voice recognition, sound based sensors for monitoring activities or other modalities could also be considered.

This report, as well as the source code for the software developed during the project, is available online from the eNTERFACE'08 web site: www.interface08.limsi.fr.

G. Ananthakrishnan is with Royal Institute of Technology, SWEDEN.

E-mail: agopal@kth.se.

H. Dibeklioğlu is with Perceptual Intelligence Laboratory, Department of Computer Engineering, Boğaziçi University, 34342 Bebek, Istanbul, TURKEY.

E-mail: hamdi.dibeklioglu@cmpe.boun.edu.tr.

M. Lojka is with Technická Univerzita v Košiciach, SLOVAKIA.

E-mail: martin.lojka@tuke.sk.

A. Lopez is with Technical University of Catalonia, Barcelona, SPAIN.

E-mail: alopez@gps.tsc.upc.edu.

S. Perdikis, A. Vogiannou and D. Tzovaras are with CERTH/ITI, GREECE.

E-mail: {perdik,tvog}@iti.gr.

U. Saeed is with EURECOM, FRANCE.

E-mail: usman.saeed@eurecom.fr.

A.A. Salah is with Centrum Wiskunde & Informatica, 1090 GB Amsterdam, THE NETHERLANDS.

E-mail: a.a.salah@cwi.nl.

In this report, we look at some of these modalities in a specific fixed-seat pilot. Our experimental setup is described in Section II, including the details of the collected database. The individual modalities are investigated in separate sections, starting with model-based motion analysis in Section III, which tracks the user via calibrated cameras during ordinary activities. The sounds that ensue during these activities are analysed for robust activity classification. This part is exposed in Section IV. Once speech is detected among the sound events, it can be further used for authentication. Section V deals with speaker authentication. Our model flexibly integrates data coming from seemingly unrelated modalities. Section VI exemplifies this by making use of an advanced interface for recognizing activity, namely a Cyberglove, which is used to collect and analyse grasping patterns. The more common face modality is used to serve as a benchmark. Continuous authentication from captured static face images is explained in Section VII, and the optical-flow based analysis of facial motion for authentication is detailed in Section VIII. Section IX builds on the motion analysis to recognize types of activities, and evaluates the authentication potential of each of these activities.

The mathematical framework we establish here is employed to seamlessly integrate an arbitrary number of sources that provide partial authentication information. Our experimental results are given in Section XI. The report concludes with a discussion of these results and on possible future directions in Section XII.

## II. THE EXPERIMENTAL SETUP

The proposed biometric system is evaluated in a fixed-seat office pilot, where the user is able to move his arms, head and torso and manipulate objects on a desk while seated. This experimental setup selection serves multiple aspects of the problem of activity-related biometric authentication:

* It is portable and easy to setup

* It can be part of a normal authentication system scenario (e.g. secured indoor premises)

* It can easily incorporate all the equipment for selected modalities

* An office environment is involved in many work-related activities, which makes the pilot ideal for testing the activity-related authentication module

* It is fully unobtrusive to the user

The selected pilot consists of a desk upon which a number of objects is placed, in stable predefined positions. This constraint implies a static environment, which slightly affects the generality of the setup, but significantly facilitates the activity recognition task. The objects are: a) desk phone, b) glass (on a pad), c) keyboard, d) mouse, e) computer screen, f) pencil (in a pencil case) g) a piece of paper for writing. The sensorial equipment is as inexpensive and unobtrusive as possible. It comprises of three Logitech QuickCam webcam (two for body motion tracking and one for continuous face authentication and facial motion analysis) and a regular low-budget microphone. Two cameras are mounted on the desktop screen facing the user (these are the frontal motion tracking camera and the face camera, which is zoomed on the user's head area), while the third camera (lateral motion tracking camera) is placed on a tripod on the left side of the desk. The microphone is mounted on the desk, next to the keyboard. Fig. 1 illustrates the actual pilot setup.

### A. Recording Scenario and Data Gathering

Within the project a database of 15 persons performing a number of actions has been recorded. Each person was asked to execute six actions in a particular order, responding to the environmental stimuli (phone ringing, instructions on the screen or on a writing form). A recording scenario has been prepared so as to enhance the database's consistency, to meet the requirements and constraints of every modality and to ensure the user's concentration and relaxation, so that he performs the required
<figure><img src="image_74.png" /><figcaption>Fig. 1. The pilot setup, shown during one of the recordings. The frontal cameras are mounted on the display, and the side camera is mounted on a pod to the left of the subject.</figcaption></figure>

actions in his natural (and therefore consistent) way. The six recorded actions were:

* Mouse manipulation (playing a computer game)

* Phone Conversation (real dialogue with a team member)

* Typing in the keyboard (filling in a given questionnaire)

* Writing (filling in a questionnaire in a writing form)

* Drinking (taking the cup, and leaving it back to its place)

* Reading (specific texts provided in the screen).

Every session consisted of one repetition of the six actions and 10 sessions were recorded for each user in order to provide enough training and testing data for all the modalities. The database size was limited to 15 persons due to limited time available for recordings.

During the data gathering users were asked to act in their natural way, without any further instructions or constraints. The selected activities are common work - related activities involving usual office objects, there was no previous knowledge about their suitability for authentication. The evaluation of their discriminative power is among the objectives of this project.

### III. MODEL-BASED MOTION ANALYSIS

Markerless human motion capture is a challenging problem that involves the estimation of a high-dimensional configuration of a three-dimensional non-rigid and self-occluding object. Since a wide range of applications are derived from the unobtrusive characterization of human activity, this research area has recently undergone several advances due to the yielded interest.

A common approach is to consider an articulated body model with several degrees of freedom per joint, depending on the complexity of the possible poses and the quality of the available data. This representation implies the use of kinematic constraints on the motion. Additional assumptions and motion constraints can be adopted at the cost of

<table><caption>TABLE I
ARTICULATED BODY MODEL JOINTS</caption><thead><tr><th>Angle</th><th>Joint</th><th>Rotation Axis</th><th>Range</th></tr></thead><tbody><tr><td>θ<sub>1</sub></td><td>Base of the Neck</td><td>y</td><td>[-π/4, π/4]</td></tr><tr><td>θ<sub>2</sub></td><td>Right Shoulder</td><td>x</td><td>[-π/4, π]</td></tr><tr><td>θ<sub>3</sub></td><td>Left Shoulder</td><td>x</td><td>[-π/4, π]</td></tr><tr><td>θ<sub>4</sub></td><td>Right Shoulder</td><td>y</td><td>[-π/4, π]</td></tr><tr><td>θ<sub>5</sub></td><td>Left Shoulder</td><td>y</td><td>[-π/4, π]</td></tr><tr><td>θ<sub>6</sub></td><td>Right Shoulder</td><td>z</td><td>[-π/4, π/2]</td></tr><tr><td>θ<sub>7</sub></td><td>Left Shoulder</td><td>z</td><td>[-π/4, π/2]</td></tr><tr><td>θ<sub>8</sub></td><td>Right Elbow</td><td>y</td><td>[0, π]</td></tr><tr><td>θ<sub>9</sub></td><td>Left Elbow</td><td>y</td><td>[0, π]</td></tr></tbody></table>

generality of the solution which we intend to preserve. To this end, Particle Filters [2] have become a relevant technique due to their ability to handle multi-modal non-linear and non-Gaussian distributions. Several approaches such as partitioned sampling [37], hierarchical sampling [41] and annealing particle filter [15] have been developed to cope with high-dimensional limitations of the classical Condensation algorithm [21].

We present a particular implementation of the annealing particle filter for a simplified body model in order to retrieve the human body poses of a subject performing different actions in a multi-view scenario. We propose simplifications of the body tracking problem without almost no loss of generality in the given pilot and with the capability of coping with realistic scenarios.

#### A. Body Model

A simplistic articulated body model will fulfill the requirements of the scenario presented in section II. This model is based on the kinematic chain framework and comprises a set of joints. In our case, this set of joints are the base of the neck, shoulders and elbows. Every joint has a maximum of three degrees of freedom according to the complexity of the motions that we want to capture. Each degree of freedom is represented by an axis of rotation defined in a default body configuration, where all the angles are set to zero (see fig. 2). The range of joint angles is also defined according to this default body pose. In our model, a total of nine degrees of freedom are defined (see table I). In order to set the model in a world position, a three-dimensional coordinate system built with the base of the neck as origin and a body orientation are defined. Our model reference point is set to be the base of the neck. Therefore, our body model defines a thirteen-dimensional state vector:

$$x_t = \{x_0, y_0, z_0, \theta_0, \dots, \theta_9\} \quad (1)$$

Angle $θ_0$ is the orientation of the whole body model while all the other angles are designed following basic kinematic constraints. The use of angles ensures a compact representation in front of a state defined by only 3D coordinates. Knowing the limbs' dimensions we can go from a set of angles to Cartesian coordinates by means of exponential twists formulation [7]; every point of interest can be computed from its initial location with respect to the reference point in the default body configuration and the product of the exponential maps affecting the motion of this point:

$$p(x_t) = \prod_i M_i(x_t) p_0 \quad (2)$$

$$M_i = \begin{bmatrix} R_i(x_t) & t_i(x_t) \\ 0 & 1 \end{bmatrix} \quad (3)$$

where $p(x_t)$ represents a point of interest as a function of the state vector, that encodes model position, model orientation and joint angles, and $M_i(x_t)$ is the exponential map in the chain where $p$ is found. The exponential map comprises the rotation matrix $R$ and the translation vector $t$. The whole notation is being presented in homogeneous coordinates due to its compactness.

#### B. Particle Filter

Particle Filters (PF) [2] are recursive Bayesian estimators derived from Monte Carlo sampling techniques which can handle non-linear and non-Gaussian processes. Commonly used in tracking problems, they are used to estimate the posterior density $p(x_t|z_t)$ by means of a set of
<figure><img src="image_75.png" /><figcaption>Fig. 2. Simple articulated model for body tracking</figcaption></figure>

$N_s$ weighted samples or particles. Given a Bayesian recursive estimation problem:

$$p(\mathbf{x}_{0:t}|\mathbf{z}_{1:t}) = \frac{p(\mathbf{z}_{1:t}|\mathbf{x}_t)p(\mathbf{x}_t|\mathbf{x}_{t-1})p(\mathbf{x}_{0:t-1}|\mathbf{z}_{1:t-1})}{p(\mathbf{z}_t|\mathbf{z}_{1:t-1})} \quad (4)$$

we want to draw samples from the posterior such that:

$$p(\mathbf{x}_{0:t}|\mathbf{z}_{1:t}) \approx \sum_{i}^{N_s} w_t^i \delta(\mathbf{x}_t - \mathbf{x}_t^i) \quad (5)$$

where $w_t^i$ is the weight associated to the i-th particle. This discrete approximation of the posterior requires the weights evaluation. This is done by means of the importance sampling principle [16], with a probability density function (pdf) $q(\mathbf{x}_{0:t}|\mathbf{z}_{1:t})$ from which we can generate samples that can be evaluated with the posterior (up to proportionality). Applying the importance sampling principle in Eq. 4:

$$w_t^i \propto \frac{p(\mathbf{x}_{0:t}|\mathbf{z}_{1:t})}{q(\mathbf{x}_{0:t}|\mathbf{z}_{1:t})} \quad (6)$$

$$w_t^i \propto \frac{p(\mathbf{z}_{1:t}|\mathbf{x}_t)p(\mathbf{x}_t|\mathbf{x}_{t-1})}{p(\mathbf{z}_t|\mathbf{z}_{1:t-1})q(\mathbf{x}_{0:t}|\mathbf{z}_{1:t})} p(\mathbf{x}_{0:t-1}|\mathbf{z}_{1:t-1}) \quad (7)$$

and choosing this importance distribution in a way that factors appropriately we have:

$$w_t^i \propto \frac{p(\mathbf{z}_{1:t}|\mathbf{x}_t)p(\mathbf{x}_t|\mathbf{x}_{t-1})p(\mathbf{x}_{0:t-1}|\mathbf{z}_{1:t-1})}{p(\mathbf{z}_t|\mathbf{z}_{1:t-1})q(\mathbf{x}_t|\mathbf{x}_{0:t-1}, \mathbf{z}_t)q(\mathbf{x}_{0:t-1}|\mathbf{z}_{1:t-1})} \quad (8)$$

$$w_t^i \propto w_{t-1}^i \frac{p(\mathbf{z}_{1:t}|\mathbf{x}_t)p(\mathbf{x}_t|\mathbf{x}_{t-1})}{p(\mathbf{z}_t|\mathbf{z}_{1:t-1})} \quad (9)$$

Moreover, if we apply the Markov assumption the expression is simplified regarding the fact that observations and current state only depend on the previous time instant. Therefore, the PF is a sequential propagation of the importance weights.

Two major problems affect the PF design. The first is the choice of the importance distribution. This is crucial since the samples drawn from $q()$ must hit the posterior's typical set in order to produce a good set of importance weights. It has been shown in [16] that $q(\mathbf{x}_t|\mathbf{x}_{t-1}, \mathbf{z}_t) = p(\mathbf{x}_t|\mathbf{x}_{t-1}, \mathbf{z}_t)$ is optimal in terms of variance of the weights. The second problem deals with particle degeneracy in terms of variance of the weights. After several iterations the majority of the particles have negligible weights and as a consequence of this the estimation efficiency decays. An effective measure for the particle degeneracy is the survival rate [34] given by:

$$\alpha = \frac{1}{N_s \sum_{i=1}^{N_s} (w_t^i)^2} \quad (10)$$

In order to avoid the estimator degradation the particle set is resampled. After likelihood evaluation a new particle set must be drawn from the posterior estimation, hence particles with higher weights are reproduced with higher probability. Once the new set has been drawn all the weights are set to $\frac{1}{N_s}$, leading to a uniformly weighted sample set concentrated around the higher probability zones of the estimated posterior.

The Sampling Importance Resampling (SIR) Particle Filter proposed by Gordon et. al [18] is a method commonly used in computer vision problems. It's characterized by applying resampling at every iteration and by defining the importance distribution as the prior density $p(x_t|x_{t-1})$. By substituting this importance density in 8, it's easy to realize that weight computation only depends on the likelihood. Consequently, the design of the particle filter is basically a problem of finding an appropriate likelihood function.

### C. Likelihood Evaluation

In computer vision problems probability density functions usually are not directly accessible, thus an observation model is required to approximate the likelihood function. It is necessary to determine which image features are more correlated with the true body configuration. Therefore, finding the appropriate likelihood approximation involves both image and body model. Deutscher et al. [15] proposed a matching of the model projection with foreground segmentation and edges. Their flesh model consists of conic sections with elliptical cross-sections surrounding virtual skeleton segments. Raskin et al. [50] add the body part histogram as an additional feature. Other authors use Visual Hull approaches [27] to work with voxel data. In that case, they can use three-dimensional flesh models, like ellipsoids [40] or three-dimensional Gaussian mixtures [9].

Our challenge is to produce a likelihood approximation able to deal with moving objects, clothing, limited number of views and low frame rate. In our approach we should not rely on a 3D reconstruction because only a few views are available, thus a projection of the model onto the images is required. Our proposal is to avoid the computational cost of projecting the whole set of sampling points of a 3D flesh model by projecting a reduced set of points per body part. Our flesh model will be set of cylinders around all the skeleton segments except the head, which will be modelled by a sphere (see Fig. 2). Therefore, our reduced set of projected points will be defined by the vertices of the trapezoidal section resulting from the intersection of a plane, approximately parallel to the image plane, with the cylindrical shapes modelling the limb (or spherical shape in the case of the head).

To define an intersecting plane for a given cylinder, we compute the vectors going from the camera center towards one of the limit points of the limb. Then the cross product of these vectors with the one defined by the limb itself is computed to determine two normal vectors that lie on the intersecting plane and along which we will find the key points to project. The head template is handled with a similar procedure using as limb vector the one going from the body model reference point to the head center. The norm of the cross product, as well as the area of the projected trapezoid, can be used as a quality measure in order to determine whether the limb is properly aligned with the view (this does not apply for the head). If this quality measure is above a certain threshold, we can change the trapezoidal projected shape by a circle or an ellipse. However, in our scenario the views are set so that they capture good limb alignments in most of the frames, thus we can obviate the computation of this measure.

Regarding the image features, we have seen that common likelihood approximations like [15] do not perform well in our scenario with the described body model. We propose modifications on this approximation while keeping common features that are easy to extract, like foreground silhouettes, contours and detected skin. We extract foreground silhouettes by means of a background learning technique based on Stauffer and Grimson's method [56]. A single multivariate Gaussian $N(\mu_t, \Sigma_t)$ with diagonal covariance in the RGB space is used to model every pixel value $I_t$. The algorithm learns the background model for every pixel using a set of background images and then, for the rest of the sequence, evaluates the likelihood of a pixel color value to belong to the background. With every pixel that matches the background the pixel model is updated, adaptively learning smooth illumination changes:

$$\mu_t = (1 - \rho)\mu_{t-1} + \rho I_t \quad (11)$$

$$\Sigma_t = (1 - \rho)\Sigma_{t-1} + \rho(I_t - \mu_{t-1})^T (I_t - \mu_{t-1}) \quad (12)$$
<figure><img src="image_76.png" /><figcaption>Fig. 3. Projection of the flesh model associated to a given particle</figcaption></figure>

A shadow removal algorithm [65], based on the color and brightness distortion, is used to enhance the segmentation.

<figure><img src="image_77.png" /><figcaption>(a) Foreground Mask</figcaption></figure>

<figure><img src="image_78.png" /><figcaption>(b) Contours Mask</figcaption></figure>

<caption>Fig. 4. Extracted Image features</caption>

Contour detection is performed by means of the Canny edge detector [10]. The result is dilated with a 8-connectivity and 5x5 structuring element, and smoothed with a Gaussian mask. In order to avoid spurious contours, we subtract the background contours. This implies also deleting some pixels in the edges of interest but the body structure it's in general preserved. Finally, a simple skin detection method based on evaluating the likelihood ratio between skin and non-skin hypothesis is performed. The likelihood functions are estimated by 8-bins color histograms of several skin and non-skin samples.

The likelihood evaluation procedure involves the projection of the flesh model of every particle onto the image coordinate system. The resulting shape is scanned and matched with the foreground segmentation. The weight is computed as follows:

$$ \omega^{fl} = \frac{1}{N} \sum_{n=1}^{N} (1 - I_{n}^{f}) \qquad (13) $$

Since pixel intensities in the foreground masks ($I_t^f$) have 0 or 1 as possible values, the weighting function is obtained by a normalized sum of the background pixels falling inside the projected flesh model. In the head model case, we add skin detection information:

$$ \omega^{fh} = \frac{1}{N} \sum_{n=1}^{N} (1 - I_{n}^{f} I_{n}^{s}) \qquad (14) $$

Therefore, the final foreground weight $\omega^f$ is the averaged sum of all the limbs and head weights. Foreground segmentation provides data that are generally invariant to clothing and most of the background conditions.

Since many configurations can be explained via this feature, foreground information is used to penalize false poses rather than to single out the correct one. Moreover, the proposed measure shows how well the model fits the observation, but doesn't evaluate how well the observations are being explained by the model. Suppose the likelihood $p(z_t|x_t)$ is available and that a given pose generates a pdf. A measure that can be used to assess the similarity of the likelihood and the generated pdf is the Kullback-Leibler divergence. At this point is important to remark that the KL divergence will provide different results depending on the factor order

(except if both pdfs are identical). We can establish an analogy with our likelihood approximation. We are trying to determine the mutual information of the model and the observations. Therefore, we propose to include an additional divergence measure between the projection of the flesh model and the foreground masks to see how well a particle explains the observations.

$$ \omega^d = \frac{1}{N_f} \sum_{n=1}^{N_f} (I_n^f (1 - B_n)) \qquad (15) $$

This divergence basically aims for projecting a given particle and computing the overlap between the pixels $B_n$ of this projection and the $N_f$ foreground pixels of the observation.

Contours found in the body usually provide good information on the location of the arms and the legs. However, in some cases, clothing and background can introduce spurious contours that reduce the reliability of this feature. As mentioned above, we try to minimize the background impact by subtracting the background contours. The proposed weighting function for this feature is a sum of squared differences between the contour pixels and the edges of the flesh model aligned with the axis of the limb:

$$ \omega^e = \frac{1}{N} \sum_{n=1}^{N} (1 - I_{n}^{e})^2 \qquad (16) $$

Finally, all these weights are combined for every camera:

$$ \omega = \exp \left( \sum_{c=1}^{C} (\lambda_c^f \omega^f + \lambda_c^e \omega^e + \lambda_c^d \omega^d) \right) \qquad (17) $$

We use a set of weights for every camera and measure to adjust the importance of every feature according to its importance and visibility. Since in our scenario the subject stays in his seat, we assume that the visibility component can be determined beforehand.

## D. Annealing Particle Filter

It has been shown in several works that SIR Particle Filters are a good approach for tracking in low dimensional spaces, but they become inefficient in high-dimensional problems. Deutscher et. al [15] proposed a variation of the SIR framework by introducing the concept of Annealing PF. In body pose tracking problems, the likelihood approximation often is a function which has several peaked local maxima. Annealing PF deals with this problem by evaluating the particles in several smoothed versions of the likelihood approximation. After the weights are computed via the modified likelihood, particles are resampled and propagated with Gaussian noise with zero mean and a covariance that decreases at every step. Each one of this steps (weighting with a smoothed function, resampling and propagation) is called an annealing run. In the last annealing run the estimation is given by means of the Monte-Carlo approximation of the posterior mean:

$$ \hat{x}_t = \sum_{i=1}^{N_s} w_i^t x_t^i \qquad (18) $$

The most usual way to smooth the weighting function is by means of an annealing rate, an exponent $\beta < 1$. In the first layer $\beta$ is minimum but it progressively increases with each layer, sharpening the likelihood approximation. In [15] a method for tuning $\beta$ with the survival rate after each annealing run is proposed.

The sharpness of the likelihood function is due to the high dimensional space in which is defined, the use of a hierarchical model [11] is another possible strategy in order to have annealing layers. Since our model is quite simple, a hierarchical approach is not justified. We have implemented an annealing particle filter in which the smoothing is done by means of an exponent $\beta$. In our case, the annealing rate is updated according to the survival rate of the preceding layer $\alpha(\beta_{t-1})$. Given a desired survival rate $\alpha_T$:

$$ \beta_t = \beta_{t-1} - \lambda(\alpha_T - \alpha(\beta_{t-1})) \qquad (19) $$

Due to the image feature characteristics, we also introduce $\beta$ in (17), giving higher importance to the foreground-based measures in the first layers and to the contour-based measures in the last layers.

$$ \omega = \exp \left( \sum_{c=1}^{C} (\lambda_c^f(\frac{1}{\beta})\omega^f + \lambda_c^e(\beta)\omega^e + \lambda_c^d(\frac{1}{\beta})\omega^d) \right) \qquad (20) $$
<figure><img src="image_79.png" /><figcaption>Fig. 5. Annealing Layers. The covariance used in the propagation step is progressively reduced through four annealing layers while the estimator gets closer to the true pose</figcaption></figure>

Therefore, we propose to work with overall smoothing and feature-based smoothing. However, more work needs to be done in this area in order to show that this approach can help to efficiently reach the true pose.

## IV. SOUND-BASED EVENT DETECTION

This section deals with detection of sound activity and classification of sounds into the typical events that would be encountered. In the first step, any sort of sound activity is detected and in the second step it is classified. The details of each step are explained below.

### A. Sound Activity Detection

The field of Sound Activity Detection has been researched for several years. Most of the research has been in the field of Voice activity detection in noisy conditions. This is essentially different from the current experiment in which all sound activity needs to be detected. This makes it a slightly difficult problem in a way, because a threshold on the length of the activity cannot be provided. The detection has to be made on short bursts of sounds like clicks of mouse as well as continuous speech. So a dynamic threshold needs to be provided, based on the current noise level.

Previous work done in voice activity detection was mainly by Mak *et al.* [38] and Nemer *et al.* [45]. Nemer *et al.* proposed a method based on the residual of the signal, and used higher order statistics of the noise in order to set the threshold to detect sound activities. Renevey and Drygajlo [52], proposed an Entropy based threshold for activity detection. The method used in this experiment uses the entropy found on the residual as a measure to detect activity.

The following steps are taken to detect sound activity

* The signal is windowed with a window size of 40 ms and a shift of 20 ms.

* The signal within one window is approximated by 2 Linear Prediction Coefficients (LPC). This is done to grossly approximate the frequency spectrum and calculate the bias.

* The residual of the signal, which is the error between the estimated LPC and the true signal is calculated. Fig. 6 shows the spectrum of the signal and Fig. 7 shows the corresponding residual. One can observe that the bias has been canceled and the spectrum has been whitened.

* The Entropy is calculated for the residual, assuming a gaussian distribution, since whitening has been performed. The evidence of activity is given by the entropy. A higher entropy indicates a higher level of activity.

* A dynamic threshold is calculated, which decides whether the entropy is high enough to be classified as noise.

The biggest problem with sound activity detection is the hysteresis associated with detection. After detecting a certain sound, we cannot hear

<figure><img src="image_80.png" /><figcaption>Fig. 6. The log-frequency spectrum of a typical signal</figcaption></figure>

<figure><img src="image_81.png" /><figcaption>Fig. 7. The log-frequency spectrum of the residual</figcaption></figure>

other less louder sounds occurring after it. Hence a dynamic threshold has to be calculated based on the statistics of the past. Since the distribution of the sound activity entropy is unknown, a histogram is calculated, for the entropy over a history of around 10 seconds. If the entropy level is in the highest L% range of the histogram, then it considered as activity. However, the entropy level has to go below the 50% range of the past activity to be classified as background noise. Fig. 8 shows the Entropy variation of a short segment of the signal. The two dynamic thresholds are also indicated along with the decision.

The value of L decides the region in the Detection Error Trade-off (DET) curve as shown in Fig. 9. Most of the errors that occur are due to the fact that length of the detected activity is either shorter or longer than the annotated activity. Often what is annotated as a contiguous activity is split into several activities or what is annotated as different activities, is detected as a single activity. The DET curve for length independent detection is shown in Fig. 10.

### B. Sound Event Classification

Sound event classification has been commonly called auditory scene analysis in literature. The most seminal work on auditory scene analysis is discussed by Bregman [8]. Several methods and several features have been tried for this purpose. Among the most common features used are Bark-filter coefficients, wavelet coefficients, Linear Prediction Coefficients etc. Similarly, Support Vector Machines (SVM), Self Organizing Maps (SOM), Artificial Neural Networks (ANN) and Gaussian Mixture Models (GMM) and their combinations have been used for this purpose.

In our experiments Bark filter coefficients are used as features for classification, because the Bark filters mimic the subjective measurements
<table><caption>TABLE II
SOUND EVENT RECOGNITION RESULTS</caption><thead><tr><td>Sound</td><td>Accuracy</td><td>False Alarm</td><td>Most confused</td></tr></thead><tbody><tr><td>Voice</td><td>67.3%</td><td>12.3%</td><td>Pencil keep</td></tr><tr><td>Telephone ringing</td><td>54.3%</td><td>0%</td><td>Voice</td></tr><tr><td>Writing sound</td><td>5.2%</td><td>0%</td><td>Silence</td></tr><tr><td>Keyboard typing</td><td>45.0%</td><td>15.3%</td><td>Mouse click</td></tr><tr><td>Glass use</td><td>89.3%</td><td>56.3%</td><td>-</td></tr><tr><td>Mouse click</td><td>43.3%</td><td>19.5%</td><td>Typing</td></tr><tr><td>Phone receiver</td><td>63.3%</td><td>32.4%</td><td>Keyboard Typing</td></tr><tr><td>Pencil use</td><td>54.3%</td><td>12.8%</td><td>Voice</td></tr><tr><td>Overall</td><td>53.8%</td><td>23.7%</td><td>-</td></tr></tbody></table>

<figure><img src="image_82.png" /><figcaption>Fig. 8. The Entropy variation for a short segment of the signal. The dynamic thresholds are also shown</figcaption></figure>

<figure><img src="image_83.png" /><figcaption>Fig. 9. The log scale plot of the Detection Error Trade-off Curve</figcaption></figure>

<figure><img src="image_84.png" /><figcaption>Fig. 10. The log scale plot of the Detection Error Trade-off Curve, independent of length</figcaption></figure>

of loudness of the human ear. Since we have sound events with different durations, and since we are classifying contiguous blocks of signals, one-state HMMs are used for event classification where the observation probability distribution is expressed with a GMM. This helps in coupling the likelihoods of each of the frames of the signal to give a single likelihood value.

The most important question in these models is to decide how many mixture components will be employed. This is a difficult problem, especially because there are only a few available sounds, with varying length and duration. The number of Gaussians for each sound class is decided by maximizing the Bayesian Information Criterion (BIC). The sound classes that we used are as follows:

1) Voice

2) Telephone Ringing

3) Typing Sound

4) Writing sound (with a pencil)

5) Placing the glass on the table

6) Clicking of the Mouse

7) Picking up the phone receiver or putting it back

8) Picking up or placing the pencil

One can see that, a few of these sounds are quite similar and difficult to distinguish even for human beings. However, since the experiment is set-up in a controlled environment, one can expect a decent performance. Table II denotes the results of the recognition of each of the sounds in the list.

As we can see, the accuracy is higher for detection of voice and glass use, but the false alarm is also high for the same two sounds. There is a very high confusion rate between mouse click and typing for example. It is expected that we assume higher priors for more probable events and lower priors for less probable events. However in that case most of the sounds would be classified as voice, because the voice includes sounds similar to each of these mentioned sounds. So the classification is done assuming an equal prior. The overall accuracy may be boosted if the priors were selected according to their probability of occurrence, but then the overall accuracy evaluation would be biased. It does not make sense to use a weighted average for calculation of accuracy, because one wrongly classified event with low probability would affect the overall accuracy greatly.

More work can be done in the direction of a better classifier, using combination of GMM with classifiers like ANNs or SVM. More evaluation is necessary to deal with different time lengths of each of these sound events. Different number of models and modeling the dynamics of the sounds could be other options. A varying length window in order to calculate the Bark coefficients maybe another direction of research.

## V. SPEAKER VERIFICATION

The speaker verification system provides a Boolean authentication decision based on the analysis of a speech fragment. Speech-based verification systems can be classified into two main types. In the first approach, the speaker utters a word or a sentence, which is fixed for all authentication attempts. This is called the text-dependent approach. In the more difficult text-independent approach, which is more appropriate for this scenario, the speaker can utter any sentence, and the textual content is not known a priori. For a good survey of speaker verification systems, the reader is referred to [47]. Suffice it to say that all such systems need a speaker model, and an impostor model to determine the decision for authentication. Frequently employed methods for modeling the speaker as well as the impostor include dynamic time warping (DTW),
vector quantization (VQ), Gaussian mixture models (GMM), and hidden Markov models (HMM).

The DTW is used for non-linear aligning of two time sequences and computing the minimum distance between them. Use of DTW in speaker verification system is based on assumption that every speaker is uttering the same word or sentence approximately in the same manner but differently from other speakers. Here the speaker is represented by a template of one or limited set of words or sentences. As such, this method is not adequate for text-independent verification. Vector quantization methods are based on the assumption that the acoustic space of a speaker's speech output can be divided into non-overlapping classes, representing different kinds of sounds, for example phonemes. Each class is defined by one vector, centroid and so each speaker is represented by a set of these classes, thus by his own codebook of vectors. In the GMM approach, the codebook vectors are the means of the Gaussian distributions. Here, the noise around each mean is assumed to be normally distributed. Each speaker is represented by a Gaussian mixture density, which is weighted linear combination of Gaussian distributions of each speaker's acoustic class. Thus speaker is represented by a set of weigths, means and variances. In the HMM approach, the speech dynamic is modeled by a Markov model, where the states are modeled by codebooks of the VQ (discrete HMM) or by Gaussian mixture densities (continuous and semi-continuous HMM). In the particular case of text-independent verification systems, ergodic models are preferred where all interstate-transitions have non-zero probabilities.

In this work, we follow the GMM approach based on the results reported in [4], [51]. First, a number of features are extracted from the input signal. Following [20], we use Mel-filter cepstral coefficients (MFCC) by applying the following transformations:

* Preemphasis filter

* Division of signal into frames

* Fast Fourier transformation for obtaining frequency spectrum

* Logarithmic transform

* Application of Mel-filter banks to the spectrum

* Discrete cosine transform

In speech recognition, usually 13 coefficients are selected from the MFCC. The first and second derivatives (i.e. velocity and acceleration) are added to these coefficients to indicate the history and evolution of the signal, resulting in 39-dimensional feature vectors. N-dimensional feature vector implies using N-dimensional Gaussian distributions, thus the N-dimensional mean and NxN covariance matrix. Because of sufficient effectiveness in modeling the components are restricted to have diagonal covariances.

Once a speaker model is learned, there are two ways of authenticating a particular speaker [47]. In the first approach, a threshold is selected for the probability $P(\lambda^t|O)$, where $\lambda$ denotes the model parameters of the target speaker, and $O$ is the observed signal. In the second approach is a threshold selected to the ratio of probability of the genuine speaker and probability of the impostor model, which is trained on all speakers in the system except the genuine speaker class. This implies that for every person in the system, two models will be trained. In the case of sufficiently many subjects, a single and generic impostor model can be employed. The implementation of the GMM approach is done by using the Hidden Markov Models Toolkit (HTK) [66]. The GMM was built as an HMM with just one state, as shown in Fig. 11.

<figure><img src="image_85.png" /><figcaption>Fig. 11. One-state HMM</figcaption></figure>

## VI. CONTACT-BASED BIOMETRICS

The concept of Contact-Based Biometrics derives from the simple observation that every person handles the objects of the surrounding environment quite differently. For example, the action of picking up a glass or holding a knife depends on the physiological characteristics of each person and the way that this person is used to manipulate objects. Contact-Based Biometrics can also been thought as a specialized part of Activity-Related Biometrics for every activity which involves an object.

In the context of this project we intend to investigate the feasibility of such biometric features in user authentication applications. The proposed approach exploits methods from different scientific fields, such as collision detection and pattern classification, to solve the problem of authentication. The major parts of the final implementation scheme are the setup of a 3D virtual environment, the registration of the user and the objects in this environment, the extraction of collision features during an action between the user and an object and the classification procedure.

### A. 3D Environment Setup and Model Registration

Collision detection algorithms can only be used in a 3D environment with full knowledge of the geometry of each object. The virtual environment of the presented pilot requires only the 3D representation of the user's hand and each object that is of interest. The user's hand is modeled as a set of five fingers connected to the palm, which is modeled as a simple rectangle (Figure 12(a)). Each finger has four degrees of freedom (DOF) and consists of three phalanxes which are modeled as simple capsules.

For the registration of the hand we used the CyberGlove® (http://www.immersion.com/3d/). The CyberGlove® (Figure 12(b)) provides the angles between the phalanxes of the hand, so it is possible to reconstruct the 3D representation of the hand. Note, that the virtual representation of the hand is not perfectly accurate because the size of the fingers and the phalanxes are not known. In order to satisfy the requirements of a realistic pilot we cannot make any assumptions or measures on the user so this inaccuracy is considered as noise.

<figure><img src="image_86.png" /><figcaption>Fig. 12. (a) The 3D representation of the hand. (b) The CyberGlove®.</figcaption></figure>

The objects of the environment can be registered using computer vision techniques for object tracking. However, it is not absolutely necessary to have an accurate representation of the object in the virtual environment. In particular for rigid objects, which are typically encountered in an office environment, we can simplify the geometry of the object using a priori information. This simplification is possible as the real shape of each object is mostly related to the specific action that is used and not to the way it is handled. For example, a glass can be represented by a cylinder since the user grabs only the outer surface of the glass.

### B. Collision Feature Extraction

The classification features consist of any information that can be acquired by employing state-of-the-art algorithms for proximity queries. These include penetration depth [24], closest distance [26], [32], contact points etc. The literature in the field is vast and there are numerous algorithms to accurately perform queries in real-time. The interested reader is directed to [17], [33], [57], [58], [60] for further details. For our purposes we used the algorithms for rigid convex objects [59], [60] of the software package SOLID (http://www.dtecta.com/).

Proximity queries are performed between the object and every finger of the user's hand. Each query refers to either of two states, collision or no collision between the two virtual shapes. For example penetration depth can only be calculated when two objects intersect since it is always zero otherwise. However, in a user-object interaction scheme it is necessary to continuously produce discriminant feature samples. Thus, any proximity query as a single feature would not provide adequate information to a classifier.
In the proposed method we employ the combination of the penetration depth and the closest distance, depending on the collision state, to define the feature space. The penetration depth and the closest distance are usually described as 3D vectors in virtual simulations. However, in our case we prefer to describe them as the pair of points ($p_{finger}$, $p_{object}$), one on the finger and the other one on the object, that define the respective vector $v = p_{finger} - p_{object}$. This way the 3D position of each finger affects the values of the feature vector, while $v$ would only describe the relative direction which is most probable to be similar even for different fingers. Let $pd_k$ and $cd_k$ denote the points of the penetration depth and the closest distance respectively for either the finger or the object $k$. The feature sample $f_e(i, O)$ for the finger $e$ and the object $O$ on the $i$-th frame is

$$f_e(i, O) = \begin{cases} (pd_e, pd_O), & e \text{ and } O \text{ collide} \\ (cd_e, cd_O), & e \text{ and } O \text{ do not collide} \end{cases}$$

The final feature vector $F = \cup_e \{f_e\}$ is formed using the collision information from all the five fingers and is a 30-dimensional vector.

## VII. CONTINUOUS FACE AUTHENTICATION

With the rapid increase of video surveillance equipment and webcam usage, it became necessary to develop robust recognition algorithms that are able to recognize people using video sequences, which not only provide abundant data for pixel-based techniques, but also record the temporal information. This project inspects two complementary approaches to face biometrics from continuous video, detailed in this section and the next.

The processing for the face and facial motion analysis modules starts with detecting the face. We use the OpenCV face detection module that relies on the adaboosted cascade of Haar features, i.e. the Viola-Jones algorithm for this purpose [61]. The face camera is positioned so that the face image roughly covers a 150 × 150 pixel area, which changes greatly as the subject moves around.

One of the assumptions we have in the face authentication module is that the statistical models that incorporate general face information are trained offline, prior to the actual experimental setup. This means that the bulk of the training database should consist of external data. For this purpose, we have used the world model of 300 face images that accompany the BANCA database [3], enriched with one gallery image per enrolled person. This is a realistic assumption, and since the gallery is acquired with different illumination conditions as well, the actual experimental environment presents a formidable challenge, with completely uncontrolled illumination under ordinary (and poor) office lighting.

For continuous face authentication, we take a straightforward approach. The detected faces are cropped, rescaled to a fixed size, projected to a previously computed subspace, and compared to the templates residing in the gallery. For controlling the illumination, we apply a image enhancement procedure proposed by Savvides and Kumar [54]. In this procedure, the pixel intensities are mapped to a logarithmic range, which nonlinearly allocates a broader range to dark intensity levels, increasing the visibility.

The subspace is found by applying the Karhunen-Loeve transform to the enhanced training set. The matching of a claim with a gallery image can be achieved by thresholding a Mahalanobis-cosine distance between projected vectors. If the subspace-projected query is denoted by $u = [u_1 u_2 \dots u_p]'$ and the subspace-projected gallery template is denoted by $v = [v_1 v_2 \dots v_p]'$, denote their corresponding vectors in the Mahalanobis space with unit variance along each dimension as:

$$m_i = \frac{u_i}{\sigma_i} \qquad (21)$$

and

$$n_i = \frac{v_i}{\sigma_i} \qquad (22)$$

where $\sigma_i$ is the standard deviation for the $i$-th dimension of the p-dimensional eigenspace. Then the Mahalanobis cosine distance is given by [49]:

$$d_{MC}(\mathbf{u}, \mathbf{v}) = \cos(\theta_{mn}) = \frac{mn}{|m||n|} \qquad (23)$$

### A. Adaptive Cropping

The preprocessing of the external database is not replicated in our acquisition conditions. This means that the eigenspace projection that models the variation in aligned face images is not necessarily the ideal projection for a given query image. To remedy this situation, we apply an adaptive cropping algorithm that fine-tunes the face detection result

so as to minimize the reprojection error $e$. Assume the eigenspace is denoted with $[\lambda, e]$, where $\lambda$ stands for the sorted eigenvalues and $e$ are the corresponding eigenvectors. The projection of query $x$ to the eigenspace is:

$$\mathbf{u}_{(p \times 1)} = e'_{(p \times d)} (\mathbf{x}_{(d \times 1)} - \mu_{(d \times 1)}), \qquad (24)$$

where $\mu$ denotes the data mean, and the subscripts indicate dimensionality. The reprojection error is given by:

$$e = \|\mathbf{x}_{(d \times 1)} - \mathbf{e}_{(d \times p)}\mathbf{u}_{(p \times 1)} + \mu_{(d \times 1)}\| \qquad (25)$$

The pseudocode of the algorithm is given in Fig. 13. Fig. 14 shows the cumulative effect of illumination correction and adaptive cropping on a sample frame.

```
algorithm Adaptive_Cropping(faceImg)
    cropping ← [0,0,0,0]
    oldError ← Infinity
    found = False
    cropDir = 1
    while NOT found
        oldError ← newError
        /*Crop the image in one of four directions*/
        cropping(cropDir) ← cropping(cropDir) + 1
        croppedImg ← crop(faceImg, cropping)
        /*Scale to fixed size*/
        scaledImg ← scale(croppedImg)
        /*Illumination normalization*/
        normalizedImg ← logTransform(scaledImg)
        /*Projection*/
        projImg ← eigenVectors*(normalizedImg-meanImg)
        /*Re-projection into the original space*/
        reprojImg ← (eigenVectors*projImg)+meanImg
        /*Update the error*/
        reprojError = norm(reprojImg - normalizedImg)
        if reprojError < oldError
            newError ← reprojError
        else
            end
        /*Reverse the cropping*/
        cropping(cropDir) ← cropping(cropDir) - 1
        end
        /*Update the next cropping direction*/
        cropDirection ← mod(cropDirection+1,4)
        found ← (updated in the last cycle of four directions)
    end
    return cropping
end
```

<figure><img src="image_87.png" /><figcaption>Fig. 14. a) The original captured frame. b) The illumination compensated image. c) The result of the adaptive cropping</figcaption></figure>

### B. Probabilistic Matching

The activity model necessitates a short video sequence to be recorded for training purposes. This allows us to use a larger training set for the face authentication module as well. For each subject in the gallery, one sequence of recordings is processed with the face detection and adaptive
cropping modules. The ensuing cropped images are projected to the Mahalanobis space, and modeled with a mixture distribution.

The general expression for a *mixture model* is written as

$$p(x) = \sum_{j=1}^{J} p(x|G_j)P(G_j) \qquad (26)$$

where $G_j$ stand for the components, $P(G_j)$ is the prior probability, and $p(x|G_j)$ is the probability that the data point is generated by component j. In a *mixture of Gaussians* (MoG), the components in Eq. 26 are Gaussian distributions:

$$p(x|G_j) \sim \mathcal{N}(\mu_j, \Sigma_j) \qquad (27)$$

Typically, the covariance expression is restricted in MoG models to control the complexity of the model, as a diagonal covariance scales linearly with dimensionality, whereas a full covariance scales quadratically. In this work we use the factor analysis approach to model the covariance, where the high dimensional data $x$ are assumed to be generated in a low-dimensional manifold, represented by latent variables $z$. The *factor space* spanned by the latent variables is similar to the principal space in the PCA method, and the relationship is characterized by a *factor loading matrix* $\Lambda$, and independent Gaussian noise $\epsilon$:

$$x - \mu_j = \Lambda_j z + \epsilon_j \qquad (28)$$

The covariance matrix in the $d$-dimensional space is then represented by $\Sigma_j = \Lambda_j \Lambda_j^T + \Psi$, where $\Psi$ is a diagonal matrix and $\epsilon_j \sim \mathcal{N}(0, \Psi)$ is the Gaussian noise. We obtain a *mixture of factor analysers* (MoFA) by replacing the Gaussian distribution in Eq. 26 with its FA formulation.

To learn the distribution of training faces of a single class, we use the incremental mixtures of factor analysers (IMoFA) algorithm, which automatically determines the number of components in the mixture, and tunes the latent variable dimensionality for each mixture component separately. For more details, the reader is referred to [53]. The ensuing model for the subject is $(\Lambda_j, \mu_j, \epsilon_j, \pi_j)$, with $\pi_j$ being the component prior, and $j$ is the index for mixture components. The authentication of a normalized and projected image $x_t$ is effected by checking a pre-fixed threshold:

$$p(x_t|G_j) \ge \tau \qquad (29)$$

At any point in time, the continuous face authentication module evaluates the most recent frame, and returns a Boolean decision. The threshold $\tau$ depends on the Mahalanobis space dimensionality, and scales approximately linearly with it. For a 300-dimensional Mahalanobis space, we have used a threshold of -400 for the log-likelihood, a higher value will reject more frames and ensure a more secure system, whereas a lower value will favour user convenience over security. It is also possible to base the decision on all the frames up to time $t$, by using any classifier combination method.

## VIII. BEHAVIORAL FACE BIOMETRICS

The previous section dealt with the static facial appearance, ignoring the behavioral cues that can be potentially useful for discriminating identities. Recently there is much attention to biometric systems that exploit temporal information in videos, and most of the proposed approaches involve a heterogeneous mixture of techniques. These approaches can roughly be classified into the following categories:

* Holistic approach: This family of techniques analyze the head as a whole, by extracting the head displacements or the pose evolution. In [30] Li et al. propose a model-based approach for dynamic object verification and identification using videos. In 2002, Li and Chellappa were the first to develop a generic approach for simultaneous object tracking and verification in video data, using posterior probability density estimation through sequential Monte Carlo methods [29]. Huang and Trivedi in [19] describe a multi-camera system for intelligent rooms, combining PCA based subspace feature analysis with Hidden Markov Models (HMM). Liu and Cheng proposed a recognition system based on adaptive HMMs [35]. They first compute low-dimensional feature vectors from the individual video frames by applying a Principal Component Analysis (PCA); next they model the statistics of the sequences and the temporal dynamics using a HMM for each subject. In [1] Aggarwal et al. have modeled the moving face as a linear dynamical system using an autoregressive and moving average (ARMA) model. The parameters of the ARMA model are estimated for the entire database using the closed form solution. Recently, Lee et al. developed a unified

framework for tracking and recognition, based on the concept of appearance manifold [28]. In this approach, the tracking and recognition components are tightly coupled: they share the same appearance model.

* Feature based approach: The second group of methods exploits the individual facial features, like the eyes, nose, mouth and eyebrows. One of the first attempts to exploit facial motion for identifying people is presented by Chen et al. in [12]. In their work, they propose to use the optical flow extracted from the motion of the face for creating a feature vector used for identification.

* Hybrid approach: These techniques use both holistic and local features. Colmenarez et al. in [14] have proposed a Bayesian framework which combines face recognition and facial expression recognition to improve results; it finds the face model and expression that maximizes the likelihood of the test image.

This section proposes a new person recognition system based on temporal features from facial video. As in the previous section the face area is first detected in each frame of the video. The registration, or the alignment problem, however, has different criteria to satisfy. Since we will track the features, the alignment is not absolute, but relative to the previous frame, minimizing a mean square error measure. For aligned faces, the optical flow is calculated from consecutive frames, and used as feature vectors for person recognition.

Once the faces are detected with the Viola-Jones method, a representation called the "integral image" is created using Haar-like features.

The learning algorithm is based on AdaBoost, which can efficiently select a small number of critical visual features from a larger set, thus increasing performance considerably.

Next the resulting image is cropped as shown in Fig. 15 based on anthropological measures to limit the image to facial features that exhibit more motion.

<figure><img src="image_88.png" /><figcaption>Fig. 15. Detected and cropped face images in two frames.</figcaption></figure>

Face alignment was required due to the simple fact that we wanted to focus our attention on motion of local features from the face such as the lips and the eyes. If this step is not performed before feature extraction, global motion of the head significantly affects the results. Alignment of the faces detected in two different frames was carried out by minimizing the mean square error of the integral image difference:

$$\arg\min \frac{1}{M \times N} \sum_{i=1}^{M} \sum_{j=1}^{N} (I_1(i, j) - I_2(i, j))^2 \qquad (30)$$

where, ... Fig. 16 shows two facial images found in consecutive frames aligned with this method.

<figure><img src="image_89.png" /><figcaption>Fig. 16. Two facial images aligned and superimposed.</figcaption></figure>

We have decided to use optical flow vectors for person recognition, calculated by the Lucas-Kanade technique [36], which uses the spatial intensity gradient of the images to guide the search for matching locations, thus requiring much less comparisons with respect to algorithms that use a predefined search pattern or search exhaustively. Then block means are
taken to reduce the size of the feature vector to standard dimensionality of 200. Fig. 17 shows the optical flow computed from the images aligned in Fig. 16.

<figure><img src="image_90.png" /><figcaption>Fig. 17. Optical flow from consecutive frames.</figcaption></figure>

## IX. CONTINUOUS ACTIVITY - RELATED BIOMETRIC AUTHENTICATION

Among the project's prominent objectives is to investigate the effectiveness and applicability of activity - related biometric technologies. Activity - related biometrics is a novel and innovative concept in biometric user authentication and refers to biometric signatures extracted by analyzing the response of the user to specific stimuli, while performing predefined but natural work - related activities. The novelty of the approach lies in the employment of dynamic features extracted by the moving human model as biometric signal, as well as in the fact that the biometric measurements will correspond to the user's response to specific events, being, however, fully unobtrusive and fully integrated in the user's workspace. The activity - related biometric authentication module evaluates the fundamental assumption that each user's dynamic behavioral profile contains unique intrinsic characteristics that can be used for authentication. Furthermore, a reliable implementation of an activity - related biometric authentication system is ideal for continuous user authentication, thus alleviating the main limitation of some successful state-of-the-art approaches (fingerprint, iris etc.) which cannot be recovered once forged.

In the following, the modules and methods that were implemented to perform activity - related authentication will be described. In addition to that, the pilot setup and the experimental procedures followed in order to evaluate activity - related biometrics will be presented.

### A. Activity Detection and Recognition Module

As stated above, the user's dynamic profile extraction is based on the response to specific environment - generated stimuli. Any human behavior is associated to some action or activity. The aim of stimuli generation is to trigger the execution of specific actions by the user, upon which his behavioral profile can be then calculated. It is therefore clear that the extraction of the activity - related features must be preceded by an action detection, segmentation and recognition procedure. This goal is achieved by means of a multimodal approach that uses the output of the sound event recognition Module, the Object Occlusion Tracking Module and the body motion tracking module along with a Coupled Hidden Markov Model formulation in order to detect the generation of the stimuli and segment the user's response (action). The segmentation output of the Activity Recognition Module can be then fed to the Activity - Related Biometric Authentication Module. Fig. 18 illustrates the above inter - module relationships.

Numerous relevant approaches for activity recognition have been reported in the literature using object manipulation context information [43], [46], [64] and/or object trajectory information in the given scene [5], [31]. Sound event detection has also been previously employed to assist inference of ongoing activities [55], [63].

The proposed method for Activity Recognition is based on the detection of three different kinds of Scene Events occurring in the scene: Sound Events (e.g. Phone Ringing), detected by the sound event recognition Module, Proximity Events (e.g. "Hand close to Glass"), detected by the

<figure><img src="image_91.png" /><figcaption>Fig. 18. Module cooperation for Activity Recognition</figcaption></figure>

Human Body Tracking Module along with predefined knowledge of the object positions on the controlled workspace and Object Occlusion Events detected by the respective tracker. An Object Occlusion Event is emitted when some object in the scene is missing from its "normal" position.

In order to achieve action recognition, a two - stream Coupled HMM is associated to every action class and trained on two sets of discrete observation symbols (one for each stream) extracted by the primitive events described above (i.e. second layer events). The first set of second layer symbols is a subset of the Sound Event set that can be associated to a particular action. For example, the Phone Conversation Coupled HMM only handles relative sound events (Ringing, Speech, Silence etc.) and disregards the rest (e.g. Writing sound). The observation symbols of the second stream are formed as meaningful (for the particular action class) combinations of the Object Occlusion and Proximity Events of the first layer. For instance, the state "Phone receiver missing" AND "Left Hand close to Head" forms a single second layer event that is used as observation symbol of the second stream of the Phone Conversation CHMM to represent the state of "talking on the phone".

At every timestamp of some activity sequence, first and second layer events are detected and form N double - stream discrete observation sequences, where N the number of actions to be recognized and segmented. Each CHMM uses an overlapping sliding window that goes through its own observation sequence. The size of the sliding windows and the size of overlapping are experimentally defined. The CHMM of each action is trained on manually annotated sequences and a probability threshold is defined, above which the respective action is recognized and a portion in the size of the sliding window is segmented and fed to the Activity Biometrics Module. Fig. 20 graphically depicts the Activity Recognition Module. The reason for performing the mapping from first layer events to second layer events is to impose a smaller size on the final observation sets and process the three initial streams of events into only two - stream Coupled HMMs, which results in making training more efficient.

### B. Coupled Hidden Markov Models

The need of a Coupled Hidden Markov Model formulation is justified by the fact that Scene Event detection is often erroneous, producing many false alarms, wrong inferences and multiple occlusions over time. Consequently, detected event symbols would better be thought of as the probabilistic output of some underlying process, rather than as deterministic events. Furthermore, Coupled HMMs offer a robust mathematical background for integrating multimodal observations and fusing different but correlated processes (sound events + human activity based events).

Our Coupled HMM implementation is based on the formulation presented by Ara V. Nefian et al. [44], where the hidden nodes of each stream interact and at the same time have their own observations (Fig. 21). The elements of the CHMM (Initial, Transition and Observation probabilities) are described as:

$$ \pi(i) = \prod_{s} \pi^{s}(i_{s}) = \prod_{s} P(q_{1}^{s} = i_{s}) \quad (31) $$

$$ b_{t}(i) = \prod_{s} b_{t}^{s}(i_{s}) = \prod_{s} P(O_{t}^{s} | q_{t}^{s} = i_{s}) \quad (32) $$

$$ \alpha(i|\mathbf{j}) = \prod_{s} \alpha^{s}(i_{s}|\mathbf{j}) = \prod_{s} P(q_{t}^{s} = i_{s}|\mathbf{q}_{t-1} = \mathbf{j}) \quad (33) $$

The CHMMs are trained using an EM algorithm, based on the calculation of the forward and backward variables, $a_t(i) = P(O_1, ..., O_t, q_t = i)$ and $\beta_t(i) = P(O_{t+1}, ..., O_T, q_t = i)$ respectively, where T the length of the observation sequence:
<figure><img src="image_92.png" /><figcaption>Fig. 19. Coupled Hidden Markov Model structure. Squares denote the hidden nodes of each interacting process and circles the associated observable outputs</figcaption></figure>

$$a_1(\mathbf{i}) = \pi(\mathbf{i})b_1(\mathbf{i}) \quad (34)$$

$$a_t(\mathbf{i}) = b_t(\mathbf{i}) \sum_j \alpha(\mathbf{i}|\mathbf{j})a_{t-1}(\mathbf{j}) \quad (35)$$

for $t = 2, 3, ..., T$

$$\beta_1(\mathbf{i}) = 1 \quad (36)$$

$$\beta_t(\mathbf{j}) = \sum_i b_{t+1}(\mathbf{i})\alpha(\mathbf{i}|\mathbf{j})\beta_{t+1}(\mathbf{i}) \quad (37)$$

for $t = T, T-1, ..., 2$

The probability of the $r_{th}$ observation sequence $O_r$ of length $T_r$ is computed as $a_{r,T}(N_1, N_2, ..., N_S) = \beta_{r,1}(1, ..., 1)$

The scaled version of the forward and backward variables ($\hat{\alpha}_s, \hat{\beta}_s$) [48] obtained in the E step are used to re-estimate the transition and observation parameters as follows:

$$\tilde{\alpha}^s(i|\mathbf{j}) = \frac{\sum_r \sum_{i,s,t} \hat{a}_{r,t}(j) \alpha(i|\mathbf{j}) b_{r,t+1}(i) \hat{\beta}_{r,t+1}(i)}{\sum_r \sum_t \hat{a}_{r,t}(j) \hat{\beta}_{r,t}(j) \frac{1}{c_t}} \quad (38)$$

$$\tilde{b}_i^s(k) = \frac{\sum_r \sum_{i,s,t} \hat{a}_{r,t} \alpha_{i,s,t} = i \sum_{t,s,t} O_t^s = k \hat{a}_t(i) \hat{\beta}_t(i) \frac{1}{c_t}}{\sum_r \sum_t \sum_{i,s,t} \hat{a}_t(i) \hat{\beta}_t(i) \frac{1}{c_t}} \quad (39)$$

where $c_t$ the scaling coefficient for time t.

The number of states has been defined taking into consideration the inherent structure of each action. For instance, the Phone Conversation action consists of the “natural” states “Ringing” - “Reach Phone” - “Bring close to Head” - “Speech” - “Hang Up”, upon which various second layer events can be defined.

### C. Activity - Related Biometric Authentication Module

The aim of the activity - related biometric authentication module is to receive the dynamics of the human posture produced by the body motion tracking module on some user action segmented by the Activity Recognition Module and output some authentication results (Fig 18). Within this project we would like to evaluate the assumption that behavior can be employed as biometric signal as well as the hypothesis that our belief measure on the user's identity increases with time. Furthermore, various work - related motions should be tested with regard to their discriminative power.

Related work includes several model - based and feature - based methods for human gait identification and authentication [62], [13]. Key stroke dynamics have been also employed for activity - related person authentication [42]. To our knowledge, activity - related person authentication based on environment generated stimuli and work - related activities is a completely novel concept and has never been implemented before.

The output of this module for a particular action could either be a strict authentication result (Accepted/Rejected) or a belief measure that can be integrated with future partial inferences of the same modality and/or inferences of other modalities to converge in a final authentication result at later time stamps (Continuous Authentication). The latter approach seems more promising, as the user's “natural” behavior can be more reliably confirmed on multiple action instances. In general, a user's way

<figure><img src="image_93.png" /><figcaption>Fig. 20. a) Event Extraction b) Sliding Window for CHMM</figcaption></figure>

of execution of some motion can diverge from its usual dynamics on single instances depending on various factors (psychological condition, unusual environmental conditions etc.). Despite that, it can be assumed that over longer periods of times where multiple instances of many actions take place, the user's identity could be reliably inferred.

The Activity - Related Biometric Authentication Module assumes a mapping of a user's behavior to his identity, therefore tools, methods and features that have been used for action and gesture recognition can be applied. In this implementation the body joint angles and position of the central point of the human model (III) and their derivatives are used as features for modeling the user's natural way of executing some action, since those features can powerfully represent the human model posture and its dynamics. Principal Component Analysis for each action class is used to reduce the dimensionality of the feature vector.

For biometric authentication Hidden Markov Models with Multivariate Gaussian outputs are used to capture the spatio - temporal dynamics of the human behavior. Standard HMM classification is performed by assigning one model to every individual enrolled in the authentication system. Given some extracted observation sequence $O_{1:T}$ of length T associated to a segmented action, and the set of HMMs $\lambda_i, i = 1, ..., N$ where N the number of enrolled users, the probability $P(O|\lambda_i)$ is calculated for all HMMs. By assigning an authentication threshold to each user's HMM, direct authentication results based on single actions can be obtained. A more promising option is propagating all the above probabilities to an integration module that emits authentication results on longer periods of activity. Fig. 21 graphically represents the Activity - Related Biometric Authentication Module.

## X. INTEGRATION OF DECISIONS

A typical authentication system presents a DET (Detection Error Trade-off) curve which enables a system to select a point on the curve to trade off between security and ease of use of a system. However, a continuous authentication system needs to traverse this DET curve based on the current situation. If the system is confident based on past inferences, temporary drops in the probability of the target class should not cause the rejection of the user. However if there is an elongated period of diffidence about the authenticity of the target person, then the system should be able to reject the person eventually.

The second problem is the integration of the inferences from the different modalities. Each mode produces different inferences with a
<figure><img src="image_94.png" /><figcaption>Fig. 21. Activity - related biometric feature extraction and authentication</figcaption></figure>

different probability and these inferences are available at different points in time. There is the additional complication of assessing the reliability (and consequently the relative weight) of each modality. This problem is termed in the literature as “Holistic Fusion”.

Among previous work done on holistic fusion, the most significant are Zhang et al. [67] and Kittler et al. [25]. Zhang et al. suggested a two state Hidden Markov Model, where the two states are “safe” and “attacked”. A decay factor was proposed, which exponentially weighted over the previous observations, as well as weighted sums to integrate over modalities, where the weights were the assessed reliabilities of the modalities. The area under the Receiver Operating Characteristics (ROC) curve for each modality is used to quantify reliability. The approach we will present now is similar in some respects to this method, but it does not use HMMs.

Let $\lambda_{\Omega}$ be the model of 'target' person, the person whom we want to authenticate. Let $\lambda_{m}$ be one among the M impostor models. Let $O_{t}^{n}$ be the $t^{th}$ observation in time, among $\Gamma$ observations from the $n^{th}$ modality among N modalities. Each module produces the likelihood of $\lambda_{\Omega}$ given $O_{t}^{n}$, i.e. $p(O_{t}^{n}|\lambda_{\Omega})$. Since the likelihoods from different modalities have the inherent problem of being in different scales, it becomes difficult to find suitable weights. So the posterior is calculated as follows

$$P(\lambda_{\Omega}|O_{t}^{n}) = \frac{p(O_{t}^{n}|\lambda_{\Omega}) * P(\lambda_{\Omega})}{p(O_{t}^{n})} \quad (40)$$

The next question is about calculating the prior $P(\lambda_{\Omega})$ and the observation probability $p(O_{t}^{n})$. The observation probability can be given by

$$p(O_{t}^{n}) = p(O_{t}^{n}|\lambda_{\Omega}) * P(\lambda_{\Omega}) + \sum_{m=1}^{M} p(O_{t}^{n}|\lambda_{m}) * P(\lambda_{m}) \quad (41)$$

How to estimate $P(\lambda_{\Omega})$ is an interesting problem. This value is tunable and different points on the DET curve can be achieved by changing this value. Increasing this value makes the system more confident about the authenticity of the subject and thereby increases the false acceptance rate (FAR). Reducing this value increases the false rejection rate (FRR) while decreasing the FAR.

A continuous authentication system is typically used after the authenticity is verified by an independent system. The initial estimate of the prior, $P_{0}(\lambda_{\Omega})$, can be received from this entry system or taken to be an arbitrarily high value. The subsequent values of this prior are calculated as shown below

$$P_t(\lambda_{\Omega}) = \frac{\sum_{n=1}^{N} P_t(\lambda_{\Omega}|O_t^n) * P(O_{1:t}^n)}{\sum_{n=1}^{N} P(O_{1:t}^n)} \quad (42)$$

where

$$P(O_{1:t}^{n}) = \frac{\left( \frac{p(O_{i}^{n}|\lambda_{\Omega}) + \sum_{m=1}^{M} p(O_{i}^{n}|\lambda_{m})}{M+1} \right)}{\frac{1}{t} \sum_{i=1}^{t} \left( \frac{p(O_{i}^{n}|\lambda_{\Omega}) + \sum_{m=1}^{M} p(O_{i}^{n}|\lambda_{m})}{M+1} \right)} \quad (43)$$

Now with a time-varying estimate of prior available equation 40 can be combined with 41 and written as shown below.

$$P_t(\lambda_{\Omega}|O_t^n) = \frac{p(O_t^n|\lambda_{\Omega}) * P_{t-1}(\lambda_{\Omega})}{p(O_t^n|\lambda_{\Omega}) * P_{t-1}(\lambda_{\Omega}) + \sum_{m=1}^{M} p(O_t^n|\lambda_m) * P_{t-1}(\lambda_m)} \quad (44)$$

where $\forall m$

$$P_{t-1}(\lambda_{m}) = \frac{1 - P_{t-1}(\lambda_{\Omega})}{M} \quad (45)$$

The prior is updated at every calculation and the confidence of the system depends on all the different modalities. In a system such as the one described in the experiment, it may not be possible to get a new inference from each modality at each instance of time. So the latest inference from each modality is used for re-computing the estimate of the prior, $\lambda_{\Omega}$.

The strategy proposed builds a confidence value about the identity of a person. This confidence is in terms of the updated posterior probability. If the different modalities ascribe low confidence to the authenticity of the person, then the overall confidence drops down. But if the modalities provide high confidence to the authenticity, then the overall confidence in the person builds up. At some point, if one of the modalities ascribes low confidence to the authenticity of the target, then it is weighed by how probable the occurrence of such an observation is. So if an observation is not very probable in the model of the entire system, then a lower weight is given in the overall confidence calculation.

If at any point, the user is switched with an impostor, it will take some time for the system to bring down the confidence levels due to the high confidence levels initially built on the user, and the impostor is likely to be authenticated for some time. But the overall confidence will drop eventually, with a speed that depends on the confidence scores of each modality. Using a window-approach that takes into account the last k frames in assessing probabilities may be useful in providing a fast decrease under switched persons.

Further testing needs to be done in the case of impostor switching and hysteresis of the system under these circumstances.

# XI. EXPERIMENTAL RESULTS

## A. Continuous Face Authentication

The face authentication module is tested with the recordings of 11 individuals. The first session is used to construct the statistical models for each person. The remaining nine sessions are used for reporting the success of the algorithm. For 99 sessions, the face detection module locates faces 92.3 per cent of the total recording time, with a standard deviation equal to 7.4 per cent. This means that for a 1000 frame session, about 923 face images are processed for authentication. Some of these faces are false alarms, caused by the failure of the Viola-Jones face detector.

In general, the face detection module is robust enough to correctly localize faces during activities like phone conversations. This implies that for these frames, the cropped face area contains the hand and the phone itself. We have observed that the face authentication module frequently stays below authentication threshold for these cases. Fig. 22 shows the authentication result for a single session. The horizontal axis is the time, and the vertical axis is the likelihood value obtained by the class models. Each face is shown as a dot on this plot. We only report the likelihood from the genuine class and the best impostor claim for that frame. The threshold is selected as -400, and the shown sequence justifies this choice nicely. In fact the threshold is optimized on a separate set, but since it strictly depends on the subspace dimensionality, it produces uniformly good results across the test sessions, as shown by the low variance of the results. At the bottom of the figure, a coloured band indicates when faces are not detected in the video (with red), when they are detected but the true class authentication does not follow (with yellow) and correct authentications (with green). The parts with longer bands of yellow are the activities where the face is not isolated or completely frontal.

The complete testing data consists of 91250 frames, recorded from nine sessions per subject, and 11 subjects. For each frame, the best impostor access is selected by evaluating the remaining 10 models. We demonstrate the effect of selecting different thresholds in Fig. 23, where the false accept rate and the false rejection rate of the system are plotted for a range of threshold values. For the selected threshold of -400, the system has 0.3 per cent false acceptance rate and 30.1 false rejection rate. This means that for a video sequence with 1000 detected faces, roughly 3 frames would admit impostors, and 700 frames would indicate the presence of the true user. At this level, there is no interpretation of these results. In practice, a session of continuous authentication can
<figure><img src="image_95.png" /><figcaption>Fig. 22. The output of the continuous face authentication module during one session. (a) The likelihood of the genuine and best impostor claims. The band shows correct authentications (green), no authentication (yellow), and no detection (red) cases. (b) The likelihood ratio of the genuine class to the best impostor class for the same session.</figcaption></figure>

operate on a sliding window of frames, where the genuine and impostor
likelihoods are compared, and the system outputs a decision at every
time slot. Under these controlled conditions (i.e. difficult but similar
illumination conditions in training and test sessions), it is obvious that
the face modality provides very robust authentication.

<figure><img src="image_96.png" /><figcaption>Fig. 23. The receiver operator characteristic curve for a range of thresholds of authentication. The genuine class is evaluated against the best impostor model for each frame. The average values for 99 sessions are reported. The cross indicates the selected threshold for the operating point of the system.</figcaption></figure>

<figure><img src="image_97.png" /><figcaption>Fig. 24. DET curve for speaker verification module</figcaption></figure>

B. Speaker verification

For purposes of training and testing, approximately 20 seconds of
speech is recorded during each session in form of a telephone conversation
in addition to 40 seconds of speech in form of reading a paragraph

of written text. 15 subjects have contributed to the database, with 10
recording sessions per subject. The results reported in this section are
obtained by training with sessions one to five, and testing with the session
six and seven, for 10 subjects. We have evaluated GMMs with different
numbers of components.

Fig. 24 shows that the best results are achieved using 128 components for Gaussian mixture densities.

C. Contact-based Biometrics

The experimental setup includes one testing action and eight subjects. In particular, the right hand of each user and the glass of the office were registered in the virtual environment for the action notated as "grabbing the glass". For the classification we implemented standard techniques of pattern recognition. PCA was used to reduce the dimensionality of the feature space while neural networks were trained for the final classification. Each person performed the action 10 times which produced
1000 sample frames on average for each subject due to the high sampling frequency of the CyberGlove®. From these samples 70% were used to train the network and 30% for testing. Fig. 25 displays the final ROC curve of the FAR and FRR rates for the testing data of the eight subjects.

<figure><img src="image_98.png" /><figcaption>Fig. 25. ROC curve for the action "grabbing the glass" and eight subjects.</figcaption></figure>

The results show that collision features are comparable to other Activity-Related biometrics and therefore comprise a very interesting approach for user authentication.

## D. Body Motion Tracking

The body tracker was tested in the office pilot. Two webcam, one frontal and one lateral, recording at 9.5 fps provided the frames onto which the 3D articulated model was projected. 3D body part locations (head, shoulders, elbows and wrists) have been manually annotated in one subject sequence in order to test the tracker performance. The error is expressed as the mean distance between the annotated and the estimated joints. Comparative results between the APF with the common likelihood approach (comprising edges and foreground matching) and our proposal are shown in Fig. 26. In both cases we used the body model and the projection procedure explained in section III-C. Final mean error obtained by our approach for this sequence was 85 mm. Common likelihood evaluation makes the tracker vulnerable to track loss, leading to higher mean error. On the other hand, the divergence measure and the feature-based smoothing of the likelihood approximation make the tracker more robust under our experimental conditions.

<figure><img src="image_99.png" /><figcaption>Fig. 26. Comparative results using 3 layers and 200 particles per layer with the normal likelihood approximation and our proposal.</figcaption></figure>

We found out that some spurious contours due to clothing and objects caused our tracker to fail in its estimation. The aparent motion recorded in the images was very fast in some of the actions required for activity- based recognition. These aparent fast motions caused blurs in the image

and abrupt translation of body parts. Since the implemented annealing PF works with contours as most determinant feature, the algorithm was not able to track several of these fast motions. However, it was able to recover some poses after a tracking error. Similarly, we detected that some poses couldn't be retrieved due to self-occlusions caused by the lack of additional views. Therefore, for some of the actions and poses, the problem becomes ill-posed and, as a consequence, more information is needed.

After testing several sequences, it was found that for several non-fast motions good results can be obtained with 3 layers and between 100 and 200 particles per layer. However, a more exhaustive study with ground truth angles must be done under similar conditions in order to refine the likelihood approximation, the annealing parameters and the number of particles.

## E. Other Modules

The results of the Sound-based Event Detection module are illustrated in the respective section IV. Testing of the Activity Recognition module and the Behavioral Face Biometrics module remain as future work.

Preliminary results for the Activity-related biometric module reveal a potential of using work-related activities as biometric signals. Experimenting on 7 manually segmented sequences (5 for training and 2 for testing) of the action classes Writing and Phone Conversation, we found out that the true person receives good HMM log-likelihood ranking. Despite that, the need for more accurate and stable 3D Motion Tracking was obvious, as it is the case for most state of the art model-based techniques. Future work includes testing on larger sets and more action classes, with improved motion tracking data. A feature-based approach (direct feature extraction on segmented human blobs) will also be implemented.

# XII. CONCLUSIONS AND FUTURE DIRECTIONS

In this project we have evaluated several activity related biometric modalities for their relative success in continuously determining and verifying the identity of a user in a typical and non-obtrusive work environment scenario. Apart from more traditional face and speech based verification, facial actions and movement patterns were assessed for authentication. A pilot setup with different action scenarios is defined, and a large database is collected from 15 subjects. Each subject contributed 10 sessions, which are manually annotated by the project group for further evaluation.

The experimental evaluation of all the modalities is not achieved exhaustively, and their possible integration remains to be a future endeavor. The latter is partly due to the success of individual modalities on the restricted pilot setup, which suggests that under closely resembling training and testing conditions there will be no marked benefit under fusion scenarios. However, the results demonstrate that activity-based biometrics is a promising venue for further study.

# XIII. ACKNOWLEDGEMENTS

The authors thank Christophe D'Alessandro and the organization team of the eNTERFACE'08 for all their efforts. Albert Ali Salah is supported by the Dutch BRICKS/BSIK project, and a scientific mission grant from EU COST 2101 Action. Martin Lojka is supported by Ministry of education of Slovak Republic under research project VEGA 1/4054/07 and Slovak Research and Development Agency under research project APVV-0369-07. This work was supported in part by the EC under contract FP7-215372 ACTIBIO.

# REFERENCES

[1] G. Aggarwal, A. Chowdhury, and R. Chellappa. A system identification approach for video-based face recognition. Proc. Int. Conf. on Pattern Recognition, 4:176-178, 2004.

[2] M. Arulampalam, S. Maskell, N. Gordon, T. Clapp, D. Sci, T. Organ, and S. Adelaide. A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking. Signal Processing, IEEE Transactions on [see also Acoustics, Speech, and Signal Processing, IEEE Transactions on], 50(2):174-188, 2002.

[3] E. Bailly-Bailliere, S. Bengio, F. Bimbot, M. Hamouz, J. Kittler, J. Mariethoz, J. Matas, K. Messer, V. Popovici, F. Poree, et al. The BANCA Database and Evaluation Protocol. LNCS, pages 625-638, 2003.

[4] P. Balucha. Automatic speaker recognition with gaussian mixture models. Master's thesis, Technical University of Košice, 2006.
[5] F. Bashir, A. Khokhar, and D. Schonfeld. Object trajectory-based activity classification and recognition using hidden markov models. Image Processing, IEEE Transactions on, 16(7):1912-1919, July 2007.

[6] N. Boulgouris and Z. Chi. Gait Recognition Using Radon Transform and Linear Discriminant Analysis. IEEE Transactions ON Image Processing, 16(3):731, 2007.

[7] C. Bregler and J. Malik. Tracking People with Twists and Exponential Maps. IEEE Computer Society Conference On Computer Vision And Pattern Recognition, 1998.

[8] A. Bregman. Auditory scene analysis. MIT Press Cambridge, Mass, 1990.

[9] F. Caillette, A. Galata, and T. Howard. Real-Time 3-D Human Body Tracking using Variable Length Markov Models. British Machine Vision Conference, 1:469-478, 2005.

[10] J. Canny. A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 8(6):679-698, 1986.

[11] C. Canton-Ferrer, J. Casas, and M. Pardas. Exploiting Structural Hierarchy in Articulated Objects Towards Robust Motion Capture. Lecture Notes in Computer Science, pages 82-91, 2008.

[12] L. Chen, H. Liao, and J. Lin. Person identification using facial motion. Image Processing, 2001. Proceedings. 2001 International Conference on, 2, 2001.

[13] M.-H. Cheng, M.-F. Ho, and C.-L. Huang. Gait analysis for human identification through manifold learning and hmm. Pattern Recogn., 41(8):2541-2553, 2008.

[14] A. Colmenarez, B. Frey, and T. Huang. A Probabilistic Framework for Embedded Face and Facial Expression Recognition. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1:592-597, 1999.

[15] J. Deutscher, A. Blake, and I. Reid. Articulated body motion capture by annealed particle filtering. PROC IEEE Comput Soc Conf Comput Vision Pattern Recognit, 2:126-133, 2000.

[16] A. Doucet, S. Godsill, and C. Andrieu. On sequential Monte Carlo sampling methods for Bayesian filtering. Statistics and Computing, 10(3):197-208, July 2000.

[17] C. Ericson. Real-Time Collision Detection. Morgan Kaufmann, 2004.

[18] N. Gordon, D. Salmond, and A. Smith. Novel approach to nonlinear/non-Gaussian Bayesian state estimation. Radar and Signal Processing, IEE Proceedings F, 140(2):107-113, 1993.

[19] K. Huang and M. Trivedi. Streaming face recognition using multicamera video arrays. Proc. Int. Conf. on Pattern Recognition, 4:213-216, 2002.

[20] X. Huang, A. Acero, and H. Hon. Spoken Language Processing. Prentice Hall, 2001.

[21] M. Isard and A. Blake. CONDENSATION-Conditional density propagation for visual tracking. Int. Journal of Computer Vision, 29(1):5-28, 1998.

[22] A. Jain, S. Dass, and K. Nandakumar. Soft biometric traits for personal recognition systems. Lecture notes in computer science, pages 731-738.

[23] A. Kale, N. Cuntoor, and R. Chellappa. A framework for activity-specific human recognition. Proceedings of the International Conference on Acoustics, Speech and Signal Processing (Orlando, FL), 706, 2002.

[24] Y. J. Kim, M. C. Lin, and D. Manocha. Incremental penetration depth estimation between convex polytopes using dual-space expansion. IEEE Transactions on Visualization and Computer Graphics, 10(2):152-163, 2004.

[25] J. Kittler, M. Hatef, R. Duin, and J. Matas. On combining classifiers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20:226-239, 1998.

[26] E. Larsen, S. Gottschalk, M. Lin, and D. Manocha. Fast distance queries with rectangular swept sphere volumes. volume 4, pages 3719-3726, 2000.

[27] A. Laurentini. The visual hull concept for silhouette-based image understanding. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 16(2):150-162, 1994.

[28] K. Lee, J. Ho, M. Yang, and D. Kriegman. Visual tracking and recognition using probabilistic appearance manifolds. Computer Vision and Image Understanding, 99:303-331, 2005.

[29] B. Li and R. Chellappa. A generic approach to simultaneous tracking and verification in video. IEEE Transactions on Image Processing, 11:530-544, 2002.

[30] B. Li, R. Chellappa, Q. Zheng, and S. Der. Model-based temporal object verification using video. IEEE Transactions on Image Processing, 10(6):897-908, 2001.

[31] Z. Li, S. Wachsmuth, J. Fritsch, and G. Sagerer. View-adaptive manipulative action recognition for robot companions. Intelligent Robots and Systems, 2007. IROS 2007. IEEE/RSJ International Conference on, pages 1028-1033, 29 2007-Nov. 2 2007.

[32] M. C. Lin and J. F. Canny. A fast algorithm for incremental distance calculation. pages 1008-1014, 1991.

[33] M. C. Lin and S. Gottschalk. Collision detection between geometric models: A survey. In In Proc. of IMA Conference on Mathematics of Surfaces, pages 37-56, 1998.

[34] J. Liu and R. Chen. Sequential Monte Carlo methods for dynamical systems. Journal of the American Statistical Association, 93(5):1032-1044, 1998.

[35] X. Liu and T. Chen. Video-Based Face Recognition Using Adaptive Hidden Markov Models. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1, 2003.

[36] B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. DARPA Image Understanding Workshop, pages 121-130, 1981.

[37] J. McCormick and M. Isard. Partitioned Sampling, Articulated Objects, and Interface-Quality Hand Tracking. Lecture Notes in Computer Science, pages 3-19, 2000.

[38] B. Mak, J.-C. Junqua, and B. Reaves. A robust speech/non-speech detection algorithm using time and frequency-based features. Proc. ICASSP, 1:269-272, 1992.

[39] F. Matta and J. Dugelay. A behavioural approach to person recognition. Proceedings of IEEE International Conference on Multimedia and Expo (ICME2006), pages 9-12, 2006.

[40] I. Mikic. Human Body Model Acquisition and tracking using multi-camera voxel Data. PhD. Thesis, University of California, San Diego, 2003.

[41] J. Mitchelson and A. Hilton. Simultaneous pose estimation of multiple people using multiple-view cues with hierarchical sampling. In Proc. of BMVC, September, 2003.

[42] F. Monrose and A. Rubin. Authentication via keystroke dynamics. In CCS '97: Proceedings of the 4th ACM conference on Computer and communications security, pages 48-56, New York, NY, USA, 1997. ACM.

[43] D. J. Moorey, I. A. Essaz, and M. H. H. Iiiy. Objectspaces: Context management for human activity recognition. In Second International Conference on Audio- Vision-based Person Authentication, 1999.

[44] A. V. Nefian, L. Liang, X. Pi, X. Liu, and K. Murphy. Dynamic bayesian networks for audio-visual speech recognition. EURASIP J. Appl. Signal Process., 2002(1):1274-1288, 2002.

[45] E. Nemer, R. Goubran, and S. Mahmoud. Robust voice activity detection using higher-order statistics in the LPC residual domain. IEEE Transactions on Speech and Audio Processing, 9:217-231, 2001.

[46] D. Patterson, D. Fox, H. Kautz, and M. Philipou. Fine-grained activity recognition by aggregating abstract object usage. Wearable Computers, 2005. Proceedings. Ninth IEEE International Symposium on, pages 44-51, Oct. 2005.

[47] P. Psutka, L. Müller, J. Matoušek, and V. Radová. Mluvíme s počítačem česky. Academia, 2004.

[48] L. R. Rabiner. A tutorial on hidden markov models and selected applications in speech recognition. pages 267-296, 1990.

[49] N. Ramanathan, R. Chellappa, and A. Roy Chowdhury. Facial similarity across age, disguise, illumination and pose. Proc. Int. Conf. on Image Processing, 3, 2004.

[50] L. Raskin, E. Rivlin, and M. Rudzsky. Using Gaussian Process Annealing Particle Filter for 3D Human Tracking-Volume 2008, Article ID 592081, 13 pages. EURASIP Journal on Advances in Signal Processing, 2008.

[51] A. Raynolds. Speaker identification and verification using Gaussian mixture speaker models.

[52] P. Renevey and A. Drygajlo. Entropy Based Voice Activity Detection in Very Noisy Conditions. In Seventh European Conference on Speech Communication and Technology. ISCA, 2001.

[53] A. Salah and E. Alpaydn. Incremental mixtures of factor analyzers. Int. Conf. on Pattern Recognition, 1:276-279, 2004.

[54] M. Savvides and B. Vijaya Kumar. Illumination normalization using Logarithm transforms for face authentication. Lecture notes in computer science, pages 549-556.

[55] M. Stager, P. Lukowicz, and G. Troster. Implementation and evaluation of a low-power sound-based user activity recognition system. In ISWC '04: Proceedings of the Eighth International Conference on Image Processing, IEEE Transactions on Image Processing, 11(6):1507-1514, 2002.
*Symposium on Wearable Computers*, pages 138–141, Washington, DC, USA, 2004. IEEE Computer Society.

[56] C. Stauffer and W. Grimson. Learning Patterns of Activity Using Real-Time Tracking. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, pages 747–757, 2000.

[57] M. Teschner, S. Kimmerle, G. Zachmann, B. Heidelberger, L. Raghupathi, A. Fuhrmann, M.-P. Cani, F. Faure, N. Magnetat-Thalmann, and W. Strasser. Collision detection for deformable objects. In *Eurographics State-of-the-Art Report (EG-STAR)*, pages 119–139. Eurographics Association, 2004.

[58] F. Thomas and C. Torras. 3d collision detection: A survey. *Computers and Graphics*, 25:269–285, 2001.

[59] G. Van and D. Bergen. Efficient collision detection of complex deformable models using aabb trees. *J. Graphics Tools*, 2, 1997.

[60] G. van den Bergen. *Collision Detection in Interactive 3D Environments*. Morgan Kaufmann, 2003.

[61] P. Viola and M. Jones. Rapid Object Detection Using a Boosted Cascade of Simple Features. *IEEE Computer Society Conference On Computer Vision And Pattern Recognition*, 1, 2001.

[62] L. Wang, T. Tan, H. Ning, and W. Hu. Silhouette analysis-based gait recognition for human identification. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 25(12):1505–1518, 2003.

[63] J. A. Ward, P. Lukowicz, G. Troster, and T. E. Starner. Activity recognition of assembly tasks using body-worn microphones and accelerometers. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 28(10):1553–1567, 2006.

[64] J. Wu, A. Osuntogun, T. Choudhury, M. Philipose, and J. Rehg. A scalable approach to activity recognition based on object use. *Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on*, pages 1–8, Oct. 2007.

[65] L. Xu, J. Landabaso, and M. Pardas. Shadow Removal with Blob-Based Morphological Reconstruction for Error Correction. *Acoustics, Speech, and Signal Processing, 2005. Proceedings.(ICASSP'05).* IEEE International Conference on, 2, 2005.

[66] S. Young, G. Evermann, M. Gales, T. Hain, D. Kershaw, X. Liu, G. Moore, J. Odell, D. Ollason, D. Povey, V. Valtchev, and P. Woodland. *The HTK Book*. Cambridge University Engineering Department, 2006.

[67] S. Zhang, R. Janakiraman, T. Sim, and S. Kumar. Continuous Verification Using Multimodal Biometrics. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 29:687–700, 2007.

<figure><img src="image_100.png" /></figure>

Martin Lojka was born in Snina, Slovakia, in 1984. He received engineer's degree in 2007. He is currently a PhD student in department of Electronics & Multimedia Communications of Technical University of Košice under the supervision of Doc. Ing. Jozef Juhar, PhD. His research interests focus on algorithms of audio processing in embedded systems.

<figure><img src="image_101.png" /></figure>

Adolfo López was born in Barcelona in 1982. He received his BS in Telecommunications Engineering from Universitat Politècnica de Catalunya (UPC) in Barcelona in 2007. He is currently a PhD student in the Image and Video Processing Group of the UPC under the supervision of professor Josep Ramon Casas. His research interests include Markerless Body Pose and Motion Estimation, Gesture and Motion Recognition and Particle Filter based Visual Tracking.

<figure><img src="image_102.png" /></figure>

Serafeim Perdikis received his Diploma in Electrical and Computer Engineering from Aristotle University of Thessaloniki, Thessaloniki, Greece, in 2007. Currently, he is a PhD candidate in the Aristotle University of Thessaloniki and a research fellow with the Informatics and Telematics Institute, Centre for Research and Technology Hellas, Thessaloniki. His main research interests include activity recognition, gesture recognition, human - computer interaction, dynamic biometrics and signal processing. He is also a member of the Technical Chamber of Greece.

<figure><img src="image_103.png" /></figure>

Gopal Ananthakrishnan received his Master of Science (Engg) from the Indian Institute of Science, Bangalore, India, in 2007. Currently, he is a PhD candidate in the Royal Institute of Technology, Stockholm, Sweden. His main research interests include Auditory to Articulatory Inversion, Speech production, Analysis of Speech and Audio signals, Perceptual and Neuro-Physiological analysis of hearing, Modeling and Simulation of Perceptual properties of human hearing, Pattern Recognition, On-line Handwriting Recognition.

<figure><img src="image_104.png" /></figure>

Usman Saaed was born in Lahore, Pakistan in 1981. He received a BS in Computer System Engineering from GIK Institute (Topi, Pakistan) in 2004. After graduation he was associated with the Electrical Engineering Dept. of Comsats Institute (Lahore, Pakistan) as a research associate. In 2005, he joined the University of Nice-Sophia Antipolis (Sophia Antipolis, France) for a Master of Research in Image Processing. He is currently a PhD student in the Multimedia Communication department of Institut Eurecom (Sophia Antipolis, France) under the supervision of Prof. Jean-Luc Dugelay. His current research interests focus on facial analysis in video.

<figure><img src="image_105.png" /></figure>

Hamdi Dibeklioğlu was born in Denizli, Turkey in 1983. He received his B.Sc. degree from Yeditepe University Computer Engineering Department, in June 2006, and his M.Sc. degree from Boğaziçi University Computer Engineering Department, in July 2008. He is currently a research assistant and a Ph.D. student at Boğaziçi University Computer Engineering Department. His research interests include 3D face recognition, computer vision, pattern recognition and intelligent human-computer interfaces. He works with Professor

<figure><img src="image_106.png" /><figcaption>Lale Akarun on 3D Face Recognition.</figcaption></figure>

Albert Ali Salah received his PhD in 2007 from the Dept. of Computer Engineering of Boğaziçi University, with a dissertation on biologically inspired 3D face recognition. This work was supported by two FP6 networks of excellence: BIOSECURE on multimodal biometrics, and SIMILAR on human-computer interaction. His research areas are pattern recognition, biometrics, and multimodal information processing. He received the inaugural EBF Biometrics Research Award in 2006, and joined with the Signals and Images group at CWI, Amsterdam as a BRICKS scholar. Recent scientific assignments include program committee memberships for BIOID'08, biometrics track of ICPR'08, and ICB'09.
<figure><img src="image_107.png" /></figure>

Dr. Dimitrios Tzovaras is a Senior Researcher (Grade B) at the Informatics and Telematics Institute. He received the Diploma in Electrical Engineering and the Ph.D. in 2D and 3D Image Compression from the Aristotle University of Thessaloniki, Greece in 1992 and 1997, respectively. Prior to his current position, he was a senior researcher on the Information Processing Laboratory at the Electrical and Computer Engineering Department of the Aristotle University of Thessaloniki. His main research interests include information management, multimodal data fusion and knowledge management. His involvement with those research areas has led to the co-authoring of over thirty articles in refereed journals and more than eighty papers in international conferences. He has served as a regular reviewer for a number of international journals and conferences. Dr. Tzovaras is an Associate Editor of the Journal of Applied Signal Processing (JASP).

<figure><img src="image_108.png" /></figure>

Athanasios Vogiannou received his Diploma degree in electrical and computer engineering from Aristotle University of Thessaloniki, Thessaloniki, Greece, in 2007. Currently, he is a PhD candidate in the Aristotle University of Thessaloniki and a research fellow with the Informatics and Telematics Institute, Centre for Research and Technology Hellas, Thessaloniki. His main research interests include virtual reality, collision detection, physically based modeling, real time simulations and computer vision. He is also a member of the Technical Chamber of Greece.
# Tracking-dependent and interactive video projection

Matei Mancas (1), Donald Glowinski (2), Pierre Bretéché (3), Jonathan Demeyer (1), Thierry Ravet

(1), Gualtiero Volpe (2), Antonio Camurri (2), Paolo Coletta (2)

(1) FPMS, Mons, Belgium (2) Casa Paganini/InfoMus Lab, Genova, Italy

(3) Laseldi Lab, Montbéliard, France

## Abstract

Gestures' expressivity, as perceived by humans, may be related to the amount of attention they attract. In this paper, we present three experiments that quantify behavior saliency by the rarity of selected motion and gestural features in a given context. The first two ones deal with the current quantity of motion of a person's silhouette compared to a brief history of his quantity of motion values and with the current speed compared to a brief history of the person's speed. The third one focuses on the motion speed of a person compared to the motion speed of other persons around him. Considering both features (speed and quantity of motion) and contexts (space and time), we compute an attention index providing cues on the behavior novelty. This can be considered as a preliminary step to an expressive gesture analysis based on behavioral. In order to achieve accurate tracking, a fusion between color and IR camera streams is achieved. This fusion let us have a robust tracking system with respect to illumination and partial occlusions issues.

**Index Terms**—computational attention, saliency, rarity, data fusion, tracking, gestures.

## I. PRESENTATION OF THE PROJECT

### A. Introduction: towards a context-based gestural analysis

A lot of research effort has been devoted to robustly track humans in a scene and to analyze their gestures in order to individuate and characterize their behavior. Gestural analysis, often applies in situations where either the human on which the analysis is carried on is previously selected or the same kind of analysis is performed to all the subjects that can be distinguished in the scene. A recent field of research aims at investigating collective behaviors [1]. Still, the object of the analysis is already defined and the work mainly focuses on characterizing collective displacements. The possibility of dynamically selecting the person to carry analysis on or to adapt analysis to the current behavior of a person in a context-dependent way would open new directions for gesture research. Human beings naturally show the capacity to dedicate limited perceptual resources to what is of particular interest. However, computers capacity to exhibit a behavior worthy of attention remains very limited.

### B. Computational attention interest in Human-Computer Interfaces (HCI)

In many real situations, where people interact freely together, it can be difficult to select the participants that exhibit a behavior worthy of attention.

The design of (expressive) gestures interface can gain from a better understanding of individual- and context-dependent human behavior. It can ensure their usability in a more naturalistic environment. Automatic attention cues are also able to simplify information access in those complex-situations which leads, in the HCI domain at foster interaction, anticipating focus of attention as automatic zoom on the Region Of Interest (ROI).

### C. Work overview

The project consisted in two main steps:

* *A robust tracking system*

This system uses both color video cameras and infra-red cameras to be robust enough to illumination changes and partial occlusions. Infra-red (IR) camera is much less sensitive to light changes if those lights are not directly in the camera field of view (FOV). The color camera FOV is larger and it is able to keep on the tracking process if infra-red markers are occluded or out of the infra-red camera FOV.

* *Motion attention: human-like reactions*

Once participant tracking is robust enough to handle naturalistic scenarios, an automatic attention index can be computed which highlights which movements should be the most "interesting" for a human observer. Attention is computed both in a spatial and in a temporal context on several features: speed and quantity of motion.

In section II, after a hardware and software overview, we will describe the video data acquisition and processing (blob segmentation) for each one of the two video modalities. In section III, the fusion mechanism which leaded to a robust tracking system will be explained. Section IV deals with attention computation both in spatial and temporal context. Finally, we will conclude by a discussion in section V. The source code of this project and some video demos can be found on the eNTERFACE 2008 workshop website [2].
## II. SIGNAL ACQUISITION

### A. Material and system overview

We developed a setup that analyzes human behavior in a flexible environment with regard to illumination changes. Three cameras were used to capture video: two “Eneo VKC-1354” color analogical cameras with a 752*582 pixel resolution at 25 frames per second (fps) and one “Imaging Source DMK 31BF03” monochrome digital camera delivering a 1024*768 pixel resolution at 30 fps. They were equally placed on the side of a 3 x 3 meters area, at a height of 2.5 meters, downward looking above the participants and recording with constant shutter, manual gain and focus. The participants used each one a red hat (for color segmentation) and a halogen light which emits visible but also infra-red light (for IR segmentation) in all space directions. Figure 1 shows the setup configuration.

<figure><img src="image_109.png" /><figcaption>Fig. 1: Experimental setup</figcaption></figure>

As described on Figure 2, two computers were used to perform the IR video and color video stream acquisition and processing (IR-and Color-based blob detection and tracking). The third PC was used to achieve data fusion between the IR and color stream data and to further higher-level processing and rendering.

<figure><img src="image_110.png" /><figcaption>Fig. 2: System overview</figcaption></figure>

A non-invasive video-based approach was adopted based on the EyesWeb XMI free software platform. We were interested in automatically extracting the displacement of people moving in front of the camera and computing their motion features. Head detection and tracking solutions were privileged to fully exploit the reduced available space, obtain depth information and avoid occlusions due to the interactions between participants.

The EyesWeb XMI (www.eyesweb.org) is a free software platform [3]. It consists of two main components: a kernel and a graphical user interface (GUI). The GUI manages interaction with the user and provides all features needed to design patches. It allows hand fast development of custom interfaces for use in artistic performances and interactive multimedia installations. The kernel manages real-time data processing, synchronization of multimodal signals. It supports the integration of user-developed plugins; an SDK (Software Development Kit) is provided to simplify the creation of such plugins by means of Microsoft Visual C++. The user-developed plugins, together with the ones provided with EyesWeb are the building blocks that the end user can interconnect to create an application (patch).

### B. Blob Detection

The present system's analysis of human activity starts from foreground segmentation based on the analysis of the color and infra-red video streams. This analysis provides a binary mask of the spatial extension of the region of interest through time (blob detection).

* Color video stream

A skin color detection algorithm was used on the signal coming from the color camera. We developed a modified version of the Continuously Adaptive Mean Shift Algorithm (CamShift) which is itself an adaptation of the Mean Shift algorithm for object tracking. Our method consisted in manually selecting the color of interest (COI) which is converted into a HSV colorimetric system. The set of colored pixel is quantized in a one-dimensional histogram to create the COI's model. Furthermore, a bandwidth of acceptable Hue and Saturation values are defined to allow the tracker to compute the probability that any given pixel value corresponds to the selected color. In order to enhance the system robustness to illumination changes, the color model was updated in several areas of the scene which have different illumination. In that way the color model was resistant to moderate illumination changes as those present on our scenario visible on Figure 3 between the left-side and the right-side of the scene.
<figure><img src="image_111.png" /><figcaption>Fig. 3: Color-based red hat blob detection</figcaption></figure>

### • Infra-red video stream

The signal coming from the IR camera is not affected by illumination variations but might be artefacted by light reflections or some static infrared sources. We processed the video stream with a background subtraction to eliminate static elements. Then, we binarized the signal with an empirically-tested threshold value to extract the moving regions of interest (blobs). Figure 4 shows that the IR lights located on the top of the red hats are very clearly detected.

<figure><img src="image_112.png" /><figcaption>Fig. 4: Infra-red lights located on the top of the red hats detection</figcaption></figure>

### C. Single and Multi-Blob Tracking

Resulting from the pre-processing step (color or IR based), we obtained a binary image where white represents the foreground objects. The next step is to assign a label that identifies the different white blobs, and to track them. The tracking result can be seen on Figure 5.

<figure><img src="image_113.png" /><figcaption>Fig. 5: Multi-blob tracking: from left to right the labeling (identification) of (1) the three blobs of the IR video stream (2) the three blobs of the color video stream</figcaption></figure>

To achieve the image tracking, we defined an adjacency measure based on the n-connectivity of two pixels (Figure 6).

<figure><img src="image_114.png" /><figcaption>Fig. 6. From left to right, illustration of (1) a 4-connectivity: the four filled circles (pixels) are connected to the one of interest (the cross); we defined this case as adjacency 1. (2) 8-connectivity, defined as adjacency = 2; (3) 20-connectivity defined as adjacency = 3. This adjacency measure can be generalized to n-connectivity</figcaption></figure>

Basing on this definition, we have used the following algorithm for image segmentation and tracking:

### • Definitions

**Image segmentation** `image_sgm` (video frame `f`) returns the set of distinct connected components `cc(f)` such that each pixel of an item in `cc(f)` is at distance `x` ≥ threshold from any other pixel belonging to other items in `cc(f)`

**Valid region** `valid_reg` is user defined as the maximum Euclidean distance between two blobs' baricentres in two consecutive frames.

**Minimum difference** `min_diff` returns the following:

$$ \min\_diff = \alpha \times \text{dist}(b_1, b_2) + \beta \times |\text{area}(b_1) - \text{area}(b_2)| \quad (1) $$

where `dist` (`b1`, `b2`) is the Euclidean distance between the baricentres of blobs `b1` and `b2` and `α` and `β` are the weights of area and position, comprised between:

$$ A = [0,1] \text{ and } \beta = [0,1] $$

These values are manually decided according to the cam's FOV and the foreground objects to track (e.g. humans)

### • Procedure

**Initialization:** `t = 0`

cc(t=0) = image_sgm(frame(0));
store cc(0) and assign a new label to each item in cc(0)

**For** `t = 1, 2, ...`

cc(t) = image_sgm(frame(t)); i.e. the set of distinct blobs in current frame (frame(t));

store cc(t)
for each item x in cc(t)
find x in cc(t-1) which minimizes min_diff
if x is in valid_reg, then

{ 
assign y the same label as x
if x is recognized in N consecutive frames,
then the item is considered to be trackable (i.e.,
we can measure its velocity, direction, etc.)

else
the item is unstable (recognized but not tracked)

}

else
y is assigned a new distinct label
### III. DATA FUSION

#### A. A common reference

Prior to any fusion algorithm, we need to provide a common reference to the signals to be fused. As we worked with different cameras, their fields of view (FOV) were different. A robust transformation was necessary to match the position of a point computed on a frame from one video camera, to the position of the same point computed on a frame from another video camera. The projective transformation meets our need because the perspective is subject to change with respect to the camera focal distance. A correction of the radial distortion was not necessary as the distortion due to wide-angle lenses was not very important.

The projective transformation conserves the proportions between two sets of two points: a quadrilateral is projected onto another one. Since it is not a linear transformation, we used homogeneous coordinates to compute the transformation with a matrix product.

The transformation matrix $H_{ab}$ performing the projection of the points $P_a$ in image “a” to the points $P_b$ in image “b” can be written in homogeneous coordinates as follows (points are located on the same subjective plane):

$$P_a = \begin{pmatrix} x_a \\ y_a \\ 1 \end{pmatrix}, P'_b = \begin{pmatrix} x'_b \\ y'_b \\ w_b \end{pmatrix}, H_{ab} = \begin{bmatrix} h_{11} & h_{12} & h_{13} \\ h_{21} & h_{22} & h_{23} \\ h_{31} & h_{32} & h_{33} \end{bmatrix}$$

Then:

$$P'_{b} = H_{ab} \cdot P_{a} \quad \text{and} \quad P_{b} = P'_{b} / w'_{b} = \begin{pmatrix} x_{b} \\ y_{b} \\ 1 \end{pmatrix} \qquad (2)$$

In order to visually test the accuracy of our transformation, we developed EyesWeb blocks which perform projective image transformation according to an input matrix $H_{ab}$. We also developed a block which composes this projective matrix from the correspondence of four points in the original and the referent image.

<figure><img src="image_115.png" /><figcaption>Fig. 7. IR and color video stream registration process. From left to right: selection of the four corresponding points in a color and IR snapshot in Matlab, computation of the transformation matrix in EyesWeb, image warping in EyesWeb</figcaption></figure>

Figure 7 displays the entire transformation process. We first selected four points in a snapshot of the IR and color video streams using Matlab. Then, the coordinates of those points are used by EyesWeb to compute the transformation matrix. Finally, this matrix is used to warp the initial image: the superposition of the color and IR image show a good registration in the plane where the points in the two images were chosen. Figure 8 shows how the initial video stream is transformed into a new video stream according to a projection matrix $H_{ab}$.

<figure><img src="image_116.png" /><figcaption>Fig. 8. Left: initial video frames, right: real-time projective warping using OpenCV-based EyesWeb block</figcaption></figure>

Once the registration was visually validated, the matrix $H_{ab}$ was used to compute the projective transformation for the blob baricenters and not for the whole image. The use of a newly implemented EyesWeb block which performs matrix multiplication greatly reduced the computational cost: this solution avoided the computation of the projective transformation for all pixels in each frame of the video stream. Figure 9 shows the rendering of the final video stream with the superposition of IR (green markers) and color video tracking (red markers) which converges onto the participants hats.

<figure><img src="image_117.png" /><figcaption>Fig. 9. IR tracking of the lights on top of the red hats (green), color tracking of the top of the red hats (red)</figcaption></figure>

The projective transform blocks were achieved by using some functions already implemented in OpenCV (Open Computer Vision) [4]. OpenCV is a library developed by Intel with a BSD license. EyesWeb can easily wrap OpenCV functions to handle them as blocks in the software platform.
## B. Confidence Level

The second step performed before the fusion was to compute a confidence level on each modality (IR and color).

The weight (confidence level) changes according to the participants' visibility with respect to the camera's field of view (FOV) and obstruction occurrences. Data range from 0 when the participants are not visible by the camera to 1 when they are visible. For the color modality, the confidence level also gradually changes between 0 and 1 depending on blob area variations: abrupt variations due for example to sudden illumination changes decrease the confidence level whereas stable blob areas over time increase it.

* Color video stream confidence level

Blob tracking with skin color detection can have a lack of accuracy due to illumination variation or even a loss of coordinates due to visibility issues (obstruction or disappearance from the camera's FOV).

This confidence level is built on two hypotheses. The first one is to consider location information obsolete after a short delay (mobile objects tracking). The second hypothesis is that blob's area abrupt variation can be related either to some undetected surface or to unwanted detections. According to these assumptions, the confidence level $CL_{VID}$ is computed as the conjunctions of two terms:

$$CL_{VID} = (blob in FOV) \land (stable blob's area) \quad (3)$$

where $CL_{VID} \in [0, 1]$.

For the "blob in FOV" indicator computation, we used a clock generator to check that the elapsed time since the last coordinates' acquisition keeps below an acceptable delay. The "stable blob's area" indicator is computed on a temporal sliding window. We compared the current blob's area value with its mobile average and returned an inverse variation rate (minimum value between the ratio and its inverse).

* Infra-red video stream confidence level

The IR light tracking might be interrupted in two situations: the blobs might be obstructed by the occlusion of one participant with respect to the others or they can come out from the camera FOV. The blob detection could also be corrupted when the tracked participants move closely. A binary confidence level was developed to handle these tracking issues. We measured the elapsed time since the last valuable detection. If this delay exceeded an empirically tested threshold, the confidence degree was set to 0 until a new detection occurred. The confidence level $CL_{IR}$ is computed as:

$$CL_{IR} = (blob in FOV) \quad (4)$$

where $CL_{IR} \in \{0, 1\}$.

We measured the elapsed time since the last valuable detection. If this delay is over a threshold that we fixed, "blob in FOV" is fixed at 0 until a new detection occurs.

## C. Fusion algorithm

* Single blob fusion

In order to fuse the 2D coordinates coming from the IR and the color video stream, a weighted mean rule was applied. The weights are the respective confidence levels previously computed for both modalities.

* Extension to multi-blob fusion

In the case of multi-blob tracking, each blob coordinates set was extracted and computed in each modality (color and IR) together with their confidence level.

Nevertheless, the fact that the blobs from both modalities will remain linked to the same target during the experiment is not obvious. To prevent this issue, our method needed to test continuously the relation between the coordinates of the same blobs located in the two modalities. We created a new fusion EyesWeb block where we considered only the non null confidence level elements and we matched each one of them in a modality with the nearest neighbor point (following a Euclidian distance) in the other modality. We fused these couples with the same weighted mean rule described in the previous section. Figure 10 shows the fusion results.

<figure><img src="image_118.png" /><figcaption>Fig. 10: Top-left: color image tracking, bottom-left: IR image tracking, right: modality fusion</figcaption></figure>

# IV. SALIENT GESTURES: AN ATTENTION FILTER

## A. Computational Attention

The aim of computational attention is to automatically predict human attention on different kinds of data such as sounds, images, video sequences, smell or taste, etc... This domain is of a crucial importance in artificial intelligence and its applications are numberless from signal coding to object recognition. Intelligence is not due only to attention, but there is no intelligence without attention.

Attention is also very closely related to memory through a continuous competition between a bottom-up or unsupervised approach which uses the features of the acquired signal and a top-down or supervised approach which uses observer's a priori knowledge about the observed signal. We focused here only on bottom-up attention due to motion.
While numerous models were provided for attention on still images, time-evolving two-dimensional signals as videos have been much less investigated.

Nevertheless, some of the authors providing static attention approaches generalized their models to the time dimension: Dhale and Itti [5], Yee and Pattanaik [6], Parkhurst and Niebur [7], Itti and Baldi [8], Le Meur [9] and Liu [10]. Motion has a predominant place and the temporal contrast of its features is mainly used to highlight important movements. Zhang and Stentiford [11] provided a motion analysis model based on comparing image neighborhoods in time. The limited spatial comparison led to a “block-matching”-like approach providing information on motion alone more than on motion attention. Boiman and Irani [12] provided an outstanding model which is able to compare the current movements with others from the video history or video database. Attention is related to motion similarity. The major problem of this approach is in its high computational cost.

As we already stated in [13] and [14], a feature does not attract attention by itself: bright and dark, locally contrasted areas or not, red or blue can equally attract human attention depending on their context. In the same way, motion can be as interesting as the lack of motion depending on the scene configuration. The main cue which involves attention is the rarity or the contrast of a feature in a given context. A pre-attentive analysis is achieved by humans in less than 200 milliseconds. How to model rarity in a simple and fast manner?

The most basic operation is to count similar areas in the context. Within information theory, this simple approach based on the histogram is close to the so-called self-information. Let us note $m_i$ a message containing an amount of information. This message is part of a message set $M$. A message self-information $I(m_i)$ is defined as:

$$I(m_i) = -\log(p(m_i)) \qquad (5)$$

where $p(m_i)$ is the probability that a message $m_i$ is chosen from all possible choices in the message set $M$ or the occurrence likelihood. We obtain an attention map by replacing each message $m_i$ by its corresponding self-information $I(m_i)$. The self-information is also known to describe the amount of surprise of a message inside its message set: rare messages are surprising, hence they attract our attention.

We estimate $p(m_i)$ as a two-terms combination:

$$p(m_i) = A(m_i) \times B(m_i) \qquad (6)$$

The $A(m_i)$ term is the direct use of the histogram to compute the occurrence probability of the message $m_i$ in the context $M$:

$$A(m_i) = \frac{H(m_i)}{Card(M)} \qquad (7)$$

where $H(m_i)$ is the value of the histogram $H$ for message $m_i$ and $Card(M)$ the cardinality of $M$. The $M$ set quantification provides the sensibility of $A(m_i)$: a smaller quantification value will let messages which are not the same but quite close to be seen as the same.

$B(m_i)$ quantifies the global contrast of a message $m_i$ on the context $M$:

$$B(m_i) = 1 - \frac{\sum_{j=1}^{Card(M)} |m_i - m_j|}{(Card(M) - 1) \times Max(M)} \qquad (8)$$

If a message is very different from all the others, $B(m_i)$ will be low so the occurrence likelihood $p(m_i)$ will be lower and the message attention will be higher. $B(m_i)$ was introduced to avoid the cases where two messages have the same occurrence value, hence the same attention value using $A(m_i)$ but in fact one of the two is very different from the others while the other one is just a little different.

In order to get a fast model of motion attention we have here a three-level rarity-based approach. While the first two approaches are bottom-up and use motion features context to attract computer's attention, the last one is mostly top-down and it learns a model of the scene which will be able to modify bottom-up attention by inhibiting some movements for example. The three levels of motion attention we propose here are:

*   **Low-level instantaneous motion attention:**

Motion features are compared in the spatial context of the current perceived frame. Rare motion behaviors should immediately pop-out and attract attention. This low-level approach is pre-attentive (reflex) and it uses no memory capacities.

*   **Middle-level short-term motion attention:**

Once a moving object was selected using low-level attention, its behavior into a short temporal context is than observed. Short-term memory (STM) is here used to save an object motion during 2 or 3 seconds (for longer time periods, motion details of an object are forgotten). Rare behaviors of an object through time will be quoted as interesting while repeating motion will be less important.

*   **High-level long-term motion attention:**

This third top-down attention approach uses long-term memory capacities and it is a first step through motion and scene comprehension. The attention level of each pixel through time is accumulated which leads in areas of the scene which concentrate attention more than the others : a street accumulates more attention through time than a grassy area close to it, a tree which moves because of the wind or a flickering light will also accumulate attention through time. The scene can thus be segmented in several areas of attention accumulation and the motion in these areas can be summarized by only one motion vector per area. If a moving object passes through one of these areas and it has a motion vector similar to the one summarizing this area, its attention will be inhibited. If this object is outside those segmented attention areas or its motion vector is different from the one summarizing the area where it passes through, the moving object will be assigned a very high attention score. This third attention step builds an attention model learnt from instantaneous and short-term attention steps which is able to inhibit bottom-up attention if it corresponds to the model or to enhance it if the motion does not match with this model.
Within this project we developed an implementation of the two bottom-up motion attention comparing current motion features with a spatial and a short-term temporal context. The third top-down attention model is further discussed in section V of this article.

## B. Instantaneous motion attention

An implementation of the spatial motion rarity was achieved as an EyesWeb XMI patch by using Equation (5) with $p(m_i)=B(m_i)$. In the scenario three people were tracked and their instantaneous speed was used. As only 3 motion vectors were available, the computation of the rarity $A(m_i)$ had not much sense from a statistical point of view.

Fig. 11 shows part of the tested scenario. Three people moving or not are present behind the cameras. Their instantaneous velocity vectors $V_1$, $V_2$, and $V_3$ are computed.

<figure><img src="image_119.png" /><figcaption>Fig. 11: Left: fastest moving object ($V_1$) is the most important (hot red), Right: slower moving object ($V_3$) is the most important.</figcaption></figure>

On the left-side $V_1$ is very different from $V_2$ and $V_3$: $V_1$ has high speed amplitude while $V_2$ and $V_3$ are very slow or stopped. In this case the faster object has a higher attention score (hot red) compared with the lower attention score (darker red) of $V_2$ and $V_3$. This situation can be compared with the one which often occurs if a classical motion detection module is used and where faster motion is well highlighted. That is not the case on the right image where the most different speed is $V_3$ (stopped) while $V_1$ and $V_2$ are quite the same (very fast). In this case, the attention score is higher (hot red) on $V_3$ which does not move while fast moving objects do not attract a large amount of attention. The result of this approach shows a different behavior compared to a simple motion detection algorithm. This first step enables the computer to choose the most “interesting” moving object very efficiently and then to apply to it short-term motion attention.

## C. Short-term motion attention

### Trajectory-related features

The moving object speed was computed on a history of 3 seconds and the speed mean was computed on a 10 frames sliding window in order to avoid a too high variability of the speed due to segmentation and tracking noise.

The speed range was divided into 3 bins: static or very low speed, normal speed and high speed. The Equation (5) was applied with $p(m_i)=A(m_i)$ because there was enough data within the 3 seconds history to get a statistically reliable occurrence likelihood $p(m_i)$. Moreover, the contrast between the 3 speed bins is always very high so the $B(m_i)$ term is here less important.

<figure><img src="image_120.png" /><figcaption>Fig. 12: The brutal speed change is more important (red) than the speed value: the amplitudes of $V(T1)$ and $V(T4)$ are the same but they have a different attention score. Similar behavior can be seen with $V(T2)$ and $V(T3)$.</figcaption></figure>

Fig. 12 shows the tested scenario. One participant moves his head from left to right very fast, then normally and finally he stops. When a change in the speed is detected (stop to normal speed, normal speed to high speed, high speed to stop, etc...), the attention score is very high, but it decreases exponentially when a stable speed occurs.

### Silhouette-related features

The same algorithm as in the previous point was applied but the feature used here was the Quantity of Motion (QoM). This measure is obtained by integrating in time the variations of the body silhouette (called Silhouette Motion Images - SMI).

<figure><img src="image_121.png" /><figcaption>Fig. 13: The brutal QoM change is more important (red) than its value: the QoM at $T1$ and $T2$ are the same but they have a different attention score. The same behavior can be seen between $T3$ and $T4$.</figcaption></figure>
Fig. 13 shows the test case. One participant is stopped, moves normally and moves a lot. If he moves fast while within his temporal context he is majorly stopped, this will be interesting. But if he moves fast while within his temporal context he also majorly moves fast the computer will classify this movement as uninteresting because it is repetitive and no new information is brought.

As in section IV.B, the short-term motion attention algorithm was implemented as an EyesWeb XMI patch.

### D. Spatial and temporal motion attention fusion?

The previous two points (B and C) provide rarity-based attention indexes for moving objects based on spatial and on short-term temporal contexts. An interesting point is about a possible fusion of the results of those two approaches: how simultaneously take into account attention based on two contexts which do not have the same nature?

It is impossible to compare a given temporal context with a spatial one on the same basis: time and space are orthogonal which is also confirmed by the fact that time and space features are processed in two separate regions of the brain [15]. Instead of trying to fuse those two kinds of attention indexes, it seems more realistic to state that one of them (spatial context) is pre-attentive and it occurs first while the second one (short-term) is attentive and it focuses on the temporal context of an object previously selected by the spatial attention. Once this object is analyzed by the temporal attention, another one which may spatially pop-out can be tracked and its attention computed. Thus, spatial attention selects potential interesting moving targets while temporal attention verifies if those targets have also interesting behaviors through time.

In our opinion it is much more relevant to fuse the motion spatial attention map with a static image attention map based on color and gray level rarity [16]. As the context is in this case the same, it is realistic to compute rarity cues and compare static images (as background) and motion rarity (foreground) to get a final spatial attention which takes into account motion but also the other static pre-attentive features as colors or shape.

## V. RESULTS AND DISCUSSION

### A. Results and rendering

We developed a solution to track human movement based on the fusion of two video modalities: a color and IR system. This solution showed robustness with respect to the light changing conditions and partial occlusions.

We analyzed spatio-temporal profiles of people activities at different levels of details. At gross level, human activity was analyzed in terms of the spatial trajectory of moving blobs corresponding to heads. However trajectory, by itself, hardly provided detailed information about the performed gestures [17]. A more-detailed level of person's activity was analyzed in terms of full-body quantity of motion (QoM) and found more relevant to characterize motion sequences and related gestures.

An attention index was developed to highlight motion saliency. Both the spatial and temporal contexts were used to

compute rarity, thus attention indexes which provides additional information compared to a simple motion detection algorithm. We showed that motion perception can be radically different from simple motion detection: depending of the context, the lack of motion appeared to be more "interesting" than motion itself. Spatial context is used to select the participant which exhibits the highest saliency. Then, the selected participant can be tracked on a short period of time in order to see if he also has a salient behavior over time. This project is a first step towards systems which have more human-like reactions and perceive signals instead of only detecting them.

### B. Further improvements

Several improvements could be achieved in addition to improve the already implemented system:

* **Video stream synchronization**

The blob detection data flows were not synchronized. A delay could be found between the tracking achieved on the IR video stream and the one achieved on the color video stream. This is due to the fact that the analog cameras and the digital camera had not the same time response. The output signal of the first ones must be digitalized with an extern FireWire/AD converter. We could observe a time lag between the video output and the filmed scene.

Moreover, the computation time for the color video processing is longer than the one needed for the IR video processing. Finally, the computers we used for the two video stream processing had not the same performances in terms of both CPU and graphical card.

The data flow synchronization could be improved if both cameras are digital and have the same characteristics. The synchronization could be even simpler if both cameras are acquired on the same computer using several FireWire ports. In this case, the EyesWeb XMI possibility to synchronize several processes on a single platform could be used and the two data flows would be perfectly synchronized. EyesWeb XMI highly improved the multimodal synchronization possibilities [3]: each block has two additional pins called "Sync-in" and "Sync-out" which can be used to propagate synchronization clock signals between blocks.

* **Confidence level improvement**

The present confidence level (CL) characterizes how much we should be confident in the tracked object position in each modality (IR and color video stream) in order to fuse them using a weighted mean. For the current implementation, we only used two assumptions: time since the last valuable detected position and smoothness of the detected blob variation over time. The first way to improve the modality confidence level could consist in adding other assumptions. For example, as we are tracking human heads, we could assume that abrupt speed variation is correlated with tracking inaccuracy and to decrease the positioning CL.

An additional improvement could consist in defining a confidence level for the fusion itself that we call fusion confidence level (FCL). A tracked object position could have a high CL in both modalities but the final fusion may not be
necessarily accurate if the two positions are very far one from the other for example. Other elements like speed and direction could help in defining if two positions from two different modalities can be fused. This FCL is able to point out if the data fusion should be considered or not.

* Additional features: getting closer to human attention

An interesting improvement could also be achieved by computing attention indexes for many other features. We can think first about motion direction which was not taken into account during this implementation: only the speed attention was computed. If the speed of a moving object is very low, the motion direction information is not very reliable because it can be due only to detection noise. Nevertheless, if the speed is high enough the information brought by motion direction is very important. From an attentional point of view, if many people have a common direction while one participant has a different direction, this is a very important cue on objects saliency. The same idea can be developed for temporal attention where brutal direction changes are very interesting. A future work will consist in reaching a more complete description of human activities. The head tracking should be integrated by information about overall body motion. Camurri and all [18] revealed that bounding box variations or ellipse inclination, that approximate 2D translation of body, can account for expressive communication. A more detailed analysis of upper body-part should also be accomplished. Glowinski et al. [19] show that color-based tracking of head and hand can reveal expressive information related to emotion portrayals. On the basis of this refined description of human movement, we could consider new motor cues (e.g. symmetry, directivity, contraction, energy, smoothness...) accounting for the communication of an expressive content [18] that could be integrated, processed by the attention index for a more pertinent context-based analysis of expressivity.

If the fusion between spatial and temporal attention is not interesting (as discussed in section IV.D), it is crucial to fuse attention information coming from several features in the same context (space or time). This fusion can simply be done by using the maximum operator: if a feature highly attracts attention in a specific place, this area is very interesting at least from this feature point of view.

* High-level motion attention

In real life, when we observe a scene which does not change every time (fixed camera for example), we build knowledge models about those scenes. That model will have important influences on the final attention score and it will be able to modify attention coming from the bottom-up attention mechanisms described in this paper. A first simple implementation of this high level model is to use attention accumulation as a threshold [13]: only the objects which imply a bottom-up attention which is higher than the attention accumulation of the model are really interesting, all the other objects are inhibited.

Figure 14 displays an example of the results of this implementation: a frame of the video is extracted on the top-left image. The top-right image is an attention map of the same frame. The bottom-left image is a model of the scene:

the trajectory of the moving person is visible and also is the tree which often moves because of the wind. The bottom-right image show the final attention map after the high-level attention inhibition: some noisy areas (grass which also moves because of the wind, moving person's trajectory) are inhibited. The moving tree area has also been inhibited and only few attention is focused on the tree area. The moving person remains the only area of high attention score.

<figure><img src="image_122.png" /><figcaption>Fig. 14: Top-left: current video frame, top-right: bottom-up attention, bottom-left: scene model (top-down attention), bottom-right: final attention map after high-level attention inhibition</figcaption></figure>

This simple approach already showed that it can inhibit some attention areas due to noise or repetitive movements (trees which move because of the wind, flickering lights ...).

A more complex implementation of the high-level attention model should segment the areas where a lot of attention accumulated and summarize them with a single motion vector. An object moving into these areas should be inhibited only if its motion vector is close to the motion vector which summarizes that area and its bottom-up attention score is below the model local amplitude. If the motion vector of this object is very different from the segmented attention area it passes through, the object will not be inhibited.

## C. Discussion and Conclusion

We developed a context-based framework working in a non-controlled environment. It can deal with multiple heterogeneous situations caused by environmental disturbances and focus on relevant/rare events.

This system is more robust comparing to other applications based on video-tracking commonly developed for controlled environment. Stable environmental factors such as constant illumination and stable background greatly facilitate human activity monitoring. The present system can be a response to the growing demands, human monitoring systems in many application fields (e.g. video-surveillance, elder-people ambient assistant living, performing arts, museum spaces, etc.) Moreover, our system opens new research perspective for affecting computing and analysis of human expressivity.