Analyzing Free-standing Conversational
Groups: A Multimodal Approach
Xavier Alameda-Pineda

University of Trento
Trento, Italy
xavier.alamedapineda@unitn.it

Yan YanUniv. of Trento / ADSC, UIUC
Trento, Italy / Singapore
yan.yan@unitn.it

Oswald Lanz

Fondazione Bruno Kessler
Trento, Italy
lanz@fbk.eu

Elisa Ricci

FBK / Univ. of Perugia
Trento, Italy / Perugia, Italy
eliricci@fbk.eu

Nicu Sebe

University of Trento
Trento, Italy
sebe@disi.unitn.it

ABSTRACTDuring natural social gatherings, humans tend to organize themselves in the so-called free-standing conversationalgroups. In this context, robust head and body pose estimates can facilitate the higher-level description of the ongoing interplay. Importantly, visual information typically obtained with a distributed camera network might not suﬃce
to achieve the robustness sought. In this line of thought, recent advances in wearable sensing technology open the door
to multimodal and richer information ﬂows. In this paper we
propose to cast the head and body pose estimation problem
into a matrix completion task. We introduce a framework
able to fuse multimodal data emanating from a combinationof distributed and wearable sensors, taking into account thetemporal consistency, the head/body coupling and the noise
inherent to the scenario. We report results on the noveland challenging SALSA dataset, containing visual, auditory
and infrared recordings of 18 people interacting in a regular indoor environment. We demonstrate the soundness ofthe proposed method and the usability for higher-level tasks
such as the detection of F-formations and the discovery ofsocial attention attractors.

1.

INTRODUCTION

Several wearable sensing devices became available for thegeneral public in the past few years. Often, such consumer platforms include several sensors, e.g. microphone,accelerometer or blue-tooth. Investigating methods to process the data gathered with wearable devices is worthwhile
for many reasons. First of all, considering that modern consumer platforms, such as smart-phones or smart-watches,
are at the reach of the general public, many are already using them for data acquisition. Second, these platforms are
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor proﬁt or commercial advantage and that copies bear this notice and the full citation on the ﬁrst page. Copyrights for components of this work owned by others thanACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior speciﬁc permission
and/or a fee. Request permissions from Permissions@acm.org.
MM’15, October 26–30, 2015, Brisbane, Australia.
c 2015 ACM. ISBN 978-1-4503-3459-4/15/10 ...$15.00.
�
DOI: http://dx.doi.org/10.1145/2733373.2806238.

Figure 1: People interacting naturally organized infree-standing conversational groups during a poster
session. The social interplay is captured by a distributed camera network and by sociometric badges
worn by each participant (small white boxes).
inherently multimodal, thus they provide a rich descriptionof the environment. Third, since these devices are easy-towear, the continuous ﬂow of information emanating fromthem is a precious resource to study casual situations of thereal life, far from laboratory-controlled conditions.
Importantly, wearable technologies are complementary to
distributed sensor networks. For example, when analyzingsocial interactions, wearable sensing devices can be exploitedinter alia to localize people and to roughly estimate theiractivities. However, a ﬁne-grained analysis of the social
scene requires additional information gathered with alternative distributed sensing networks, e.g. visual data [29]. Developing methods that robustly fuse data from wearable devices and distributed networks remains largely unexplored,
and many challenges arise in this context. As with any consumer device, data gathered with wearable technology arecorrupted by severe noise. Therefore, unimodal approaches
fail to provide a robust and accurate representation of theenvironment and smart multimodal fusion strategies need to
be developed [27]. Moreover, these strategies should also account for the speciﬁcities of the combination of a distributed
sensor network and the wearable technology.
The focus of this research study is the analysis of spontaneous social interactions in natural indoor environments,
see Figure 1. In such social events, human beings tend to�������������������

������������������

��������������

�������������������������
���������������������

�������������

���������
�������

������������
������
�����������������������������������
������������
���������������������������

Figure 2: Overview of the proposed approach for detection of attentional patterns. The scene is captured
by a distributed camera network and sociometric badges worn by each participant. The estimation of thehead and body poses, required to further extract F-formations and social attention attractors, is cast into amatrix completion problem. The HBPE are used to provide a higher-level description of the social interplay.
organize themselves in free-standing conversational groups
and body pose of all people, is cast into a matrix completion problem. Within this framework, the joint head and(FCGs). Contrary to the staticity proper to round-table
body pose estimation adds an additional coupling term tomeetings, FCGs are dynamic by nature (they increase andthe formulation. Similarly, the temporal structure inherent
decrease in size, move and split) and therefore their analysis
to the problem is induced by means of a Laplacian matrix
is inherently diﬃcult [11, 33]. In this context, robust humanpose estimates can ease the further mining of higher-levelthat regularizes the matrix completion task. The resulting
HBPE are then used to infer the F-formations and to furdescriptions, such as F-formations or attentional patterns.
ther retrieve the social attention attractors. Figure 2 shows
In this paper we are expressly interested in analyzing FCGs
an overview of the proposed approach.
using head and body pose estimates obtained through theprocessing of the multimodal ﬂow of data emanating from
The main contributions of this study are: (i) address multimodal HBPE within the analysis of FCGs (ii) propose aa distributed camera network and sociometric badges [10]framework versatile enough to handle not only data coming
worn by each participant.
from diﬀerent modalities, but also to process features exProviding a complete description of the human pose (potracted with both wearable sensors and a distributed camera
sitions of the limbs) is chimeric in video surveillance settings, i.e. when people are monitored with distant and large
network (iii) cast HBPE into a matrix completion problem,
accounting for the noisy multimodal labels, and the inherent
ﬁeld-of-view cameras. Indeed, the low-resolution images andtemporal structure of the head and body poses (iv) induce
the numerous occlusions have a relentless negative eﬀect on
joint HBPE by coupling the two incomplete matrices (v) decurrent approaches. Fortunately, the head and body posecan be used as a surrogate of the full pose when analyzing
rive a novel optimization approach to solve for the matrix
completion task and (vi) conduct an extensive evaluation of
free-standing conversational groups. Indeed, all features dethe proposed method on the novel SALSA dataset, showing
lineating group social interplays, are often extracted fromthe usability of the HBPE for detecting F-formations andhead and body pose estimates (HBPE). More precisely, acdiscovering social attention attractors.
curate and robust HBPEs are the prelude to detect face-toface interactions, identify the potential addressee and adThe rest of the paper is structured as follows. Related
approaches for head and body pose estimation are discussed
dresser, describe the geometric structure of the group (or Fin the next section. The proposed matrix completion model,
formation) and recognize personality traits. Many researchaccounting for noise, temporal regularization and joint headstudies focused on HBPE from visual data in the recent
past, and demonstrated the synergistic eﬀect of joint headand body pose estimation is presented in Section 3, together
with the associated optimization algorithm. Extensive evaland body pose estimation, when compared to independent
uations of the proposed approach, and comparisons with theestimation [9]. Furthermore, as previously discussed, wearstate of the art are reported in Section 4, before concluding
able sensing devices provide a complementary source of inand delineating future research directions in Section 5.
formation to the surveillance distributed camera network.
Nowadays, up to the authors’ knowledge, there are no existing research studies that address the challenging task of
2. RELATED WORK
head and body pose estimation in crowded indoor environThe applicative context of the present study is the multiments using multimodal data gathered with a combination
modal analysis of social interactions in informal gatherings.
of distributed visual sensors and wearable devices.
Special focus is given to the estimation of the head and bodyIn this paper we propose to integrate multimodal signals
pose, easing the automatic study of free-standing conversaacquired in regular indoor environments to robustly extract
tional groups. Moreover, the proposed solution is based on
HBPE of several people naturally arranged in FCG. Morea matrix completion framework.
precisely, we use auditory and proximity (infrared) information to automatically label visual features of head and
2.1 Multimodal analysis of social interactions
body respectively. All this information, together with the
Combining data from heterogeneous modalities is of utvisual features extracted from the surveillance camera netmost importance for understanding human behaviors [36].
work, are pulled together into a linear classiﬁcation problem.
Approaches relying on audio-visual cues are probably the
Seeking for the non-available labels, i.e. estimating the headmost popular and successful examples [18, 30, 37, 1]. In[18], the role of non-verbal cues for the automatic analysis of face-to-face social interactions is investigated. Social
interactions in political debates are studied in [37]. A multimodal approach for detecting laughter episodes from audiovisual data and a novel dataset for studying this problem
are introduced in [30]. In the last few years, mobile andwearable devices opened novel opportunities for multimodal
analysis of social interactions [6, 16, 15, 28]. In [15] a probabilistic approach is proposed to automatically discover interaction types from large-scale dyadic data (e.g. proximity
blue-tooth data, phone call network or email network). In[28] social interactions on a small spatio-temporal scale combining proximity data and accelerometers are analyzed.
On the one hand, traditional distributed camera and microphone networks are able to capture subtle features ofhuman behavior, but their performance drops signiﬁcantly
when dealing with a crowded scene. On the other hand,
wearable devices permit the ubiquitous localization of people for a long time span, as they typically embed proximity
sensors, but they fail to provide accurate and detailed descriptions of the ongoing interplay. Therefore, it is obviousthat the nuances and the complexity of social interactions
require leveraging information from both wearable and traditional sensors. This paper develops from this intuition
and, to our knowledge, it is the ﬁrst work jointly employing
sociometric badges and external cameras for HBPE.

2.2

Head and body pose estimation

As already outlined, the analysis of the human behavior
will deﬁnitely beneﬁt from automatic human pose estimation. Multimodal approaches based on RGB-D data [35,42], eventually combined with audio [17], and with visualinertial sensors [13], have recently proved very successful intracking human limbs. However, the former methods areappropriate when only one or few people move in proximity of the camera, while the latter approaches are often notpractical solutions, as they require to instrument the person’s body with multiple sensors. In addition, the robust
estimation of the full human pose is chimeric in crowded
scenes, which is typically the case of informal social gatherings. Consequently, when considering social interactions
among several people, the head and body pose can be used
as a surrogate of the full human pose.
Previous research has demonstrated that head and bodyorientations can be successfully detected from visual data [3,9, 41, 31] and used as primary cues to infer high level information, such as visual focus of attention [38] and conversational groups [11]. In order to alleviate the humanannotation eﬀort, previous research has focused on using related cues to estimate head and body pose. For example,
head pose labels generated using walking direction are considered by Benfold and Reid [3]. Similarly, coupling of headand body pose due to anatomical constraints is enforced byChen and Odobez [9]. The body orientation is considered
as a link between walking direction and head pose in [32,25]. A transfer learning framework for head pose estimation
under target motion is proposed in [31]. In [41] a multitask learning approach is introduced for accurate estimatesof the head pose from large ﬁeld-of-view surveillance cameras. More recently, approaches to speciﬁcally cope withlabel noise [19] and integrate temporal consistency [14] areintroduced. Audio-visual head pose estimation is attemptedin [8]. However, up to the authors’ knowledge, there are no

research studies combining visual features from a distributed
camera network with multimodal cues extracted from wearable sensors to jointly estimate the head and body poses.

2.3

Matrix Completion

We propose to cast multimodal head and body pose estimation into a matrix completion problem, which has beenshown to be equivalent to learning a classiﬁer in a transductive setting [21]. This formulation is particularly advantageous when data and labels are noisy or in the case of
missing data. In the computer vision and multimedia communities, this fact has been exploited in several applications,such as multi-label image classiﬁcation [5], image retrieval[40] and facial expression analysis [39]. A recent study [24]extends the matrix completion problem to take into accountan underlying graph structure inducing a weighted relationship between the columns and between the rows of the matrix. We also utilize this structure to model the temporalsmoothness of the head and body pose estimates. However,from the technical point of view our approach is novel because: (i) we address two matrix completion problems (forthe head and body poses) that are coupled and cannot be
reduced to a joint matrix completion problem, (ii) the proposed framework is able to deal with data coming from multiple modalities, handling signals generated from diﬀerent
types of sensing devices (distributed and wearable).

3.

MATRIX COMPLETION FOR
MULTIMODAL POSE ESTIMATION

The technical challenge of the present study is the robust
estimation of the body and head pose in crowded scenarios
for further analysis of FCGs. We assume a regular indoor
environment populated with K people and captured with a
distributed camera network and by sociometric badges [10]worn by the participants. On the one hand, we extract visual features of the head and body of all the participants,
localized in the scene thanks to a multi-target visual tracking algorithm. On the other hand, since manual annotation
is tedious and resource-demanding, we automatically labelpart of the sequence using the information gathered withthe sociometric badges. The full preprocessing pipeline is
described in Section 4.2, but we outline here the main insights and notations used in our research.
The visual descriptors of the head and body of person khb
at time t are denoted by vkt∈ Rdh and by vkt∈ Rdb respectively, where dh and db are the features’ dimension. Similarly, the head and body orientations of person k at time thb
are denoted by θkt
and θkt. Inspired by previous research on
the topic [9], we discretize the space of angular orientations
into C sectors. Therefore theand head orientations
� b body�should be understood as
are C-dimensional vectors. θktcthe probability that the associated body visual feature liesin the cth sector (analogously for the head). For instance,bb= [1, 0, . . . , 0], means that θktbelongs to the ﬁrst sector.
θkt
As outlined before, we assume the existence of a labeledtraining set L containing the raw features and noisy labelsof the head and body pose of all the K persons involvedin the interaction for T0 < T frames, where T is the total
� b b h h �K,T0, θkt , vkt , θkt k=1,t=1 .
number of frames. That is: L = vkt
Complementary, the set of unlabeled features is deﬁned as:� b h �K,T
, vkt k=1,t=T +1 . It is important to remark thatU = vkt
0part of the training set is manually annotated and the rest

optimization problem for ΘbU :�����������������

Θb∗
U = argmin rank (Jb ) .

����������
��������
������������

Θb
U

����
��������

����

����

������

��������

Minimizing the rank of a matrix is a NP-hard problem, but
it can be exactly relaxed by means of the nuclear norm [7].
Indeed, since the nuclear norm � · �∗ is the tightest convex
envelope of the rank, the previous optimization problem is
equivalent to:Θb∗
U = argmin �Jb �∗ .
ΘbU��������������������

��������������������

�������������
�����������������������
���������������������
�������������
�����������������������
���������������������
��������������

Figure 3: Illustration of the proposed matrix completion framework for multimodal head and bodypose estimation (MC-HBPE). Two matrix completion problems, for the head and body poses, are regularized to account for the temporal consistency andfor head/body coupling.
is automatically labeled. More precisely, we use the infrared
detections as a proxy for the body pose and auditory correlations to automatically label the head pose. Indeed, it isreasonable to assume that the audio signals have a strong
inﬂuence on the gaze direction of the targets, while the infrared detections naturally correlate with the body orientation. Finally, it is worth noticing that while visual data arecontinuously available, audio and infrared signals are sparse
observations. In the following we describe the proposed approach for fusing these multiple modalities, so to robustly
estimate the head and body poses of all participants.

3.1

The Model

Recent theoretical results and practical advances in matrix completion theory [5, 21], motivated us to consider thehead and body problem from this perspective. We thoroughly cast the estimation into a matrix completion problem, thus introducing Matrix Completion for Head and Body
Pose Estimation (MC-HBPE), a graphical illustration ofwhich is shown in Figure 3. In the following, in order to ease
the exposition, all the claims, deﬁnitions and properties arestated only for the body pose, but they remain true for thehead pose, unless it is explicitly written otherwise. Before
describing the model, we set a few useful notations: the matrices of labeled and unlabeled body features of person k,
b
b T0
b T
VL,k
= [vkt]t=1 and VUb ,k = [vkt]t=T0 +1 ; their concatenabb T
VUb ,k ] = [vkt]t=1 ; and the concatenation
tion, Vk = [VL,kover all persons, i.e. , the concatenation of all body visualbbb
features, Vb = [Vkb ]K
k=1 . The matrices ΘL,k , ΘU ,k , Θk andb.
Θb are analogously built from θkt
The objective is to estimate the matrix ΘU = [ΘbU ,k ]K
k=1 ,under the assumption of a linear classiﬁer:
� b �
V
,
(1)
Θb = W b
1�
with Wb ∈ RC×(db +1) . Remarkably, the joint feature-label���
matrix Jb = Θb� Vb� 1∈ R(C+db +1)×KT is low rank,since the linear classiﬁer in (1) imposes a linear dependency
between the rows of Jb . Therefore, we set the following

(2)

(3)

In real applications both the observations and the traininglabels are noisy. Therefore, it is of crucial importance toaccount for the observational and label noise. In practice, we�b represents the noisy features�b = Jb +Eb , where J
assume J
and labels, and Eb represents the noise. The objective is toestimate the low-rank matrix Jb that best approximates the�b . This is done by constraining (3), andobserved matrix J
relaxing it into [21]:min νb �Jb �∗ +
Jb

�
��2
λb � b � ���PO Jb − Jb � ,
2
F

(4)

2
where
� νb �and λb are regularization parameters, �X�bF =deTr X� X denotes the Frobenius norm of X and PO
notes the projection onto the “set of observations”, deﬁned������ � � b,� Kb= [ΘL,k 0]k=1 Vb,� 1 , and thusas POΘb,� Vb,� 1

setting to zero all elements associated to ΘbU . We remarkthat this unitary regularization term, encompasses the error
on the visual features as well as on the training labels.
One of the prominent factors alleviating the matrix completion problem with noisy observations is the inherent temporal structure of the problem. Intuitively, we would expectbb
the labels to be smooth in time, that is θkt≈ θkt+1, ∀ k, t.
We reinforce this intuition through a set of binary regularization terms, grouped in the following loss function:�1 �
Tb (Jb ) = Tr PΘb (Jb )Tb PΘb (Jb )� ,
(5)
2

b
where
onto the labels, Θb , deﬁned as�� PΘ is the projection�� � � b,���bb,� b,�
= Θ 0 0 , and Tb ∈ RKT ×KT isPΘ Θ V 1
the Laplacian matrix associated to the graph encoding therelations between the variables. In our case, Tb = IK ⊗ L,
where IK is the K-dimensional identity matrix, ⊗ is the
Kronecker product and L ∈ RT ×T is a tri-diagonal matrix
deﬁned as follows: the elements of the two subdiagonals are
set to −1, and the elements of the main diagonal are set to2, except L11 = LT T = 1. In practice this means that only
contiguous pose labels of the same person are embolden tobe equal.
Until this point, we discussed the problem of body andhead pose estimation separately. However, previous researchstudies in head and body pose estimation demonstrated thesynergistic eﬀect of joint estimation [9]. Subsequently, theestimation of head and body pose beneﬁts from a coupling
structure, in addition to the temporal structure already disbhcussed. We write, θkt≈ θkt
and formalize the coupling
structure by means of the following regularization term:
�
�2
1�
�
C (Jb , Jh ) = �PΘb (Jb ) − PΘh (Jh )� ,(6)2F

where PΘh is deﬁned analogously1 as PΘb .
Summarizing, the head and body pose estimation is castinto a matrix completion problem encompassing the temporal and coupling structural constraints, by considering thefollowing optimization problem:
�
��2
λb �
� b �
�
(7)
�PO Jb − Jb �
Jb ,JhF2����λh � h �
�2+ νh �Jh �∗ +
�PO Jh − Jh �
F
2
� τ
�
�
τb � bhbhh+(Jb ) +(Jh )� Th PΘ(Jh )Tr PΘ (Jb )� Tb PΘ
Tr PΘ
2
2
�2
λc �
� b
�
h
+
(Jh )� .
�PΘ (Jb ) − PΘF2minνb �Jb �∗ +This is a non-linear convex optimization problem whosecritical points cannot be expressed in closed-form solution.
In the following we describe the proposed optimization algorithm to ﬁnd the optimal solution for (7).

3.2

Optimization method

The alternating-direction method of multipliers (ADMM)
is a well-known optimization procedure, and a review of theoretical results, practical implementation considerations andapplications can be found in [4]. Recently, an ADMM-based
optimization procedure for the matrix completion problem
has been presented [24]. We also considered ADMM to solve
the matrix completion problem. However, the proposed
method is intrinsically diﬀerent from previous approaches
because of two reasons:• First, we consider a coupled matrix completion problem, which is diﬀerent from solving for the concatenated head/body matrix. Absolutely, completing theconcatenation matrix implicitly assumes, not only thatthe body and head visual features have the same dimension, but also that the linear classiﬁer for the bodyand head poses are equal Wb = Wh . Therefore, themethod developed in [24] cannot be used in our case.
Instead, we propose and develop a Coupled Alternating
Direction Method of Multipliers (C-ADMM).
• Second, the temporal structure of the problem is modeled with a column-wise regularizer on the matrices,which is a particular case of [24]. Importantly, as will
be shown in the following, column-wise regularization
can be solved in closed-form, while the formulationof [24] cannot. Consequently, all the steps of the proposed C-ADMM have a closed-form solution, and theconvergence is guaranteed.
Classically, the ADMM arises from the construction of theaugmented Lagrangian [4], which in our case writes:
�
��2
λb �
� b �
�
(8)
�PO Jb − Kb �
F
2
�
�
��
2
λh � h ��+ νh �Jh �∗ +
�PO Jh − Kh �
F
2
� τ
�
�
τb � bh�bhh+(Kh )� Th PΘ(Kh )Tr PΘ (Kb ) Tb PΘ (Kb ) +
Tr PΘ
2
2
�2
λc �
φb
φh
�
� b
h
+
(Kh )� +�Kb − Jb �2F +�Kh − Jh �2F�PΘ (Kb ) − PΘF222+ �Mb , Jb − Kb � + �Mh , Jh − Kh � .

L=νb �Jb �∗ +1
Even if PΘb and PΘh are conceptually equivalent, they areformally diﬀerent since the features’ dimension is diﬀerent
in general, db �= dh .

The Lagrangian L has to be minimized with respect to Jb ,Jh , Kb , Kh , Mb and Mh , where the last two are matrices of
Lagrangian multipliers,
Kb and Kh are auxiliary variables,�
and �A, B� = ij Aij Bij is the scalar product in the matrix
space.
At iteration r + 1, the C-ADMM updates are:
�

�

�
r+1
Jr+1
= arg min L(Jb , Jh , Krb , Krh , Mrb , Mrh ) ,b , Jhr+1
Kr+1
b , Kh�Jb ,Jh��r+1
rr= arg min L Jr+1
b , J h , K b , K h , Mb , M h ,Kb ,Kh

�

�

Mr+1
,
= Mrb + φb Jr+1
− Kr+1
b
b
b
�
�
Mr+1
.
= Mrh + φh Jr+1
− Kr+1
h
h
h

(9)
(10)
(11)
(12)

From (8) it is straightforward that the coupling term does
not have any eﬀect on (9), but only on (10). Therefore,
Jr+1
can be computed
solving for (9) the two matrices Jr+1bhindependently. Furthermore, the optimal solution of (9) is:
Jr+1= Ub S νb (Db ) Vb
b
φb

(13)

where Ub Db Vb is the singular-value decomposition of thematrix Krb − φ1b Mrb and Sλ (x) = max(x − λ, 0) is the shrinkage operator with constant λ applied element-wise to theis found analdiagonal matrix of singular values, Db . Jr+1hogously. The details of the derivation can be found in the
supplementary material.
The second C-ADMM update, i.e. (10), is also solved inclosed-form. The critical point is computed by canceling thederivative of the objective function in (10) with respect to
Kb and Kh . As shown in the supplementary material, thesederivatives lead to block-diagonal linear systems that canbe eﬃciently solved. In order to sketch the proof, we deﬁne
kb = vec(Kb ) as the row-vectorization of Kb . The same notation stands for all other matrices involved. Moreover, webdeﬁne kb,kc = [(θkt)c ]Tt=1 , as the T -dimensional vector composed of the coordinate c of the angle labels of person k overtime. Likewise, kb,v stands for the KT (db + 1)-dimensionalrow vectorization of the rest of the matrix Kb , that is thepart corresponding to the visual features Vb . With thesenotations, the vector kb rewrites:�����kb = k�,b,11 . . . kb,KC kb,v
and analogously for all other row-vectorized matrices.
Each of the KC variables for body and head, kb,kc andkh,kc , are independently solved:
��
kb,kc = (Lh Lb − IT )−1 Lh kb,kc + kh,kc ,(14)kh,kc = Lb kb,kc − kh,kc ,(15)where all matrices and vectors are precisely deﬁned in the
supplementary material. The vector kb,kc is computed fromj̃b,kc , mrb,kc and jr+1
b,kc and thus depends on the iteration(analogously for the head). Importantly, the matrices Lb
and Lh , computed from L and the regularization parameters, do not depend on the iteration. Therefore, the inversion can be computed before the iterative procedure of
the C-ADMM starts, leading to an eﬃcient algorithm. The
system involving kb,v and kh,v is simpler since there is noAlgorithm 1 The coupled alternating-directions methodof multipliers solving for the derived matrix completion for
multimodal HBPE.
1: Input:�b and Laplacian Th , Tb matrices.
�h , J
Observation J
Regularization parameters.
2: Output:
Optimized Jh , Jb .
3: Randomly initialize Mb , Mh , Kb , Kh ;4: Repeat
Solve for Jh , Jb in (9) using (5);Solve for Kh , Kb in (10) using (14)-(17);Solve for Mh , Mb using (11)-(12);Until Convergence

coupling between body and head features and there is no
temporal regularization:(λb + φb )kb,v = λb j̃b,v + mrb,v + φb jr+1
b,v

(16)

(λh + φh )kh,v = λh j̃h,v + mrh,v + φh jr+1
h,v .

(17)

Equations (14)-(17) provide the critical point sought in (10)and conclude the derivation of the proposed C-ADMM. As
it is classically the case, the complexity bottle-neck of thematrix-completion solver is in the computation of the singularvalue decomposition required in (5). The complete C-ADMM
algorithm is shown in Algorithm 1.

4.

EXPERIMENTS

In this section, we show the results of our experimental
evaluation conducted on the novel SALSA (Synergistic sociAL Scene Analysis) dataset [2]. Then, we provide a detailed description of our experimental setup and show theresults of our extensive evaluation, demonstrating the eﬀectiveness of the proposed MC-HBPE method. Finally, we
perform further experiments on the analysis of social interaction in SALSA, showing that the computed head and bodyorientations are accurate and robust so to eﬃciently detect
F-formations and discover social attention attractors.

4.1

SALSA

The SALSA dataset records a two-part social event involving 18 participants. While the ﬁrst half consists on aposter presentation session (see Fig.1), the second half consists on a cocktail party over food and beverages. The sceneof SALSA is captured by a distributed camera network andby sociometric badges worn by the participants. Speciﬁcally,visual data are recorded by four synchronized static RGB
cameras operating at 15 FPS. The sociometric badges areequipped with a microphone and an infrared (IR) beam anddetector. Importantly, these wearable devices are batterypowered and store the data on a USB card without the needof any wired connection, thus guaranteeing a natural social
interplay. SALSA also comprises manual annotations every
3 seconds of position, head and body orientation of eachtarget as well as F-formations.

4.2

Experimental setup

In order to extract the visual features for body and head,bhvtk
and vtk, the participants can be tracked on the visualstream with multi-target tracking algorithms with state-ofthe-art occlusion handling such as [26]. The estimates of

the ground positions are used together with the camera calibration to retrieve the bounding box of the head and bodyof all participants for each camera view. We used thoseframes in which all participants are seen from all four cameras. Inspired by [9], we choose to describe each boundingbox with a Histogram of Oriented Gradient (HOG) descriptors [12]. More precisely, the head and body images are
ﬁrst normalized to 20 × 20 and 80 × 60 pixel images, respectively, from which we extract HOG in non-overlappingcells of 4 × 4 pixels. Finally, the descriptors for all the four
views are concatenated and the dimensionality of the headand body descriptors is reduced using Principal Component
Analysis, keeping 90% of the variance. In all, the head andbody visual features extracted from the four cameras forbheach participant (vtk
and vtk) are roughly 100–dimensional.
As outlined in Section 3, the infrared and auditory information from the sociometric badges are processed in order
to automatically extract body and head pose labels, respectively. Indeed, the diﬀerent nature of the infrared and auditory signals carries a rich representation of the badge’s
owner with respect to the other participants. On one side, if
the badge of person k detects the infrared beam of person l,most likely k’s body orientation points to l, since the badge
settles on its owner’s chest. On the other side, if the auditory signal of k’s badge is highly correlated with the one ofl, it is likely that l is speaking towards k, and that therefore
the head of l points towards k. It is worth noticing that,whilst visual features are extracted in a continuous manner,
infrared and auditory labels are sparse. Finally, inspiredby [9], we discretize the angular space to C = 8 sectors (corresponding to an angular resolution of 45◦ ) for both headand body labels.

4.3

Head and body pose estimation

To experimentally validate the technical core of the present
study, i.e. the MC-HBPE, we considered those frames of
SALSA in which all the participants are in the ﬁeld of viewof the four cameras. In practice, we use 340 frames of theposter session (17 minutes approximately), resulting in adata set comprising 6120 head/body samples. We split thisset into two equal parts, and use only the annotations of theﬁrst part for training: 10% of manual annotations and 90%
of weak annotations obtained with the sociometric badges,
when available. In essence, we have 5% of manual annotations for head and body pose, and therefore the labels aresparse. We report the classiﬁcation accuracy of the test samples, i.e. the second half of the data set. In all our experiments, and for both our approach and the baseline methods,
the regularization parameters are set with cross-validation.
Figure 4 shows the performance obtained with our method
when the training labels come from diﬀerent modalities. At
the light of the results, we can clearly state that using multimodal labels signiﬁcantly improves the accuracy of bothhead and body pose estimates. Interestingly, when considering only ground truth and auditory labels (i.e. no additional labels for body pose), not only the accuracy of thehead pose estimates improves, but also the accuracy of bodypose estimates. This is an important and remarkable eﬀectof the head/body coupling term introduced in the proposed
formulation. We notice the analogous eﬀect with the performance rise of, not only body pose estimates, but also headpose estimates, when adding the infrared labels (i.e. withno additional labels for head pose). It is worth noticing that

Head	  Pose	  Es7ma7on	
  Accuracy	
  
0.7	
  

Table 1: Classiﬁcation accuracy of diﬀerent methodsfor HBPE.

0.65	
  

MethodTb , T h

Body

Head

✓
✓

0.515
0.548
0.597

0.465
0.515
0.567

Chen et al. [9]

0.524

0.475

TSVM [23]

0.481

0.429

Golberg et al. [21]

0.479

0.439

0.6	
  

C
✓

MC-HBPE

0.55	
  

✓

0.5	
  
0.45	
  
0.4	
  
GT	
  

GT-­‐A	  GT-­‐IR	
  

ALL	
  

Body	
  Pose	  Es7ma7on	
  Accuracy	
  
0.7	
  
0.65	
  
0.6	
  
0.55	
  
0.5	
  
0.45	
  
0.4	
  
GT	
  

GT-­‐A	  GT-­‐IR	
  

ALL	
  

Figure 4: Accuracy of the head (top) and body(bottom) pose estimates obtained with the proposed
MC-HBPE approach using visual features and labelsfrom diﬀerent modalities: ground truth annotations(GT), GT and audio data (GT-A), GT and infrared
data (GT-IR), all modalities (ALL).
infrared labels have a signiﬁcantly more positive eﬀect than
auditory labels. We ascribe this fact to the higher accuracy of the infrared labels (87%) compared to the auditory
labels (70%). when compared to the manual annotations.
Importantly, we highlight the synergistic eﬀect of jointly using auditory, infrared and visual data to robustly solve thehead and body pose estimation problem by means of matrix
completion techniques. Finally, the implementation of the
proposed methodology needs roughly 0.5 seconds to estimate
the head and body poses of one person.
Table 1 reports the classiﬁcation accuracy of several methods. First of all, the proposed MC-HBPE method considering diﬀerent conﬁgurations regarding the head/body coupling and the temporal consistency (from top to bottom): (i)only with head-body coupling τb , τh = 0 in (7); (ii) only withtemporal consistency λc equal to 0 in (7) and (iii) with bothtypes of constraints. The comparison with the state-of-theart is particularly diﬃcult in our case, since there is no existing methodology for head and body pose estimation using
multimodal data. That is why we chose to compare with thestate-of-the-art on visual-only HBPE [9]. Notably, the conversational ﬂavor of SALSA is challenging for [9], since whenanalyzing FCG the direction of motion cannot be considered
as a proxy for the body pose, as done in [9]. Moreover, giventhat the proposed learning methodology belongs to the classof transductive learning approaches, and [9] does not, we
perform additional comparisons with Transductive Support
Vector Machines (TSVM) [23]. It is worth noting that neglecting the coupling and the temporal consistency terms
in MC-HBPE, i.e. τb , λc , τh = 0 in (7), is similar to thework of Goldberg et al. [21]. Therefore, MC-HBPE could
be seen as a generalization of the transductive methodology
presented in [21]. Results in Table 1 demonstrate the ad-

Figure 5: Frame example: (left) the original imageof one of the cameras, (right) the head–green andbody–red pose estimates plot in bird view.
vantageous performance obtained when enforcing temporalconsistency or coupling the head and body pose estimates.
Importantly, the eﬀect of both regularizations is impressive,increasing by approximately 20% the classiﬁers’ accuracy.
Our understanding, is that this is the main reason for which
the complete method we presented outperforms the baselines up to a large extent. In order to show the complexityof the task, we show the original image of one of the cameras
and the estimated poses plot in a bird’s eye view in Figure 5.
Finally, we evaluate the robustness of the approach to theamount of training and of manually annotated data in Figure 6, where we remark the 40% accuracy obtained with
only 1% of manually annotated data (plus the noisy labelsfrom the sociometric badges).

4.4

F-formation detection

One of the prominent applications of robust estimation of
the head and body poses when analyzing FCGs is the detection of F-formations. We evaluate three state-of-the-art
approaches for F-formation detection, namely: the Houghvoting method (HVFF-lin) in [11], its multi-scale extension (HVFF-ms) [33] and the graph cut approach in [34].
We chose these methods because their implementation is
publicly available2 . All these approaches compute the Fformation independently frame by frame, from the targets’
position and pose. The rationale behind the Hough voting
methods is that each target generates a set of samples infront of him/her, which are candidates of the F-formation
centers. By aggregating these samples into a discrete number of cells, F-formations are detected by ﬁnding local maxima in the discretized space. Conversely, in the graph-cut
algorithm [34], an optimization problem is solved, discovering directly the coordinates of the F-formations.
Previous works [22] have acknowledged the importanceof accurate body pose estimates for detecting F-formations.
However, one of the inherent diﬃculties within the analysis
of FCGs in crowded indoor environments is the adverse im2

http://profs.sci.univr.it/~cristanm/ssp/

No	
  Coupling,	
  No	
  Temporal	
  Consistency	  Temporal	  Consistency	
  Coupled	  head/body	
  	
  
Full	
  Model	  Table 2: Performance of state-of-the-art approaches
on F-formation detection from diﬀerent estimationsof the head and body pose. Each table cell contains
precision, recall and F1-measure, in this order.

0.6	
  

���������

0.55	
  

0.45	
  
0.4	
  

�����

0.35	
  
0.3	
  
0.25	
  
���������

���������

���������

����������������������������������������������������
0.65	
  

���������

0.6	
  
0.55	
  
0.5	
  
0.45	
  

�����

0.4	
  
0.35	
  
0.3	
  
0.25	
  
10%	
  [1%]	
  20%	
  [2%]	
  50%	
  [5%]	
  ����������������������������������������������������

Figure 6: Classiﬁcation accuracy for diﬀerent percentage of training samples and of manually annotated samples (in square brackets).
pact of highly occluded body images, which empower strong
noise and outliers in the estimated body poses. Classically,
the head pose is used as a surrogate to the body pose, andthe F-formation are detected from the head pose. In Section 4.3, we demonstrate that the proposed MC-HBPE approach can be eﬀectively used to accurately infer the bodypose, thus making possible to use the body pose for Fformation detection and therefore improving the quality of
the analysis of FCGs.
We implement the validation protocol deﬁned in [11], andreport F-formation precision, recall and F1-measure in Table 2. At each frame, we consider a group as correctly estimated if at least T · |G| of the members are correctly found,and if no more than 1 − T · |G| non-members are wrongly
identiﬁed, where |G| is the cardinality of the group G, andT = 2/3. Table 2 shows the performance measures using
body (ﬁrst three rows) and head (fourth and ﬁfth rows) pose.
We compare the estimates obtained with MC-HBPE to thebody pose ground truth and the body estimates computedwith [9]. Regarding the head pose, in addition to our approach, we also used the transductive SVM approach in [23]to emulate previous works on F-formation detection thatused head pose estimates. The results clearly conﬁrm that
accurate inference of the body pose using the MC-HBPE
framework is advantageous for the detection of F-formations.

4.5

HVFF lin [11]

HVFF ms [33]

GC [34]

Our-B
[9]-B
GT-B

0.62 / 0.61 / 0.62
0.56 / 0.58 / 0.58
0.66 / 0.65 / 0.65

0.66 / 0.64 / 0.66
0.59 / 0.62 / 0.61
0.72 / 0.69 / 0.71

0.69 / 0.66 / 0.67
0.62 / 0.61 / 0.62
0.74 / 0.72 / 0.73

Our-H[23]-H
GT-H

0.61 / 0.59 / 0.60
0.58 / 0.58 / 0.58
0.65 / 0.63 / 0.64

0.64 / 0.63 / 0.63
0.59 / 0.58 / 0.59
0.70 / 0.67 / 0.69

0.65 / 0.64 / 0.64
0.62 / 0.59 / 0.61
0.72 / 0.70 / 0.71

0.5	
  

Retrieving Social Attention Attractors

Automatically discovering attentional patterns from multimodal data is an unexplored area of research. Interestingly, retrieving the person who holds the attention of theother members in the group can be helpful for other higherlevel tasks. In the short-term it can be used to describe
who is leading the conversation. In the mid-term, it can describe the role of the person during the interaction. And in

the long-term it is related to personality. In this particularstudy we are interested in interactions that last several minutes, thus we focus on role analysis. In practice, we analyze
the scientiﬁc poster session of SALSA to automatically ﬁndout who are the poster presenters.
In order to do that we introduce the ﬁrst computationalmodel for the social attention holding power (SAHP), which
is deﬁned as “the ability to direct favorable attention to theself” [20]. Intuitively, in our scenario the people with high
SAHP are likely to be the sought poster presenters, or someone interacting with the presentation e.g. asking a question.
Within the context of FCGs, SAHP can be computed as thepercentage of the F-formation members that are looking towards a particular person.
We consider that if person k is looking to person l, thehestimated head pose of person k at time t (θ̂tk) lies withinmaxdegrees of the direction from k to l at time t. This proθvides us a binary vector Akl of size T , in which (Akl )t is one
if k is looking at l at time t and zero otherwise. However thisbinary matrix is insuﬃcient, since we are interested in theability of holding the attention. Therefore, we consider thatholding someone’s attention for more than TN = 5 seconds
is positive, but holding it for less than the same amount,meaning that the attention was not held long enough, isnegative. These allows us to split the attentional periods
into two groups Tkl+ ⊂ {t|(Akl )t = 1} the temporal frameswith an ongoing long held attention (positive SAHP) andTkl+ ⊂ {t|(Akl )t = 1} the temporal frames with an ongoingshort held attention (negative SAHP). Formally, the social
attention holding power of person l is computed as:
SAHPl =

T
�
�
1 � �
I(t ∈ Tkl+ ) − I(t ∈ Tkl− ) ,
|G
tl |
t=1

(18)

k∈Gtl

where I is the indicator function and Gtl is the F-formation
to which person l belongs at time t.
In the addressed scenario, however, part of the attention
is held by the posters themselves. Intuitively, the attention
given to the poster corresponds to the poster presenter orto an attendee who just took part into the discussion (e.g.
asking a question). Therefore, we add the attention given to
the poster after participation of person l, to the previously
computed SAHPl . Moreover, since we know that there are
four posters, we seek the four highest values of SAHP, andwe refer to these participants as social attention attractors.
We performed an extensive set of experiments to systematically evaluate the robustness of the proposed computational formulation of SAHP against errors in the estimation of the head pose and the detection of F-formations.
First, we computed the attention vectors Akl with an angular threshold of θmax ∈ {5◦ , 10◦ , 15◦ , 20◦ }, and the positive

Table 3: Accuracy results of the social attention attractor detector built from the proposed computational model for the SAHP.
Parameters

Simulated Data

Real Data

W

θmax

5◦

20◦

55◦

HP

F-f.

Both

Pos

1�

5◦
10◦
15◦
20◦

0.92
0.92
0.94
0.95

0.77
0.82
0.87
0.87

0.74
0.76
0.78
0.82

0.71
0.72
0.75
0.75

0.90
0.92
0.90
0.90

0.71
0.73
0.73
0.75

0.73
0.70
0.70
0.70

2�

5◦
10◦
15◦
20◦

0.93
0.92
0.94
0.96

0.78
0.83
0.89
0.90

0.77
0.77
0.83
0.87

0.70
0.70
0.77
0.79

0.91
0.88
0.92
0.92

0.68
0.69
0.76
0.79

0.70
0.69
0.71
0.74

5◦
10◦
15◦
20◦

0.96
0.96
0.96
0.98

0.81
0.88
0.94
0.93

0.82
0.76
0.85
0.88

0.70
0.76
0.79
0.79

0.89
0.91
0.96
0.98

0.68
0.72
0.78
0.77

0.71
0.67
0.69
0.76

4

�

and negative held attentional periods from them. Thirdly,
we calculated the SAHP within a sliding window of lengthW ∈ {1, 2, 4} minutes. Finally we compare the retrieved social attention attractors with the ones obtained applying thesame procedure to the ground truth data. This evaluation
procedure is applied to simulated and to real data. In theformer case, we corrupted the head pose ground truth values with uniform random noise of amplitude 5◦ , 20◦ and 55◦ .
In the latter case, we evaluate the eﬀect of the estimation
procedure of the proposed MC-HBPE together with the Fformation detection state-of-the-art graph-cut method [34](since it provided the best performance with our data). Finally, we compare our computational model to the straightforward strategy of selecting the person closest to the poster.
We report the accuracy for all combinations of the parameter values in Table 3. The two group columns correspond to the experiments with simulated and real data
respectively. In the case of real data we consider three subcases: when only the head pose is estimated (HP), when
only the F-formation is estimated (F-f.) and when both are
estimated (Both). Last single column corresponds to thestrategy of selecting the person closest to the poster (Pos).
We ﬁrst notice that, in the case of simulated data, the noisenotably decreases the performance of the social attention
attractor detector: in severe noise conditions, the accuracy
drops down roughly to 0.75. Importantly, we also remarkthat part of this performance loss can be recovered if wechoose the right value for the parameters. Indeed, in severe noise conditions, the right choice of parameters makethe performance increase by 0.14. Moreover, the eﬀect ofthe parameter choice in the robustness against noise is intuitive. On the one side, when the integration window W
is longer, we expect short errors to be smoothed out. On
the other side, a larger angular threshold allows to accountfor low-accuracy estimates of the head pose. We performedexperiments with larger values of W (up to 30� ) and of θmax
(up to 40◦ ), and the accuracy results follow the same trend.
Regarding the real data, we remark that the eﬀect of estimating the head pose is more critical than detecting theF-formations. In addition, the proposed model outperforms
the Pos strategy probably because: (i) the person closest to
the poster is not necessarily the social attractor at a given
time (ii) the poster presenter is not systematically the social
attractor. The results conﬁrm this fact since the accuracy ofthe head estimates is lower than the accuracy of the detected

F-formations. Remarkably, the eﬀect of the parameters follows the same trend as in the case of simulated data. At
the light of this results we conclude that, there is still room
for improvement in the estimation of the head and bodyposes within FCGs in crowded environments. Furthermore,since robust estimation of head and body pose is useful forhigher-level analysis, and no previous methodologies exist to
fuse multimodal data ﬂows emanating from a combinationof distributed and wearable sensor networks, we believe it isworth pursuing further this line of research.

5.

CONCLUSIONS AND FUTURE WORKThe focus of this research study is the analysis of freestanding conversational groups from multimodal estimatesof head and body pose. We propose a framework able to (i)combine data emanating from a distributed camera networkand from sociometric badges worn by each participant, (ii)fuse data from diﬀerent modalities to boost the estimationof head and body pose, (iii) alleviate the annotation process
thanks to the multimodal labels provided by the sociometricbadges, (iv) handle noisy features and labels by means of thematrix completion framework, (v) account for the temporalconsistency and head/body coupling inherent to the scenarioand (vi) provide robust estimates of the head and body posesthat have proven to be useful for higher-level tasks, such as
the detection of F-formations and the discovery of social
attention attractors.
At the light of the results obtained through the extensive
experimental evaluation of the presented methodology, we
believe that this study opens the door to several interesting
future research directions. First of all, the links with otherrelated tasks should be exploited in a bidirectional way. Indeed, the estimation of the head and body poses can ease
the tracking of multiple people. Similarly, the recognitionof F-formations and discovery of social attention attractors
can help the estimation of head and body poses. Second,
the spectrum of applications should be enlarged to address
personality recognition. Importantly, the analysis of personality traits could require longitudinal studies (i.e. extend therecordings to days or weeks), and the current methodology
should be adapted to this scenario. Finally, from the technical point of view, one could focus on speeding up the current
MC-HBPE algorithm and develop incremental matrix completion strategies, so to adapt the proposed methodology toreal-time processing.

Acknowledgments
This work has been partially supported by the MIUR Cluster project Active Ageing at Home, the EC project ACANTO
and the A*STAR Singapore Human-Centered Cyber-physical
Systems (HCCS) grant.

6.

REFERENCES[1] X. Alameda-Pineda and R. Horaud. Vision-guided
robot hearing. IJRR, 34(4-5):437–456, 2015.
[2] X. Alameda-Pineda, J. Staiano, R. Subramanian,
L. Batrinca, E. Ricci, B. Lepri, O. Lanz, and N. Sebe.
Salsa: A novel dataset for multimodal group behavior
analysis. arXiv:1506.06882, 2015.
[3] B. Benfold and I. Reid. Unsupervised learning of ascene-speciﬁc coarse gaze estimator. In ICCV, 2011.

[4] S. Boyd, N. Parikh, E. Chu, B. Peleato, and
J. Eckstein. Distributed optimization and statisticallearning via the alternating direction method ofmultipliers. Foundations and Trends in Machine
Learning, 3(1):1–122, 2011.
[5] R. Cabral, F. De la Torre, J. Costeira, and
A. Bernardino. Matrix completion for multi-label
image classiﬁcation. TPAMI, 37(1):121–135, 2014.
[6] A. T. Campbell, S. B. Eisenman, N. D. Lane,
E. Miluzzo, and R. A. Peterson. People-centric urban
sensing. In Int. Work. on Wireless Internet, 2006.
[7] E. J. Candès and T. Tao. The power of convex
relaxation: Near-optimal matrix completion. IEEE
Trans. Inf. Theory, 56(5):2053–2080, 2010.
[8] C. Canton-Ferrer, C. Segura, J. R. Casas, M. Pardas,
and J. Hernando. Audiovisual head orientation
estimation with particle ﬁltering in multisensor
scenarios. EURASIP Jour. on Adv. in Sig. Proc., 2008.
[9] C. Chen and J.-M. Odobez. We are not contortionists:
coupled adaptive learning for head and bodyorientation estimation in surveillance video. In CVPR,
2012.
[10] T. Choudhury and A. Pentland. Sensing and modelinghuman networks using the sociometer. In Inter. Sym.
on Wearable Comp., 2003.
[11] M. Cristani et al. Social interaction discovery bystatistical analysis of F-formations. In BMVC, 2011.
[12] N. Dalal and B. Triggs. Histograms of oriented
gradients for human detection. In CVPR, 2005.
[13] F. De la Torre et al. Guide to the Carnegie Mellon
University multimodal activity (CMU-MMAC)
database. Technical report, 2009.
[14] M. Demirkus, D. Precup, J. J. Clark, and T. Arbel.
Probabilistic temporal head pose estimation using ahierarchical graphical model. In ECCV, 2014.
[15] T. M. T. Do and D. Gatica-Perez. Human interaction
discovery in smartphone proximity networks. Personaland Ubiquitous Computing, 17(3):413–431, 2013.
[16] N. Eagle and A. Pentland. Reality mining: sensing
complex social systems. Personal and Ubiquitous
Computing, 10(4):255–268, 2006.
[17] S. Escalera, X. Bar, J. Gonzlez, M. A. Bautista,
M. Madadi, M. Reyes, V. Ponce, H. J. Escalante,
J. Shotton, and I. Guyon. Chalearn looking at people
challenge 2014: Dataset and results. In ECCV
Workshops, 2014.
[18] D. Gatica-Perez. Automatic nonverbal analysis ofsocial interaction in small groups: A review. Imageand Vision Computing, 27(12):1775–1787, 2009.
[19] X. Geng and Y. Xia. Head pose estimation based on
multivariate label distribution. In CVPR, 2014.
[20] P. Gilbert, J. Price, and S. Allan. Social comparison,social attractiveness and evolution: How might they
be related? New Ideas in Psychology, 13(2), 1995.
[21] A. Goldberg, B. Recht, J. Xu, R. Nowak, and X. Zhu.
Transduction with matrix completion: Three birds
with one stone. In NIPS, 2010.
[22] H. Hung and B. Kröse. Detecting F-formations as
dominant sets. In ICMI, 2011.
[23] T. Joachims. Transductive inference for textclassiﬁcation using SVM. In ICML, 1999.

[24] V. Kalofolias, X. Bresson, M. Bronstein, andP. Vandergheynst. Matrix completion on graphs.
arXiv:1408.1717, 2014.
[25] N. Krahnstoever, M.-C. Chang, and W. Ge. Gaze andbody pose estimation from a distance. In AVSS, 2011.
[26] O. Lanz. Approximate bayesian multibody tracking.
TPAMI, 28:1436–1449, 2006.
[27] F. Lingenfelser, J. Wagner, E. André, G. McKeown,
and W. Curran. An event driven fusion approach forenjoyment recognition in real-time. In ACMMM, 2014.
[28] A. Matic, V. Osmani, A. Maxhuni, and O. Mayora.
Multi-modal mobile sensing of social interactions. In
PervasiveHealth, 2012.
[29] M. Pantic, N. Sebe, J. F. Cohn, and T. Huang.
Aﬀective multimodal human-computer interaction. In
ACMMM, 2005.
[30] S. Petridis, B. Martinez, and M. Pantic. The mahnob
laughter database. Image and Vision Computing,31(2):186–202, 2013.
[31] A. K. Rajagopal, R. Subramanian, E. Ricci, R. L.
Vieriu, O. Lanz, and N. Sebe. Exploring transfer
learning approaches for head pose classiﬁcation frommulti-view surveillance images. IJCV,
109(1-2):146–167, 2014.
[32] N. Robertson and I. Reid. Estimating gaze direction
from low-resolution faces in video. In ECCV, 2006.
[33] F. Setti, O. Lanz, R. Ferrario, V. Murino, and
M. Cristani. Multi-scale F-formation discovery forgroup detection. In ICIP, 2013.
[34] F. Setti, C. Russell, C. Bassetti, and M. Cristani.
F-formation detection: Individuating free-standing
conversational groups in images. arXiv:1409.2702,
2014.
[35] J. Shotton et al. Real-time human pose recognition inparts from single depth images. Communications of
the ACM, 56(1):116–124, 2013.
[36] Y. Song, L.-P. Morency, and R. Davis. Multimodal
human behavior analysis: learning correlation andinteraction across modalities. In ICMI, 2012.
[37] A. Vinciarelli, A. Dielmann, S. Favre, and H. Salamin.
Canal9: A database of political debates for analysis ofsocial interactions. In ICACII, 2009.
[38] M. Voit and R. Stiefelhagen. 3d user-perspective,
voxel-based estimation of visual focus of attention in
dynamic meeting scenarios. In ICMI, 2010.
[39] B. Wu, S. Lyu, B.-G. Hu, and Q. Ji. Multi-label
learning with missing labels for image annotation andfacial action unit recognition. Pattern Recognition,
2015.
[40] L. Wu, R. Jin, and A. K. Jain. Tag completion forimage retrieval. TPAMI, 35(3):716–727, 2013.
[41] Y. Yan, E. Ricci, R. Subramanian, O. Lanz, and
N. Sebe. No matter where you are: Flexible
graph-guided multi-task learning for multi-view headpose classiﬁcation under target motion. In ICCV,2013.
[42] Y. Yan, E. Ricci, R. Subramanian, G. Liu, and
N. Sebe. Multitask linear discriminant analysis forview invariant action recognition. IEEE Transactions
on Image Processing, 23(12):5599–5611, 2014.