Social Processes: Self-Supervised Forecasting of Nonverbal Cues in Social Conversations
Authors: Chirag Raman
Hayley Hung
Marco Loog
Venue: NeurIPS 2021 Submitted (2021)
Keywords: Social Behavior Forecasting
Situated Interactions
Neural Processes
Free-standing Conversations
Self-supervised Learning

Abstract: The default paradigm for the forecasting of human behavior in social conversations is characterized by top-down approaches. These involve identifying predictive relationships between low level nonverbal cues and future semantic events of interest (e.g. turn changes, group leaving). A common hurdle however, is the limited availability of labeled data for supervised learning. In this work, we take the first step in the direction of a bottom-up self-supervised approach in the domain. We formulate the task of Social Cue Forecasting to leverage the larger amount of unlabeled low-level behavior cues, and characterize the modeling challenges involved. To address these, we take a meta-learning approach and propose the Social Process (SP) models—socially aware sequence-to-sequence (Seq2Seq) models within the Neural Process (NP) family. SP models learn extractable representations of non-semantic future cues for each participant, while capturing global uncertainty by jointly reasoning about the future for all members of the group. Evaluation on synthesized and real-world behavior data shows that our SP models achieve higher log-likelihood than the NP baselines, and also highlights important considerations for applying such techniques within the domain of social human interactions.

Social Processes: Self-Supervised Forecasting of Nonverbal Cues in Social Conversations
Anonymous Author(s)
Affiliation
Address
ema i
Abstract
The default paradigm for the forecasting of human behavior in social conversations is characterized by top-down approaches. These involve identifying predictive relationships between low level nonverbal cues and future semantic events of inter- est (e.g. turn changes, group leaving). A common hurdle however, is the limited availability of labeled data for supervised learning. In this work, we take the first step in the direction of a bottom-up self-supervised approach in the domain. We formulate the task of Social Cue Forecasting to leverage the larger amount unlabeled low-level behavior cues, and characterize the modeling challenges in- volved. To address these, take a meta-leaming approach and propose the Social Process (SP) models-sociall, aware sequence-to- toweence-t--seuencee (Seq2Seq) models within the Neural Process (NP) family. SP models learn extractable representations of non-semantic future cues for each participant, while capturing global uncertainty by jointly reasoning about the future for all members of the group. Evaluation on synthesized and world behavior data shows that our SP models achieve higher log- ligeelklioood than the NP baselines, and also highlights important considerations for applying such techniques within the domain of social human interactions.
2
11
13
15
Introduction
17
18
Picture a situated interactive agent such as a social robot conversing with a group of people. How agents act in such setting? We sustain conversations spatially and temporally through explicit behavioral cues examples include locations of partners, their orientation, gestures, gaze, and floor control actions [1-3]. Evidence suggests that employ anticipation of these and other cues to navigate daily social interactions [1, 4-8]. Consequently, the ability to forecast the future constitutes natural objective towards the realization of machines with social skills. As such, interactive agents
20
22
23
24 typically contend with uncertainties in inferences surrounding cues [3]. So beyond making real-time
25 inferences. such systems may achieve more fluid interactions by leveraging the ability to forecast future states of the conversation [9].
28
In addition the development of social agents, behavior forecasting is also of significance in social psychology, where the focus on gaining insight into human behavior. Since human- Jaman-interpretabbliyy of essence, top-down approaches largely constitute the default paradigm, where specific events of semantic interest are selected first for consideration and their relationship to potentially predictive cues studied in isolation-eti either in controlled interactions in lab settings, in subsequent statistical analyses [10, 11]. Examples of such semantic events include speaker turn transitions [5, 12, 13), mimicry episodes [14], the termination of interaction [9, 15]. However, one hurdle in the tow-down paradigm is limited data. The events (that constitute the labels or the dependent variables) often occur infrequently over longer interaction, reducing the effective amount of labeled data. This precludes the use of neural supervised learning techniques that tend to be data intensive.
28
30
31
32
33
34
35
38
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

a.
group leaving
b.
01
tobs
f1
fT
tut
Figure 1: Conceptual illustration of forecasting approaches on an in-the-wild conversation from the Match- NMingle dataset [16]. a. The top-down approach entails predicting semantic event or action interest for the observed window tobs = [ol.. oT' ool'.. Here we illustrate group leaving [15]; the cireled individual in the center leaves group in the future. b. In contrast, we propose bottom-up approach in the social conversation forecasting domain through the task of Social Cue Forecasting. This entails using the non-semantic low-level cues over tobs to regress the same cues over future window Eute == [fl... ... . JT]. the cues of head pose (solid normal), body pose (hollow normal), and speaking status (speaker in orange). The hypothetical uncertainty estimates over thut are also depicted as shaded spreads.
this example we depict
37 In this work, we take an initial step towards bottom-up approach forecasting human behavior for
38 free standing conversational groups. Our guiding motivation is to learn predictive representations of
39 general future social behavior by utilizing unlabeled streams low-level behavioral features. We do this by regressing future sequences of these features from observed sequences of the same features in
41 a self-supervised manner. We term this task of non-semantic future behavior forecasting as Social Cue Forecasting (SCF).
43
Our approach is built on the observation that the social signal [17] the high-level attitudes and social meaning transferred in interactions-is already embedded in the low-level cues [18]. To conceptually illustrate the contrasting top-down and bottom-up approaches on an example task, Figure depicts an instance of group leaving event in naturalistic social conversation. Evidence suggests that such events can be anticipated from certain preceding rituals [15] reflected in the postural changes of conversing members [1]. van Doom [15] built predictor using 200 instances of group leaving found in over 90 minutes of mingling interaction and hand-crafted features. In contrast, our bottom-up approach would entail learning task agnostic representations of future behavior using the entire 90 minutes of data, and then training simpler predictors for group leaving using the learnt representations input. The figure also illustrates the complexity of naturalistic interactions where cross- cross-group social influence exists. In this work we focus on the simpler setting of a single group in a scene.
44
45
46
47
49
51
53
54
There are several challenges intrinsic to computationally modeling future behavior in social conversa- 55 tions. The future is intrinsically uncertain, the forecasts for interaction partners are inter-dependent, and the social dynamics is unique for each grouping of individuals. We address these through the following contributions:
56
57
58
We formalize the task of SCF. We characertize the modeling challenges involved, and cast the problem into the meta-learning paradigm, allowing for data-efficient generalization to unseen groups at evaluation without learning group-specific models.
59
60
61
We propose and evaluate two socially aware within the Neural Process (NP) family [19] for SCF in social conversations. Our method encodes complex social dynamics informative of future group behavior into extractable representations for each individual.
(Seq2Seq) models
82
63
64
85
This paper is organized as follows. In Section 2 we formally define and characterize the task of SCF. We situate this work within broader literature in Section 3, and review background concepts in Section 4. We propose the Social Process models in Section 5 and describe our experiments in Section 6, concluding with discussion of our findings in Section 7.
67
2 Social Cue Forecasting
70 The objective of SCF to predict future behavioral cues of all people involved in a social encounter given an observed sequence of their behavioral features. More formally, let denote window

---

..... 72 of observed timesteps as tobe = [ol, o2.., .T), and an unobserved future time window as
73 tha == [fl, f2.... fT], fl > oT. Note that tiut and tabs are typically con-overlapping, can be of
74 different lengths, and frut need not immediately follow Lobs- Given a set of n interacting participants,
75 let us denote their social cues over a labs and tfut respectively as
X = b E tobe]==1 Y = [b]; E turl-=1
(Ia, b)
76 The vector by encapsulates the multimodal cues of interest from participant i at time 1. These can
77 include head and body pose, speaking status, facial expressions, gestures, and verbal content-any information stream that combine to transfer social meaning.
78
79 In its simplest form, given X, the objective of SCF is to learn single function f such that
80 Y = f(X). However, an inherent challenge in forecasting behavior is that an observed sequence
81 of interaction does not have a deterministic future and result in multiple socially valid ones window of overlapping speech between people both may and may not result in a change of speaker
82
[12, 20], change in head orientation may continue into sweeping glance across the room a darting
84
glance stopping at a recipient of interest [21]. In some cases certain observed behaviors- Sllaviors-iitonatiinn intonation and gaze cues [5, 13] synchronization in speaker-listnner speech [22] for turntaaking-night make some outcomes more likely than others. Given that there are both supporting and challenging arguments for how these observations influence subsequent behaviors [22, p. 5; 13, p. 22), it would be beneficial if a data-driven model expresses a measure of uncertainty in its forecasts. We do this by modeling the distribution over possible futures (X) rather than forecasting a single future.
85
86
87
88
89
Another design consideration arises from a defining characteristic of focused interactions- the participants' behaviors interdependent. Participants in group sustain equal access to the shared interaction space through cooperative maneuvering [1, p. 220]. Moreover, when multiple groups co- co-located, outsiders unengaged in these intra- group maneuvers may also influence the behavior 94 of those within the group [23, p. 91;1, P. 233], sometimes causing them to leave (see Figure 1). It
90
91
92
93
85 is therefore essential to capture uncertainty in forecasts at the global level-jointly forecasting one future for all participants at a time, rather than at a local output level-one future for each individual independent of the remaining participants' futures.
96
97
96
How participants coordinate their behaviors is a function of several individual factors [24, Chap. I; I, p. 237]. Consequently, the social dynamics guiding interaction also has unique attributes for every unique grouping of individuals. Rather than learning group-specifee models to capture these unique dynamics, we formulate the forecasting problem in terms of meta-learning, or few-shor function estimation. We interpret each unique group of individuals as the meta-learning notion of a task. The
99
100
102
103
core idea is that we can learn to predict distribution over futures for target sequence X having captured the group's unique behavioral tendencies from a context set C of their observed-future sequences. We can then generalize to unseen groups at evaluation by conditioning
104
105
short observed slice of their interaction. We believe that this approach is especially suitable for social conversation forecasting-a setting that involves limited data regime where good uncertainty estimates are
108
107
108
desirable. Note that when conditioning on context is removed (C = Ø), we simply revert to the formulation p(Y ).
109
3 Related Work
110
111
Free-standing conversations are an example of what social scientists call focused interactions, said to arise when a "group of persons gather close together and openly cooperate to sustain single focus of attention, typically by taking turns at talking" [23, 24]. A long long-sianding topic of study has been the systematic organization of turn-take taking [25-27]. with a particular interest in the event of upcoming
112
113
114
115
speaking turns [5-8]. There has also been some interest in the forecasting task itself, to anticipate disengagement from an interaction [9, 15], the splitting or merging of groups [28], the time-evolving
116
117
size of a group [29] or semantic social action labels [30, 31]. Most of these works use heuristics, either to generate semantic labels [9], model the dynamics itself [29], or hand-craft features [15].
118
119
Although not forecasting task, the closest work that shares our motivation in predicting non-semantic
120
lewelevel features the recently introduced task of Social Signal Prediction (SSP) [32]. The objective
121
is to predict the social cues' of target person using cues from the communication partners as
IIn the domain of Social Signal Processing, social signal [17] refers to the relational attitudes displayed by people. It is high-level construct resulting from the perception of cues (see Fig. 1 in [18). From this
3

---

122
input (Joo et al. focus on predictions within the same time window [32, Eq. 6)). While the most general formulation of SSP involves forecasting a single timestep for a target person given the
123
124
entire group's 's past behavior [32, Eq. 3]. generalizing this formulation runs into an inherent problem; applying the definition to forecasting entails iteratively treating each individual as target, learning separate functions for every person. However, as we discuss in Section 2, these futures of interacting individuals are not independent given observed group behavior. Furthermore, constrained definition of forecasting that predicts immediate step into the future is limiting, since forecasting an event that occurs after a delay (e.g. a time lagged synchrony [33] or mimicry [14] episode) might be of interest. Operationalizing this definition would entail sliding window iteratively using predictions over the offset between tobs and tut as input, which would cascade prediction errors.
125
126
127
129
131
132 A related social setting where forecasting has been of interest is that of unfocused interactions. These occur when individuals find themselves by circumstance in the immediate presence of each other, such as pedestrians walking in proximity. Early approaches for forecasting pedestrian trajectories were heuristic based, involving hand-crafted energy potentials to describe the influence pedestrians have on each other [34-41]. More recent approaches encode the relative positional information directly into a neural architecture [42-46].
133
134
135
137
138
In a broad sense, the self-supervised learning aspects of this work has some overlap with recent approaches focusing on the non non-interaction task of visual forecasting. These works have taken a
139
140
non-semantic approach to predict low level pixel-based features or intermediate representations
141
[38, 47-52], and demonstrated a utility of the learned representation for other tasks like semi- supervised classification [53], or training agents in immersive environments [54].
142
4 Preliminaries
144
Meta-learning. A supervised learning algorithm can be viewed as a function mapping a dataset C= (Xc, Yc)= == ((0;, [eNNe to predictor f((z). Here No is the number of datapoints in
145
148
C. and [No] := .Nc). The key idea of meta-lcarning is to learn the learning process itself, modeling this function representing the initial algorithm using another supervised learning algorithm;
147
148
hence the name meta-learning. In meta-learmin literature, a task refers to each dataset in collection
149
M- = (T.) i=1
Nupk: of related datasets [55]. For each task T, a meta-learner is episodically trained to fit subset of target points D = () {Ix, [e) given another subset of context observations C. At meta-test time, the resulting predictor f(x, C) uses the information obtained during meta-learning to make predictions for unseen target points conditioned on context sets unseen meta-training.
150
151
152
153
154
Neural Processes Sharing the same core motivations, NPs are a family of latent variable models that extend the idea of meta-learning to situations where uncertainty in the predictions fax C)) are desirable. They do this by meta-learning a map from datasets to stochastic processes, estimating a distribution over the predictions p(Y [X, C'). To capture this distribution, NPs model the conditional latent distribution p( (C) from which task representation E Idd is sampled. This constitutes the model's 's latent path. The context can also be incorporated through a deterministic path, via a representation rc E Rd aggregated over C. An observation model p(ya (2;; TC, 2) then fits the target observations in D. The generative process for the NP is written as
155
156
157
158
159
160
161
p O ()= DOA.X.C.DEEOOE//XXX.........
C,
162
where p (Y rc, 2) = (lEE P(y, (T,, The latent is modeled by a factorized Gaussian parameterized by &c = (C). with fs being deterministic function invariant to order permutation over C. When the conditioning on context is removed (C = Ø), we have g(2[8>) == p(z), the zero-information prior on z. C is encoded on the deterministic path using a function f. similar to f.. so that re == fC). In practice this is implemented as rc = Eie[Ne] MLP(x;, p.)/Nc. The observation model is referred to as the decoder, and q, fr. fo comprise the encoders. The parameters of the NP are learned for random subsets C and D by maximizing the evidence lower bound (ELBO)
163
164
165
168
log p( Y X, C) 2 ( (log p X C, x)] (L(((>))D)) ( (=]sc)))
(3)
perspective, the task of Social Signal Prediction [32] is a misnomer since it still relates to the prediction of cues and not signals, a distinction we preserve in this work.

---

Process Encoder
Process Decoder
pocibl encoderg
rng decoder
social encoder
decoder
Sekernee mrmble Parth
Jatort Path
decoddr
Modial encodde
MNG diocoder
social incodier
cioooder
Rog pognggton
@1
Pintaddr
Figure Architecture of the SP ASP family.
Social Processes
169
170
In this section we present our socially aware Seq2Seq models within the NP family that is agnostic to group member identities and group size. To setup the task, we split the contextual interaction on which
171
172
we condition into pairs of observed and future sequences, writing the context as C' = (Xc, Yc) = (X,, , Ae) where every X, occurs before the corresponding Yr. As discussed in
173
Section 3, domain experts focusing on behavior analysis might be interested in settings where tobs and fot are offset by an arbitrary delay. Consequently, the jth bobs can have multiple associated brut windows. Denoting the set of target window pairs D (X,Y)= = (Xj, (e) our focus in the rest of this work is to model the distribution XYX,C))
175
178
The generative process for model we call the Social Process (SP) follows Eq. 2, which extend to social forecasting in two ways, We embed an observed sequence æ for an individual into a condensed encoding
179
IR that is then decoded into the future sequence using Seq2Seq architecture [56, 57]. Our intuition is that this would cause the representation to encode temporal information about the future. Further, for every individual we model this e as a function of their own behavior, and that of their partners as viewed by them. The intuition is that this captures the spatial influence partners have on the participant over the tubs- Using notation we established in Section we define the observation model for the SP for a single participant p.
180
181
182
183
184
185
(4) decoding is carried out in an auto-regressive manner, we can further write the right hand side of Eq. 4 as | p
ply''=² .) p(b b'y b bOr, C, x) p(b) byrle' (yrle',ro,2))
186
by e' e', rc,, x). Following the standard NP setting, we implement the observation model as a set of Gaussian distributions factorized over time and feature dimensions. We also incorporate the cross- coss-attention mechanism from the Attentive Neural Process (ANP) [58] to define the variant Attentive Social Process (ASP). Following Eq. 4 and the definition of the ANP, the corresponding observation model of the ASP for single participant is defined
187
188
189
190
191
ply xe C, = pbb), byrle', rorlee,r"(C rC, x x)
(5)
192
Here each target query sequence æi attends to the context sequences Xc to produce a query-specific representation r. == (C, x E Rdd The model architectures are illustrated in Figure 2.
193
Encoding Partner Behavior. solely a compact representation of the observed sequence, we'd like to condition an individual's forecast the observed behavior of both, themselves and their partners. We do this using pair of sequence encoders: one to encode the temporal dynamics of participant p, 's features, e = f and another to encode the dynamics of transformed representation of the features of p, partners, Gartner = Spartner ( ()) (Z),,)))))))) Using a separate network to encode partner behavior grants the practical advantage of being able sample individual's and partners' features at different sampling rates.
While typical Seq2Seq setup conditions the sequence decoder on
195
197
199
201
How do we model ( We want the partners' representation to possess two properties: per- mulation invariance chaninn changing the order of the partners should not affect the representation; and group size independence want to compactly represent all partners independent of the group size.
203
5

---

205
Beyond coordinate space invariance, we wish to intuitively capture a view of the interaction from
206
P/ perspective. We extend the approach Qi et al. [59] applied point clouds to focused interactions by computing pooled embeddings of relative behavioral features. Since most commonly considered nonverbal cues in literature (see Section 6.3) include the attributes of orientation or location (e-g. head/body pose or keypoints) or binary indicator (such as speaking status), we specify how we transform these. The 3D pose (orientation, location) of every partner pj is transformed to a frame of
207
208
209
210
reference defined by p:'s pose, At timestep f, denoting orientation, location, and binary speaking
status for P. as bj 1q'; ;;;1sss] s²), and those for Pý as b/ [q P"s]. we have
212
qida = q' (q")-1 rre = 1 - I", g ss
(6a-c)
213
Note that we use unit quaternions (denoted q) for representing orientation due their various benefits over other representations of rotation [60, Sec. 3.2]. The operator denotes the Hamilton product of the quaternions. These transformed features for each Pj are encoded using embedder MLP. The
214
216
outputs are concatenated with e and processed by pre-pooler MLP, which is followed by the symmetric element-wise Max-pooling function to obtain xxx>)) at each timestep. We capture the dynamics in the pooled representation over loss using fpartner. Finally, we combine eaetr and eparter for pj through linear projection (defined by a weight matrix W) to obtain the individual's embedding ind W partner] Our intuition is that with information about both themselves, and of
217
218
219
220
p.'s ss partners from p ss point-of-view, e now contains the information required to predict p; 's
222
future behavior.
223
Encoding Future Window Offset. As we've ve discussed at the start of this section, a single tobs might have multiple associated lus windows at different offsets. Our intuition that training a sequence decoder to decode the same enn into multiple sequences (corresponding to the multiple fut) in the absence of any timing information might cause an averaging effect in either the decoder or the information encoded in ennd- One way around this would be to start decoding one timestep following the end of tobs and discard the predictions in the gap between tobs and tiut. However, if decoding is done auto-regressively this might lead to cascading errors over the gap. Instead, we address this one-to-many issue by injecting the offset information into eind so that the decoder receives unique encoded representation for every tiut to decode over. We do this by repurposing the idea of sinusoidal positional encodings [61] to encode offsets rather than relative positions in sequences. For given tobs and tuu, and de- dedimmeniional eind define the offset At fl of, and the corresponding offset encoding OFst as
224
225
226
227
228
230
232
234
( = sin( 100002m/d.)) O.( = cos (40000/00000/m/44
(7a, b)
Here m refers to the dimension index in the encoding. We finally compute the representation e for Eqs. and as
236
e' e'= eind + OES..
237
Auxiliary Loss Functions. We incorporate geometric loss function that improves performance in
238
pose regression tasks. For ps at time t, given the ground truth bj = [q: s), and the predicted mean
b- qq: i we denote the tuple (b), b() Bi. We then have the location loss in Eucliden space and we can regress the quaternion values using
L1 (B) 1-1 -i
cq = q â
(9)
241 Kendall and Cipolla [60] show how these losses can be combined using the homoscedastic uncertain- 242 ties in position and orientation, of and o²:
L. (B:) (B:) exp(- + $ + LB(B:) exp() + 8qz
(10)
243 where 8 = log 82. Using the binary cross-entropy loss for speaking status La (B)), we have the 244 overall auxiliary loss over & E truti
Laux Laux 8) - C.B)).
(11)
245 The parameters of the SP and ASP are trained by maximizing the ELBO in Eq. and minimizing this auxiliary loss function for each of our sequence decoders.

---

Tarpet Contest Set, phaseu4.2
Thrnet hot Contest Set, phaseud
Table 1: Mcan (Std.) Negative Log- Likelihood (NLL) on the Haggling Test Sets. The reported mean and std. are over individual sequences in the test sets. Lower is better. The superscript" indicates best NI. within family, boldface hest overall.
Context
Random
Fixed-Initial
Baselines
NP-latent
38.34 (19.1)
37.61 (18.1)
NP-latent-dot
40.11 (23.9)
40.15 (23.0)
ANP-dot
35.00" (20.8)
38.00 38.06" (20.6)
ANP-multhhed
(19.2)
41.11 (19.2)
Ours (MLP)
SP-latent
74.06 (6.0)
-74199 (5.9)
SP-latent+det
77.49 (7.8)
76,90 (8.4)
ASP-dot
76633 (6.5)
73.15 (6.3)
ASP-multhead
83.77" (10.3)
83.43" (9.7)
Ours (GRU)
SP-Iatenl
4.233 (27.4)
-3.72 (30.7)
SP-latent+det
-17.38 (50.5)
-16.08* (52.2)
ASP-dot
19.91 (16.7)
31.39 (77.0)
Figure 3: Ground truths and model predictions for the toy task simulating the forecasting of glancing behavior.
ASP-multhhcad
-7.11 (26.9)
-0.51 (28.8)
Experiments and Results
247
248
6.1
Models and Baselines
249
Our modeling assumption is that the underlying stochastic process generating the behaviors does not evolve over time. Stated differently, we assume that the individual factors determining how participants coordinate ageev cultural background, personality variables [24, Chap.
250
251
are likely to remain the same over the short duration of single interaction. This is in contrast related line of work that deals with meta-transfer learning, where the stochastic process itself changes over time [62-65]. We therefore compare against the NP and ANP family which share our model assumptions and meta learning attributes. Note that in contrast to our methods, these baselines have direct access to the future sequences in the context, and therefore constitute strong baseline. We consider two variants: -latent denoting only the latent path; and -latent+det, containing both deterministic and stochastic paths. We further consider two attention mechanisms for the cross- attention module: -dor with dot attention, and -multihead with wide multi-heal attention [58]. We operationalize the original definitions of the baseline models sequences by collapsing the timestep and feature dimensions. While the ANP-RNN model [66] shares our model assumptions, locations, and cannot be operationalized to forecasting in any simple way (see Section 3 discussing the distinction). We experiment with two choices of architectures for the sequence encoders and decoders in our proposed models: multi-layer perceptrons (MLP), and Gated Recurrent Units (GRU). Implementation and training details for our experiments can be found Appendix C.
253
255
257
259
261
defined for task analogous SSP for concurrent
263
264
285
286
287
6.2
Evaluation on Synthesized Behavior: Forecasting Glancing Behavior
288
With limited behavioral data availability, a common practice in the domain is to train and evaluate methods on synthesized behavior dynamics [31, 67]. In keeping with this practice,
289
construct
270
synthesized dataset simulating two glancing behaviors in social settings [21]. We use 1D sinusoid
271
represent horizontal head rotation over 20 timesteps. The sweeping Type glance is represented by
272
pristine sinusoid, while the gaze fixating Type III glance is denoted by clipping the amplitude for
273
the last six timesteps. The task to forecast the signal over the last 10 timesteps (Ira) by observing the first 10 (f.rs). Consequently, the first half of lut is certain, while uncertainty over the last half
275
results from every observed sinusoid having two ground-truths. It is impossible infer from an
observed sequence alone if the head rotation will stop partway through the future. We describe
7

---

277
additional data setup, model details, and quantitative results for this setting in Appendices A. C and D.I, respectively. Figure 3 illustrates the ground truths. predicted means and std. deviations for a sequence within and outside the context set. We observe that all models estimate the mean reasonably well, although our proposed SP models learn a slightly better fit. More erucially, the SP models especially the SP-GRU-Learn much better uncertainty estimates over the certain and uncertain parts of the future compared to the NP baseline.
278
280
282
283
6.3
Real-World Behavior: The Haggling Dataset
284
We also evaluate our models on real-world behavior data, using the Haggling dataset of triadic interactions [32]. Participants are engaged in an unscripted game where two sellers compete to sell a fictional product to a buyer who has to choose between the two. We use the same split of 79 training
285
286
287
sets (groups) and 28 test sets used by Joo et al. [32]. our experiments we consider the following social cues: head pose described by the 3D location of the nose keypoint and a face normal; body pose described by the location of the mid-point of the shoulders and a body normal; and binary speaking status. Apart from being the most commonly considered cues in computational analyses of such conversations [68-70], pose and turn taking are found to be crucial in the sustaining of conversation
289
291
292
[1, 12, 18]. We specify the dataset preprocessing details in Appendix D.2.
6.4
Evaluation
294
Context Regimes. We evaluate all models on two context regimes: random, and fixed-initial. The random regime follows the standard NP setting that the models are trained in. Context samples (sequence-pairs) are selected as a random subset target samples, so the model is exposed to behaviors from any phase of the interaction lifecycle. Here we ensure that batches contain unique toe to prevent any single observed sequence from dominating the aggregation of representations take 50% of the batch as context. In the fixed-initial context
296
298
over the context split. At evaluation, regime, we investigate how the model can generalize knowledge of group specific characteristics from observing the initial dynamics of an interaction where certain gestures and patterns are more distinctive [1, Chap. 6]. This matches what a social agent might face in a real- world scenario. Here
300
302
303
we treat the first 20% of the entire interaction as context, treating sequences from the rest as target.
304
Evaluation Metrics. We report the negative log-likelihood (NLL) log (Y | in Table 1 (computed by summing over feature dimensions and people, and averaging over timesteps). Beyond the NLL, we also report the error in the predicted means over test sequences in Table 2: mean-squared error (MSE) for the head and body keypoint locations; mean absolute error (MAE) in orientation
305
306
307
308
in degrees; and speaking status accuracy. Note that while the ground truth orientation normals are constrained in the horizontal plane, we don't 't constrain our predicted quaternions. We therefore report the absolute error in rotation in 3D. The reported mean and std. deviation of all metrics are over sequences in the test sets. We further report the metries for every timestep over tiut in Appendix A.2, and qualitative visualizations of the forecasts in Appendix B.
309
310
311
312
313
6.5 Ablations
314 Encoding Partner Behavior. Modeling the interaction from the perspective of each individual is a central idea in our approach. We investigate the influence of encoding partner behavior into individual representations r'n on the performance. We train the SP-latent+det GRU variant in two configurations: no-pool, where we do not encode any partner behavior; and pool-oT where we pool over partner representations only at the last timestep (similar to [44]). We choose the SP-GRU model since it achieves the best trade-off between minimizing NLL and forecasting cues consistent with human behavior. Both configurations lead worse NLL and location errors (Appendix A.3).
315
316
317
318
319
320
321
Deterministic Decoding and Social Encoder Sharing. Error gradients can flow back into our sequence encoders through two paths: from the final stochastic sequence decoder, as well as the deterministic decoders on the latent and deterministic paths. We investigate the effect of the determin-
322
323
324
istic decoders by training the SP-latent+det GRU model without them. We also investigate sharing a single social encoder between the Process Encoder and Process Decoder in Figure 2. We find that removing the decoders only improves lig-likelihood if the encoders are shared, and at the cost of
326
head orientation errors (Appendix A.3).
8

---

Table 2: Mean (Std.) Errors in Predicted Means over Sequences in the Haggling Test Sets. Lower is better for all metrics except for speaking status accuracy. indicates best measure within family, boldface hest overall.
Randou Contest
Fited-lattial Coutest
Head Loe. (cm)
Body Loc. (cm)
Head
Body Ovi
Spenking Accemey
Head
Rody Loc. (cm)
Head Orl. MARC22
Body Ocl.
Speoking Accrneyy
Baselimes
Niviatm
10.21 (8.5)
15.06 (6.1)
16.70 (13.8)
12:87 (13 T)
7877 (0 103)
13,85,00 (6:12
10:71 (5:T)
16.22 (14.1)
12:0 (13 9)
0.774' (0.21)
NPlatzutdeet
15.0
15.07 (7.2)
17.45 (15.3)
14.03 (30
0.715 (0
15:03 (7.0)
10.00 (7.5)
27.29 (15.0)
14.68 (16 73
0.701 (0.24)
ANPh-dee
80 (5.4)
22" (5.5)
15.48 (13.3)
56 (16.0)
500'
12.65 13. 9)
13 (6.0)
16, (13.7)
12.50 (17 (5)
0.717 (0:23)
ANP-miithadd
(T:1)
(3.3)
(20-4)
21))
(7:7)
(31.2)
(20 -33
(0.71)
Ours (MLP)
SPilaest
25.56 (10.1)
(13.55 (9)
03635
11.97 (1)
1M633" (8:9)
91:14 (83.5)
(83 53)
0.540
SPNlaienede:
31.98 (8.2)
36.53 (7-3)
(13.5)
91.39 (73.9
11071 (1
331553
16 (8.5)
91:15 (18.9)
81 11 (83-9)
0.833 (0.14)
ASP-ax
27.16 (2:1)
31.19 (2:8)
90s6 (23.5)
91:43 (25.6)
11201 (6 18)
04 (7.66
13. (7:1)
90.50 (15.9)
01.43 (231k)
0,526 (1.20)
ASP ASP-maltizad
23,58' (2.50
22.15
50.007 (20.9)
91.04" (24.1)
0,792' (6.26)
24.00" (5.33
27.55 (5.5)
00.537 (23.3)
91.001 (24.1)
0.700 (0.20)
Ours (GRU)
SPiaicai
(6.5)
17.41 (6.2)
17.00 [16.8]
14.75" (20.7)
0.215
10:06
17.47" (16.0)
14.54" (20.5)
0.706
SPlatcst ids
(5.0)
17.76 (7.5)
30:55 (15.5)
21.78 (29.0)
0.073 22)
16.00" (6.6)
18.20 (5.0)
20.74 (19.5)
21.01 (26-0)
0.874 (0.22)
ASP Oxt
22.00 (8.7)
22.61 (1.11)
17.59 (12.8)
15.56 (106))
722 (0 25)
23.06 (8:7)
24.30 (11.7)
19:22 (14.8)
16.82 (10 ()
0.620 (1.27)
ASP molthead
15.35" (6.7)
15.07 (6.0)
24:76 (2)6))1).))
35. 05 (38.5) 5)
0.778" (6.73)
16.54 M1
16.507 (6:3)
25.37 (21
25.41 (28-0)
0,713" (0.30)
7 Discussion and Conclusion
What qualifies as the best performing model for SCF? Our SP-GRU learns the best fit for synthesized behavior. On the commonly used metric of NLL [19, 58, 62]. our SP-MLP models perform the best for real-world data. However, they fare the worst at estimating the mean. On the other hand, the SP-GRU models estimate a better likelihood than the NP baselines with comparable errors in mean forecast. While the NP baselines attain the lowest errors in predicted means, they also achieve the worst NLL. From the qualitative visualizations and ablations, it seems that the models minimize NLL at the cost of orientation errors; in the case of SP-MLP seemingly by predicting the majority orientation of the two sellers who face the same direction. Also, the NP models forecast largely static
330
332
334
335
336
337
futures. In contrast, while being more dynamic, the SP-GRU forecasts also contain some smoothing.
338
Our synthesized glancing behavior is grounded social literature, and matches the head pose features in the real-world data (horizontal orientation). Why do we see large discrepancy in qualitative forecasts? One crucial distinction between the synthetic and real data is the subtlety and sparsity of motion. Our synthesized data makes the common implicit assumption that head pose is proxy for gaze [31, 67, 68, 70-72]. In real-world data, attention shifts through changes in gaze are not always accompanied by similar head rotations [73, Fig. 5], and gaze is harder to record -on-invasivelly ih-the-wild with reasonable accuracy. The consequence of this approximation is exacerbated in the triadic Haggling setting where people are arranged roughly in triangle and within each other's field of vision, making head movements even more subtle. In natural settings, groups occupy varied formations such as side-by-side, or L-arrangement [60, p. 213]. Here the more accentuated pose changes could aid in anticipating behavior. From this perspective, the combination of limited data and our simplifying assumption of a single group in a scene is a primary limitation of this work. The only publicly available dataset meeting our assumptions is the Haggling dataset, where all interactions follow similar patterns. As targeted development of techniques for recording such datasets in-the- wild gain momentum [74], evaluating these models in the different interaction settings would yield increased insight. Nevertheless, our aim in evaluating on synthesized as well as real-world data
340
342
343
344
345
346
347
348
349
350
351
352
353
354
was to highlight the influence that such common implicit assumptions can have on performance when applying methods. As an aside, we believe that this subtlety and sparsity of motion is also an important distinction between forecasting in focused and unfocused interactions. While the same techniques can be applied in both scenarios, pedestrian location a perpetually changing data stream.
355
357
358
The broader goal of this paper is to take a step towards bridging a gap we perceive between research domains; on one hand, we notice that there growing trend of applying deep learning techniques in the small data regime that is social behavior data [30, 75J. Without citing specific works as negative exemplars, this is occasionally accompanied by surface treatment of social science literature.
359
380
361
362
On the other hand, in our conversations we have also perceived a preemptive resistance to deep learning methods precisely due to limited data. We believe that our work here- specificalll our conceptualization of conversations groups as meta-learning tasks grounded in extensive considerations from social literature; our approach of learning extractable task-agnostic representations of predictive behavior; and the distinction between real-world and synthesized dynamics commonly used for evaluation-iis of value in stimulating a broader community discussion about the considerations when applying machine learning approaches within the domain of free-standing social conversations.
363
364
365
367
9

---

References
389
[1] Adam Kendon. Conducting Interaction: Patterns of Behavior in Focused Encounters. Number 7 in Studies in Interactional Sociolinguistics. Cambridge University Press, Cambridge: New York, 1990. ISBN 778-0-521-38036- 978-0-521-38938-9 1, 2, 3, 7,8
371
372
373
[2] Alessandro Vinciarelli, Maja Pantic, and Hervé Bourlard. Social signal processing: Survey of an emerging domain.
374
375
[3]
Dan Bohus and Fric Horvitz. Models for multiparty engagement in open-world dialog. In Proceedings of the SIGDIAL 2009 Conference on The 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue- SIGDIAL '09. pages 225-234, London, United Kingdom, 2009. Association for Computational Linguisties. ISBN 978- 778-1-932432-64-0 doi: 10.3115/1708376. 1708409.
378
378
379 [4] Ryo Ishii, Shiro Kumano, and Kazuhiro Otsuka. Prediction of Next-Utterance Timing using Head Movement in Multi-Party Meetings. In Proceedings of the 5th International Conference Human Agent Interaction, HAT 17. pages 181-187, New York. NY, USA, Octoher 2017. Association for Computing Machinery. ISBN 978-1-4503-5113-3. doi: 10.1145/125739..1255
382
[5] Anne Keitel and Moritz M Daum. The use of intonation for turn anticipation in observed conversations without visual signals as source of information. Frontiers in psychology. 6:108, 2015. 1,3
385
[6] Simon Garrod and Martin J Pickering. The use of content and timing to predict turn transitions. Frontiers in psychology, 6:751, 2015.
386
387
7 Amélie Sochet-Capelllan and Susanne Fuchs. Take a breath and take the turn: how breathing meets turns in
388
spoutaneous dialogue. Philosophical Transactions of the Royal Society B: Biological Sciences, 369((668)) 201 30399, 2014.
389
390
[8] M. Wlodarczak and M. Heldner. Respiratory tum-taking eues. In INTERSPEECH, 2016. 1,3
391
[9] Dan Bohus and Eric Horvitz. Managing Human-Robot Engagement with Forecasts and... um... Hesita-
392
tions. page 8. 1,3
383
[10] Cynthia C. S. Liem, Markus Langer, Andrew Demetriou, Annemarie M. F. Hiemstra, Achmadnoer Sukma Wicaksana, Marise Ph. Born, and Cornelius J. König. Psychology Meets Machine Learning: Interdisciplinary Perspectives on Algorithmic Job Candidate Screening, In Hugo Jair Escalante, Sergio Escalera, Isabelle Guyon, Xavier Baró, Yağmur Güęçlüüürk, Umut Güçlü, and Marcel Gerven, editors, Explainable and Interpretable Models in Computer Vision and Machine Learning, pages 197-253. Springer 118-3-319-981311
394
395
387
396
International Publishing, Cham, 2018. ISBN 978-3-319-981300 978-3 088-3-319-9811 119-91131-499.
doi: 10.1007/
389
400
[11] Erlend Nilsen, Diana Bowler, and John Linnell. Exploratory and confirmatory research in the open science era. Journal of Applied Ecology, 57, February 2020. doi: 1111/1365-2664113353
401
402
[12] Starkey Duncan. Some signals and rules for taking speaking turns in conversations. Journal of Personality and Social Psychology, 23(2):283-292 1972 ISSN (1239313333
403
doi: 10.
404
1037/h003001. 1,3,8
405
[13] Akko Kalma. Gazing in triads: A powerful signal in floor apportionment. British Journal of Social Psychology. 31(1):1111333 -3,, March 1992. 1.3
407
[14]
Sanjay Bilakhia, Stavros Petridis, and Maja Pantic. Audiovisual Detection of Behavioural Mimicry. In 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction, pages 123-128, Geneva, Switzerland, September 2013. IEEE ISBN 17780-7695-5008-00 doi: 10. 10.1109/ACI..011 1, 4
408
410
[15] Felix van Doorn. Rituals of Leaving: Predictive Modelling of Leaving Behaviour Conversation. Master of Science Thesis, Delft University of Technology. 2018. 2,
411
[16] Laura Cabrera- Quirra- Andrew Demetriou, Ekin Gedik, Leander der Meij. and Hayley Hung. The matchnmingle dataset: novel multi- multi-sensor resource for the analysis of social interactions and group dynamics in-the-wild during free-standing conversations and speed dates. IEEE Transactions on Affective Computing, 2018. 2
413
414
415
416
[17] Nalini Ambady, Frank J Bernieri, and Jennifer A Richeson. Toward histology of social behavior: Judgmental accuracy from thin slices of the behavioral stream. In Advances in experimental social psychology, volume 32, pages 201-271. 1. Elsevier, 2000. 2,
417
418
419
[18] Alessandro Vinciarelli, H Salamin, and M Pantic. Social Signal Processing: Understanding social interactions through nonverbal behavior analysis (PDF). 2009 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2009, June 2009. doi: 10. 10.11////22 WWW.PPRRR.200992222299009 2,
420
421
422
[19]
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, S. M. Ali Eslami, and Yee Whye Teh. Neural Processes. arXiv 01622 (cs, stat), 2018. 2,9
423
424
[20]
Mattias Heldner and Jens Edlund, Pauses, gaps and overlaps in conversations. Journal of Phonetics, 38(4):
355-668 October 2010. ISSN 0095-4470. doi: 1o..016/..ocn.2210000 002. 3, 17
10

---

428
[21] Monica M. Moore. Nonverbal courtship patterns in women: Context and consequences. Ethology and Sociobiology, 6(4):237-247, January 1985. ISSN 0162-3095. doi: 10.
427
3,
428
[22] Stephen C. Levinson and Francisco Torreira. Timing in tur--tkking and its implications for processing models of language. Frontiers in Psychology, 6, June 2015. ISSN 1664-1078 doi: 10.3389/fpsyg.2011.
429
00731. 3
431
[23] Erving Goffiman. Behavior in Public Places: Notes on the Social Organization of Gatherings. The Free Press, 1. paperback ed., 24. printing edition. ISBN 778-0-02-911940-- 3
432
433
[24]
Nina-Jo Moore, Hickson Mark III, and W Don. Stacks. Nonverbal communication: Studies and applications.
434
2013. 3,7 7
[25]
Harvey Sacks, Emanuel A. Schegloff, and Gail Jefferson. A Simplest Systematics for the Organization of Turn-Taking for Conversation. 50(4):40, 1974. 3
436
437
[26]
Charles Goodwin. Conversational Organization: Interaction Between Speakers and Hearers. Jamuary
1981.
439
[27]
Carole Edelsky. Who's Got the Floor? Language in Society, 10(3):383 421, 1981. ISSN 0047-4045, 3
[28]
Allan Wang and Aaron Steinfeld. Group Split and Merge Prediction With 3D Convolutional Networks. IEFF Robotics and Automation Letters, 5(2):1923-1930, April 2020. ISSN 2377-3766. doi: 10. R09/LRA 2020.2969947 3
442
[29] Massimo Mastrangeli, Martin Schmidt, and Lucas Lacasa. The roundtable: An abstract model of conversa- tion dynamics. arXiv:1000.294: [physicx]] October 2010. 3
445
[30] Louis Airale, Dominique Vaufreydaz, and Xavier Alameda-Pineda. SociallnteractionGGAAN Multi-person
446
Interaction Sequence Generation. arXiv:2103.0550 [cs, stat], March 2021. 3,9
447
[3I] Navyata Sanghvi, Ryo Yonetani, and Kris Kitani. Mgpi: A computational model of multiagent group perception and interaction. arXiv preprint arXiv:
448
2019. 7,7
[32] Hanbyul Joo, Tomas Simon, Mina Cikara, and Yaser Sheikh. Towards Social Artificial Intelligence: Nonverbal Social Signal Prediction a Triadic Interaction. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10865-10875, Long Beach, CA, USA, June 2019. IEEE. ISBN
450
451
452
doi: 10.1 10.1109/CVPR.20119 2019.0111 3, 4,
453
[33]
Emilie Delaherche, Mohamed Chetouani. Ammar Mahdhaoui. Catherine Saint- Georges. Sylvie Viaux. and David Cohen. Interpersonal Synchrony: A Survey of Evaluation Methods across Disciplines. JEEE Transactions on Affective Computing, 3(3):349-36) 365, July 2012. ISSN 1949-3045. doi: 10. 109/TAFF...
454
455
2012.12. 4
457
[34] Dirk Helbing and Peter Molnar. Social Force Model for Pedestrian Dynamics. Physical Review E, 51(5): 4282-4286, May 1995. ISSN 1063-651X, 1095-3787. doi: 10.1 1033 1.2.22.. 4
458
459
[35] Jarostaw Was, Bartlomiej Gudowski, and Pawet J. Matuszyk. Social Distances Model of Pedestrian Dynamics. In David Hutchison, Takeo Kanade, Josef Kittler, Jon M. Kleinberg, Friedemann Mattern John C. Mitchell, Moni Naor, Oscar Nierstrasz, C. Pandu Rangan, Bernhard Steffen, Madhu Sudan, Demetri Terzopoulos. Dough Tygar, Moshe Y. Vardi, Gerhard Weikum, Samira El Yacoubi, Bastien Chopard, and Stefania Bandini, editors, Cellular Automata, volume 4173. pages 492-501. Springer Berlin Heidelberg, Berlin, Heidelberg. 2006. ISBN 978-3-540-40929-88 978-3-540-40932-8. doi:
460
461
462
484
57.
485
[36] Gianluca Antonini, Michel Bierlaire, and Mats Weber. Discrete Choice Models for Pedestrian Walking September 2006. doi: 10.1016/j.
Behavior. Transportation Research Part B: Methodological, 00.22055.0.0006
467
488
[37] Adrien Treuille. Seth Cooper, and Zoran Popović. Continuum crowds. ACM Transactions on Graphics / SIGGRAPH 2006, 25(3): 1 160-1168, July 2006.
470
[38] Alexandre Robiequet, Amir Sadeghian, Alexandre Alahi, and Silvio Savarese. Learning Social Etiquette: Human Trajectory Understanding In Crowded Scenes. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision ECCV 2016, volume 9912, pages 549-565. Springer International Publishing, Cham, 2016. ISBN 978-3-39--46483-1 978-3-319-4648488 doi: 10.100779788-3311-444444 33.
471
472
474
475
[39] J. M. Wang, J. J. Fleet, and A. Hertzmann. Gaussian Process Dynamical Models for Human Motion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(2):283-296 February 2008. ISSN 1939-3539. doi: 10. 109.T/..............
477
478
[40]
Christopher Tay and Christian Laugier. Modelling Smooth Paths Using Gaussian Processes. In Proc. of the Int. Conf. on Field and Service Roboties, 2007.
480
[41]
Andrew Patterson, Arun Lakshmanan, and Naira Hovakimyan. Intent-Aware Probabilistic Trajectory Estimation for Collision Prediction with Uncertainty Quantification. arXi::1904.0275 [cs, mathj, April 2019. 4
481

---

483
[42] Alexandre Alahi, Kratarth Goel. Vignesh Ramanathan, Alexandre Robicquet. Li Fei-Fei, and Silvio Savarese. Social LSTM: Human Trajectory Prediction in Crowded Spaces. In 2016 IEEE Conference on Computer Vision and Puttern Recognition (CVPR). pages 961-971, Las Vegas, NV. USA, June 2016. IEEE ISBN 978-1-4673-8851-1 doi: 10.1 10.1109/CVPR.2016
484
485
486
4
[43] Pu Zhang, Wanli Ouyang, Pengfei Zhang. Jianru Xue, and Nanning Zheng. SR-LSTM: State Refinement for LSTM towards Pedestrian Trajectory Prediction. arXiv:1903.00 1903.02793 [cs], March 2019.
488
[44] Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi, Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks. arXiv:1803.10892 [cs]. March 2018. 8
490
[45] Yingfan Huang, Huikun Bi, Zhaoxin Li, Tianlu Mao, and Zhaoqi Wang. STGAT: Modeling Spatial- Temporal Interactions for Human Trajectory Prediction. In 2019 JEEE/CVF International Conference on Computer Vision (ICCV). pages 627 1-6280. Seoul, Korea (South), October 2019. IEEE. ISBN 978-1-72814-803-8. doi: 10.1 000110//CCV22219.0006377
482
493
494
495
[46] Abduallah Mohamed, Kun Qian, Mohamed Elhoseiny, and Christian Claudel. Social-STGCCI Social Opatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction. arXiv:2002.11927 [cs). February 2020. 4
496
497
488
[47] Murelio Ranzato, Arthur Szlam, Joan Bruna, Michael Mathieu, Ronan Collobert, and Sumit Chopra. Video (language) modeling: A baseline for generative models of natural videos, arXiv:1412.666: [cs], December 2014. 4
489
500
501
[88] Jacob Walker, Kenneth Marino, Abhinav Gupta, and Martial Hebert. The Pose Knows: Vidco Forecasting by Generating Pose Futures. arXiv:/705.000 [es), April 2017.
502
503
449
Jacob Walker, Abhinav Gupta, and Martial Hebert. Dense Optical Flow Prediction from a Static Image. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 2443-2451, Santiago, Chile, December 2015, IEFE. ISBN 978-1 178-1-467338831-1 doi: 10.1 109/ICCV.20552811
504
505
506
500
Alexey Dosovitskiy, Philipp Fischer. Eddy 1lg. Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, and Thomas Brox. FlowNet: Learning Optical Flow with Convolutional Networks. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 2758-2766, Santiago, December 2015. IEBE. ISBN 978- 177-1-4673--391-2.
507
509
doi: 10. 109/CCCV2201..31
510
[SI] Jacob Walker, Abhinav Gupta, and Martial Hebert. Patch to the Future: Unsupervised Visual Prediction. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 3302-3309, Columbus, OHI, USA, June 2014. IEEE, ISBN 17781-4779-51188
512
doi: 10. 19.100922P2...014441
[52] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Anticipating Visual Representations from Unlaheled Video. In 2016 IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pages 98-106, Las Vegas, NV, USA, June 2016. IEEE. ISBN 978- -4773-8817-8811111
515
doi: 10. 00.1109/CVP..2016.118
516
517
[53] Nitish Srivastava. Elman Mansimov, and Ruslan Salakhutdinov. Unsupervised Learnin of Video Repre- sentations using LSTMs. arXiv: 15220400284 /es), February 2015. 4
518
519
[54]
Alexey Dosovitskiy and Vladlen Koltun. Learning to Act by Predicting the Future. arXiv:1611.01779 [cs), November 2016. 4
520
[55] Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-Learning in Neural Networks: A Survey, arXiv:2004.0544 [cs, stat], November 2020.
522
523
[56] Nya Sutskever, Oriol Vinyals, and Quoe V Le. Sequence to Sequence I Learning with Neural Networks. In Z.. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 3104-3112. Curran Associates, Inc., 2014. 5
524
$25
[57] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning Phrase Representations using RNN Encoder-Decodee for Statistical Machine Translation. arXiv: 1406. 1478 [cs, stat), September 2014. 5
527
528
$29
[58] Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Gamelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive Neural Processes. arXiv: / 9010100007661 [cs, stat], July 2019. 5,7,9
530
[59]
Charles R. Qi. Hao Su, Kaichun Mo. and Leonidas J. Guibas. PointNet: Deep Learning Point Sets for 3D Classification and Segmentation. arXiv: [cs], April 2017.
532
[60|
Alex Kendall and Roberto Cipolla. Geometric Loss Functions for Camera Pose Regression with Deep Learning. 174.:0970 [cs], May 2017. 6, 9
534
535
[61]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. a7Xiv:1706.03762 [cs), June 2017. 6
536
537
[62]
Gautam Singh, Jaesik Yoon, Youngsung Son, and Sungjin Ahn. Sequential Neural Processes. URL http://azee org/ abs/ 190./ 7.9
12

---

539
[63]
Jaesik Yoon. Gautam Singh, and Sungjin Ahn. Robustifying Sequential Neural Processes. In International Conference on Machine Learning, pages 10861-10870. PMLR, November 2020.
541
[64] Timon Willi, Jonathan Masci, Jürgen Schmidhuber, and Christian Osendorfer. Recurrent Neural Processes. arXiv: (ww../9069999999 [cs, stat], November 2019.
542
543
[65]
Sumit Kumar. Spatiotemporal Modeling using Recurrent Neural Processes. Master of Science Thesis, Carnegie Mellon University, page 43. 7
544
[66]
Shenghao Qin, Jiacheng Zhu, Jimmy Qin, Wenshuo Wang, and Ding Zhao. Recurrent Attentive Neural Process for Sequential Data. arXiv:/910.09523 [es, stat]. October 2019. 7
548
547
[67]
Marynel Vazquez, Aaron Steinfeld, and Scott E. Hudson. Maintaining awareness of the focus of attention of conversation: A Symposium Robot and Human Interactive Communication (RO-MAN), pages 36-43, New York, NY, doi: 10.
reinforcement learning approach. In 2016 25th IEEE International
549
550
USA, August 2016. IEFF, ISBN 978- 7781--500-3992--6
A...ROMMA.2206.775508888 7,9
[68] Xavier Alameda-Pinedas Pineda, Yan Yan, Elisa Ricci, Oswald Lanz, and Nicu Sebe. Analyzing Free-standing Conversational Groups: A Multimodal Approach. pages 5-14. ACM Press, 2015. ISBN 978-1-4503-3355-4
553
doi: 10.1145/27333332886622
8, 9
554
[69]
Lu Zhang and Hayley Hung. On Social Involvement in Mingling Scenarios: Detecting Associates of F- Formattions in Still Images. IEEE Transactions Affective Computing, 2018.
556
[70]
Stephanie Tan, David M. J. Tax, and Hayley Hung. Multimodal Joint Head Orientation Estimation in Interacting Groups via Proxemics and Interaction Dynamics. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Téchnologies, 5-1):1-22, March 2021. ISSN 2474-9567, doi: 10.1145/ 3448122. 8,
557
$59
580
[71] Rutger Rienks, Ronald Poppe, and Mannes Poel. Speaker Prediction based on Head Orientations. page 7.
581
[72] M. Farenzena, A. Tavano, L. Bazzani, D. Tosato, G. Paggetti, Menegaz, V. Murino, and M. Cristani. Social interactions by visual focus of attention in a thrce-dimensional environment. Expert Systems, 30(2):
562
563
115-127, May 2013. ISSN 02664720. doi: 10.1 1/j. 1468-03394 201.0444001.0022222 2.00622.x 9
564
[73] S.O. Ba and J.-M. Odobez. Recognizing Visual Focus of Attention From Head Pose in Natural Mcetings. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cyberneticx), 39(1):16-33, February 2009.
565
566
ISSN 1083-4419. doi: 101100/TTCBB..00...... 9272747.7 9
[74] Chirag Raman, Stephanic Tan, and Hayley Hung. A modular approach for synchronized wireless multi- modal multisensor data acquisition in highly dynamic social settings, arXiv preprint arXiv:200s.0377 03715.
588
569
2020. 9
[75] Mason Swofford, John Charles Peruzzi. Nathan Tsoi, Sydney Thompson, Roberto Martin-Martin Silvio Savarese, and Marynel Vázquez. Improving Social Awareness Through DANTE: Deep Affinity Network for Clustering Conversational Interactants. Proceedings of the ACM on Human Human-Compitter Interaction, 4 (CSCW1):1 1):1-2, May 2020. ISSN 2573-0142. doi: 10.1 (0.1145/3392824, 9
571
572
573
574
[66]
Tuan Anh Le, Hyunjik Kim, and Marta Garnelo, Empirical Evaluation of Neural Process Objectives.
575
page 9. 16
177] Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv: 1412.6688 [csl,
577
January 2017. 16
578
[78] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019. 16
579
581
582
583
504
[79]
WA Falcon al. Pytorch lightning. GütHub. Note: http:/ghhub..oooofoooooehehhnn/e highttinn
585
3, 2019. 16
[80]
Chirag Raman and Hayley Hung. Towards automatic estimation of conversation floors within f- f-fommations. In 2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIW), pages 175-181. IEEE, 2019. 17
587
588
13

---

Checklist
589
580
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We mention the contributions of the task formulation and method in Section 1, along with our simplifying assumption for the setting, and circle back to discussing its implications in Section 7.
592
694
585
(b) Did you describe the limitations of your work? [Yes] Please refer to the discussion in Section 7.
596
597
(C) Did you discuss any potential negative societal impacts of your work? [N/A]
598
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
600
2. If you are including theoretical results...
601
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
603
3. If you ran experiments..
604
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [Yes]] Code, processed data, splits, and test batches are provided for reproduction in the Supplementary mate- rial.
606
608
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] Please refer to Appendices C and D.
609
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] We have reported mean and std. for the metrics over individual sequences in the test sets.
611
613
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please refer to Appendix C.2. We specify the specific GPUs used for our experiments and their corresponding memory capacities.
614
615
616
617
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
618
(a) If your work uses existing assets, did you cite the creators? [Yes] We have cited and discussed the original work proposing the Haggling dataset.
(b) Did you mention the license of the assets? [No] The Haggling dataset is freely available for non-commercial and research purpose only.
621
622
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you' 'r using/curating? [No] The Haggling dataset is freely available for non-commercial and research purpose only.
624
626
(e) Did you discuss whether the data you are using/curating contains personally identifi- able information or offensive content? [No] However, we visualized the features in Blender3D from scratch not display original videos with the subjects.
627
628
5. If you used crowdsourcing conducted research with human subjects...
630
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
631
632
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
634
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
635
14


Reviews:
[{"review_id": "xSylz7W0xkb", "reviewer": "Reviewer_u6ot", "summary": "This work proposes the Social Process (SP) models to predict human behaviors in social conversations.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- First of all, I don't have sufficient knowledge to evaluate this work from the perspective of psychology, so most of the following evaluation will focus on the machine learning aspects\n- The notations and technical details used in this work are very hard to follow. For example, in the paragraph of \"Neural Processes\", I assume this paragraph is mainly about variational inference, at least judging from equation 3. However, I quickly got lost in the technical terms, such as \n  - Line 162 - 153: \"The latent z is modeled by a factorized Gaussian parameterized by $s_C:=f_s(C)$, with $f_s$ being a deterministic function invariant to order permutation over $C$\", I have no idea what this line means\n  - Line 165: \"C is encoded on the deterministic path ...\", what is the definition of a deterministic path?\n  - Equation 3, what are $q(z|s_D)$ and $q(z|s_C)$ respectively? what are the parameters of this optimization problem?\n- In section 5\n  - Line 173: \"every $X_j$ occurs before the corresponding $Y_k$\", are $X_j$ and $Y_k$ pairwised? \n  - Line 180: \"Our intuition is that this would cause the representation to encode temporal information about the future\", I am not sure I agree with this statement. A seq2seq model can learn a mapping between inputs and outputs, but it may not be powerful enough to encode the future\n  - The use of positional encoding in the paragraph of \"Encoding Future Window Offset\" is really out of nowhere. In BERT, the use of positional embedding is to avoid the self-attention mechanism that confuses the word order information. While in sequential modeling, I am not sure it is necessary. At least, from the paper, I did not understand the motivation. \n\n\nThe abovementioned items are just some example issues of this work. Overall, I think this work tries to combine some psychology concepts with machine learning, by throwing out lots of machine learning techniques without sufficient understanding. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Some limitations are discussed in the last section of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "x4kebFd0vaQ", "reviewer": "Reviewer_6r9M", "summary": "This paper focuses on forecasting the certainty of nonverbal behavior cues for each individual in a group social conversation. Instead of using limited labelled data, this paper uses low-level unlabeled data to build models that estimate the certainty of individuals' future behivour, given their past behivour. This paper proposes Social Process (SP) models, socially aware sequence-to-sequence (Seq2Seq) models within the Neural Process (NP) family, to handle this task. Evaluation on synthesized and real-world behavior data shows that the proposed SP models achieve higher log-likelihood than strong baselines.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \nThe task proposed in this paper, forecasting he certainty of nonverbal behavior cues in social conversation, is a novel computational task in the context of social science. However, this task similar to language modeling task, a typical NLP task with long-standing research effort, which is to use the history sequence to predict the future sequence. The main different here is that language modeling is to calculate the probaility for the future sequences, while this task is to calculate certainty for the future sequences. In order to calculate certainty, this paper introduces seq2seq-based Nerual Processing model. Starting from this point, most of the contents here are more or less direct applications of existing technologies. For example, the cross-attention mechanism from the Attentive Neural Process (ANP) is used as the backbone of model for single participant; The approach in Qi et al. ([59] in the submission) to process point clouds is used to model partners' representation in the conversation. In summary, the technical contribution for this paper is incremental for NeurIPS. I think this submission may be more appropriate for CVPR or ICCV conference as they are interested in general CV application papers. \n\nQuality & Clarity:\nThis paper presents a fairly completed work with various experiments, including synthesized and real-world behavior data. The paper is well written and easy to follow. The task requires various of information, including people pose and positions and reproduce this research could be complicated. I suggest the author to release all preprocessed information/features to help other researchers.\n\nSignificance:\nThis paper proposes a task with strong social science motivation. It is excited to see that nerual network systems have been applied to various fields to help domain experts. However, the model proposed in this paper can be viewed as a combination of various machine learning and compter vision technologies. Researchers work for machine learning and neural models may not be interested in this submission. I suggest authors to submit this paper to more general CV conferences (e.g., CVPR, ICCV)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "n/a", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "qUKsrdOXfY1", "reviewer": "Reviewer_29nJ", "summary": "This paper presents social process models that solve a social cue forecasting problem. It is defined as predicting figure behavioral cues of participants who have social interaction given observed their behaviors. The social process (SP) is based on a neural process (NP) that can solve the meta-learning problem, and mix the seq2seq models to encode the social behaviors. The authors also suggest encoding a partner’s behaviors. Experiments on the synthetic dataset that mimics glancing behavior show that SP outperforms NP in terms of fitting in the future timestamp. Experiments on a real dataset that captures the head, body, and speaking behaviors of three people during a conversation show that SP outperforms NP in terms of predicted negative log-likelihood.\n\nThe main strength of this paper is suggesting an interesting task - social cue forecasting and quite simple models (# parameters 3.0M) to resolve the problem than baseline (NP, # parameters 2.8M).  I enjoy reading this paper since it can make many interesting questions that promise researchers to do future research.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The task is quite new and the model is also new. SP uses NP and seq2seq, so they are quite well-known models. \n\nQuality: Experiment results show that the suggested model quite performs well to predict behaviors in terms of negative log-likelihood, but baselines (NP and ANP) outperform to predict head orientation and speaking status. They are important to understand communication behaviors since head orientation is related to listening behavior. It is my big concern with this paper since we want to understand human behaviors such as who is talking to whom, not just simulating future behaviors only.\n\nClarity: This paper is well-written and well organized. \n\nSignificance: I am still curious about the results in Table 2. But other researchers are likely to use the author’s ideas and try to solve the social cue forecasting problem.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I have some questions and comments that I would like to hear from the authors.\n\n- Can we figure out the important social cues that affect other participants to change their social behaviors in the future time window?\n\n- One of the pros of SP models is being more dynamic in the future sequence, but NP models do not. Then, it would be better to test the predicted means over a different length of the test sequences. I think that SP models outperform NP models when the test sequence is quite long.\n\n- It is hard to understand the examples of qualitative visualizations in appendix B because of the shaking of the bodies (or std area). And I cannot see the difference of social cues from 60 to 111 in the ground truth. It would be better to draw the examples simply or write the text that describes the important social cues in each image.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "dKVnti1quEk", "reviewer": "Reviewer_JGHq", "summary": "The paper addresses the problem of nonverbal social cue forecasting in group social conversations, and proposes a bottom-up self-supervised approach to solve this problem. The bottom-up approach utilizes low-level behaviour cues to predict higher-level  social cues. The authors propose to use meta-learning and neural process to model the social process. Results on simulated and real data show that the proposed method is better than the baselines.\n\nContributions: \n1. The paper takes the first step towards a bottom-up self-supervised method to solve the social cue forecasting problem. Previous methods are all top-down.\n2. It utilizes the notion of neural process to formalize the concept of Social Process.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The method is novel since it's the first bottom-up method for predict human behaviour. And the intent of bottom-up is to solve the problem of limited amount of labeled data.\n\nQuality: The paper experimented on both a simulated and a real dataset, and performed necessary ablation studies to show the functionality of each module. But many experimental results are in the appendix.\n\nClarity: I think the writing could be improved. I found the paper a bit hard to follow because it covers many topics and many different techniques, it feels like the paper is constantly jumping from one topic to another without enough connections between different parts (e.g. why do we need this, and this is our assumption and later we use xx experiment to prove this).  Also, there are no enough descriptions of the experimental results (Table 1 and 2, etc).\n\nSignificance: It's not clear if statistical tests were performed from the tables and figures. If the results are statistically significant, I think this work could be a good addition to the current social cue prediction work. \n\n\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the writing could be improved. The experiments and content look good, but it might be worthwhile working a little bit more on how to organize and present them to the users.  For example, in line 181, \"Our intuition is that this would cause the representation to encode temporal information about the future.\" Is it verified by the experiments?\n\nAnd also as mentioned above, the results are not described in details.\n\nAlso, in Table1 and 4, why would NLL be negative for the models? And why is the difference between models so big? Are statistical tests performed here? ", "review_text": "", "extra_scores": {"ethical_concerns": "It's worth verifying if the synthetic dataset contains personal-identifiable information like face.", "needs_ethics_review": "Yes", "ethics_review_area": ["Privacy and Security (e.g., consent)"], "time_spent_reviewing": "1 hour"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel problem involving the prediction of nonverbal behaviors in social settings, and introduces a neural process model fit to the problem. Reviewers had no major methodological concerns, but raised concerns about novelty in the context of a machine learning methods venue, and about clarity. An extensive discussion resolved some misunderstandings, but did not yield a consensus in favor of accepting the paper to NeurIPS.
Decision: Reject