Social Processes: Self-Supervised Forecasting of Nonverbal Cues in Social Conversations
Year: 2021

Abstract:
The default paradigm for the forecasting of human behavior in social conversations 1 is characterized by top-down approaches. These involve identifying predictive 2 relationships between low level nonverbal cues and future semantic events of inter3 est (e.g. turn changes, group leaving). A common hurdle however, is the limited 4 availability of labeled data for supervised learning. In this work, we take the first 5 step in the direction of a bottom-up self-supervised approach in the domain. We 6 formulate the task of Social Cue Forecasting to leverage the larger amount of 7 unlabeled low-level behavior cues, and characterize the modeling challenges in8 volved. To address these, we take a meta-learning approach and propose the Social 9 Process (SP) models—socially aware sequence-to-sequence (Seq2Seq) models 10 within the Neural Process (NP) family. SP models learn extractable representations 11 of non-semantic future cues for each participant, while capturing global uncertainty 12 by jointly reasoning about the future for all members of the group. Evaluation on 13 synthesized and real-world behavior data shows that our SP models achieve higher 14 log-likelihood than the NP baselines, and also highlights important considerations 15 for applying such techniques within the domain of social human interactions. 16

The default paradigm for the forecasting of human behavior in social conversations1 is characterized by top-down approaches. These involve identifying predictive2 relationships between low level nonverbal cues and future semantic events of inter-3 est (e.g. turn changes, group leaving). A common hurdle however, is the limited4 availability of labeled data for supervised learning. In this work, we take the first5 step in the direction of a bottom-up self-supervised approach in the domain. We6 formulate the task of Social Cue Forecasting to leverage the larger amount of7 unlabeled low-level behavior cues, and characterize the modeling challenges in-8 volved. To address these, we take a meta-learning approach and propose the Social9 Process (SP) models—socially aware sequence-to-sequence (Seq2Seq) models10 within the Neural Process (NP) family. SP models learn extractable representations11 of non-semantic future cues for each participant, while capturing global uncertainty12 by jointly reasoning about the future for all members of the group. Evaluation on13 synthesized and real-world behavior data shows that our SP models achieve higher14 log-likelihood than the NP baselines, and also highlights important considerations15 for applying such techniques within the domain of social human interactions.16
1 Introduction17
Picture a situated interactive agent such as a social robot conversing with a group of people. How18 can agents act in such a setting? We sustain conversations spatially and temporally through explicit19 behavioral cues—examples include locations of partners, their orientation, gestures, gaze, and floor20 control actions [1–3]. Evidence suggests that we employ an anticipation of these and other cues to21 navigate daily social interactions [1, 4–8]. Consequently, the ability to forecast the future constitutes22 a natural objective towards the realization of machines with social skills. As such, interactive agents23 typically contend with uncertainties in inferences surrounding cues [3]. So beyond making real-time24 inferences, such systems may achieve more fluid interactions by leveraging the ability to forecast25 future states of the conversation [9].26
In addition to the development of social agents, behavior forecasting is also of significance in social27 psychology, where the focus is on gaining insight into human behavior. Since human-interpretability28 is of essence, top-down approaches largely constitute the default paradigm, where specific events of29 semantic interest are selected first for consideration and their relationship to potentially predictive cues30 are studied in isolation—either in controlled interactions in lab settings, or in subsequent statistical31 analyses [10, 11]. Examples of such semantic events include speaker turn transitions [5, 12, 13],32 mimicry episodes [14], or the termination of an interaction [9, 15]. However, one hurdle in the33 top-down paradigm is limited data. The events (that constitute the labels or the dependent variables)34 often occur infrequently over a longer interaction, reducing the effective amount of labeled data. This35 precludes the use of neural supervised learning techniques that tend to be data intensive.36
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.
In this work, we take an initial step towards a bottom-up approach to forecasting human behavior for37 free standing conversational groups. Our guiding motivation is to learn predictive representations of38 general future social behavior by utilizing unlabeled streams of low-level behavioral features. We do39 this by regressing future sequences of these features from observed sequences of the same features in40 a self-supervised manner. We term this task of non-semantic future behavior forecasting as Social41 Cue Forecasting (SCF).42
Our approach is built on the observation that the social signal [17]—the high-level attitudes and social43 meaning transferred in interactions—is already embedded in the low-level cues [18]. To conceptually44 illustrate the contrasting top-down and bottom-up approaches on an example task, Figure 1 depicts45 an instance of a group leaving event in a naturalistic social conversation. Evidence suggests that46 such events can be anticipated from certain preceding rituals [15] reflected in the postural changes of47 conversing members [1]. van Doorn [15] built a predictor using 200 instances of group leaving found48 in over 90 minutes of mingling interaction and hand-crafted features. In contrast, our bottom-up49 approach would entail learning task agnostic representations of future behavior using the entire 9050 minutes of data, and then training simpler predictors for group leaving using the learnt representations51 as input. The figure also illustrates the complexity of naturalistic interactions where cross-group52 social influence exists. In this work we focus on the simpler setting of a single group in a scene.53
There are several challenges intrinsic to computationally modeling future behavior in social conversa-54 tions. The future is intrinsically uncertain, the forecasts for interaction partners are inter-dependent,55 and the social dynamics is unique for each grouping of individuals. We address these through the56 following contributions:57
• We formalize the task of SCF. We characertize the modeling challenges involved, and cast58 the problem into the meta-learning paradigm, allowing for data-efficient generalization to59 unseen groups at evaluation without learning group-specific models.60
• We propose and evaluate two socially aware Sequence-to-Sequence (Seq2Seq) models61 within the Neural Process (NP) family [19] for SCF in social conversations. Our method62 encodes complex social dynamics informative of future group behavior into extractable63 representations for each individual.64
This paper is organized as follows. In Section 2 we formally define and characterize the task of65 SCF. We situate this work within broader literature in Section 3, and review background concepts66 in Section 4. We propose the Social Process models in Section 5 and describe our experiments in67 Section 6, concluding with a discussion of our findings in Section 7.68
2 Social Cue Forecasting69
The objective of SCF is to predict future behavioral cues of all people involved in a social encounter70 given an observed sequence of their behavioral features. More formally, let us denote a window71
of observed timesteps as tobs := [o1, o2, ..., oT ], and an unobserved future time window as72 tfut := [f1, f2, ..., fT ], f1 > oT . Note that tfut and tobs are typically non-overlapping, can be of73 different lengths, and tfut need not immediately follow tobs. Given a set of n interacting participants,74 let us denote their social cues over a tobs and tfut respectively as75
X := [bit; t 2 tobs]ni=1, Y := [bit; t 2 tfut]ni=1. (1a, b)
The vector bit encapsulates the multimodal cues of interest from participant i at time t. These can76 include head and body pose, speaking status, facial expressions, gestures, and verbal content—any77 information stream that combine to transfer social meaning.78
In its simplest form, given an X , the objective of SCF is to learn a single function f such that79 Y = f(X). However, an inherent challenge in forecasting behavior is that an observed sequence80 of interaction does not have a deterministic future and can result in multiple socially valid ones—a81 window of overlapping speech between people both may and may not result in a change of speaker82 [12, 20], a change in head orientation may continue into a sweeping glance across the room or a darting83 glance stopping at a recipient of interest [21]. In some cases certain observed behaviors—intonation84 and gaze cues [5, 13] or synchronization in speaker-listener speech [22] for turn-taking—might85 make some outcomes more likely than others. Given that there are both supporting and challenging86 arguments for how these observations influence subsequent behaviors [22, p. 5; 13, p. 22], it would87 be beneficial if a data-driven model expresses a measure of uncertainty in its forecasts. We do this by88 modeling the distribution over possible futures p(Y |X) rather than forecasting a single future.89 Another design consideration arises from a defining characteristic of focused interactions—the90 participants’ behaviors are interdependent. Participants in a group sustain equal access to the shared91 interaction space through cooperative maneuvering [1, p. 220]. Moreover, when multiple groups92 are co-located, outsiders unengaged in these intra-group maneuvers may also influence the behavior93 of those within the group [23, p. 91;1, p. 233], sometimes causing them to leave (see Figure 1). It94 is therefore essential to capture uncertainty in forecasts at the global level—jointly forecasting one95 future for all participants at a time, rather than at a local output level—one future for each individual96 independent of the remaining participants’ futures.97
How participants coordinate their behaviors is a function of several individual factors [24, Chap. 1; 1,98 p. 237]. Consequently, the social dynamics guiding an interaction also has unique attributes for every99 unique grouping of individuals. Rather than learning group-specific models to capture these unique100 dynamics, we formulate the forecasting problem in terms of meta-learning, or few-shot function101 estimation. We interpret each unique group of individuals as the meta-learning notion of a task. The102 core idea is that we can learn to predict a distribution over futures for a target sequence X having103 captured the group’s unique behavioral tendencies from a context set C of their observed-future104 sequences. We can then generalize to unseen groups at evaluation by conditioning on a short observed105 slice of their interaction. We believe that this approach is especially suitable for social conversation106 forecasting—a setting that involves a limited data regime where good uncertainty estimates are107 desirable. Note that when conditioning on context is removed (C = ?), we simply revert to the108 formulation p(Y |X).109
3 Related Work110
Free-standing conversations are an example of what social scientists call focused interactions, said to111 arise when a “group of persons gather close together and openly cooperate to sustain a single focus112 of attention, typically by taking turns at talking” [23, p. 24]. A long-standing topic of study has been113 the systematic organization of turn-taking [25–27], with a particular interest in the event of upcoming114 speaking turns [5–8]. There has also been some interest in the forecasting task itself, to anticipate115 disengagement from an interaction [9, 15], the splitting or merging of groups [28], the time-evolving116 size of a group [29] or semantic social action labels [30, 31]. Most of these works use heuristics,117 either to generate semantic labels [9], model the dynamics itself [29], or hand-craft features [15].118
Although not a forecasting task, the closest work that shares our motivation in predicting non-semantic119 low-level features is the recently introduced task of Social Signal Prediction (SSP) [32]. The objective120 is to predict the social cues1 of a target person using cues from the communication partners as121
1In the domain of Social Signal Processing, a social signal [17] refers to the relational attitudes displayed by people. It is a high-level construct resulting from the perception of cues (see Fig. 1 in [18]). From this
input (Joo et al. focus on predictions within the same time window [32, Eq. 6]). While the most122 general formulation of SSP involves forecasting a single timestep for a target person given the123 entire group’s past behavior [32, Eq. 3], generalizing this formulation runs into an inherent problem;124 applying the definition to forecasting entails iteratively treating each individual as target, learning125 separate functions for every person. However, as we discuss in Section 2, these futures of interacting126 individuals are not independent given observed group behavior. Furthermore, a constrained definition127 of forecasting that predicts an immediate step into the future is limiting, since forecasting an event128 that occurs after a delay (e.g. a time lagged synchrony [33] or mimicry [14] episode) might be of129 interest. Operationalizing this definition would entail a sliding window iteratively using predictions130 over the offset between tobs and tfut as input, which would cascade prediction errors.131
A related social setting where forecasting has been of interest is that of unfocused interactions. These132 occur when individuals find themselves by circumstance in the immediate presence of each other,133 such as pedestrians walking in proximity. Early approaches for forecasting pedestrian trajectories134 were heuristic based, involving hand-crafted energy potentials to describe the influence pedestrians135 have on each other [34–41]. More recent approaches encode the relative positional information136 directly into a neural architecture [42–46].137
In a broad sense, the self-supervised learning aspects of this work has some overlap with recent138 approaches focusing on the non-interaction task of visual forecasting. These works have taken a139 non-semantic approach to predict low level pixel-based features or intermediate representations140 [38, 47–52], and demonstrated a utility of the learned representation for other tasks like semi-141 supervised classification [53], or training agents in immersive environments [54].142
4 Preliminaries143
Meta-learning. A supervised learning algorithm can be viewed as a function mapping a dataset144 C := (XC ,YC) := {(xi,yi)}i2[NC ] to a predictor f(x). Here NC is the number of datapoints in145 C, and [NC ] := {1, . . . , NC}. The key idea of meta-learning is to learn the learning process itself,146 modeling this function representing the initial algorithm using another supervised learning algorithm;147 hence the name meta-learning. In meta-learning literature, a task refers to each dataset in a collection148 M := {Ti}Ntasksi=1 of related datasets [55]. For each task T , a meta-learner is episodically trained149 to fit a subset of target points D := (X,Y ) := {(xi,yi)}i2[ND] given another subset of context150 observations C. At meta-test time, the resulting predictor f(x, C) uses the information obtained151 during meta-learning to make predictions for unseen target points conditioned on context sets unseen152 at meta-training.153
Neural Processes Sharing the same core motivations, NPs are a family of latent variable models154 that extend the idea of meta-learning to situations where uncertainty in the predictions f(x, C) are155 desirable. They do this by meta-learning a map from datasets to stochastic processes, estimating a156 distribution over the predictions p(Y |X, C). To capture this distribution, NPs model the conditional157 latent distribution p(z|C) from which a task representation z 2 Rd is sampled. This constitutes158 the model’s latent path. The context can also be incorporated through a deterministic path, via a159 representation rC 2 Rd aggregated over C. An observation model p(yi|xi, rC , z) then fits the target160 observations in D. The generative process for the NP is written as161
p(Y |X, C) := Z p(Y |X, C, z)p(z|C)dz = Z p(Y |X, rC , z)q(z|sC)dz, (2)
where p(Y |X, rC , z) := Q
i2[ND] p(yi|xi, rC , z). The latent z is modeled by a factorized Gaussian162 parameterized by sC := fs(C), with fs being a deterministic function invariant to order permutation163 over C. When the conditioning on context is removed (C = ?), we have q(z|s?) := p(z), the164 zero-information prior on z. C is encoded on the deterministic path using a function fr similar to165 fs, so that rC := fr(C). In practice this is implemented as rC = P i2[NC ] MLP(xi,yi)/NC . The166 observation model is referred to as the decoder, and q, fr, fs comprise the encoders. The parameters167 of the NP are learned for random subsets C and D by maximizing the evidence lower bound (ELBO)168
log p(Y |X, C) Eq(z|sD)[log p(Y |X, C, z)] KL(q(z|sD)||q(z|sC)). (3) perspective, the task of Social Signal Prediction [32] is a misnomer since it still relates to the prediction of cues and not signals, a distinction we preserve in this work.
5 Social Processes169
In this section we present our socially aware Seq2Seq models within the NP family that is agnostic to170 group member identities and group size. To setup the task, we split the contextual interaction on which171 we condition into pairs of observed and future sequences, writing the context as C := (XC ,YC) :=172 (Xj ,Yk)(j,k)2[NC ]⇥[NC ], where every Xj occurs before the corresponding Yk. As discussed in173 Section 3, domain experts focusing on behavior analysis might be interested in settings where tobs174 and tfut are offset by an arbitrary delay. Consequently, the jth tobs can have multiple associated tfut175 windows. Denoting the set of target window pairs as D := (X,Y ) := (Xj ,Yk)(j,k)2[ND]⇥[ND], our176 focus in the rest of this work is to model the distribution p(Y |X, C).177 The generative process for our model we call the Social Process (SP) follows Eq. 2, which we178 extend to social forecasting in two ways. We embed an observed sequence x for an individual179 into a condensed encoding e 2 Rd that is then decoded into the future sequence using a Seq2Seq180 architecture [56, 57]. Our intuition is that this would cause the representation to encode temporal181 information about the future. Further, for every individual we model this e as a function of their own182 behavior, and that of their partners as viewed by them. The intuition is that this captures the spatial183 influence partners have on the participant over the tobs. Using notation we established in Section 2,184 we define the observation model for the SP for a single participant pi as185
p(yi|xi, C, z) := p(bif1, . . . , bifT |bio1, . . . , bioT , C, z) = p(bif1, . . . , bifT |ei, rC , z). (4) If decoding is carried out in an auto-regressive manner, we can further write the right hand side of186 Eq. 4 as QfT t=f1 p(b i t|bit 1, . . . , bif1, ei, rC , z). Following the standard NP setting, we implement the187 observation model as a set of Gaussian distributions factorized over time and feature dimensions.188 We also incorporate the cross-attention mechanism from the Attentive Neural Process (ANP) [58] to189 define the variant Attentive Social Process (ASP). Following Eq. 4 and the definition of the ANP, the190 corresponding observation model of the ASP for a single participant is defined as191
p(yi|xi, C, z) = p(bif1, . . . , bifT |ei, r⇤(C,xi), z). (5)
Here each target query sequence xi⇤ attends to the context sequences XC to produce a query-specific192 representation r⇤ := r⇤(C,xi⇤) 2 Rd. The model architectures are illustrated in Figure 2.193
Encoding Partner Behavior. While a typical Seq2Seq setup conditions the sequence decoder on194 solely a compact representation of the observed sequence, we’d like to condition an individual’s195 forecast on the observed behavior of both, themselves and their partners. We do this using a pair196 of sequence encoders: one to encode the temporal dynamics of participant pi’s features, eiself =197 fself(xi), and another to encode the dynamics of a transformed representation of the features of pi’s198 partners, eipartner = fpartner( (xj,(j 6=i))). Using a separate network to encode partner behavior199 grants the practical advantage of being able to sample an individual’s and partners’ features at different200 sampling rates.201
How do we model (xj)? We want the partners’ representation to possess two properties: per-202 mutation invariance—changing the order of the partners should not affect the representation; and203 group size independence—we want to compactly represent all partners independent of the group size.204
Beyond coordinate space invariance, we wish to intuitively capture a view of the interaction from205 pi’s perspective. We extend the approach Qi et al. [59] applied to point clouds to focused interactions206 by computing pooled embeddings of relative behavioral features. Since most commonly considered207 nonverbal cues in literature (see Section 6.3) include the attributes of orientation or location (e.g.208 head/body pose or keypoints) or a binary indicator (such as speaking status), we specify how we209 transform these. The 3D pose (orientation, location) of every partner pj is transformed to a frame of210 reference defined by pi’s pose. At timestep t, denoting orientation, location, and binary speaking211 status for pi as bit = [qi; li; si], and those for pj as b j t = [q j ; lj ; sj ], we have212
qrel = qi ⇤ (qj) 1, lrel = lj li, srel = sj si. (6a-c) Note that we use unit quaternions (denoted q) for representing orientation due their various benefits213 over other representations of rotation [60, Sec. 3.2]. The operator ⇤ denotes the Hamilton product of214 the quaternions. These transformed features for each pj are encoded using an embedder MLP. The215 outputs are concatenated with ejself and processed by a pre-pooler MLP, which is followed by the216 symmetric element-wise Max-pooling function to obtain (xj) at each timestep. We capture the217 dynamics in the pooled representation over tobs using fpartner. Finally, we combine eiself and e i partner218 for pi through a linear projection (defined by a weight matrix W ) to obtain the individual’s embedding219 eiind = W.[e i self ; e i partner]. Our intuition is that with information about both pi themselves, and of220 pi’s partners from pi’s point-of-view, eiind now contains the information required to predict pi’s221 future behavior.222
Encoding Future Window Offset. As we’ve discussed at the start of this section, a single tobs223 might have multiple associated tfut windows at different offsets. Our intuition is that training a224 sequence decoder to decode the same eiind into multiple sequences (corresponding to the multiple225 tfut) in the absence of any timing information might cause an averaging effect in either the decoder226 or the information encoded in eiind. One way around this would be to start decoding one timestep227 following the end of tobs and discard the predictions in the gap between tobs and tfut. However,228 if decoding is done auto-regressively this might lead to cascading errors over the gap. Instead, we229 address this one-to-many issue by injecting the offset information into eiind so that the decoder230 receives a unique encoded representation for every tfut to decode over. We do this by repurposing231 the idea of sinusoidal positional encodings [61] to encode offsets rather than relative positions in232 sequences. For a given tobs and tfut, and de-dimensional eiind we define the offset as t = f1 oT ,233 and the corresponding offset encoding OE t as234
OE( t,2m) = sin( t/10000 2m/de), OE( t,2m+1) = cos( t/10000 2m/de). (7a, b)
Here m refers to the dimension index in the encoding. We finally compute the representation ei for235 Eqs. 4 and 5 as236
ei = eiind +OE t. (8)
Auxiliary Loss Functions. We incorporate a geometric loss function that improves performance in237 pose regression tasks. For pi at time t, given the ground truth bit = [q; l; s], and the predicted mean238 b̂it = [q̂; l̂; ŝ], we denote the tuple (bit, bit) as Bit . We then have the location loss in Eucliden space239 Ll(Bit) = l l̂ , and we can regress the quaternion values using240
Lq(Bit) = q
q̂
kq̂k
. (9)
Kendall and Cipolla [60] show how these losses can be combined using the homoscedastic uncertain-241 ties in position and orientation, ̂2l and ̂ 2 q:242
L (Bit) = Ll(Bit) exp( ŝl) + ŝl + Lq(Bit) exp( ŝq) + ŝq, (10) where ŝ := log ̂2. Using the binary cross-entropy loss for speaking status Ls(Bit), we have the243 overall auxiliary loss over t 2 tfut:244
Laux(Y , Ŷ ) = X
i
X
t
L (Bit) + Ls(Bit). (11)
The parameters of the SP and ASP are trained by maximizing the ELBO in Eq. 3 and minimizing this245 auxiliary loss function for each of our sequence decoders.246
6 Experiments and Results247
6.1 Models and Baselines248
Our modeling assumption is that the underlying stochastic process generating the behaviors does249 not evolve over time. Stated differently, we assume that the individual factors determining how250 participants coordinate behaviors—age, cultural background, personality variables [24, Chap. 1; 1,251 p. 237]—are likely to remain the same over the short duration of a single interaction. This is in252 contrast to a related line of work that deals with meta-transfer learning, where the stochastic process253 itself changes over time [62–65]. We therefore compare against the NP and ANP family which254 share our model assumptions and meta-learning attributes. Note that in contrast to our methods,255 these baselines have direct access to the future sequences in the context, and therefore constitute a256 strong baseline. We consider two variants: -latent denoting only the latent path; and -latent+det,257 containing both deterministic and stochastic paths. We further consider two attention mechanisms for258 the cross-attention module: -dot with dot attention, and -multihead with wide multi-head attention259 [58]. We operationalize the original definitions of the baseline models to sequences by collapsing the260 timestep and feature dimensions. While the ANP-RNN model [66] shares our model assumptions, it261 is defined for a task analogous to SSP for concurrent car locations, and cannot be operationalized to262 forecasting in any simple way (see Section 3 discussing the distinction). We experiment with two263 choices of architectures for the sequence encoders and decoders in our proposed models: multi-layer264 perceptrons (MLP), and Gated Recurrent Units (GRU). Implementation and training details for our265 experiments can be found in Appendix C.266
6.2 Evaluation on Synthesized Behavior: Forecasting Glancing Behavior267
With limited behavioral data availability, a common practice in the domain is to train and evaluate268 methods on synthesized behavior dynamics [31, 67]. In keeping with this practice, we construct a269 synthesized dataset simulating two glancing behaviors in social settings [21]. We use a 1D sinusoid270 to represent horizontal head rotation over 20 timesteps. The sweeping Type I glance is represented by271 a pristine sinusoid, while the gaze fixating Type III glance is denoted by clipping the amplitude for272 the last six timesteps. The task is to forecast the signal over the last 10 timesteps (tfut) by observing273 the first 10 (tobs). Consequently, the first half of tfut is certain, while uncertainty over the last half274 results from every observed sinusoid having two ground-truths. It is impossible to infer from an275 observed sequence alone if the head rotation will stop partway through the future. We describe276
additional data setup, model details, and quantitative results for this setting in Appendices A.1, C277 and D.1, respectively. Figure 3 illustrates the ground truths, predicted means and std. deviations278 for a sequence within and outside the context set. We observe that all models estimate the mean279 reasonably well, although our proposed SP models learn a slightly better fit. More crucially, the280 SP models—especially the SP-GRU—learn much better uncertainty estimates over the certain and281 uncertain parts of the future compared to the NP baseline.282
6.3 Real-World Behavior: The Haggling Dataset283
We also evaluate our models on real-world behavior data, using the Haggling dataset of triadic284 interactions [32]. Participants are engaged in an unscripted game where two sellers compete to sell a285 fictional product to a buyer who has to choose between the two. We use the same split of 79 training286 sets (groups) and 28 test sets used by Joo et al. [32]. In our experiments we consider the following287 social cues: head pose described by the 3D location of the nose keypoint and a face normal; body pose288 described by the location of the mid-point of the shoulders and a body normal; and binary speaking289 status. Apart from being the most commonly considered cues in computational analyses of such290 conversations [68–70], pose and turn taking are found to be crucial in the sustaining of conversation291 [1, 12, 18]. We specify the dataset preprocessing details in Appendix D.2.292
6.4 Evaluation293
Context Regimes. We evaluate all models on two context regimes: random, and fixed-initial. The294 random regime follows the standard NP setting that the models are trained in. Context samples295 (sequence-pairs) are selected as a random subset of target samples, so the model is exposed to296 behaviors from any phase of the interaction lifecycle. Here we ensure that batches contain unique297 tobs to prevent any single observed sequence from dominating the aggregation of representations298 over the context split. At evaluation, we take 50% of the batch as context. In the fixed-initial context299 regime, we investigate how the model can generalize knowledge of group specific characteristics300 from observing the initial dynamics of an interaction where certain gestures and patterns are more301 distinctive [1, Chap. 6]. This matches what a social agent might face in a real-world scenario. Here302 we treat the first 20% of the entire interaction as context, treating sequences from the rest as target.303
Evaluation Metrics. We report the negative log-likelihood (NLL) log p(Y |X, C) in Table 1304 (computed by summing over feature dimensions and people, and averaging over timesteps). Beyond305 the NLL, we also report the error in the predicted means over test sequences in Table 2: mean-squared306 error (MSE) for the head and body keypoint locations; mean absolute error (MAE) in orientation307 in degrees; and speaking status accuracy. Note that while the ground truth orientation normals are308 constrained in the horizontal plane, we don’t constrain our predicted quaternions. We therefore report309 the absolute error in rotation in 3D. The reported mean and std. deviation of all metrics are over310 sequences in the test sets. We further report the metrics for every timestep over tfut in Appendix A.2,311 and qualitative visualizations of the forecasts in Appendix B.312
6.5 Ablations313
Encoding Partner Behavior. Modeling the interaction from the perspective of each individual314 is a central idea in our approach. We investigate the influence of encoding partner behavior into315 individual representations riind on the performance. We train the SP-latent+det GRU variant in two316 configurations: no-pool, where we do not encode any partner behavior; and pool-oT where we pool317 over partner representations only at the last timestep (similar to [44]). We choose the SP-GRU model318 since it achieves the best trade-off between minimizing NLL and forecasting cues consistent with319 human behavior. Both configurations lead to worse NLL and location errors (Appendix A.3).320
Deterministic Decoding and Social Encoder Sharing. Error gradients can flow back into our321 sequence encoders through two paths: from the final stochastic sequence decoder, as well as the322 deterministic decoders on the latent and deterministic paths. We investigate the effect of the determin-323 istic decoders by training the SP-latent+det GRU model without them. We also investigate sharing a324 single social encoder between the Process Encoder and Process Decoder in Figure 2. We find that325 removing the decoders only improves log-likelihood if the encoders are shared, and at the cost of326 head orientation errors (Appendix A.3).327
7 Discussion and Conclusion328
What qualifies as the best performing model for SCF? Our SP-GRU learns the best fit for synthesized329 behavior. On the commonly used metric of NLL [19, 58, 62], our SP-MLP models perform the best330 for real-world data. However, they fare the worst at estimating the mean. On the other hand, the331 SP-GRU models estimate a better likelihood than the NP baselines with comparable errors in mean332 forecast. While the NP baselines attain the lowest errors in predicted means, they also achieve the333 worst NLL. From the qualitative visualizations and ablations, it seems that the models minimize334 NLL at the cost of orientation errors; in the case of SP-MLP seemingly by predicting the majority335 orientation of the two sellers who face the same direction. Also, the NP models forecast largely static336 futures. In contrast, while being more dynamic, the SP-GRU forecasts also contain some smoothing.337
Our synthesized glancing behavior is grounded in social literature, and matches the head pose features338 in the real-world data (horizontal orientation). Why do we see a large discrepancy in qualitative339 forecasts? One crucial distinction between the synthetic and real data is the subtlety and sparsity340 of motion. Our synthesized data makes the common implicit assumption that head pose is a proxy341 for gaze [31, 67, 68, 70–72]. In real-world data, attention shifts through changes in gaze are not342 always accompanied by similar head rotations [73, Fig. 5], and gaze is harder to record non-invasively343 in-the-wild with reasonable accuracy. The consequence of this approximation is exacerbated in the344 triadic Haggling setting where people are arranged roughly in a triangle and within each other’s345 field of vision, making head movements even more subtle. In natural settings, groups occupy varied346 formations such as side-by-side, or L-arrangement [60, p. 213]. Here the more accentuated pose347 changes could aid in anticipating behavior. From this perspective, the combination of limited data and348 our simplifying assumption of a single group in a scene is a primary limitation of this work. The only349 publicly available dataset meeting our assumptions is the Haggling dataset, where all interactions350 follow similar patterns. As targeted development of techniques for recording such datasets in-the-351 wild gain momentum [74], evaluating these models in the different interaction settings would yield352 increased insight. Nevertheless, our aim in evaluating on synthesized as well as real-world data353 was to highlight the influence that such common implicit assumptions can have on performance354 when applying methods. As an aside, we believe that this subtlety and sparsity of motion is also an355 important distinction between forecasting in focused and unfocused interactions. While the same356 techniques can be applied in both scenarios, pedestrian location is a perpetually changing data stream.357
The broader goal of this paper is to take a step towards bridging a gap we perceive between research358 domains; on one hand, we notice that there is a growing trend of applying deep learning techniques359 in the small data regime that is social behavior data [30, 75]. Without citing specific works as360 negative exemplars, this is occasionally accompanied by surface treatment of social science literature.361 On the other hand, in our conversations we have also perceived a preemptive resistance to deep362 learning methods precisely due to limited data. We believe that our work here—specifically our363 conceptualization of conversations groups as meta-learning tasks grounded in extensive considerations364 from social literature; our approach of learning extractable task-agnostic representations of predictive365 behavior; and the distinction between real-world and synthesized dynamics commonly used for366 evaluation—is of value in stimulating a broader community discussion about the considerations when367 applying machine learning approaches within the domain of free-standing social conversations.368
References369 [1] Adam Kendon. Conducting Interaction: Patterns of Behavior in Focused Encounters. Number 7 in370 Studies in Interactional Sociolinguistics. Cambridge University Press, Cambridge ; New York, 1990. ISBN371 978-0-521-38036-2 978-0-521-38938-9. 1, 2, 3, 7, 8372
[2] Alessandro Vinciarelli, Maja Pantic, and Hervé Bourlard. Social signal processing: Survey of an emerging373 domain.374
[3] Dan Bohus and Eric Horvitz. Models for multiparty engagement in open-world dialog. In Proceedings of375 the SIGDIAL 2009 Conference on The 10th Annual Meeting of the Special Interest Group on Discourse and376 Dialogue - SIGDIAL ’09, pages 225–234, London, United Kingdom, 2009. Association for Computational377 Linguistics. ISBN 978-1-932432-64-0. doi: 10.3115/1708376.1708409. 1378
[4] Ryo Ishii, Shiro Kumano, and Kazuhiro Otsuka. Prediction of Next-Utterance Timing using Head379 Movement in Multi-Party Meetings. In Proceedings of the 5th International Conference on Human Agent380 Interaction, HAI ’17, pages 181–187, New York, NY, USA, October 2017. Association for Computing381 Machinery. ISBN 978-1-4503-5113-3. doi: 10.1145/3125739.3125765. 1382
[5] Anne Keitel and Moritz M Daum. The use of intonation for turn anticipation in observed conversations383 without visual signals as source of information. Frontiers in psychology, 6:108, 2015. 1, 3384
[6] Simon Garrod and Martin J Pickering. The use of content and timing to predict turn transitions. Frontiers385 in psychology, 6:751, 2015.386
[7] Amélie Rochet-Capellan and Susanne Fuchs. Take a breath and take the turn: how breathing meets turns in387 spontaneous dialogue. Philosophical Transactions of the Royal Society B: Biological Sciences, 369(1658):388 20130399, 2014.389
[8] M. Wlodarczak and M. Heldner. Respiratory turn-taking cues. In INTERSPEECH, 2016. 1, 3390 [9] Dan Bohus and Eric Horvitz. Managing Human-Robot Engagement with Forecasts and. . . um. . . Hesita-391 tions. page 8. 1, 3392 [10] Cynthia C. S. Liem, Markus Langer, Andrew Demetriou, Annemarie M. F. Hiemstra, Achmadnoer393 Sukma Wicaksana, Marise Ph. Born, and Cornelius J. König. Psychology Meets Machine Learning:394 Interdisciplinary Perspectives on Algorithmic Job Candidate Screening. In Hugo Jair Escalante, Sergio395 Escalera, Isabelle Guyon, Xavier Baró, Yağmur Güçlütürk, Umut Güçlü, and Marcel van Gerven, editors,396 Explainable and Interpretable Models in Computer Vision and Machine Learning, pages 197–253. Springer397 International Publishing, Cham, 2018. ISBN 978-3-319-98130-7 978-3-319-98131-4. doi: 10.1007/398 978-3-319-98131-4_9. 1399
[11] Erlend Nilsen, Diana Bowler, and John Linnell. Exploratory and confirmatory research in the open science400 era. Journal of Applied Ecology, 57, February 2020. doi: 10.1111/1365-2664.13571. 1401
[12] Starkey Duncan. Some signals and rules for taking speaking turns in conversations. Journal of Personality402 and Social Psychology, 23(2):283–292, 1972. ISSN 1939-1315(Electronic),0022-3514(Print). doi: 10.403 1037/h0033031. 1, 3, 8404
[13] Akko Kalma. Gazing in triads: A powerful signal in floor apportionment. British Journal of Social405 Psychology, 31(1):21–39, March 1992. 1, 3406
[14] Sanjay Bilakhia, Stavros Petridis, and Maja Pantic. Audiovisual Detection of Behavioural Mimicry. In407 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction, pages 123–128,408 Geneva, Switzerland, September 2013. IEEE. ISBN 978-0-7695-5048-0. doi: 10.1109/ACII.2013.27. 1, 4409
[15] Felix van Doorn. Rituals of Leaving: Predictive Modelling of Leaving Behaviour in Conversation. Master410 of Science Thesis, Delft University of Technology, 2018. 1, 2, 3411
[16] Laura Cabrera-Quiros, Andrew Demetriou, Ekin Gedik, Leander van der Meij, and Hayley Hung. The412 matchnmingle dataset: a novel multi-sensor resource for the analysis of social interactions and group413 dynamics in-the-wild during free-standing conversations and speed dates. IEEE Transactions on Affective414 Computing, 2018. 2415
[17] Nalini Ambady, Frank J Bernieri, and Jennifer A Richeson. Toward a histology of social behavior:416 Judgmental accuracy from thin slices of the behavioral stream. In Advances in experimental social417 psychology, volume 32, pages 201–271. Elsevier, 2000. 2, 3418
[18] Alessandro Vinciarelli, H Salamin, and M Pantic. Social Signal Processing: Understanding social419 interactions through nonverbal behavior analysis (PDF). 2009 IEEE Conference on Computer Vision and420 Pattern Recognition, CVPR 2009, June 2009. doi: 10.1109/CVPRW.2009.5204290. 2, 3, 8421
[19] Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, S. M. Ali Eslami,422 and Yee Whye Teh. Neural Processes. arXiv:1807.01622 [cs, stat], 2018. 2, 9423
[20] Mattias Heldner and Jens Edlund. Pauses, gaps and overlaps in conversations. Journal of Phonetics, 38(4):424 555–568, October 2010. ISSN 0095-4470. doi: 10.1016/j.wocn.2010.08.002. 3, 17425
[21] Monica M. Moore. Nonverbal courtship patterns in women: Context and consequences. Ethology and426 Sociobiology, 6(4):237–247, January 1985. ISSN 0162-3095. doi: 10.1016/0162-3095(85)90016-0. 3, 7427
[22] Stephen C. Levinson and Francisco Torreira. Timing in turn-taking and its implications for processing428 models of language. Frontiers in Psychology, 6, June 2015. ISSN 1664-1078. doi: 10.3389/fpsyg.2015.429 00731. 3430
[23] Erving Goffman. Behavior in Public Places: Notes on the Social Organization of Gatherings. The Free431 Press, 1. paperback ed., 24. printing edition. ISBN 978-0-02-911940-2. 3432
[24] Nina-Jo Moore, Hickson Mark III, and W Don. Stacks. Nonverbal communication: Studies and applications.433 2013. 3, 7434
[25] Harvey Sacks, Emanuel A. Schegloff, and Gail Jefferson. A Simplest Systematics for the Organization of435 Turn-Taking for Conversation. 50(4):40, 1974. 3436
[26] Charles Goodwin. Conversational Organization: Interaction Between Speakers and Hearers. January437 1981.438
[27] Carole Edelsky. Who’s Got the Floor? Language in Society, 10(3):383–421, 1981. ISSN 0047-4045. 3439 [28] Allan Wang and Aaron Steinfeld. Group Split and Merge Prediction With 3D Convolutional Networks.440 IEEE Robotics and Automation Letters, 5(2):1923–1930, April 2020. ISSN 2377-3766. doi: 10.1109/LRA.441 2020.2969947. 3442
[29] Massimo Mastrangeli, Martin Schmidt, and Lucas Lacasa. The roundtable: An abstract model of conversa-443 tion dynamics. arXiv:1010.2943 [physics], October 2010. 3444
[30] Louis Airale, Dominique Vaufreydaz, and Xavier Alameda-Pineda. SocialInteractionGAN: Multi-person445 Interaction Sequence Generation. arXiv:2103.05916 [cs, stat], March 2021. 3, 9446
[31] Navyata Sanghvi, Ryo Yonetani, and Kris Kitani. Mgpi: A computational model of multiagent group447 perception and interaction. arXiv preprint arXiv:1903.01537, 2019. 3, 7, 9448
[32] Hanbyul Joo, Tomas Simon, Mina Cikara, and Yaser Sheikh. Towards Social Artificial Intelligence:449 Nonverbal Social Signal Prediction in a Triadic Interaction. In 2019 IEEE/CVF Conference on Computer450 Vision and Pattern Recognition (CVPR), pages 10865–10875, Long Beach, CA, USA, June 2019. IEEE.451 ISBN 978-1-72813-293-8. doi: 10.1109/CVPR.2019.01113. 3, 4, 8452
[33] Emilie Delaherche, Mohamed Chetouani, Ammar Mahdhaoui, Catherine Saint-Georges, Sylvie Viaux,453 and David Cohen. Interpersonal Synchrony: A Survey of Evaluation Methods across Disciplines. IEEE454 Transactions on Affective Computing, 3(3):349–365, July 2012. ISSN 1949-3045. doi: 10.1109/T-AFFC.455 2012.12. 4456
[34] Dirk Helbing and Peter Molnar. Social Force Model for Pedestrian Dynamics. Physical Review E, 51(5):457 4282–4286, May 1995. ISSN 1063-651X, 1095-3787. doi: 10.1103/PhysRevE.51.4282. 4458
[35] Jarosław Wąs, Bartłomiej Gudowski, and Paweł J. Matuszyk. Social Distances Model of Pedestrian459 Dynamics. In David Hutchison, Takeo Kanade, Josef Kittler, Jon M. Kleinberg, Friedemann Mattern,460 John C. Mitchell, Moni Naor, Oscar Nierstrasz, C. Pandu Rangan, Bernhard Steffen, Madhu Sudan, Demetri461 Terzopoulos, Dough Tygar, Moshe Y. Vardi, Gerhard Weikum, Samira El Yacoubi, Bastien Chopard, and462 Stefania Bandini, editors, Cellular Automata, volume 4173, pages 492–501. Springer Berlin Heidelberg,463 Berlin, Heidelberg, 2006. ISBN 978-3-540-40929-8 978-3-540-40932-8. doi: 10.1007/11861201_57.464
[36] Gianluca Antonini, Michel Bierlaire, and Mats Weber. Discrete Choice Models for Pedestrian Walking465 Behavior. Transportation Research Part B: Methodological, 40:667–687, September 2006. doi: 10.1016/j.466 trb.2005.09.006.467
[37] Adrien Treuille, Seth Cooper, and Zoran Popović. Continuum crowds. ACM Transactions on Graphics /468 SIGGRAPH 2006, 25(3):1160–1168, July 2006.469
[38] Alexandre Robicquet, Amir Sadeghian, Alexandre Alahi, and Silvio Savarese. Learning Social Etiquette:470 Human Trajectory Understanding In Crowded Scenes. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max471 Welling, editors, Computer Vision – ECCV 2016, volume 9912, pages 549–565. Springer International472 Publishing, Cham, 2016. ISBN 978-3-319-46483-1 978-3-319-46484-8. doi: 10.1007/978-3-319-46484-8_473 33. 4474
[39] J. M. Wang, D. J. Fleet, and A. Hertzmann. Gaussian Process Dynamical Models for Human Motion.475 IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(2):283–298, February 2008. ISSN476 1939-3539. doi: 10.1109/TPAMI.2007.1167.477
[40] Christopher Tay and Christian Laugier. Modelling Smooth Paths Using Gaussian Processes. In Proc. of478 the Int. Conf. on Field and Service Robotics, 2007.479
[41] Andrew Patterson, Arun Lakshmanan, and Naira Hovakimyan. Intent-Aware Probabilistic Trajectory480 Estimation for Collision Prediction with Uncertainty Quantification. arXiv:1904.02765 [cs, math], April481 2019. 4482
[42] Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio483 Savarese. Social LSTM: Human Trajectory Prediction in Crowded Spaces. In 2016 IEEE Conference on484 Computer Vision and Pattern Recognition (CVPR), pages 961–971, Las Vegas, NV, USA, June 2016. IEEE.485 ISBN 978-1-4673-8851-1. doi: 10.1109/CVPR.2016.110. 4486
[43] Pu Zhang, Wanli Ouyang, Pengfei Zhang, Jianru Xue, and Nanning Zheng. SR-LSTM: State Refinement487 for LSTM towards Pedestrian Trajectory Prediction. arXiv:1903.02793 [cs], March 2019.488
[44] Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social GAN: Socially489 Acceptable Trajectories with Generative Adversarial Networks. arXiv:1803.10892 [cs], March 2018. 8490
[45] Yingfan Huang, Huikun Bi, Zhaoxin Li, Tianlu Mao, and Zhaoqi Wang. STGAT: Modeling Spatial-491 Temporal Interactions for Human Trajectory Prediction. In 2019 IEEE/CVF International Conference492 on Computer Vision (ICCV), pages 6271–6280, Seoul, Korea (South), October 2019. IEEE. ISBN493 978-1-72814-803-8. doi: 10.1109/ICCV.2019.00637.494
[46] Abduallah Mohamed, Kun Qian, Mohamed Elhoseiny, and Christian Claudel. Social-STGCNN: A Social495 Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction. arXiv:2002.11927496 [cs], February 2020. 4497
[47] MarcAurelio Ranzato, Arthur Szlam, Joan Bruna, Michael Mathieu, Ronan Collobert, and Sumit Chopra.498 Video (language) modeling: A baseline for generative models of natural videos. arXiv:1412.6604 [cs],499 December 2014. 4500
[48] Jacob Walker, Kenneth Marino, Abhinav Gupta, and Martial Hebert. The Pose Knows: Video Forecasting501 by Generating Pose Futures. arXiv:1705.00053 [cs], April 2017.502
[49] Jacob Walker, Abhinav Gupta, and Martial Hebert. Dense Optical Flow Prediction from a Static Image.503 In 2015 IEEE International Conference on Computer Vision (ICCV), pages 2443–2451, Santiago, Chile,504 December 2015. IEEE. ISBN 978-1-4673-8391-2. doi: 10.1109/ICCV.2015.281.505
[50] Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick506 van der Smagt, Daniel Cremers, and Thomas Brox. FlowNet: Learning Optical Flow with Convolutional507 Networks. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 2758–2766, Santiago,508 December 2015. IEEE. ISBN 978-1-4673-8391-2. doi: 10.1109/ICCV.2015.316.509
[51] Jacob Walker, Abhinav Gupta, and Martial Hebert. Patch to the Future: Unsupervised Visual Prediction.510 In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 3302–3309, Columbus, OH,511 USA, June 2014. IEEE. ISBN 978-1-4799-5118-5. doi: 10.1109/CVPR.2014.416.512
[52] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Anticipating Visual Representations from513 Unlabeled Video. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages514 98–106, Las Vegas, NV, USA, June 2016. IEEE. ISBN 978-1-4673-8851-1. doi: 10.1109/CVPR.2016.18.515 4516
[53] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov. Unsupervised Learning of Video Repre-517 sentations using LSTMs. arXiv:1502.04681 [cs], February 2015. 4518
[54] Alexey Dosovitskiy and Vladlen Koltun. Learning to Act by Predicting the Future. arXiv:1611.01779 [cs],519 November 2016. 4520
[55] Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-Learning in Neural521 Networks: A Survey. arXiv:2004.05439 [cs, stat], November 2020. 4522
[56] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to Sequence Learning with Neural Networks.523 In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in524 Neural Information Processing Systems 27, pages 3104–3112. Curran Associates, Inc., 2014. 5525
[57] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger526 Schwenk, and Yoshua Bengio. Learning Phrase Representations using RNN Encoder-Decoder for Statistical527 Machine Translation. arXiv:1406.1078 [cs, stat], September 2014. 5528
[58] Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals,529 and Yee Whye Teh. Attentive Neural Processes. arXiv:1901.05761 [cs, stat], July 2019. 5, 7, 9530
[59] Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. PointNet: Deep Learning on Point Sets for531 3D Classification and Segmentation. arXiv:1612.00593 [cs], April 2017. 6532
[60] Alex Kendall and Roberto Cipolla. Geometric Loss Functions for Camera Pose Regression with Deep533 Learning. arXiv:1704.00390 [cs], May 2017. 6, 9534
[61] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz535 Kaiser, and Illia Polosukhin. Attention Is All You Need. arXiv:1706.03762 [cs], June 2017. 6536
[62] Gautam Singh, Jaesik Yoon, Youngsung Son, and Sungjin Ahn. Sequential Neural Processes. URL537 http://arxiv.org/abs/1906.10264. 7, 9538
[63] Jaesik Yoon, Gautam Singh, and Sungjin Ahn. Robustifying Sequential Neural Processes. In International539 Conference on Machine Learning, pages 10861–10870. PMLR, November 2020.540
[64] Timon Willi, Jonathan Masci, Jürgen Schmidhuber, and Christian Osendorfer. Recurrent Neural Processes.541 arXiv:1906.05915 [cs, stat], November 2019.542
[65] Sumit Kumar. Spatiotemporal Modeling using Recurrent Neural Processes. Master of Science Thesis,543 Carnegie Mellon University, page 43. 7544
[66] Shenghao Qin, Jiacheng Zhu, Jimmy Qin, Wenshuo Wang, and Ding Zhao. Recurrent Attentive Neural545 Process for Sequential Data. arXiv:1910.09323 [cs, stat], October 2019. 7546
[67] Marynel Vazquez, Aaron Steinfeld, and Scott E. Hudson. Maintaining awareness of the focus of attention547 of a conversation: A robot-centric reinforcement learning approach. In 2016 25th IEEE International548 Symposium on Robot and Human Interactive Communication (RO-MAN), pages 36–43, New York, NY,549 USA, August 2016. IEEE. ISBN 978-1-5090-3929-6. doi: 10.1109/ROMAN.2016.7745088. 7, 9550
[68] Xavier Alameda-Pineda, Yan Yan, Elisa Ricci, Oswald Lanz, and Nicu Sebe. Analyzing Free-standing551 Conversational Groups: A Multimodal Approach. pages 5–14. ACM Press, 2015. ISBN 978-1-4503-3459-4.552 doi: 10.1145/2733373.2806238. 8, 9553
[69] Lu Zhang and Hayley Hung. On Social Involvement in Mingling Scenarios: Detecting Associates of554 F-formations in Still Images. IEEE Transactions on Affective Computing, 2018.555
[70] Stephanie Tan, David M. J. Tax, and Hayley Hung. Multimodal Joint Head Orientation Estimation in556 Interacting Groups via Proxemics and Interaction Dynamics. Proceedings of the ACM on Interactive,557 Mobile, Wearable and Ubiquitous Technologies, 5(1):1–22, March 2021. ISSN 2474-9567. doi: 10.1145/558 3448122. 8, 9559
[71] Rutger Rienks, Ronald Poppe, and Mannes Poel. Speaker Prediction based on Head Orientations. page 7.560 [72] M. Farenzena, A. Tavano, L. Bazzani, D. Tosato, G. Paggetti, G. Menegaz, V. Murino, and M. Cristani.561 Social interactions by visual focus of attention in a three-dimensional environment. Expert Systems, 30(2):562 115–127, May 2013. ISSN 02664720. doi: 10.1111/j.1468-0394.2012.00622.x. 9563
[73] S.O. Ba and J.-M. Odobez. Recognizing Visual Focus of Attention From Head Pose in Natural Meetings.564 IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 39(1):16–33, February 2009.565 ISSN 1083-4419. doi: 10.1109/TSMCB.2008.927274. 9566
[74] Chirag Raman, Stephanie Tan, and Hayley Hung. A modular approach for synchronized wireless multi-567 modal multisensor data acquisition in highly dynamic social settings. arXiv preprint arXiv:2008.03715,568 2020. 9569
[75] Mason Swofford, John Charles Peruzzi, Nathan Tsoi, Sydney Thompson, Roberto Martín-Martín, Silvio570 Savarese, and Marynel Vázquez. Improving Social Awareness Through DANTE: A Deep Affinity Network571 for Clustering Conversational Interactants. Proceedings of the ACM on Human-Computer Interaction, 4572 (CSCW1):1–23, May 2020. ISSN 2573-0142. doi: 10.1145/3392824. 9573
[76] Tuan Anh Le, Hyunjik Kim, and Marta Garnelo. Empirical Evaluation of Neural Process Objectives.574 page 9. 16575
[77] Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv:1412.6980 [cs],576 January 2017. 16577
[78] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor578 Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang,579 Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie580 Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In581 H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and R. Garnett, editors, Advances in582 Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019. 16583
[79] WA Falcon et al. Pytorch lightning. GitHub. Note: https://github.com/PyTorchLightning/pytorch-lightning,584 3, 2019. 16585
[80] Chirag Raman and Hayley Hung. Towards automatic estimation of conversation floors within f-formations.586 In 2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and587 Demos (ACIIW), pages 175–181. IEEE, 2019. 17588
Checklist589 1. For all authors...590
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s591 contributions and scope? [Yes] We mention the contributions of the task formulation592 and method in Section 1, along with our simplifying assumption for the setting, and593 circle back to discussing its implications in Section 7.594
(b) Did you describe the limitations of your work? [Yes] Please refer to the discussion in595 Section 7.596
(c) Did you discuss any potential negative societal impacts of your work? [N/A]597 (d) Have you read the ethics review guidelines and ensured that your paper conforms to598 them? [Yes]599 2. If you are including theoretical results...600
(a) Did you state the full set of assumptions of all theoretical results? [N/A]601 (b) Did you include complete proofs of all theoretical results? [N/A]602
3. If you ran experiments...603 (a) Did you include the code, data, and instructions needed to reproduce the main experi-604 mental results (either in the supplemental material or as a URL)? [Yes] Code, processed605 data, splits, and test batches are provided for reproduction in the Supplementary mate-606 rial.607
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they608 were chosen)? [Yes] Please refer to Appendices C and D.609
(c) Did you report error bars (e.g., with respect to the random seed after running exper-610 iments multiple times)? [Yes] We have reported mean and std. for the metrics over611 individual sequences in the test sets.612
(d) Did you include the total amount of compute and the type of resources used (e.g., type613 of GPUs, internal cluster, or cloud provider)? [Yes] Please refer to Appendix C.2. We614 specify the specific GPUs used for our experiments and their corresponding memory615 capacities.616
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...617 (a) If your work uses existing assets, did you cite the creators? [Yes] We have cited and618 discussed the original work proposing the Haggling dataset.619 (b) Did you mention the license of the assets? [No] The Haggling dataset is freely available620 for non-commercial and research purpose only.621 (c) Did you include any new assets either in the supplemental material or as a URL? [No]622 (d) Did you discuss whether and how consent was obtained from people whose data you’re623 using/curating? [No] The Haggling dataset is freely available for non-commercial and624 research purpose only.625
(e) Did you discuss whether the data you are using/curating contains personally identifi-626 able information or offensive content? [No] However, we visualized the features in627 Blender3D from scratch to not display original videos with the subjects.628
5. If you used crowdsourcing or conducted research with human subjects...629 (a) Did you include the full text of instructions given to participants and screenshots, if630 applicable? [N/A]631 (b) Did you describe any potential participant risks, with links to Institutional Review632 Board (IRB) approvals, if applicable? [N/A]633 (c) Did you include the estimated hourly wage paid to participants and the total amount634 spent on participant compensation? [N/A]635

Review summaries:
This paper presents social process models that solve a social cue forecasting problem. It is defined as predicting figure behavioral cues of participants who have social interaction given observed their behaviors. The social process (SP) is based on a neural process (NP) that can solve the meta-learning problem, and mix the seq2seq models to encode the social behaviors. The authors also suggest encoding a partner’s behaviors. Experiments on the synthetic dataset that mimics glancing behavior show that SP outperforms NP in terms of fitting in the future timestamp. Experiments on a real dataset that captures the head, body, and speaking behaviors of three people during a conversation show that SP outperforms NP in terms of predicted negative log-likelihood.  The main strength of this paper is suggesting an interesting task - social cue forecasting and quite simple models (# parameters 3.0M) to resolve the problem than baseline (NP, # parameters 2.8M).  I enjoy reading this paper since it can make many interesting questions that promise researchers to do future research. 

This paper focuses on forecasting the certainty of nonverbal behavior cues for each individual in a group social conversation. Instead of using limited labelled data, this paper uses low-level unlabeled data to build models that estimate the certainty of individuals' future behivour, given their past behivour. This paper proposes Social Process (SP) models, socially aware sequence-to-sequence (Seq2Seq) models within the Neural Process (NP) family, to handle this task. Evaluation on synthesized and real-world behavior data shows that the proposed SP models achieve higher log-likelihood than strong baselines.

The paper addresses the problem of nonverbal social cue forecasting in group social conversations, and proposes a bottom-up self-supervised approach to solve this problem. The bottom-up approach utilizes low-level behaviour cues to predict higher-level  social cues. The authors propose to use meta-learning and neural process to model the social process. Results on simulated and real data show that the proposed method is better than the baselines.  Contributions:  1. The paper takes the first step towards a bottom-up self-supervised method to solve the social cue forecasting problem. Previous methods are all top-down. 2. It utilizes the notion of neural process to formalize the concept of Social Process.