Social Processes: Self-Supervised Forecasting of Nonverbal Cues in Social Conversations

The default paradigm for the forecasting of human behavior in social conversations1 is characterized by top-down approaches . These involve identifying predictive2 relationships between low level nonverbal cues and future semantic events of inter-3 est ( e.g . turn changes , group leaving ) . A common hurdle however , is the limited4 availability of labeled data for supervised learning . In this work , we take the first5 step in the direction of a bottom-up self-supervised approach in the domain . We6 formulate the task of Social Cue Forecasting to leverage the larger amount of7 unlabeled low-level behavior cues , and characterize the modeling challenges in-8 volved . To address these , we take a meta-learning approach and propose the Social9 Process ( SP ) models—socially aware sequence-to-sequence ( Seq2Seq ) models10 within the Neural Process ( NP ) family . SP models learn extractable representations11 of non-semantic future cues for each participant , while capturing global uncertainty12 by jointly reasoning about the future for all members of the group . Evaluation on13 synthesized and real-world behavior data shows that our SP models achieve higher14 log-likelihood than the NP baselines , and also highlights important considerations15 for applying such techniques within the domain of social human interactions.16 1 Introduction17 Picture a situated interactive agent such as a social robot conversing with a group of people . How18 can agents act in such a setting ? We sustain conversations spatially and temporally through explicit19 behavioral cues—examples include locations of partners , their orientation , gestures , gaze , and floor20 control actions [ 1–3 ] . Evidence suggests that we employ an anticipation of these and other cues to21 navigate daily social interactions [ 1 , 4–8 ] . Consequently , the ability to forecast the future constitutes22 a natural objective towards the realization of machines with social skills . As such , interactive agents23 typically contend with uncertainties in inferences surrounding cues [ 3 ] . So beyond making real-time24 inferences , such systems may achieve more fluid interactions by leveraging the ability to forecast25 future states of the conversation [ 9 ] .26 In addition to the development of social agents , behavior forecasting is also of significance in social27 psychology , where the focus is on gaining insight into human behavior . Since human-interpretability28 is of essence , top-down approaches largely constitute the default paradigm , where specific events of29 semantic interest are selected first for consideration and their relationship to potentially predictive cues30 are studied in isolation—either in controlled interactions in lab settings , or in subsequent statistical31 analyses [ 10 , 11 ] . Examples of such semantic events include speaker turn transitions [ 5 , 12 , 13 ] ,32 mimicry episodes [ 14 ] , or the termination of an interaction [ 9 , 15 ] . However , one hurdle in the33 top-down paradigm is limited data . The events ( that constitute the labels or the dependent variables ) 34 often occur infrequently over a longer interaction , reducing the effective amount of labeled data . This35 precludes the use of neural supervised learning techniques that tend to be data intensive.36 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . In this work , we take an initial step towards a bottom-up approach to forecasting human behavior for37 free standing conversational groups . Our guiding motivation is to learn predictive representations of38 general future social behavior by utilizing unlabeled streams of low-level behavioral features . We do39 this by regressing future sequences of these features from observed sequences of the same features in40 a self-supervised manner . We term this task of non-semantic future behavior forecasting as Social41 Cue Forecasting ( SCF ) .42 Our approach is built on the observation that the social signal [ 17 ] —the high-level attitudes and social43 meaning transferred in interactions—is already embedded in the low-level cues [ 18 ] . To conceptually44 illustrate the contrasting top-down and bottom-up approaches on an example task , Figure 1 depicts45 an instance of a group leaving event in a naturalistic social conversation . Evidence suggests that46 such events can be anticipated from certain preceding rituals [ 15 ] reflected in the postural changes of47 conversing members [ 1 ] . van Doorn [ 15 ] built a predictor using 200 instances of group leaving found48 in over 90 minutes of mingling interaction and hand-crafted features . In contrast , our bottom-up49 approach would entail learning task agnostic representations of future behavior using the entire 9050 minutes of data , and then training simpler predictors for group leaving using the learnt representations51 as input . The figure also illustrates the complexity of naturalistic interactions where cross-group52 social influence exists . In this work we focus on the simpler setting of a single group in a scene.53 There are several challenges intrinsic to computationally modeling future behavior in social conversa-54 tions . The future is intrinsically uncertain , the forecasts for interaction partners are inter-dependent,55 and the social dynamics is unique for each grouping of individuals . We address these through the56 following contributions:57 • We formalize the task of SCF . We characertize the modeling challenges involved , and cast58 the problem into the meta-learning paradigm , allowing for data-efficient generalization to59 unseen groups at evaluation without learning group-specific models.60 • We propose and evaluate two socially aware Sequence-to-Sequence ( Seq2Seq ) models61 within the Neural Process ( NP ) family [ 19 ] for SCF in social conversations . Our method62 encodes complex social dynamics informative of future group behavior into extractable63 representations for each individual.64 This paper is organized as follows . In Section 2 we formally define and characterize the task of65 SCF . We situate this work within broader literature in Section 3 , and review background concepts66 in Section 4 . We propose the Social Process models in Section 5 and describe our experiments in67 Section 6 , concluding with a discussion of our findings in Section 7.68 2 Social Cue Forecasting69 The objective of SCF is to predict future behavioral cues of all people involved in a social encounter70 given an observed sequence of their behavioral features . More formally , let us denote a window71 of observed timesteps as tobs : = [ o1 , o2 , ... , oT ] , and an unobserved future time window as72 tfut : = [ f1 , f2 , ... , fT ] , f1 > oT . Note that tfut and tobs are typically non-overlapping , can be of73 different lengths , and tfut need not immediately follow tobs . Given a set of n interacting participants,74 let us denote their social cues over a tobs and tfut respectively as75 X : = [ bit ; t 2 tobs ] ni=1 , Y : = [ bit ; t 2 tfut ] ni=1 . ( 1a , b ) The vector bit encapsulates the multimodal cues of interest from participant i at time t. These can76 include head and body pose , speaking status , facial expressions , gestures , and verbal content—any77 information stream that combine to transfer social meaning.78 In its simplest form , given an X , the objective of SCF is to learn a single function f such that79 Y = f ( X ) . However , an inherent challenge in forecasting behavior is that an observed sequence80 of interaction does not have a deterministic future and can result in multiple socially valid ones—a81 window of overlapping speech between people both may and may not result in a change of speaker82 [ 12 , 20 ] , a change in head orientation may continue into a sweeping glance across the room or a darting83 glance stopping at a recipient of interest [ 21 ] . In some cases certain observed behaviors—intonation84 and gaze cues [ 5 , 13 ] or synchronization in speaker-listener speech [ 22 ] for turn-taking—might85 make some outcomes more likely than others . Given that there are both supporting and challenging86 arguments for how these observations influence subsequent behaviors [ 22 , p. 5 ; 13 , p. 22 ] , it would87 be beneficial if a data-driven model expresses a measure of uncertainty in its forecasts . We do this by88 modeling the distribution over possible futures p ( Y |X ) rather than forecasting a single future.89 Another design consideration arises from a defining characteristic of focused interactions—the90 participants ’ behaviors are interdependent . Participants in a group sustain equal access to the shared91 interaction space through cooperative maneuvering [ 1 , p. 220 ] . Moreover , when multiple groups92 are co-located , outsiders unengaged in these intra-group maneuvers may also influence the behavior93 of those within the group [ 23 , p. 91 ; 1 , p. 233 ] , sometimes causing them to leave ( see Figure 1 ) . It94 is therefore essential to capture uncertainty in forecasts at the global level—jointly forecasting one95 future for all participants at a time , rather than at a local output level—one future for each individual96 independent of the remaining participants ’ futures.97 How participants coordinate their behaviors is a function of several individual factors [ 24 , Chap . 1 ; 1,98 p. 237 ] . Consequently , the social dynamics guiding an interaction also has unique attributes for every99 unique grouping of individuals . Rather than learning group-specific models to capture these unique100 dynamics , we formulate the forecasting problem in terms of meta-learning , or few-shot function101 estimation . We interpret each unique group of individuals as the meta-learning notion of a task . The102 core idea is that we can learn to predict a distribution over futures for a target sequence X having103 captured the group ’ s unique behavioral tendencies from a context set C of their observed-future104 sequences . We can then generalize to unseen groups at evaluation by conditioning on a short observed105 slice of their interaction . We believe that this approach is especially suitable for social conversation106 forecasting—a setting that involves a limited data regime where good uncertainty estimates are107 desirable . Note that when conditioning on context is removed ( C = ? ) , we simply revert to the108 formulation p ( Y |X ) .109 3 Related Work110 Free-standing conversations are an example of what social scientists call focused interactions , said to111 arise when a “ group of persons gather close together and openly cooperate to sustain a single focus112 of attention , typically by taking turns at talking ” [ 23 , p. 24 ] . A long-standing topic of study has been113 the systematic organization of turn-taking [ 25–27 ] , with a particular interest in the event of upcoming114 speaking turns [ 5–8 ] . There has also been some interest in the forecasting task itself , to anticipate115 disengagement from an interaction [ 9 , 15 ] , the splitting or merging of groups [ 28 ] , the time-evolving116 size of a group [ 29 ] or semantic social action labels [ 30 , 31 ] . Most of these works use heuristics,117 either to generate semantic labels [ 9 ] , model the dynamics itself [ 29 ] , or hand-craft features [ 15 ] .118 Although not a forecasting task , the closest work that shares our motivation in predicting non-semantic119 low-level features is the recently introduced task of Social Signal Prediction ( SSP ) [ 32 ] . The objective120 is to predict the social cues1 of a target person using cues from the communication partners as121 1In the domain of Social Signal Processing , a social signal [ 17 ] refers to the relational attitudes displayed by people . It is a high-level construct resulting from the perception of cues ( see Fig . 1 in [ 18 ] ) . From this input ( Joo et al . focus on predictions within the same time window [ 32 , Eq . 6 ] ) . While the most122 general formulation of SSP involves forecasting a single timestep for a target person given the123 entire group ’ s past behavior [ 32 , Eq . 3 ] , generalizing this formulation runs into an inherent problem ; 124 applying the definition to forecasting entails iteratively treating each individual as target , learning125 separate functions for every person . However , as we discuss in Section 2 , these futures of interacting126 individuals are not independent given observed group behavior . Furthermore , a constrained definition127 of forecasting that predicts an immediate step into the future is limiting , since forecasting an event128 that occurs after a delay ( e.g . a time lagged synchrony [ 33 ] or mimicry [ 14 ] episode ) might be of129 interest . Operationalizing this definition would entail a sliding window iteratively using predictions130 over the offset between tobs and tfut as input , which would cascade prediction errors.131 A related social setting where forecasting has been of interest is that of unfocused interactions . These132 occur when individuals find themselves by circumstance in the immediate presence of each other,133 such as pedestrians walking in proximity . Early approaches for forecasting pedestrian trajectories134 were heuristic based , involving hand-crafted energy potentials to describe the influence pedestrians135 have on each other [ 34–41 ] . More recent approaches encode the relative positional information136 directly into a neural architecture [ 42–46 ] .137 In a broad sense , the self-supervised learning aspects of this work has some overlap with recent138 approaches focusing on the non-interaction task of visual forecasting . These works have taken a139 non-semantic approach to predict low level pixel-based features or intermediate representations140 [ 38 , 47–52 ] , and demonstrated a utility of the learned representation for other tasks like semi-141 supervised classification [ 53 ] , or training agents in immersive environments [ 54 ] .142 4 Preliminaries143 Meta-learning . A supervised learning algorithm can be viewed as a function mapping a dataset144 C : = ( XC , YC ) : = { ( xi , yi ) } i2 [ NC ] to a predictor f ( x ) . Here NC is the number of datapoints in145 C , and [ NC ] : = { 1 , . . . , NC } . The key idea of meta-learning is to learn the learning process itself,146 modeling this function representing the initial algorithm using another supervised learning algorithm ; 147 hence the name meta-learning . In meta-learning literature , a task refers to each dataset in a collection148 M : = { Ti } Ntasksi=1 of related datasets [ 55 ] . For each task T , a meta-learner is episodically trained149 to fit a subset of target points D : = ( X , Y ) : = { ( xi , yi ) } i2 [ ND ] given another subset of context150 observations C. At meta-test time , the resulting predictor f ( x , C ) uses the information obtained151 during meta-learning to make predictions for unseen target points conditioned on context sets unseen152 at meta-training.153 Neural Processes Sharing the same core motivations , NPs are a family of latent variable models154 that extend the idea of meta-learning to situations where uncertainty in the predictions f ( x , C ) are155 desirable . They do this by meta-learning a map from datasets to stochastic processes , estimating a156 distribution over the predictions p ( Y |X , C ) . To capture this distribution , NPs model the conditional157 latent distribution p ( z|C ) from which a task representation z 2 Rd is sampled . This constitutes158 the model ’ s latent path . The context can also be incorporated through a deterministic path , via a159 representation rC 2 Rd aggregated over C. An observation model p ( yi|xi , rC , z ) then fits the target160 observations in D. The generative process for the NP is written as161 p ( Y |X , C ) : = Z p ( Y |X , C , z ) p ( z|C ) dz = Z p ( Y |X , rC , z ) q ( z|sC ) dz , ( 2 ) where p ( Y |X , rC , z ) : = Q i2 [ ND ] p ( yi|xi , rC , z ) . The latent z is modeled by a factorized Gaussian162 parameterized by sC : = fs ( C ) , with fs being a deterministic function invariant to order permutation163 over C. When the conditioning on context is removed ( C = ? ) , we have q ( z|s ? ) : = p ( z ) , the164 zero-information prior on z . C is encoded on the deterministic path using a function fr similar to165 fs , so that rC : = fr ( C ) . In practice this is implemented as rC = P i2 [ NC ] MLP ( xi , yi ) /NC . The166 observation model is referred to as the decoder , and q , fr , fs comprise the encoders . The parameters167 of the NP are learned for random subsets C and D by maximizing the evidence lower bound ( ELBO ) 168 log p ( Y |X , C ) Eq ( z|sD ) [ log p ( Y |X , C , z ) ] KL ( q ( z|sD ) ||q ( z|sC ) ) . ( 3 ) perspective , the task of Social Signal Prediction [ 32 ] is a misnomer since it still relates to the prediction of cues and not signals , a distinction we preserve in this work . 5 Social Processes169 In this section we present our socially aware Seq2Seq models within the NP family that is agnostic to170 group member identities and group size . To setup the task , we split the contextual interaction on which171 we condition into pairs of observed and future sequences , writing the context as C : = ( XC , YC ) : =172 ( Xj , Yk ) ( j , k ) 2 [ NC ] ⇥ [ NC ] , where every Xj occurs before the corresponding Yk . As discussed in173 Section 3 , domain experts focusing on behavior analysis might be interested in settings where tobs174 and tfut are offset by an arbitrary delay . Consequently , the jth tobs can have multiple associated tfut175 windows . Denoting the set of target window pairs as D : = ( X , Y ) : = ( Xj , Yk ) ( j , k ) 2 [ ND ] ⇥ [ ND ] , our176 focus in the rest of this work is to model the distribution p ( Y |X , C ) .177 The generative process for our model we call the Social Process ( SP ) follows Eq . 2 , which we178 extend to social forecasting in two ways . We embed an observed sequence x for an individual179 into a condensed encoding e 2 Rd that is then decoded into the future sequence using a Seq2Seq180 architecture [ 56 , 57 ] . Our intuition is that this would cause the representation to encode temporal181 information about the future . Further , for every individual we model this e as a function of their own182 behavior , and that of their partners as viewed by them . The intuition is that this captures the spatial183 influence partners have on the participant over the tobs . Using notation we established in Section 2,184 we define the observation model for the SP for a single participant pi as185 p ( yi|xi , C , z ) : = p ( bif1 , . . . , bifT |bio1 , . . . , bioT , C , z ) = p ( bif1 , . . . , bifT |ei , rC , z ) . ( 4 ) If decoding is carried out in an auto-regressive manner , we can further write the right hand side of186 Eq . 4 as QfT t=f1 p ( b i t|bit 1 , . . . , bif1 , ei , rC , z ) . Following the standard NP setting , we implement the187 observation model as a set of Gaussian distributions factorized over time and feature dimensions.188 We also incorporate the cross-attention mechanism from the Attentive Neural Process ( ANP ) [ 58 ] to189 define the variant Attentive Social Process ( ASP ) . Following Eq . 4 and the definition of the ANP , the190 corresponding observation model of the ASP for a single participant is defined as191 p ( yi|xi , C , z ) = p ( bif1 , . . . , bifT |ei , r⇤ ( C , xi ) , z ) . ( 5 ) Here each target query sequence xi⇤ attends to the context sequences XC to produce a query-specific192 representation r⇤ : = r⇤ ( C , xi⇤ ) 2 Rd . The model architectures are illustrated in Figure 2.193 Encoding Partner Behavior . While a typical Seq2Seq setup conditions the sequence decoder on194 solely a compact representation of the observed sequence , we ’ d like to condition an individual ’ s195 forecast on the observed behavior of both , themselves and their partners . We do this using a pair196 of sequence encoders : one to encode the temporal dynamics of participant pi ’ s features , eiself =197 fself ( xi ) , and another to encode the dynamics of a transformed representation of the features of pi ’ s198 partners , eipartner = fpartner ( ( xj , ( j 6=i ) ) ) . Using a separate network to encode partner behavior199 grants the practical advantage of being able to sample an individual ’ s and partners ’ features at different200 sampling rates.201 How do we model ( xj ) ? We want the partners ’ representation to possess two properties : per-202 mutation invariance—changing the order of the partners should not affect the representation ; and203 group size independence—we want to compactly represent all partners independent of the group size.204 Beyond coordinate space invariance , we wish to intuitively capture a view of the interaction from205 pi ’ s perspective . We extend the approach Qi et al . [ 59 ] applied to point clouds to focused interactions206 by computing pooled embeddings of relative behavioral features . Since most commonly considered207 nonverbal cues in literature ( see Section 6.3 ) include the attributes of orientation or location ( e.g.208 head/body pose or keypoints ) or a binary indicator ( such as speaking status ) , we specify how we209 transform these . The 3D pose ( orientation , location ) of every partner pj is transformed to a frame of210 reference defined by pi ’ s pose . At timestep t , denoting orientation , location , and binary speaking211 status for pi as bit = [ qi ; li ; si ] , and those for pj as b j t = [ q j ; lj ; sj ] , we have212 qrel = qi ⇤ ( qj ) 1 , lrel = lj li , srel = sj si . ( 6a-c ) Note that we use unit quaternions ( denoted q ) for representing orientation due their various benefits213 over other representations of rotation [ 60 , Sec . 3.2 ] . The operator ⇤ denotes the Hamilton product of214 the quaternions . These transformed features for each pj are encoded using an embedder MLP . The215 outputs are concatenated with ejself and processed by a pre-pooler MLP , which is followed by the216 symmetric element-wise Max-pooling function to obtain ( xj ) at each timestep . We capture the217 dynamics in the pooled representation over tobs using fpartner . Finally , we combine eiself and e i partner218 for pi through a linear projection ( defined by a weight matrix W ) to obtain the individual ’ s embedding219 eiind = W. [ e i self ; e i partner ] . Our intuition is that with information about both pi themselves , and of220 pi ’ s partners from pi ’ s point-of-view , eiind now contains the information required to predict pi ’ s221 future behavior.222 Encoding Future Window Offset . As we ’ ve discussed at the start of this section , a single tobs223 might have multiple associated tfut windows at different offsets . Our intuition is that training a224 sequence decoder to decode the same eiind into multiple sequences ( corresponding to the multiple225 tfut ) in the absence of any timing information might cause an averaging effect in either the decoder226 or the information encoded in eiind . One way around this would be to start decoding one timestep227 following the end of tobs and discard the predictions in the gap between tobs and tfut . However,228 if decoding is done auto-regressively this might lead to cascading errors over the gap . Instead , we229 address this one-to-many issue by injecting the offset information into eiind so that the decoder230 receives a unique encoded representation for every tfut to decode over . We do this by repurposing231 the idea of sinusoidal positional encodings [ 61 ] to encode offsets rather than relative positions in232 sequences . For a given tobs and tfut , and de-dimensional eiind we define the offset as t = f1 oT ,233 and the corresponding offset encoding OE t as234 OE ( t,2m ) = sin ( t/10000 2m/de ) , OE ( t,2m+1 ) = cos ( t/10000 2m/de ) . ( 7a , b ) Here m refers to the dimension index in the encoding . We finally compute the representation ei for235 Eqs . 4 and 5 as236 ei = eiind +OE t. ( 8 ) Auxiliary Loss Functions . We incorporate a geometric loss function that improves performance in237 pose regression tasks . For pi at time t , given the ground truth bit = [ q ; l ; s ] , and the predicted mean238 b̂it = [ q̂ ; l̂ ; ŝ ] , we denote the tuple ( bit , bit ) as Bit . We then have the location loss in Eucliden space239 Ll ( Bit ) = l l̂ , and we can regress the quaternion values using240 Lq ( Bit ) = q q̂ kq̂k . ( 9 ) Kendall and Cipolla [ 60 ] show how these losses can be combined using the homoscedastic uncertain-241 ties in position and orientation , ̂2l and ̂ 2 q:242 L ( Bit ) = Ll ( Bit ) exp ( ŝl ) + ŝl + Lq ( Bit ) exp ( ŝq ) + ŝq , ( 10 ) where ŝ : = log ̂2 . Using the binary cross-entropy loss for speaking status Ls ( Bit ) , we have the243 overall auxiliary loss over t 2 tfut:244 Laux ( Y , Ŷ ) = X i X t L ( Bit ) + Ls ( Bit ) . ( 11 ) The parameters of the SP and ASP are trained by maximizing the ELBO in Eq . 3 and minimizing this245 auxiliary loss function for each of our sequence decoders.246 6 Experiments and Results247 6.1 Models and Baselines248 Our modeling assumption is that the underlying stochastic process generating the behaviors does249 not evolve over time . Stated differently , we assume that the individual factors determining how250 participants coordinate behaviors—age , cultural background , personality variables [ 24 , Chap . 1 ; 1,251 p. 237 ] —are likely to remain the same over the short duration of a single interaction . This is in252 contrast to a related line of work that deals with meta-transfer learning , where the stochastic process253 itself changes over time [ 62–65 ] . We therefore compare against the NP and ANP family which254 share our model assumptions and meta-learning attributes . Note that in contrast to our methods,255 these baselines have direct access to the future sequences in the context , and therefore constitute a256 strong baseline . We consider two variants : -latent denoting only the latent path ; and -latent+det,257 containing both deterministic and stochastic paths . We further consider two attention mechanisms for258 the cross-attention module : -dot with dot attention , and -multihead with wide multi-head attention259 [ 58 ] . We operationalize the original definitions of the baseline models to sequences by collapsing the260 timestep and feature dimensions . While the ANP-RNN model [ 66 ] shares our model assumptions , it261 is defined for a task analogous to SSP for concurrent car locations , and can not be operationalized to262 forecasting in any simple way ( see Section 3 discussing the distinction ) . We experiment with two263 choices of architectures for the sequence encoders and decoders in our proposed models : multi-layer264 perceptrons ( MLP ) , and Gated Recurrent Units ( GRU ) . Implementation and training details for our265 experiments can be found in Appendix C.266 6.2 Evaluation on Synthesized Behavior : Forecasting Glancing Behavior267 With limited behavioral data availability , a common practice in the domain is to train and evaluate268 methods on synthesized behavior dynamics [ 31 , 67 ] . In keeping with this practice , we construct a269 synthesized dataset simulating two glancing behaviors in social settings [ 21 ] . We use a 1D sinusoid270 to represent horizontal head rotation over 20 timesteps . The sweeping Type I glance is represented by271 a pristine sinusoid , while the gaze fixating Type III glance is denoted by clipping the amplitude for272 the last six timesteps . The task is to forecast the signal over the last 10 timesteps ( tfut ) by observing273 the first 10 ( tobs ) . Consequently , the first half of tfut is certain , while uncertainty over the last half274 results from every observed sinusoid having two ground-truths . It is impossible to infer from an275 observed sequence alone if the head rotation will stop partway through the future . We describe276 additional data setup , model details , and quantitative results for this setting in Appendices A.1 , C277 and D.1 , respectively . Figure 3 illustrates the ground truths , predicted means and std . deviations278 for a sequence within and outside the context set . We observe that all models estimate the mean279 reasonably well , although our proposed SP models learn a slightly better fit . More crucially , the280 SP models—especially the SP-GRU—learn much better uncertainty estimates over the certain and281 uncertain parts of the future compared to the NP baseline.282 6.3 Real-World Behavior : The Haggling Dataset283 We also evaluate our models on real-world behavior data , using the Haggling dataset of triadic284 interactions [ 32 ] . Participants are engaged in an unscripted game where two sellers compete to sell a285 fictional product to a buyer who has to choose between the two . We use the same split of 79 training286 sets ( groups ) and 28 test sets used by Joo et al . [ 32 ] . In our experiments we consider the following287 social cues : head pose described by the 3D location of the nose keypoint and a face normal ; body pose288 described by the location of the mid-point of the shoulders and a body normal ; and binary speaking289 status . Apart from being the most commonly considered cues in computational analyses of such290 conversations [ 68–70 ] , pose and turn taking are found to be crucial in the sustaining of conversation291 [ 1 , 12 , 18 ] . We specify the dataset preprocessing details in Appendix D.2.292 6.4 Evaluation293 Context Regimes . We evaluate all models on two context regimes : random , and fixed-initial . The294 random regime follows the standard NP setting that the models are trained in . Context samples295 ( sequence-pairs ) are selected as a random subset of target samples , so the model is exposed to296 behaviors from any phase of the interaction lifecycle . Here we ensure that batches contain unique297 tobs to prevent any single observed sequence from dominating the aggregation of representations298 over the context split . At evaluation , we take 50 % of the batch as context . In the fixed-initial context299 regime , we investigate how the model can generalize knowledge of group specific characteristics300 from observing the initial dynamics of an interaction where certain gestures and patterns are more301 distinctive [ 1 , Chap . 6 ] . This matches what a social agent might face in a real-world scenario . Here302 we treat the first 20 % of the entire interaction as context , treating sequences from the rest as target.303 Evaluation Metrics . We report the negative log-likelihood ( NLL ) log p ( Y |X , C ) in Table 1304 ( computed by summing over feature dimensions and people , and averaging over timesteps ) . Beyond305 the NLL , we also report the error in the predicted means over test sequences in Table 2 : mean-squared306 error ( MSE ) for the head and body keypoint locations ; mean absolute error ( MAE ) in orientation307 in degrees ; and speaking status accuracy . Note that while the ground truth orientation normals are308 constrained in the horizontal plane , we don ’ t constrain our predicted quaternions . We therefore report309 the absolute error in rotation in 3D . The reported mean and std . deviation of all metrics are over310 sequences in the test sets . We further report the metrics for every timestep over tfut in Appendix A.2,311 and qualitative visualizations of the forecasts in Appendix B.312 6.5 Ablations313 Encoding Partner Behavior . Modeling the interaction from the perspective of each individual314 is a central idea in our approach . We investigate the influence of encoding partner behavior into315 individual representations riind on the performance . We train the SP-latent+det GRU variant in two316 configurations : no-pool , where we do not encode any partner behavior ; and pool-oT where we pool317 over partner representations only at the last timestep ( similar to [ 44 ] ) . We choose the SP-GRU model318 since it achieves the best trade-off between minimizing NLL and forecasting cues consistent with319 human behavior . Both configurations lead to worse NLL and location errors ( Appendix A.3 ) .320 Deterministic Decoding and Social Encoder Sharing . Error gradients can flow back into our321 sequence encoders through two paths : from the final stochastic sequence decoder , as well as the322 deterministic decoders on the latent and deterministic paths . We investigate the effect of the determin-323 istic decoders by training the SP-latent+det GRU model without them . We also investigate sharing a324 single social encoder between the Process Encoder and Process Decoder in Figure 2 . We find that325 removing the decoders only improves log-likelihood if the encoders are shared , and at the cost of326 head orientation errors ( Appendix A.3 ) .327 7 Discussion and Conclusion328 What qualifies as the best performing model for SCF ? Our SP-GRU learns the best fit for synthesized329 behavior . On the commonly used metric of NLL [ 19 , 58 , 62 ] , our SP-MLP models perform the best330 for real-world data . However , they fare the worst at estimating the mean . On the other hand , the331 SP-GRU models estimate a better likelihood than the NP baselines with comparable errors in mean332 forecast . While the NP baselines attain the lowest errors in predicted means , they also achieve the333 worst NLL . From the qualitative visualizations and ablations , it seems that the models minimize334 NLL at the cost of orientation errors ; in the case of SP-MLP seemingly by predicting the majority335 orientation of the two sellers who face the same direction . Also , the NP models forecast largely static336 futures . In contrast , while being more dynamic , the SP-GRU forecasts also contain some smoothing.337 Our synthesized glancing behavior is grounded in social literature , and matches the head pose features338 in the real-world data ( horizontal orientation ) . Why do we see a large discrepancy in qualitative339 forecasts ? One crucial distinction between the synthetic and real data is the subtlety and sparsity340 of motion . Our synthesized data makes the common implicit assumption that head pose is a proxy341 for gaze [ 31 , 67 , 68 , 70–72 ] . In real-world data , attention shifts through changes in gaze are not342 always accompanied by similar head rotations [ 73 , Fig . 5 ] , and gaze is harder to record non-invasively343 in-the-wild with reasonable accuracy . The consequence of this approximation is exacerbated in the344 triadic Haggling setting where people are arranged roughly in a triangle and within each other ’ s345 field of vision , making head movements even more subtle . In natural settings , groups occupy varied346 formations such as side-by-side , or L-arrangement [ 60 , p. 213 ] . Here the more accentuated pose347 changes could aid in anticipating behavior . From this perspective , the combination of limited data and348 our simplifying assumption of a single group in a scene is a primary limitation of this work . The only349 publicly available dataset meeting our assumptions is the Haggling dataset , where all interactions350 follow similar patterns . As targeted development of techniques for recording such datasets in-the-351 wild gain momentum [ 74 ] , evaluating these models in the different interaction settings would yield352 increased insight . Nevertheless , our aim in evaluating on synthesized as well as real-world data353 was to highlight the influence that such common implicit assumptions can have on performance354 when applying methods . As an aside , we believe that this subtlety and sparsity of motion is also an355 important distinction between forecasting in focused and unfocused interactions . While the same356 techniques can be applied in both scenarios , pedestrian location is a perpetually changing data stream.357 The broader goal of this paper is to take a step towards bridging a gap we perceive between research358 domains ; on one hand , we notice that there is a growing trend of applying deep learning techniques359 in the small data regime that is social behavior data [ 30 , 75 ] . Without citing specific works as360 negative exemplars , this is occasionally accompanied by surface treatment of social science literature.361 On the other hand , in our conversations we have also perceived a preemptive resistance to deep362 learning methods precisely due to limited data . We believe that our work here—specifically our363 conceptualization of conversations groups as meta-learning tasks grounded in extensive considerations364 from social literature ; our approach of learning extractable task-agnostic representations of predictive365 behavior ; and the distinction between real-world and synthesized dynamics commonly used for366 evaluation—is of value in stimulating a broader community discussion about the considerations when367 applying machine learning approaches within the domain of free-standing social conversations.368 References369 [ 1 ] Adam Kendon . Conducting Interaction : Patterns of Behavior in Focused Encounters . Number 7 in370 Studies in Interactional Sociolinguistics . Cambridge University Press , Cambridge ; New York , 1990 . ISBN371 978-0-521-38036-2 978-0-521-38938-9 . 1 , 2 , 3 , 7 , 8372 [ 2 ] Alessandro Vinciarelli , Maja Pantic , and Hervé Bourlard . Social signal processing : Survey of an emerging373 domain.374 [ 3 ] Dan Bohus and Eric Horvitz . Models for multiparty engagement in open-world dialog . In Proceedings of375 the SIGDIAL 2009 Conference on The 10th Annual Meeting of the Special Interest Group on Discourse and376 Dialogue - SIGDIAL ’ 09 , pages 225–234 , London , United Kingdom , 2009 . Association for Computational377 Linguistics . ISBN 978-1-932432-64-0. doi : 10.3115/1708376.1708409 . 1378 [ 4 ] Ryo Ishii , Shiro Kumano , and Kazuhiro Otsuka . Prediction of Next-Utterance Timing using Head379 Movement in Multi-Party Meetings . In Proceedings of the 5th International Conference on Human Agent380 Interaction , HAI ’ 17 , pages 181–187 , New York , NY , USA , October 2017 . Association for Computing381 Machinery . ISBN 978-1-4503-5113-3. doi : 10.1145/3125739.3125765 . 1382 [ 5 ] Anne Keitel and Moritz M Daum . The use of intonation for turn anticipation in observed conversations383 without visual signals as source of information . Frontiers in psychology , 6:108 , 2015 . 1 , 3384 [ 6 ] Simon Garrod and Martin J Pickering . The use of content and timing to predict turn transitions . Frontiers385 in psychology , 6:751 , 2015.386 [ 7 ] Amélie Rochet-Capellan and Susanne Fuchs . Take a breath and take the turn : how breathing meets turns in387 spontaneous dialogue . Philosophical Transactions of the Royal Society B : Biological Sciences , 369 ( 1658 ) :388 20130399 , 2014.389 [ 8 ] M. Wlodarczak and M. Heldner . Respiratory turn-taking cues . In INTERSPEECH , 2016 . 1 , 3390 [ 9 ] Dan Bohus and Eric Horvitz . Managing Human-Robot Engagement with Forecasts and . . . um . . . Hesita-391 tions . page 8 . 1 , 3392 [ 10 ] Cynthia C. S. Liem , Markus Langer , Andrew Demetriou , Annemarie M. F. Hiemstra , Achmadnoer393 Sukma Wicaksana , Marise Ph . Born , and Cornelius J. König . Psychology Meets Machine Learning:394 Interdisciplinary Perspectives on Algorithmic Job Candidate Screening . In Hugo Jair Escalante , Sergio395 Escalera , Isabelle Guyon , Xavier Baró , Yağmur Güçlütürk , Umut Güçlü , and Marcel van Gerven , editors,396 Explainable and Interpretable Models in Computer Vision and Machine Learning , pages 197–253 . Springer397 International Publishing , Cham , 2018 . ISBN 978-3-319-98130-7 978-3-319-98131-4. doi : 10.1007/398 978-3-319-98131-4_9 . 1399 [ 11 ] Erlend Nilsen , Diana Bowler , and John Linnell . Exploratory and confirmatory research in the open science400 era . Journal of Applied Ecology , 57 , February 2020. doi : 10.1111/1365-2664.13571 . 1401 [ 12 ] Starkey Duncan . Some signals and rules for taking speaking turns in conversations . Journal of Personality402 and Social Psychology , 23 ( 2 ) :283–292 , 1972 . ISSN 1939-1315 ( Electronic ) ,0022-3514 ( Print ) . doi : 10.403 1037/h0033031 . 1 , 3 , 8404 [ 13 ] Akko Kalma . Gazing in triads : A powerful signal in floor apportionment . British Journal of Social405 Psychology , 31 ( 1 ) :21–39 , March 1992 . 1 , 3406 [ 14 ] Sanjay Bilakhia , Stavros Petridis , and Maja Pantic . Audiovisual Detection of Behavioural Mimicry . In407 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction , pages 123–128,408 Geneva , Switzerland , September 2013 . IEEE . ISBN 978-0-7695-5048-0. doi : 10.1109/ACII.2013.27 . 1 , 4409 [ 15 ] Felix van Doorn . Rituals of Leaving : Predictive Modelling of Leaving Behaviour in Conversation . Master410 of Science Thesis , Delft University of Technology , 2018 . 1 , 2 , 3411 [ 16 ] Laura Cabrera-Quiros , Andrew Demetriou , Ekin Gedik , Leander van der Meij , and Hayley Hung . The412 matchnmingle dataset : a novel multi-sensor resource for the analysis of social interactions and group413 dynamics in-the-wild during free-standing conversations and speed dates . IEEE Transactions on Affective414 Computing , 2018 . 2415 [ 17 ] Nalini Ambady , Frank J Bernieri , and Jennifer A Richeson . Toward a histology of social behavior:416 Judgmental accuracy from thin slices of the behavioral stream . In Advances in experimental social417 psychology , volume 32 , pages 201–271 . Elsevier , 2000 . 2 , 3418 [ 18 ] Alessandro Vinciarelli , H Salamin , and M Pantic . Social Signal Processing : Understanding social419 interactions through nonverbal behavior analysis ( PDF ) . 2009 IEEE Conference on Computer Vision and420 Pattern Recognition , CVPR 2009 , June 2009. doi : 10.1109/CVPRW.2009.5204290 . 2 , 3 , 8421 [ 19 ] Marta Garnelo , Jonathan Schwarz , Dan Rosenbaum , Fabio Viola , Danilo J. Rezende , S. M. Ali Eslami,422 and Yee Whye Teh . Neural Processes . arXiv:1807.01622 [ cs , stat ] , 2018 . 2 , 9423 [ 20 ] Mattias Heldner and Jens Edlund . Pauses , gaps and overlaps in conversations . Journal of Phonetics , 38 ( 4 ) :424 555–568 , October 2010 . ISSN 0095-4470. doi : 10.1016/j.wocn.2010.08.002 . 3 , 17425 [ 21 ] Monica M. Moore . Nonverbal courtship patterns in women : Context and consequences . Ethology and426 Sociobiology , 6 ( 4 ) :237–247 , January 1985 . ISSN 0162-3095. doi : 10.1016/0162-3095 ( 85 ) 90016-0 . 3 , 7427 [ 22 ] Stephen C. Levinson and Francisco Torreira . Timing in turn-taking and its implications for processing428 models of language . Frontiers in Psychology , 6 , June 2015 . ISSN 1664-1078. doi : 10.3389/fpsyg.2015.429 00731 . 3430 [ 23 ] Erving Goffman . Behavior in Public Places : Notes on the Social Organization of Gatherings . The Free431 Press , 1. paperback ed. , 24. printing edition . ISBN 978-0-02-911940-2 . 3432 [ 24 ] Nina-Jo Moore , Hickson Mark III , and W Don . Stacks . Nonverbal communication : Studies and applications.433 2013 . 3 , 7434 [ 25 ] Harvey Sacks , Emanuel A. Schegloff , and Gail Jefferson . A Simplest Systematics for the Organization of435 Turn-Taking for Conversation . 50 ( 4 ) :40 , 1974 . 3436 [ 26 ] Charles Goodwin . Conversational Organization : Interaction Between Speakers and Hearers . January437 1981.438 [ 27 ] Carole Edelsky . Who ’ s Got the Floor ? Language in Society , 10 ( 3 ) :383–421 , 1981 . ISSN 0047-4045 . 3439 [ 28 ] Allan Wang and Aaron Steinfeld . Group Split and Merge Prediction With 3D Convolutional Networks.440 IEEE Robotics and Automation Letters , 5 ( 2 ) :1923–1930 , April 2020 . ISSN 2377-3766. doi : 10.1109/LRA.441 2020.2969947 . 3442 [ 29 ] Massimo Mastrangeli , Martin Schmidt , and Lucas Lacasa . The roundtable : An abstract model of conversa-443 tion dynamics . arXiv:1010.2943 [ physics ] , October 2010 . 3444 [ 30 ] Louis Airale , Dominique Vaufreydaz , and Xavier Alameda-Pineda . SocialInteractionGAN : Multi-person445 Interaction Sequence Generation . arXiv:2103.05916 [ cs , stat ] , March 2021 . 3 , 9446 [ 31 ] Navyata Sanghvi , Ryo Yonetani , and Kris Kitani . Mgpi : A computational model of multiagent group447 perception and interaction . arXiv preprint arXiv:1903.01537 , 2019 . 3 , 7 , 9448 [ 32 ] Hanbyul Joo , Tomas Simon , Mina Cikara , and Yaser Sheikh . Towards Social Artificial Intelligence:449 Nonverbal Social Signal Prediction in a Triadic Interaction . In 2019 IEEE/CVF Conference on Computer450 Vision and Pattern Recognition ( CVPR ) , pages 10865–10875 , Long Beach , CA , USA , June 2019 . IEEE.451 ISBN 978-1-72813-293-8. doi : 10.1109/CVPR.2019.01113 . 3 , 4 , 8452 [ 33 ] Emilie Delaherche , Mohamed Chetouani , Ammar Mahdhaoui , Catherine Saint-Georges , Sylvie Viaux,453 and David Cohen . Interpersonal Synchrony : A Survey of Evaluation Methods across Disciplines . IEEE454 Transactions on Affective Computing , 3 ( 3 ) :349–365 , July 2012 . ISSN 1949-3045. doi : 10.1109/T-AFFC.455 2012.12 . 4456 [ 34 ] Dirk Helbing and Peter Molnar . Social Force Model for Pedestrian Dynamics . Physical Review E , 51 ( 5 ) :457 4282–4286 , May 1995 . ISSN 1063-651X , 1095-3787. doi : 10.1103/PhysRevE.51.4282 . 4458 [ 35 ] Jarosław Wąs , Bartłomiej Gudowski , and Paweł J. Matuszyk . Social Distances Model of Pedestrian459 Dynamics . In David Hutchison , Takeo Kanade , Josef Kittler , Jon M. Kleinberg , Friedemann Mattern,460 John C. Mitchell , Moni Naor , Oscar Nierstrasz , C. Pandu Rangan , Bernhard Steffen , Madhu Sudan , Demetri461 Terzopoulos , Dough Tygar , Moshe Y. Vardi , Gerhard Weikum , Samira El Yacoubi , Bastien Chopard , and462 Stefania Bandini , editors , Cellular Automata , volume 4173 , pages 492–501 . Springer Berlin Heidelberg,463 Berlin , Heidelberg , 2006 . ISBN 978-3-540-40929-8 978-3-540-40932-8. doi : 10.1007/11861201_57.464 [ 36 ] Gianluca Antonini , Michel Bierlaire , and Mats Weber . Discrete Choice Models for Pedestrian Walking465 Behavior . Transportation Research Part B : Methodological , 40:667–687 , September 2006. doi : 10.1016/j.466 trb.2005.09.006.467 [ 37 ] Adrien Treuille , Seth Cooper , and Zoran Popović . Continuum crowds . ACM Transactions on Graphics /468 SIGGRAPH 2006 , 25 ( 3 ) :1160–1168 , July 2006.469 [ 38 ] Alexandre Robicquet , Amir Sadeghian , Alexandre Alahi , and Silvio Savarese . Learning Social Etiquette:470 Human Trajectory Understanding In Crowded Scenes . In Bastian Leibe , Jiri Matas , Nicu Sebe , and Max471 Welling , editors , Computer Vision – ECCV 2016 , volume 9912 , pages 549–565 . Springer International472 Publishing , Cham , 2016 . ISBN 978-3-319-46483-1 978-3-319-46484-8. doi : 10.1007/978-3-319-46484-8_473 33 . 4474 [ 39 ] J. M. Wang , D. J . Fleet , and A. Hertzmann . Gaussian Process Dynamical Models for Human Motion.475 IEEE Transactions on Pattern Analysis and Machine Intelligence , 30 ( 2 ) :283–298 , February 2008 . ISSN476 1939-3539. doi : 10.1109/TPAMI.2007.1167.477 [ 40 ] Christopher Tay and Christian Laugier . Modelling Smooth Paths Using Gaussian Processes . In Proc . of478 the Int . Conf . on Field and Service Robotics , 2007.479 [ 41 ] Andrew Patterson , Arun Lakshmanan , and Naira Hovakimyan . Intent-Aware Probabilistic Trajectory480 Estimation for Collision Prediction with Uncertainty Quantification . arXiv:1904.02765 [ cs , math ] , April481 2019 . 4482 [ 42 ] Alexandre Alahi , Kratarth Goel , Vignesh Ramanathan , Alexandre Robicquet , Li Fei-Fei , and Silvio483 Savarese . Social LSTM : Human Trajectory Prediction in Crowded Spaces . In 2016 IEEE Conference on484 Computer Vision and Pattern Recognition ( CVPR ) , pages 961–971 , Las Vegas , NV , USA , June 2016 . IEEE.485 ISBN 978-1-4673-8851-1. doi : 10.1109/CVPR.2016.110 . 4486 [ 43 ] Pu Zhang , Wanli Ouyang , Pengfei Zhang , Jianru Xue , and Nanning Zheng . SR-LSTM : State Refinement487 for LSTM towards Pedestrian Trajectory Prediction . arXiv:1903.02793 [ cs ] , March 2019.488 [ 44 ] Agrim Gupta , Justin Johnson , Li Fei-Fei , Silvio Savarese , and Alexandre Alahi . Social GAN : Socially489 Acceptable Trajectories with Generative Adversarial Networks . arXiv:1803.10892 [ cs ] , March 2018 . 8490 [ 45 ] Yingfan Huang , Huikun Bi , Zhaoxin Li , Tianlu Mao , and Zhaoqi Wang . STGAT : Modeling Spatial-491 Temporal Interactions for Human Trajectory Prediction . In 2019 IEEE/CVF International Conference492 on Computer Vision ( ICCV ) , pages 6271–6280 , Seoul , Korea ( South ) , October 2019 . IEEE . ISBN493 978-1-72814-803-8. doi : 10.1109/ICCV.2019.00637.494 [ 46 ] Abduallah Mohamed , Kun Qian , Mohamed Elhoseiny , and Christian Claudel . Social-STGCNN : A Social495 Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction . arXiv:2002.11927496 [ cs ] , February 2020 . 4497 [ 47 ] MarcAurelio Ranzato , Arthur Szlam , Joan Bruna , Michael Mathieu , Ronan Collobert , and Sumit Chopra.498 Video ( language ) modeling : A baseline for generative models of natural videos . arXiv:1412.6604 [ cs ] ,499 December 2014 . 4500 [ 48 ] Jacob Walker , Kenneth Marino , Abhinav Gupta , and Martial Hebert . The Pose Knows : Video Forecasting501 by Generating Pose Futures . arXiv:1705.00053 [ cs ] , April 2017.502 [ 49 ] Jacob Walker , Abhinav Gupta , and Martial Hebert . Dense Optical Flow Prediction from a Static Image.503 In 2015 IEEE International Conference on Computer Vision ( ICCV ) , pages 2443–2451 , Santiago , Chile,504 December 2015 . IEEE . ISBN 978-1-4673-8391-2. doi : 10.1109/ICCV.2015.281.505 [ 50 ] Alexey Dosovitskiy , Philipp Fischer , Eddy Ilg , Philip Hausser , Caner Hazirbas , Vladimir Golkov , Patrick506 van der Smagt , Daniel Cremers , and Thomas Brox . FlowNet : Learning Optical Flow with Convolutional507 Networks . In 2015 IEEE International Conference on Computer Vision ( ICCV ) , pages 2758–2766 , Santiago,508 December 2015 . IEEE . ISBN 978-1-4673-8391-2. doi : 10.1109/ICCV.2015.316.509 [ 51 ] Jacob Walker , Abhinav Gupta , and Martial Hebert . Patch to the Future : Unsupervised Visual Prediction.510 In 2014 IEEE Conference on Computer Vision and Pattern Recognition , pages 3302–3309 , Columbus , OH,511 USA , June 2014 . IEEE . ISBN 978-1-4799-5118-5. doi : 10.1109/CVPR.2014.416.512 [ 52 ] Carl Vondrick , Hamed Pirsiavash , and Antonio Torralba . Anticipating Visual Representations from513 Unlabeled Video . In 2016 IEEE Conference on Computer Vision and Pattern Recognition ( CVPR ) , pages514 98–106 , Las Vegas , NV , USA , June 2016 . IEEE . ISBN 978-1-4673-8851-1. doi : 10.1109/CVPR.2016.18.515 4516 [ 53 ] Nitish Srivastava , Elman Mansimov , and Ruslan Salakhutdinov . Unsupervised Learning of Video Repre-517 sentations using LSTMs . arXiv:1502.04681 [ cs ] , February 2015 . 4518 [ 54 ] Alexey Dosovitskiy and Vladlen Koltun . Learning to Act by Predicting the Future . arXiv:1611.01779 [ cs ] ,519 November 2016 . 4520 [ 55 ] Timothy Hospedales , Antreas Antoniou , Paul Micaelli , and Amos Storkey . Meta-Learning in Neural521 Networks : A Survey . arXiv:2004.05439 [ cs , stat ] , November 2020 . 4522 [ 56 ] Ilya Sutskever , Oriol Vinyals , and Quoc V Le . Sequence to Sequence Learning with Neural Networks.523 In Z. Ghahramani , M. Welling , C. Cortes , N. D. Lawrence , and K. Q. Weinberger , editors , Advances in524 Neural Information Processing Systems 27 , pages 3104–3112 . Curran Associates , Inc. , 2014 . 5525 [ 57 ] Kyunghyun Cho , Bart van Merrienboer , Caglar Gulcehre , Dzmitry Bahdanau , Fethi Bougares , Holger526 Schwenk , and Yoshua Bengio . Learning Phrase Representations using RNN Encoder-Decoder for Statistical527 Machine Translation . arXiv:1406.1078 [ cs , stat ] , September 2014 . 5528 [ 58 ] Hyunjik Kim , Andriy Mnih , Jonathan Schwarz , Marta Garnelo , Ali Eslami , Dan Rosenbaum , Oriol Vinyals,529 and Yee Whye Teh . Attentive Neural Processes . arXiv:1901.05761 [ cs , stat ] , July 2019 . 5 , 7 , 9530 [ 59 ] Charles R. Qi , Hao Su , Kaichun Mo , and Leonidas J. Guibas . PointNet : Deep Learning on Point Sets for531 3D Classification and Segmentation . arXiv:1612.00593 [ cs ] , April 2017 . 6532 [ 60 ] Alex Kendall and Roberto Cipolla . Geometric Loss Functions for Camera Pose Regression with Deep533 Learning . arXiv:1704.00390 [ cs ] , May 2017 . 6 , 9534 [ 61 ] Ashish Vaswani , Noam Shazeer , Niki Parmar , Jakob Uszkoreit , Llion Jones , Aidan N. Gomez , Lukasz535 Kaiser , and Illia Polosukhin . Attention Is All You Need . arXiv:1706.03762 [ cs ] , June 2017 . 6536 [ 62 ] Gautam Singh , Jaesik Yoon , Youngsung Son , and Sungjin Ahn . Sequential Neural Processes . URL537 http : //arxiv.org/abs/1906.10264 . 7 , 9538 [ 63 ] Jaesik Yoon , Gautam Singh , and Sungjin Ahn . Robustifying Sequential Neural Processes . In International539 Conference on Machine Learning , pages 10861–10870 . PMLR , November 2020.540 [ 64 ] Timon Willi , Jonathan Masci , Jürgen Schmidhuber , and Christian Osendorfer . Recurrent Neural Processes.541 arXiv:1906.05915 [ cs , stat ] , November 2019.542 [ 65 ] Sumit Kumar . Spatiotemporal Modeling using Recurrent Neural Processes . Master of Science Thesis,543 Carnegie Mellon University , page 43 . 7544 [ 66 ] Shenghao Qin , Jiacheng Zhu , Jimmy Qin , Wenshuo Wang , and Ding Zhao . Recurrent Attentive Neural545 Process for Sequential Data . arXiv:1910.09323 [ cs , stat ] , October 2019 . 7546 [ 67 ] Marynel Vazquez , Aaron Steinfeld , and Scott E. Hudson . Maintaining awareness of the focus of attention547 of a conversation : A robot-centric reinforcement learning approach . In 2016 25th IEEE International548 Symposium on Robot and Human Interactive Communication ( RO-MAN ) , pages 36–43 , New York , NY,549 USA , August 2016 . IEEE . ISBN 978-1-5090-3929-6. doi : 10.1109/ROMAN.2016.7745088 . 7 , 9550 [ 68 ] Xavier Alameda-Pineda , Yan Yan , Elisa Ricci , Oswald Lanz , and Nicu Sebe . Analyzing Free-standing551 Conversational Groups : A Multimodal Approach . pages 5–14 . ACM Press , 2015 . ISBN 978-1-4503-3459-4.552 doi : 10.1145/2733373.2806238 . 8 , 9553 [ 69 ] Lu Zhang and Hayley Hung . On Social Involvement in Mingling Scenarios : Detecting Associates of554 F-formations in Still Images . IEEE Transactions on Affective Computing , 2018.555 [ 70 ] Stephanie Tan , David M. J . Tax , and Hayley Hung . Multimodal Joint Head Orientation Estimation in556 Interacting Groups via Proxemics and Interaction Dynamics . Proceedings of the ACM on Interactive,557 Mobile , Wearable and Ubiquitous Technologies , 5 ( 1 ) :1–22 , March 2021 . ISSN 2474-9567. doi : 10.1145/558 3448122 . 8 , 9559 [ 71 ] Rutger Rienks , Ronald Poppe , and Mannes Poel . Speaker Prediction based on Head Orientations . page 7.560 [ 72 ] M. Farenzena , A. Tavano , L. Bazzani , D. Tosato , G. Paggetti , G. Menegaz , V. Murino , and M. Cristani.561 Social interactions by visual focus of attention in a three-dimensional environment . Expert Systems , 30 ( 2 ) :562 115–127 , May 2013 . ISSN 02664720. doi : 10.1111/j.1468-0394.2012.00622.x . 9563 [ 73 ] S.O . Ba and J.-M. Odobez . Recognizing Visual Focus of Attention From Head Pose in Natural Meetings.564 IEEE Transactions on Systems , Man , and Cybernetics , Part B ( Cybernetics ) , 39 ( 1 ) :16–33 , February 2009.565 ISSN 1083-4419. doi : 10.1109/TSMCB.2008.927274 . 9566 [ 74 ] Chirag Raman , Stephanie Tan , and Hayley Hung . A modular approach for synchronized wireless multi-567 modal multisensor data acquisition in highly dynamic social settings . arXiv preprint arXiv:2008.03715,568 2020 . 9569 [ 75 ] Mason Swofford , John Charles Peruzzi , Nathan Tsoi , Sydney Thompson , Roberto Martín-Martín , Silvio570 Savarese , and Marynel Vázquez . Improving Social Awareness Through DANTE : A Deep Affinity Network571 for Clustering Conversational Interactants . Proceedings of the ACM on Human-Computer Interaction , 4572 ( CSCW1 ) :1–23 , May 2020 . ISSN 2573-0142. doi : 10.1145/3392824 . 9573 [ 76 ] Tuan Anh Le , Hyunjik Kim , and Marta Garnelo . Empirical Evaluation of Neural Process Objectives.574 page 9 . 16575 [ 77 ] Diederik P. Kingma and Jimmy Ba . Adam : A Method for Stochastic Optimization . arXiv:1412.6980 [ cs ] ,576 January 2017 . 16577 [ 78 ] Adam Paszke , Sam Gross , Francisco Massa , Adam Lerer , James Bradbury , Gregory Chanan , Trevor578 Killeen , Zeming Lin , Natalia Gimelshein , Luca Antiga , Alban Desmaison , Andreas Kopf , Edward Yang,579 Zachary DeVito , Martin Raison , Alykhan Tejani , Sasank Chilamkurthy , Benoit Steiner , Lu Fang , Junjie580 Bai , and Soumith Chintala . Pytorch : An imperative style , high-performance deep learning library . In581 H. Wallach , H. Larochelle , A. Beygelzimer , F. dAlché-Buc , E. Fox , and R. Garnett , editors , Advances in582 Neural Information Processing Systems 32 , pages 8024–8035 . Curran Associates , Inc. , 2019 . 16583 [ 79 ] WA Falcon et al . Pytorch lightning . GitHub . Note : https : //github.com/PyTorchLightning/pytorch-lightning,584 3 , 2019 . 16585 [ 80 ] Chirag Raman and Hayley Hung . Towards automatic estimation of conversation floors within f-formations.586 In 2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and587 Demos ( ACIIW ) , pages 175–181 . IEEE , 2019 . 17588 Checklist589 1 . For all authors ... 590 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s591 contributions and scope ? [ Yes ] We mention the contributions of the task formulation592 and method in Section 1 , along with our simplifying assumption for the setting , and593 circle back to discussing its implications in Section 7.594 ( b ) Did you describe the limitations of your work ? [ Yes ] Please refer to the discussion in595 Section 7.596 ( c ) Did you discuss any potential negative societal impacts of your work ? [ N/A ] 597 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to598 them ? [ Yes ] 599 2 . If you are including theoretical results ... 600 ( a ) Did you state the full set of assumptions of all theoretical results ? [ N/A ] 601 ( b ) Did you include complete proofs of all theoretical results ? [ N/A ] 602 3 . If you ran experiments ... 603 ( a ) Did you include the code , data , and instructions needed to reproduce the main experi-604 mental results ( either in the supplemental material or as a URL ) ? [ Yes ] Code , processed605 data , splits , and test batches are provided for reproduction in the Supplementary mate-606 rial.607 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they608 were chosen ) ? [ Yes ] Please refer to Appendices C and D.609 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running exper-610 iments multiple times ) ? [ Yes ] We have reported mean and std . for the metrics over611 individual sequences in the test sets.612 ( d ) Did you include the total amount of compute and the type of resources used ( e.g. , type613 of GPUs , internal cluster , or cloud provider ) ? [ Yes ] Please refer to Appendix C.2 . We614 specify the specific GPUs used for our experiments and their corresponding memory615 capacities.616 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 617 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] We have cited and618 discussed the original work proposing the Haggling dataset.619 ( b ) Did you mention the license of the assets ? [ No ] The Haggling dataset is freely available620 for non-commercial and research purpose only.621 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ No ] 622 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re623 using/curating ? [ No ] The Haggling dataset is freely available for non-commercial and624 research purpose only.625 ( e ) Did you discuss whether the data you are using/curating contains personally identifi-626 able information or offensive content ? [ No ] However , we visualized the features in627 Blender3D from scratch to not display original videos with the subjects.628 5 . If you used crowdsourcing or conducted research with human subjects ... 629 ( a ) Did you include the full text of instructions given to participants and screenshots , if630 applicable ? [ N/A ] 631 ( b ) Did you describe any potential participant risks , with links to Institutional Review632 Board ( IRB ) approvals , if applicable ? [ N/A ] 633 ( c ) Did you include the estimated hourly wage paid to participants and the total amount634 spent on participant compensation ? [ N/A ] 635

Summary: This paper focuses on forecasting the certainty of nonverbal behavior cues for each individual in a group social conversation. Instead of using limited labelled data, this paper uses low-level unlabeled data to build models that estimate the certainty of individuals' future behivour, given their past behivour. This paper proposes Social Process (SP) models, socially aware sequence-to-sequence (Seq2Seq) models within the Neural Process (NP) family, to handle this task. Evaluation on synthesized and real-world behavior data shows that the proposed SP models achieve higher log-likelihood than strong baselines.