Before revision:  We present a new model architecture which we call a Nested Set Transformer which employs multi-head self-attention blocks over sets of sets that serve as a form of social attention between the elements of the sets at every timestep. Our approach can produce a distribution over  future trajectories for all agents under consideration , or focus upon the trajectory of an ego-agent.
After revision:  We present a new model architecture which we call a Nested Set Transformer which employs multi-head self-attention blocks over these sequences of sets to encode the sociotemporal relationships between the different actors of a scene. This produces either the trajectory of one ego-agent or a distribution over future trajectories for all agents under consideration , or focus upon the trajectory of an ego-agent.