# Non-Autoregressive Sparse Transformer Networks for Pedestrian Trajectory Prediction
## 1 Introduction  
Pedestrian trajectory prediction plays an important role in many fields such as autonomous driving [1, 2], robotic motion planning [3], video surveillance [4, 5], and computer vision [6, 7, 8, 9, 10]. It is a challenging task because the pedestrian motion is influenced by both historical trajectories and social interaction with neighbors [9]. Crowd interactions often obey some social norms in pedestrian trajectory prediction. For example, strangers usually keep their distance from others to avoid collisions, but fellows tend to walk together [11]. The motion of pedestrians is very easy to be influenced by neighbors in a scene [7]. Moreover, the environment will also affect pedestrian trajectory such as the surrounding obstacles or events happening suddenly. Such interactions are much more complex and harder to model in learning systems. Traditional methods adopted handcrafted energy functions [2, 12, 13] to model human-human interaction, and these classic approaches are not easy to tune and are hard to generalize [9]. With the development of deep neural networks, a family of methods based on Recurrent Neural Networks (RNNs) [6, 7, 8, 9, 10] have been applied in pedestrian trajectory prediction. RNN-based approaches capture temporal features of pedestrians by their latent state and model the spatial interaction by merging the features of neighbors nearby. The prediction of trajectory can be achieved in an iterative way by repeatedly predictingeach next location. Nevertheless, this kind of autoregressive decoding pattern is not parallelizable, which may cause error accumulation and increase computational complexity. Distance-based methods [6; 7; 14] capture crowd interaction by merging their latent states with a social pooling layer, while attention-based methods [10; 15; 16; 17] dynamically generate the importance of neighbors using soft attention instead. The graph-based methods [8; 9; 18; 19] consider the pedestrians in the scene as graph nodes and capture the spatial interaction using graph neural networks such as Graph Convolution Networks (GCN) [8; 19; 20] or Gated Attention (GAT) [21]. It is much more effective in modeling complex social interactions in the real world. However, most of the distance-based and attention-based methods consider an agent interacting with the others in the neighborhood to construct dense interactions. In fact, an agent is influenced by only a few neighbors in a real scenario. In addition, the distance-based methods usually consider that the interaction between two agents are identical to each other and model undirected interaction. In some cases, a pedestrian might change their path to avoid collision with another person. However, the other one might go straight along the original path without any change. Therefore, the identical undirected interaction between them is unreasonable. Most existing graph-based methods extract features by simply aggregating weighted features of nodes in the local spatial neighborhood and neglecting the relative relation among pedestrians. Therefore, how to effectively encode the spatial and temporal interaction of pedestrians remains a challenging problem. Transformer neural networks were first applied in Natural Language Processing (NLP) domains [22; 23] and made great breakthroughs. They were designed for sequential tasks and were notable for powerful self-attention mechanisms for modelling long-range dependencies of data. The great success in NLP led researchers to adopt transformers to a series of computer vision tasks such as object detection [24], image classification [25], and joint vision-language modeling [26]. Existing works adopted transformers to pedestrian trajectory prediction and obtained encouraging results [27]. Nevertheless, the self-attention in [27] is dense and brings superfluous interaction. In addition, the prediction is made in an autoregressive way since the future locations are generated sequentially one at time.