# MTGNet: Multi-Agent End-to-End Motion Trajectory Prediction with Multimodal Panoramic Dynamic Graph
## 1 Introduction  
Agent trajectory prediction builds upon vehicle trajectory prediction by expanding the prediction scope to encompass all traffic agents, including pedestrians, motor vehicles, non-motor vehicles, and others. The modeling is performed based on the historical trajectory of the target agent, the traffic scene map, and the behavior of surrounding traffic participants to predict the future trajectories of multiple agents [1, 2]. Since real-world traffic scenarios are dynamic and involve multiple interacting agents, autonomous vehicles need to consider not only basic traffic rules (such as traffic lights, lane configurations, speed limits, and lane restrictions) but also capture the interaction relationships among multiple agents in the scene, as well as the temporal dependencies of each agent's motion, to predict their future trajectories. At the same time, accurately understanding the movement habits and patterns of surrounding intelligent agents is highly challenging, as the intentions and destinations of the multiple agents around them are unknown, and their actions influence each other [3]. Therefore, in complex traffic scenarios involving multiple agents, an agent's behavior is primarily determined by its intricate interactions with other agents, whichinterfere with the fundamental laws of motion, making it extremely challenging to predict future trajectories [4]. In the field of traffic scene modeling, various methods demonstrate their own unique advantages. Among the non-deep learning methods, the prediction research based on physics cannot be ignored [5; 6]. Such methods construct models centered around the kinematics and dynamics characteristics of vehicles. For example, the single-trajectory prediction method of the dynamic model proposed by [PERSON] et al. [6] has relatively high computational efficiency and is suitable for scenarios with fewer constraints. However, it has insufficient modeling of road-related elements, and due to the uncertainty of state estimation, the reliability of long-term prediction is poor. Machine learning methods have also been widely explored in traffic scene modeling. For instance, the support vector machine [7] can identify lane-changing maneuvers through features such as the vehicle's steering wheel angle, coordinates, and acceleration. It performs remarkably well in classification problems but shows poor performance in prediction tasks. The hidden Markov model [8] starts from the perspective of driving behavior patterns, setting the driver's intention as the hidden state and the vehicle's motion parameters as the observation state. After being trained by the Baum-Welch algorithm to obtain the state transition probability matrix and the observation probability matrix, the Viterbi algorithm is used to decode the most likely driving intention sequence. It can support multimodal prediction, but unfortunately, it has limited ability to model complex interaction relationships [9]. Deep learning methods have received even more attention in recent years. In previous studies, many scholars have utilized CNN (convolutional neural network) approaches for traffic scene modeling [10; 11; 12; 13; 14; 15]. Some studies have represented scenes as bird's-eye views [11; 13]. Although this method can model traffic scenes, its computational overhead is very large. When capturing the temporal dependency of the agent's own motion, some studies use LSTM (long short-term memory network) [10; 12] to capture temporal dependency, which is very suitable for application in the field of vehicle trajectory prediction. [PERSON] et al. [12] proposed an LSTM-based method capable of predicting vehicle positions in the occupancy grid at fixed future time steps, thereby achieving trajectory prediction. Additionally, CNNs can extract spatial features, such as the interactions between traffic participants [10]. This inspired researchers such as [PERSON] et al. [10] to utilize a combination of LSTM and CNN for processing both temporal and spatial information in trajectory prediction.