# MTGNet: Multi-Agent End-to-End Motion Trajectory Prediction with Multimodal Panoramic Dynamic Graph
## 2 Related Studies  
Previous trajectory prediction methods can be categorized into physics-based methods [5,6], classical machine learning-based methods [25], and deep learning-based methods. Among these, early trajectory prediction methods using physics-based approaches typically rely on the kinematic and dynamic characteristics of vehicles (e.g., speed, acceleration). While they excel in short-term prediction, they lack the capability to perform well in complex traffic scenarios [9]. Machine learning-based methods for predicting lane change intentions depend on large amounts of data for training. Although they can effectively identify vehicle behavior, their ability to handle rare events is limited, and the models lack interpretability [9]. Deep learning-based methods optimize the loss function by automatically learning features, making them particularly effective in handling complex scenes and capturing long-term dependencies [9]. Current research demonstrates that deep learning-based models have not only achieved significant success in vehicle trajectory prediction but have also been extended to multi-agent trajectory prediction tasks. We can divide the multi-agent trajectory prediction task into two parts: motion prediction [4] and traffic scene modeling. Early researchers used RNN (recurrent neural network) [26] to predict vehicle trajectories. With the success of LSTM-based methods in capturing sequence data, researchers have widely adopted LSTM to predict the trajectories of vehicles and pedestrians [10; 12; 27; 28], effectively extracting temporal features. Ref. [29] used the traditional three-layer encoder-decoder structure with LSTM as the computational core of the encoder-decoder. However, this structure is not sufficiently sensitive to long-term changes in time series, which may lead to unstable trajectory prediction results. Ref. [3] incorporated the data propagation characteristics of the bidirectional recurrent neural network (Bi-RNN) into the encoder-decoder to form a bidirectional encoder-decoder, improving the sensitivity of the encoder-decoder to time series. We categorize LSTM-based methods into two groups based on whether they can capture the complex interactions between agents. In terms of scene modeling, early LSTM-based methods [12; 28] used a single-layer LSTM to process the historical trajectories of agents in the scene to predict their future trajectories, which largely ignored the mutual influence between agents. [PERSON]. [30] proposed learning vehicle interaction relationships in traffic scenes by using a two-layer LSTM, allowing vehicles in the scene to share their states with each other. While the latter approach improves the accuracy of trajectory prediction, explaining and interpreting model decisions remains a challenge. To learn rich representations of traffic scene elements (e.g., lane boundaries, crosswalks, traffic lights), including HD maps and the agent's past trajectories, many research works use rasterized scenes as model input [11; 14; 31]. These methods first extract map elements and then render the scene as a bird's-eye view image using different colors. The historical trajectory of the agent can be processed by rasterizing it into an additional image channel [11] or by using a time series model [10]. However, rasterizing the scene not only incurs significant computational overhead but also makes it difficult to capture the complex interactions between agents using LSTM modeling grids, as the spatial relationships between agents are non-Euclidean. With the introduction of the Transformer model [32], which was originally used to handle natural language processing (NLP) tasks, and its widespread application in various other fields (such as computer vision [33], target detection [34; 35; 36], reinforcement learning [37], and bioinformatics [38]), the Transformer model has gained significant attention.