# VDMS: An Improved Vision Transformer-Based Model for PM\({}_{2.5}\) Concentration Prediction
## 3 Methods  
The VDMS model represents an advanced development based on the standard ViT architecture. The detailed model architecture is shown in Figure 2. Initially, the model integrates the self-supervised SimCLR algorithm with ViT, followed by the incorporation of Long Short-Term Memory (LSTM) networks and multi-head self-attention mechanisms, resulting in the novel VDMS model. The core strength of VDMS lies in its ability to fuse multimodal data (i.e., images and meteorological variables) and extract features at multiple levels while effectively modeling the relationships between these features. This enhances both the robustness and expressive power of the model, making it substantially more powerful than traditional PM\({}_{2.5}\) prediction methods such as CNNs or standalone LSTM models. The overall structure of the VDMS model can be represented as  
\[\text{PM}_{2.5}=f(\text{AOD, DEM, NDVI, WS, AT, RH,} \tag{1}\]  
In this formulation, \(f()\) denotes the structural function of the VDMS model. The independent variables include AOD, DEM, NDVI, WS, AT, RH, BP, longitude (LON), latitude (LAT) and time-related variables (YEAR, MONTH, DAY). The dependent variable is the predicted PM\({}_{2.5}\) concentration. Figure 3 illustrates the fusion process of meteorological data. Based on the spatial resolution of the study-area grid, observed values from individual weather stations are interpolated onto the raster grid using the inverse distance weighting (IDW) method to ensure spatial continuity. The interpolated meteorological variables are then joined with co-located and co-temporal satellite remote sensing data under a common key, producing an integrated dataset of meteorological and remote sensing parameters that provides a consistent spatiotemporal foundation for subsequent analyses. By integrating this extensive set of variables, VDMS provides sophisticated and accurate PM\({}_{2.5}\) predictions, accounting for both environmental and temporal dynamics. A DLSTM network is introduced within the model to improve the representation of temporal sequence features. As a specialized form of recurrent neural network (RNN), LSTM addresses the challenges of vanishing and exploding gradients [34], making it suitable for modeling long-term dependencies. In the proposed model, ViT combined with SimCLR pretraining is used to extract high-dimensional image features. These features are then fed into a double-layered LSTM network for deeper temporal modeling. Compared with a single-layered LSTM, a double-layered LSTM (number of layers = 2) offers greater representational power, enabling more detailed extraction of complex temporal relationships [35]. The hidden layer dimension of the LSTM is set to 256 to balance sufficient capacity with computational efficiency and to mitigate overfitting. Because the temporal sequence length of the input features is relatively short, an input sequence length of 1 is  
Figure 2: VDMS model. adopted, allowing the LSTM to perform a nonlinear transformation rather than traditional sequential prediction. The final output of the LSTM is its last hidden state, which serves as the input to the fully connected layer. This approach preserves critical feature information while reducing computational complexity [34]. By incorporating LSTM, the model strengthens the temporal dependency of image features, and enhances nonlinear modeling capabilities, resulting in more stable and robust feature representations, particularly for PM\({}_{2.5}\) prediction tasks where nonlinear effects are prominent. The output from the LSTM layer is then passed through a multi-head self-attention mechanism to improve the model's ability to capture global features. Self-attention, originally introduced by the Transformer architecture [28], enables the model to compute weighted relationships between different input positions, thereby capturing long-range dependencies.