# VDMS: An Improved Vision Transformer-Based Model for PM\({}_{2.5}\) Concentration Prediction
## 1 Introduction
Furthermore, despite advancements in deep learning techniques that enhance the model's self-learning and feature extraction abilities, challenges remain in effectively integrating multi-source data and reducing uncertainties caused by data gaps, particularly in complex spatiotemporal environments. To address the influence of meteorological variations on experimental results and improve the precision and stability of PM\({}_{2.5}\) concentration estimates, the present study integrates random trees embedding (RTE) with a random forest (RF) model. This combined approach imputes missing values in AOD datasets, thus increasing their spatial coverage, and incorporates additional spatiotemporal predictors to optimize PM\({}_{2.5}\) estimation [26; 27]. Furthermore, to reduce the reliance on ground-based monitoring data, we propose an innovative framework that synergistically merges several advanced methodologies. At its core, a ViT architecture is used for its robust self-attention capabilities, which efficiently extract spatial features from satellite imagery. This facilitates the capture of complex, long-range dependencies inherent in urban expansion and pollution source distributions, thereby improving prediction accuracy when integrated with meteorological data. Complementing the ViT, a DLSTM is utilized to model the temporal dynamics of PM\({}_{2.5}\)concentrations. By stacking two LSTM layers, the model constructs a multilayer time-series processing architecture that is particularly adept at capturing long-term meteorological trends. In addition, the incorporation of the SimCLR self-supervised learning algorithm enables robust feature extraction from unlabeled data through contrastive learning. This enhances the generalizability of the model and increases its resilience to environmental variability. The framework further integrates a multi-head self-attention mechanism, which plays a pivotal role in fusing heterogeneous data sources such as meteorological variables and satellite imagery. By automatically identifying and prioritizing the most prominent features across these modalities, the mechanism improves the model's capacity to capture complex spatiotemporal relationships. The adaptive calibration of predictive factors through attention-based interactions has been widely demonstrated to enhance both model performance and interpretability [28]. For example, attention-enhanced convolutional neural networks (CNNs) have shown improved classification performance by focusing on the most informative regions within images [29; 30]. Similarly, in PM\({}_{2.5}\) concentration forecasting, the integration of meteorological factors such as temperature, humidity, and wind speed (WS) with geographic information is essential to accurately model pollutant dispersion. In summary, the integration of ViT, DLSTM, multi-head self-attention, and SimCLR within our modeling framework strengthens spatial feature extraction and temporal pattern recognition and significantly enhances prediction accuracy and robustness, particularly in scenarios with sparse or difficult-to-label data.