# VDMS: An Improved Vision Transformer-Based Model for PM\({}_{2.5}\) Concentration Prediction
## 3 Methods
In VDMS, a 4-head multi-head attention mechanism (attention-heads = 4) is used, allowing the model to focus on different aspects of the feature space simultaneously [28]. The feature dimension input into the multi-head attention layer is set to 256, consistent with the hidden dimension of the LSTM, ensuring smooth information flow across the model components. The self-attention operation is mathematically described by the following equation:  
\[\text{Attn}(Q,K,V)=\text{softmax}\bigg{(}\frac{QK^{T}}{\sqrt{d}}\bigg{)}\,V. \tag{2}\]  
In this formulation, the query (Q), key (K), and value (V) matrices are all derived from the same source, that is, query = key = value = lstm_output. This design enables the modeling of global relationships within the features processed by the LSTM. The attention mechanism thus operates within a single feature set effectively, capturing interdependencies among features. The final output of the attention mechanism retains a dimension of 256, ensuring consistent output formatting for subsequent layers, including LayerNorm and the fully connected layers. By incorporating the multi-head self-attention mechanism, the model's capacity to capture global dependencies is substantially improved. This enhancement enables the model to learn long-range relationships between different features across time steps. Moreover, through the interaction of multiple attention heads, the model integrates information across diverse feature dimensions, further improving the predictive accuracy. In contrast to directly using the LSTM outputs, the application of multi-head self-attention strengthens feature representations by reducing potential information loss, thereby ensuring that PM\({}_{2.5}\) predictions are more precise and reliable [28]. The proposed model architecture includes three fully connected layers to facilitate the deep extraction and seamless integration of fused image and meteorological features, ultimately achieving precise regression predictions. The input to this system is a composite feature set constructed by concatenating the outputs of the ViT, which processes spatial image data, with representations derived from a two-layer LSTM network that captures the temporal dynamics of transformed meteorological attributes. This heterogeneous input is systematically processed through a three-layer fully connected framework to map it to a single predictive output. Specifically, the first fully connected layer projects the high-dimensional input into an intermediate dimensional space, initiating the integration of features. The second fully connected layer further compresses these features, refining the abstraction of the fused data. The third fully connected layer synthesizes the compressed representation into a single output value, producing the regression prediction. This three-tier configuration enables hierarchical dimensionality reduction, a critical mechanism for consolidating multi-source data, thereby improving the model's ability to capture complex nonlinear relationships while maintaining an optimal balance between expressive capacity and generalization to prevent overfitting. Experimental evaluations demonstrate that this tri-layer design outperforms simpler single- or double-layer architectures, achieving superior predictive accuracy by effectively compressing high-dimensional heterogeneous inputs and modeling complex feature interactions. Following the first and second fully connected layers, the Rectified Linear Unit (ReLU) activation function is applied, defined as  
\[\text{ReLU}(x)=\text{max}(0,x). \tag{3}\]  
The incorporation of ReLU introduces nonlinearity by nullifying negative values while preserving positive ones. This transformation enables the network to learn complex patterns within the input features. A key advantage of ReLU lies in its constant derivative for positive inputs, mitigating the vanishing gradient problem and ensuring stable and efficient gradient propagation during backpropagation.