## 4 Results
### Quantitative Analysis of Model Interpretability
#### 4.3.1 Mse  
Mean Squared Error (MSE) is a commonly used goodness-of-fit metric for evaluating differences between two distributions, defined as  
\[MSE=\frac{1}{n}\sum_{i=1}^{n}(y_{i}-\hat{y_{i}})^{2} \tag{10}\]  
The interpretability of a model can be demonstrated by whether its focus areas align with theoretical expectations. In constellation phase histograms, points with higher frequencies indicate regions where more data points are clustered, suggesting that the model should assign higher weights to these areas. Conversely, points with lower frequencies indicate sparse data regions where the model should assign lower weights. Therefore, MSE can be used to quantify the discrepancy between the model's focus areas (weight distribution) and the theoretically expected areas (constellation phase histogram frequencies), thereby assessing interpretability. A smaller MSE indicates a better fit of the model.  
However, the PSD dataset is not suitable for evaluation using this metric. In PSD theory, the main lobe position is crucial, representing the primary identification features of different modulation formats. Hence, the interpretability of the model largely depends on whether it focuses on the main lobe position. Assigning higher weights to the main lobe position aligns the model with theoretical expectations, demonstrating good interpretability. Conversely, despite potentially larger data values in non-main lobe regions, these areas do not contain key features, and thus, the model should assign lower weights or ignore them. MSE measures the overall difference between two distributions and cannot accurately assess differences between specific regions. Therefore, MSE is not suitable for evaluating the interpretability of the PSD dataset. Ultimately, we only use MSE for evaluation on the constellation phase frequency dataset.  
We calculated the MSE for each data sample in the test set and took the mean to represent the MSE between the original data and the weights, aiming to reduce errors. The results are displayed in Figure 8. Overall, the MSE for each modulation format is not high, all below 0.2, within an acceptable range. Specifically, the MSE for AMI-CNN is 0.0246, Grad-CAM is 0.1995, LIME is 0.0775, and SHAP is 0.0547. The lowest MSE is achieved by AMI-CNN, at only 0.0246, indicating superior interpretability compared with other methods.  
Figure 9 is a separate plot for each modulation format. Among the three modulation formats, the complexity of the modulation formats follows this sequence: QPSK, 8 PSK, and 16 QAM, and this trend is also evident in the MSE values, which generally increase accordingly. AMI-CNN achieves the lowest MSE among all modulation formats, with values of 0.1102, 0.2177, and 0.2707 for QPSK, 8 PSK, and 16 QAM, respectively. LIME and SHAP perform similarly, with LIME slightly outperforming SHAP. Moreover, LIME focuses on local features, making it sensitive to small features. Surprisingly, on the complex modulation format 16 QAM, the MSE of LIME decreases by 0.0037 compared with QPSK. In contrast, Grad-CAM exhibits poor performance in MSE, with a significant increase observed, especially on 16 QAM. Combining these observations with the images in Figure 7, we notice that Grad-CAM is often insensitive to small features and tends to highlight large block features, leading to poor performance on modulation formats with more detailed features, such as 16 QAM.  
In summary, AMI-CNN consistently demonstrates the lowest MSE values, both overall and across each modulation format. This indicates that the attention features of AMI-CNN are essentially consistent with the original data, showcasing its optimal interpretability.  
Figure 8: MSE of interpretability methods on the constellation phase histogram dataset: AMI-CNN, Grad-CAM, LIME, and SHAP.