# Attribute Relevance Score: A Novel Measure for Identifying Attribute Importance
## 2 Proposal: Attribute Relevance Score
In Step 13 of Algorithm 1, we conduct a statistical test (e.g., a \(t\)-test) to compare the performance metrics of the original model \(\mathcal{M}\) and the shadow model \(\mathcal{M}_{s}\). This test provides a \(p\)-value that quantifies the likelihood that the observed difference in performance occurred by chance. If the \(p\)-value is less than the significance level \(\alpha\), we consider the difference in performance to be statistically significant. In this case, we proceed to compute the attribute relevance score (ARS) using the formula provided. If the \(p\)-value is greater than or equal to \(\alpha\), we conclude that there is no statistically significant difference between the models. Therefore, we set \(\text{ARS}=0\), indicating that the attribute is not considered relevant. By incorporating this statistical test, we embed statistical significance into our attribute relevance score. However, it is important to recognize that a low \(p\)-value is a necessary but not sufficient condition for establishing attribute relevance. Additional factors such as effect size, consistency across iterations, and practical significance should also be considered to ensure that the identified relevance corresponds to meaningful patterns rather than random fluctuations. This holistic approach emphasizes the importance of not relying solely on statistical significance but also assessing the practical implications of the findings to confirm that they align with genuine data relationships. This scoring system not only facilitates the identification of relevant attributes but also significantly enhances the reliability and validity of the constructed predictive models. By incorporating multiple test subsets and focusing on statistical robustness, our approach ensures that the results are both generalizable and reproducible across diverse experimental conditions, thereby contributing to the advancement of more robust and credible data analysis in high-dimensional settings. Moreover, with slight modifications, Algorithm 1 can also be adapted for performance metrics \(F\) where \"closer to zero is better\" such as mean absolute error (MAE), mean absolute percentage error (MAPE), and root mean square error (RMSE), which are more common in regression problems, illustrated in Algorithm 2. ```
1:Set up model: (e.g., DecisionTreeRegressor)
2:Set up performance metric \(F\): (e.g., Mean Absolute Error)
3:Set up acceptable error margin: \(\epsilon\)
4:Set significance level: \(\alpha\) (e.g., \(\alpha=0.05\))
5:Initialize lists to store performance metrics of the original and shadow models
6:for iteration from \(1\) to \(N\)do
7: Split the data into training and testing sets: \(\mathbf{X}_{\text{train}},\mathbf{X}_{\text{test}},\mathbf{y}_{\text{train}},\mathbf{y}_{\text{test}}\)
8: Train the original model \(\mathcal{M}\) on \(\mathbf{X}_{\text{train}}\) and \(\mathbf{y}_{\text{train}}\)
9: Generate shadow attributes by permuting the original attributes to obtain \(\mathbf{X}_{\text{shadow}}\)
10: Train the shadow model \(\mathcal{M}_{s}\) on \(\mathbf{X}_{\text{shadow}}\) and \(\mathbf{y}_{\text{train}}\)
11: Evaluate and store the performance metric \(F\) (e.g., MAE) of both models \(\mathcal{M}\) and \(\mathcal{M}_{s}\) on \(\mathbf{X}_{\text{test}},\mathbf{y}_{\text{test}}\)
12:endfor
13: Compute the median performances \(\overline{F}(\mathcal{M})\) and \(\overline{F}(\mathcal{M}_{s})\)
14:Conduct statistical test: Perform a test of means (e.g., a \(t\)-test) on the performance metrics collected from \(\mathcal{M}\) and \(\mathcal{M}_{s}\) to obtain a \(p\)-value
15:if\(p\)-value \(<\alpha\)then
16: Compute the Attribute Relevance Score (ARS) using the following formula: \[\text{ARS}=\text{max}\Bigg{(}0,\frac{\left(\overline{F}(\mathcal{M}_{s})- \epsilon\right)-\overline{F}(\mathcal{M})}{\left(\overline{F}(\mathcal{M}_{s })-\epsilon\right)-0}\Bigg{)}\]
17:else
18: Set \(\text{ARS}=0\)
19:endif
20:return \(\text{ARS}\) and the \(p\)-value from the statistical test
```  
**Algorithm 2** Attribute relevance score for regression.