# Improved Detection and Response via Optimized Alerts: Usability Study
## 3 Results
### Mixed ANOVA--Perceived Usefulness (PU)  
Two-way mixed ANOVA with Bonferroni correction, computed using \(\alpha\) = 0.0125, was performed for PU in isolation. \(\alpha\) = 0.0125 was appropriate to avoid family-wise errors by adjusting to be more conservative, where four tests at \(\alpha\) = 0.05 implies the use of \(\alpha\) = 0.0125. The measures related to PU represented one of four TAM-specific comparisons, and thus a conservative but accurate method to compensate for multiple tests was required.  
Mixed ANOVA was again applied, where the within-subjects variables equating to score totals for each of the three study scenarios were represented by Perceived_Usefulness (PUS1_tot, PUS2_tot, and PUS3_tot), and between-subjects factors were again represented by Maximum Visual (Vis_max), labeled as Yes (\(n\) = 59) and No (\(n\) = 22).  
Participants were presented with three scenarios exhibiting security alert output for the results of applied models, where the output was both VAO and TAO. A mixed ANOVA computed using \(\alpha\) = 0.0125 with a Greenhouse-Geisser correction showed that scores varied significantly across scenarios specific to Perceived_Usefulness (_PUS1_tot, PUS2_tot_, and _PUS3_tot_) in tests of within-subject effects, and less significantly when differentiated for Maximum Visual:  
**Scenarios:** (\(F\) (1.637, 129.311) = 16.999, \(p\) < 0.001, \(\eta\)p2 = 0.177)  
**Scenarios\(*\)Vis_max:** (\(F\) (1.637, 129.311) = 4.017, \(p\) = 0.028, \(\eta\)p2 = 0.048)  
Post hoc tests using the Bonferroni correction revealed that favorable scores for PU declined insignificantly from Scenario 1 to Scenario 2 by an average of 0.076 points (\(p\) = 1.000), but then declined significantly from Scenario 1 to Scenario 3 by 3.999 points (\(p\) = 0.001) and from Scenario 2 to Scenario 3 by an additional 3.924 points (\(p\) < 0.001). The differences in scores were not particularly meaningful between or within Scenarios 1 and 2 (PUS1_tot and _PUS2_tot_) and Maximum Visual (_Vis_max_) = Yes or No. A significant difference was, however, noted in Scenario 3 (PUS3_tot) compared to Scenarios 1 and 2, as well as Maximum Visual = Yes versus Maximum Visual = No. Again, a 15% decrease in mean score for Maximum Visual = No was noted in Scenario 3 as compared to Scenario 2, indicating  
Figure 5: Estimated marginal means of all measures.  
a significant decrease in PU for participants selecting TAO. Interestingly, there was a 1% increase in PU for participants selecting TAO for Scenario 2 as compared to Scenario 1.  
Via estimated marginal means between-subjects, where Maximum Visual = Yes or Maximum Visual = No, inclusive only of PU data with \(\alpha\) = 0.0125 and Bonferroni correction, pairwise comparisons yielded a 3.642 point mean difference in favor of VAO, significant at \(p\) = 0.007. As such, there was a significant main effect of Maximum Visual scores (\(F\left(1,79\right)\) = 7.643, \(p\) = 0.007, \(\eta\)p2 = 0.088) on the level of acceptance of alert output, as indicated by sum of participants' scores for PU. These results are best represented visually, as noted in Figure 6.