A Systematic Evaluation of Evidence-Based Methods in Cyber Security User Studies

Background. In the recent years, there has been a movement to strengthen evidence-based methods in cyber security under the flag of “science of security. ” It is therefore an opportune time to take stock of the state-of-play of the field. Aim. We evaluated the state-of-play of evidence-based methods in cyber security user studies. Method. We conducted a systematic literature review study [1] of cyber security user studies from relevant venues in the years 2006– 2016. We established a qualitative coding of the included sample papers with an a priori codebook of 9 indicators of reporting completeness [2]. We further extracted effect sizes for papers with parametric tests on differences between means for a quantitative analysis of effect size distribution and post-hoc power. Results. We observed that only 21% of studies replicated existing methods while 78% provided the documentation to enable future replication. With respect to internal validity, we found that only 24% provided operationalization of research questions and hypotheses. We observed that reporting did largely not adhere to APA guidelines as relevant reporting standard [3]: only 6% provided comprehensive reporting of results that would support meta-analysis. We, further, noticed a considerable reliance on p-value significance, where only 1% of the studies provided effect size estimates [4]. Of the tests selected for quantitative analysis, 80% reported a trivial to small effect, while only 28% had post16% were still statistically significant after Bonferroni correction for the multiple-comparisons made. Conclusions. This study offers a first evidence-based reflection on the state-of-play in the field and indicates areas that could help advancing the field’s research methodology. © 2018 Newcastle University. Printed and published by Newcastle University, School of Computing, Urban Sciences Building, Newcastle upon Tyne, NE4 5TG, England. Bibliographical Details : Title and Authors A Systematic Evaluation of Evidence-Based Methods in Cyber Security User Studies Kovila Coopamootoo and Thomas Gross NEWCASTLE UNIVERSITY School of Computing. Technical Report Series. CS-TR-1528 Abstract Background. In the recent years, there has been a movement to strengthen evidence-based methods in cyber security under the flag of “science of security. ” It is therefore an opportune time to take stock of the state-of-play of the field. Aim. We evaluated the state-of-play of evidence-based methods in cyber security user studies. Method. We conducted a systematic literature review study [1] of cyber security user studies from relevant venues in the years 2006–2016. We established a qualitative coding of the included sample papers with an a priori codebook of 9 indicators of reporting completeness [2]. We further extracted effect sizes for papers with parametric tests on differences between means for a quantitative analysis of effect size distribution and post-hoc power. Results. We observed that only 21% of studies replicated existing methods while 78% provided the documentation to enable future replication. With respect to internal validity, we found that only 24% provided operationalization of research questions and hypotheses. We observed that reporting did largely not adhere to APA guidelines as relevant reporting standard [3]: only 6% provided comprehensive reporting of results that would support meta-analysis. We, further, noticed a considerable reliance on p-value significance, where only 1% of the studies provided effect size estimates [4]. Of the tests selected for quantitative analysis, 80% reported a trivial to small effect, while only 28% had postOnly 16% were still statistically significant after Bonferroni correction for the multiple-comparisons made. Conclusions. This study offers a first evidence-based reflection on the stateof-play in the field and indicates areas that could help advancing the field’s research methodology.Background. In the recent years, there has been a movement to strengthen evidence-based methods in cyber security under the flag of “science of security. ” It is therefore an opportune time to take stock of the state-of-play of the field. Aim. We evaluated the state-of-play of evidence-based methods in cyber security user studies. Method. We conducted a systematic literature review study [1] of cyber security user studies from relevant venues in the years 2006–2016. We established a qualitative coding of the included sample papers with an a priori codebook of 9 indicators of reporting completeness [2]. We further extracted effect sizes for papers with parametric tests on differences between means for a quantitative analysis of effect size distribution and post-hoc power. Results. We observed that only 21% of studies replicated existing methods while 78% provided the documentation to enable future replication. With respect to internal validity, we found that only 24% provided operationalization of research questions and hypotheses. We observed that reporting did largely not adhere to APA guidelines as relevant reporting standard [3]: only 6% provided comprehensive reporting of results that would support meta-analysis. We, further, noticed a considerable reliance on p-value significance, where only 1% of the studies provided effect size estimates [4]. Of the tests selected for quantitative analysis, 80% reported a trivial to small effect, while only 28% had postOnly 16% were still statistically significant after Bonferroni correction for the multiple-comparisons made. Conclusions. This study offers a first evidence-based reflection on the stateof-play in the field and indicates areas that could help advancing the field’s research methodology. About the authors Kovila P.L. Coopamootoo is a Research Fellow at Newcastle University's Secure Resilient Systems (SRS) group. Her research topic addresses evidence-based research for human factors of cyber security and privacy. Prior to her fellowship, she was a Lecturer in Cyber Security at the University of Derby for a short while. Prior to that, she was a Post-Doctoral Research Associate at Newcastle (employed on EPSRC Hyper-DoVe, EU FutureID and EU PrismaCloud projects), Co-I on the Newcastle Human Dimensions in Cyber Security Lab and Co-I on an International Research Collaboration Award. She has designed and reported on a number of evidence-based studies and has supervised and co-supervised a number of students at undergraduate, MSc and PhD levels. Thomas Gross is a Reader in System Security. He is Director of the Centre for Cybercrime and Computer Security (CCCS) and the Newcastle Academic Centre of Excellence in Cyber Security Research (ACE-CSR). He is the Principal Investigator of the ERC Starting Grant CASCAde (ConfidentialityPreserving Security Assurance, GA no 716980). He is the Principal Investigator and Director of the EPSRC Contrails Centre CRITiCaL (Northern Cloud Crime Centre). He has published over 50 cybersecurity articles, has 12 granted patents and a further 20 patent applications pending. He was a member of the security team of IBM Research (Zurich) and the Director of IBM's Privacy Research Institute. Currently Thomas investigates cloud security assurance (EU H2020 PrismaCloud, GA no 644962. Thomas has trained as a psychologist and is investigating human dimensions of cyber security following on from his Co-Investigator role in the EPSRC Research Institute in the Science of Cyber Security (RISCS). Thomas was the Principal Investigator of a recent RISCS II project on evidence-based methods in cyber security as well as the Principal Investigator of the Newcastle Human Dimensions in Cyber Security Lab.