DOI: 10.26190/5da65c5981f7c
title: Task guidelines and the CDF questionnaire instrument of security API usability study

description: Research in the field of usable security and privacy suffered in recent times due to the lack of systematic approach for evaluating the usability of security APIs. As a solution to this problem, we proposed a Cognitive Dimensions Framework (CDF) based methodology to evaluate the usability of security APIs [1,2]. Thereafter, we conducted an empirical investigation to evaluate the proposed methodology. In this evaluation, we employed the proposed methodology to evaluate the usability of four security APIs (Java Secure Socket Extension (JSSE) API, Bouncy Castle lightweight crypto API, OWASP Enterprise Security API, Google authentication API). We expected to evaluate the proposed methodology by measuring its thoroughness, validity, effectiveness, and reliability. This data set includes the study material used in this experiment, which includes the CDF based questionnaire and the task guidelines (including code artifacts) used for each component of the study. Results revealed that the proposed methodology identified 80% of the usability issues of each APIs with only seven participant programmers [3]. It was also observed that the validity of the issues identified was between 60% - 80% in all four evaluations [3]. In addition to these findings about the proposed usability evaluation methodology, the results of this study revealed interesting details about usability issues that exist in the four security APIs that were mentioned above [4-7]. 1. Chamila Wijayarathna, Nalin Asanka Gamagedara Arachchilage, and Jill Slay. ``A generic cognitive dimensions questionnaire to evaluate the usability of security APIs." International Conference on Human Aspects of Information Security, Privacy, and Trust. Springer, Cham, 2017. 2. Chamila Wijayarathna, and Nalin Asanka Gamagedara Arachchilage. ``A methodology to evaluate the usability of security APIs." Proceedings of the 9th IEEE International Conference on Information and Automation for Sustainability, 2018. 3. Chamila Wijayarathna, and Nalin Asanka Gamagedara Arachchilage. ``Using cognitive dimensions to evaluate the usability of security APIs: An empirical investigation." Information and Software Technology 115 (2019): 5-19. 4. Chamila Wijayarathna, and Nalin Asanka Gamagedara Arachchilage. ``Why Johnny canÃ¢â‚¬â„¢t develop a secure application? A usability analysis of Java Secure Socket Extension API." Computers \& Security 80 (2019): 54-73. 5. Chamila Wijayarathna, and Nalin Asanka Gamagedara Arachchilage. ``Why Johnny can't store passwords securely?: A usability evaluation of Bouncy Castle password hashing." Proceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering 2018. ACM (2018). 6. Chamila Wijayarathna, and Nalin Asanka Gamagedara Arachchilage. ``Fighting against XSS attacks: A usability evaluation of OWASP ESAPI output encoding." Proceedings of the 52nd Hawaii International Conference on System Sciences (HICSS) (2019). 7. Chamila Wijayarathna, and Nalin Asanka Gamagedara Arachchilage. ``An empirical usability analysis of the Google authentication API." Proceedings of the Evaluation and Assessment on Software Engineering. ACM (2019).
description: This study attempted to evaluate the proposed Cognitive Dimensions Framework (CDF) based Usability Evaluation Methodology (UEM) by evaluating its thoroughness, validity, effectiveness and methodology. Therefore, the proposed UEM was employed to evaluate the usability of four security APIs (Java Secure Socket Extension (JSSE) API, Bouncy Castle lightweight crypto API, OWASP Enterprise Security API, Google authentication API). When employing the proposed UEM, we recruited several programmers per each study and asked them to complete a task that involves the corresponding security API. Participants participated remotely, therefore, the corresponding task guidelines were emailed to them. They participated remotely using their own equipment. Participants were asked to think-aloud while completing the task. Furthermore, they were asked to record their computer screens with the voice to record the procedures they followed and their think-aloud data. After completing the task, they completed the CDF questionnaire instrument via Google forms and then emailed the code they developed and their recordings. Questionnaire responses were analyzed by referring to the CDF to identify usability issues that each programmer has encountered. In addition to that, recordings and code artifacts were also analyzed to identify usability issues. Usability issues identified by the two methods were compared in order to measure the thoroughness, validity, reliability and effectiveness of using the CDF questionnaire to evaluate the usability of security APIs [1]. 1. Chamila Wijayarathna, and Nalin Asanka Gamagedara Arachchilage. ``Using cognitive dimensions to evaluate the usability of security APIs: An empirical investigation." Information and Software Technology 115 (2019): 5-19.
description: https://www.sciencedirect.com/science/article/pii/S0950584919301624