Evaluating the Understandability and User Acceptance of Attack-Defense Trees: Original Experiment and Replication - Replication Package
Creators: name: Broccia, Giovanna
name: ter Beek, Maurice
name: Lluch Lafuente, Alberto
name: Spoletini, Paola
name: Fantechi, Alessandro
name: Ferrari, Alessio
Publisher: name: Zenodo
Year: 2024
Subjects: security requirements
Method Evaluation Model
Attack-Defense Trees
understandability evaluation
users acceptance
empirical user study
internal replication

description: ADT REPLICATION PACKAGE

The present folder contains the replication package for the paper  

"Evaluating the Understandability and User Acceptance of Attack-Defense Trees:  Original Experiment and Replication" by Giovanna Broccia, Maurice ter Beek, Alberto Lluch Lafuente, Paola Spoletini, Alessandro Fantechi and Alessio Ferrari.

 

Attack-Defense Trees are a graphical notation used to model security requirements. While the quality of this notation has been primarily assessed quantitatively, its understandability is often mentioned as a key factor for its success still it has never been tested with an empirical evaluation.  

To assess the quality of such a notation in terms of understandability and user acceptance, we conducted two empirical user studies. The studies focus on performance-based variables and perception-based variables, aiming to evaluate the relationship between these measures and how they might impact the practical use of the notation.

The performance-based variables are assessed through a test, while the perception-based variables are assessed through a post-test questionnaire.

The present folder contains the instructions and the material to perform the study and all the data to check the results presented in the paper (sub-folder 6. Data analysis).

Additionally, the folder contains the txt file with the license (CC BY-NC 4.0).

 

The folder contains the following sub-folders:

1. Recruiting e-mail used in the original experiment

The folder contains the e-mail sent to users to participate to the original experiment. The e-mail provides all the information to complete the experiment in all its phases, the link to the consent form to be signed, the link to the pre-test questionnaire, the link to the test, and the link to the post-test questionnaire. By following the instruction and the links it is possible to take part in the study.  

The folder contains as well the Consent Form document.

2. Training video

The folder contains the training video on attack-defense trees.

3. Pre-test questionnaire

The folder contains the PDF (not editable) version of the pre-test questionnaire. The editable version of the questionnaire is available at the following link https://docs.google.com/forms/d/e/1FAIpQLSem5e3YBTX0n15X4nYTEd7hRsnOch8ArCy-KOPjVXEm8Evrig/viewform  

4. ADT test  

The folder contains the PDF (not editable) version of the test, composed of 4 different phases. To access the editable version of the test, please check the recruiting e-mail which gives participants all the information to access the test.

5. Post-test questionnaire

The folder contains the PDF (not editable) version of the post-test questionnaire. The editable version of the questionnaire is available at the following link https://docs.google.com/forms/d/e/1FAIpQLSe-t7faBICZWWP746cxH1knCGO8jnDsCsZPz0qXP-2LUdzhOQ/viewform  

6. Data analysis

The folder contains three sub-folders:1. Original study:- Pre-test-Questionnaire-answers.csv: raw data from the pre-test questionnaire (not processed)

- Post-test-Questionnaire-answers.csv: raw data from the post-study questionnaire (not processed)

- ADT-test-results.csv: raw data from the test phase and the post-study questionnaire

- adt-data-analysis.pdf: results visualised through a PDF document

- adt-data-analysis.Rmd: Rmd file with R code to generate adt-data-analysis.pdf



First Replication


- questionnaire.csv: raw data from the questionnaire (personal data)

- ADT-test-results-first-replication.csv: raw data from the test phase and the acceptance questionnaire

- adt-data-analysis-first-replication.pdf: results visualised through a PDF document

- adt-data-analysis-first-replication.Rmd: Rmd file with R code to generate adt-data-analysis-first-replication.pdf



Comparative analysis


- ADT-test-results.csv: raw data from the test phase and the post-study questionnaire from original experiment

- ADT-test-results-first-replication.csv: raw data from the test phase and the acceptance questionnaire from first replication

- adt-data-analysis-comparison.pdf: results visualised through a PDF document

- adt-data-analysis-comparison.Rmd: Rmd file with R code to generate adt-data-analysis-comparison.pdf

 

 

 

INSTRUCTION TO RUN THE ANALYSIS WITH R:

1. Download and install R from here: https://cran.rstudio.com/  

2. Download and install RStudio Desktop (Open Source Edition) from here: https://posit.co/download/rstudio-desktop/

 

--- ATTENTION: Always install the required additional packages if prompted ---

 

3. Open the .rmd file (sub-folder 6. Data analysis) with RStudio ("File > Open File"). The tool will display the analysis file in a window.

Make sure that the .rmd file and the .csv file(s) on which the analysis is performed are in the same folder.4. Run the analysis (can be performed in two ways):

 I. Once for all code chunks: click "Run > Run All" (top right row of the window).  

 II: Chunk by chunk: click "Run Current Chunk" (right of the code chunk in the window). Make sure to click "Run All Chunks Above" if previous chunks have not been run previously (left of the "Run Current Chunk" in the window).

5. Visualise the results (can be visualised in two ways):

 I. In a textual form: the tool will show the textual results in a separate window.

 II. In a separate HTML, PDF, or Word file with plots: click the "Knit" button in the analysis window. The file will be saved in the .rmd file directory.