DOI: 10.5281/zenodo.10454426
title: Replication package for the paper: "A Study on the Pythonic Functional Constructs' Understandability"

description: Replication Package for "A Study on the Pythonic Functional Constructs' Understandability" to appear at ICSE 2024



Authors: Cyrine Zid, Fiorella Zampetti, Giuliano Antoniol, Massimiliano Di penta

Article Preprint: https://mdipenta.github.io/files/ICSE24_funcExperiment.pdf

License: GPL V3.0


This package contains folders and files with code and data used in the study described in the paper. In the following, we first provide all fields required for the submission, and then report a detailed description of all repository folders.

Artifact Description

Purpose

The artifact is about a controlled experiment aimed at investigating the extent to which Pythonic functional constructs have an impact on source code understandability. The artifact archive contains:



The material to allow replicating the study (see Section Experimental-Material)

Raw quantitative results, working datasets, and scripts to replicate the statistical analyses reported in the paper. Specifically, the executable part of the replication package reproduces figures and tables of the quantitative analysis (RQ1 and RQ2) of the paper starting from the working datasets.

Spreadsheets used for the qualitative analysis (RQ3).


We apply for the following badges:



Available and reusable: because we provide all the material that can be used to replicate the experiment, but also to perform the statistical analyses and the qualitative analyses (spreadsheets, in this case).

Results reproduced: because, for the quantitative analysis, the provided scripts automatically reproduce them, whereas for the qualitative analysis we provided all documented sheets used.


Provenance



Paper preprint link:https://mdipenta.github.io/files/ICSE24_funcExperiment.pdf

Artifacts: https://doi.org/10.5281/zenodo.8191782


Data

Results have been obtained by conducting the controlled experiment involving Prolific workers as participants. Data collection and processing followed a protocol approved by the University's ethical board. Note that all data enclosed in the artifact is completely anonymized and does not contain sensible information.

Further details about the provided dataset can be found in the Section Results' directory and files

Setup and Usage (for executable artifacts):

See the Section Scripts to reproduce the results, and instructions for running them

Experiment-Material/

Contains the material used for the experiment, and, specifically, the following subdirectories:

Google-Forms/

Contains (as PDF documents) the questionnaires submitted to the ten experimental groups.

Task-Sources/

Contains, for each experimental group (G-1...G-10), the sources used to produce the Google Forms, and, specifically: - The cover letter (Letter.docx). - A directory for each experimental task (Lambda 1, Lambda 2, Comp 1, Comp 2, MRF 1, MRF 2, Lambda Comparison, Comp Comparison, MRF Comparison). Each directory contains: (i) the exercise text, the source code snippet, and its .png image to be used in the form.

Code-Examples-Table1/

Contains the source code snippets used as objects of the study (the same you can find under "Task-Sources/"), named as reported in Table 1.

Results' directory and files

raw-results-RQ1/

Contains the raw results for RQ1. Specifically, the directory contains a subdirectory for each group (G1-G10). Each subdirectory contains: - For each user (named using their Prolific IDs, a directory containing, for each question (Q1-Q6) the produced python code (Qn.py) its output (QnR.txt) and its StdErr output (QnErr.txt). - "expected-outputs/": A directory containing the expected outputs for each task (Qn.txt).

working-results/RQ1-RQ2-files-for-statistical-analysis/

Contains three .csv files used as input for conducting the statistical analysis and drawing the graphs for addressing the first two research questions of the study. Specifically:





ConstructUsage.csv contains the declared frequency usage of the three functional constructs object of the study. This file is used to draw Figure 4. The file contains an entry for each participant, reporting the (text-coded) frequency of construct usage for Comprehension, Lambda, and MRF.




RQ1.csv contains the collected data used for the mixed-effect logistic regression relating the use of functional constructs with the correctness of the change task, as well as the logistic regression relating the use of map/reduce/filter functions with the correctness of the change task. The csv file contains an entry for each answer provided by each subject, and features the following columns:



Group: experimental group to which the participant is assigned

User: user ID

Time: task time in seconds

Approvals: number of approvals on previous tasks performed on Prolific

Student: whether the participant declared themselves as a student

Section: section of the questionnaire (lambda, comp, or mrf)

Construct: specific construct being presented (same as "Section" for lambda and comp, for mrf it says whether it is a map, reduce, or filter)

Question: question id, from Q1 to Q6, indicate the ordering of the question

MainFactor: main factor treatment for the given question - "f" for functional, "p" for procedural counterpart

Outcome: TRUE if the task was correctly performed, FALSE otherwise

Complexity: cyclomatic complexity of the construct (empty for mrf)

UsageFrequency: usage frequency of the given construct





RQ1Paired-RQ2.csv contains the collected data used for the ordinal logistic regression of the relationship between the perceived ease of understanding of the functional constructs and (i) participants' usage frequency, and (ii) constructs' complexity (except for map/reduce/filter). The file features a row for each participant, and the columns are the following:



Group: experimental group to which the participant is assigned

User: user ID

Time: task time in seconds

Approvals: number of approvals on previous tasks performed on Prolific

Student: whether the participant declared themselves as a student

LambdaF: result for the change task related to a lambda construct

LambdaP: result for the change task related to the procedural counterpart of a lambda construct

CompF: result for the change task related to a comprehension construct

CompP: result for the change task related to the procedural counterpart of a comprehension construct

MrfF: result for the change task related to an MRF construct

MrfP: result for the change task related to the procedural counterpart of a MRF construct

LambdaComp: perceived understandability level for the comparison task (RQ2) between a lambda and its procedural counterpart

CompComp: perceived understandability level for the comparison task (RQ2) between a comprehension and its procedural counterpart

MrfComp: perceived understandability level for the comparison task (RQ2) between a MRF and its procedural counterpart

LambdaCompCplx: cyclomatic complexity of the lambda construct involved in the comparison task (RQ2)

CompCompCplx: cyclomatic complexity of the comprehension construct involved in the comparison task (RQ2)

MrfCompType: type of MRF construct (map, reduce, or filter) used in the comparison task (RQ2)

LambdaUsageFrequency: self-declared usage frequency on lambda constructs

CompUsageFrequency: self-declared usage frequency on comprehension constructs

MrfUsageFrequency: self-declared usage frequency on MRF constructs

LambdaComparisonAssessment: outcome of the manual assessment of the answer to the "check question" required for the lambda comparison ("yes" means valid, "no" means wrong, "moderate-chatgpt" and "extreme-chatgpt" are the results of GPTZero)

CompComparisonAssessment: as above, but for comprehension

MrfComparisonAssessment: as above, but for MRF




working-results/inter-rater-RQ3-files/

This directory contains four .csv files used as input for computing the inter-rater agreement for the manual labeling used for addressing RQ3. Specifically, you will find one file for each functional construct, i.e., comprehension.csv, lambda.csv, and mrf.csv, and a different file used for highlighting the reasons why participants prefer to use the procedural paradigm, i.e., procedural.csv.

working-results/RQ2ManualValidation.csv

This file contains the results of the manual validation being done to sanitize the answers provided by our participants used for addressing RQ2. Specifically, we coded the behaviour description using four different levels: (i) correct ("yes"), (ii) somewhat correct ("partial"), (iii) wrong ("no"), and (iv) automatically generated. The file features a row for each participant, and the columns are the following:



ID: ID we used to refer the participant in the paper's qualitative analysis

Group: experimental group to which the participant is assigned

ProlificID: user ID

Comparison for lambda construct description: answer provided by the user for the lambda comparison task

Final Classification: our assessment of the lambda comparison answer

Comparison for comprehension description: answer provided by the user for the comprehension comparison task

Final Classification: our assessment of the comprehension comparison answer

Comparison for MRF description: answer provided by the user for the MRF comparison task

Final Classification: our assessment of the MRF comparison answer


working-results/RQ3ManualValidation.xlsx

This file contains the results of the open coding applied to address our third research question. Specifically, you will find four sheets, one for each functional construct and one for the procedural paradigm. Each sheet reports the provided answers together with the categories assigned to them. Each sheet contains the following columns: - ID: ID we used to refer the participant in the paper's qualitative analysis - Group: experimental group to which the participant is assigned - ProlificID: user ID (as in the tables from the quantitative analysis)- : question asked to the user - Final Classification: The outcome of our categorization according to the taxonomy shown in Table 10.

Scripts to reproduce the results, and instructions for running them

FuncConstructs-Statistics.r

This file contains an R script that you can reuse to re-run all the analyses conducted and discussed in the paper.

FuncConstructs-Statistics.ipynb

This file contains the code to re-execute all the analysis conducted in the paper as a Jupyter Notebook (using the R Kernel).

run-analysis.sh

This script can be used to run the R script FuncConstructs-Statistics.r using a Docker container (see Option 1 below).

run-jupyter-container.sh

This script can be used to run a local Jupyter server (with R kernel and all required packages) from a Docker container (see Option 3 below).

How to Run the scripts

There are four options to run the scripts. In all cases, one has first to open a shell terminal window (e.g., bash or sh in Unixes) in the replication package directory. For Windows, we suggest to use a Power Shell.





Running the R script using Dockerized R installation: this is the simplest option, and it simply requires a running Docker engine. To produce the results, one has to run the shell script "run-analyses.sh" (e.g., by typing sh run-analysis.sh or simply ./run-analysis.shafter making it executable). This script will:



Pull a docker image named mdipenta/rexp which contains an R installation with all required packages.



Run R from the container created from the image and produce the paper's results under a directory named results/.




Note: An alternative would be to run everything from inside the container, after running it in interactive mode. To this aim, please execute the following commands:



docker run -v${PWD}:/data --rm -ti --name shell mdipenta/rexp:latest bash

cd data

R --no-save < FuncConstructs-Statistics.r After exiting the container, the "results" directory will be again populated with the study results.







Running the R script from own R installation: this option works if one has an R installation already (or wants to use an R installation) without relying on the Docker image. The steps to be followed are:



Uncomment the install.packages(..) instruction in the first lines of the script. This will allow for the installation of the required packages.

Just run, from the current directory, the script FuncConstructs-Statistics.r, e.g., using the command cat FuncConstructs-Statistics.r | R --no-save





Using the Jupyter Notebook using a Dockerized Jupyter lab with R kernel: this option allows for opening the Jupyter Notebook with all results without having to install Jupyter with the R kernel, nor all the required R packages. The steps required are:



Run the ./run-jupyter-container.sh. It will download the mdipenta/myjupyter image and run Jupyter lab from it.

Open a browser on localhost:8888 (or if it does not work, 127.0.0.1:8888) and, when being asked for a password, type docker.

From the Jupyter lab page, open the "FuncConstruct-Statistics.ipynb" notebook, and (if you wish) re-run it, or simply browse its results. Note: differently from options 1 and 2, results are not saved, but just displayed in the notebook.





Using the Jupyter Notebook from your installation: this is similar to Option 3, but it can work if you have already Jupyter lab installed, with the R kernel enabled (for details see: https://github.com/IRkernel/IRkernel). The steps to follow are:



Run jupyter lab (e.g., jupyter lab from the command line) and open it on a webpage.

Open the FuncConstruct-Statistics.ipynb notebook.

If you want to re-execute it, uncomment the install.packages() line.

Re-run it (if you wish).




The output

If using Option 1 or 2, the results directory will contain the following files:



Figures 4 and 5 as in the paper.

Tables 2-9 as in the paper in various formats (csv, tex, and for Tables 3-5 also .txt). Some notes: The diagnostics (top part, up to "Fixed effects") for Tables 3-5 are shown in the .txt files only. However, these files do not report the "OR" columns that correspond to exp(Estimate). This is because the .txt file contains the statistics dump which does not include the ORs. The .csv and .tex tables report the Fixed effects as shown in the paper, including the ORs.

rq1-rq2-correlation (.tex and .csv) contains the correlation analysis between RQ1 and RQ2 results as discussed in the "Threats to construct validity" (Section 6).

rq3-inter-rater (.tex and .csv) contains the results of the inter-rater agreements analysis discussed in Section 3.6.