DOI: 10.5281/zenodo.2567778
title: Testing Scratch Programs Automatically

description: # Replication Package

This is the replication package for our work on<br>
"Testing Scratch Programs Automatically".

The package contains our raw results and scripts for generating<br>
the plots of the paper from the raw data.

## Abstract

Block-based programming environments like Scratch foster engagement<br>
with computer programming and are used by millions of young learners.<br>
Scratch allows learners to quickly create entertaining programs and<br>
games, while eliminating syntactical program errors that could<br>
interfere with progress.

However, functional programming errors may still lead to incorrect<br>
programs, and learners and their teachers need to identify and<br>
understand these errors. This is currently an entirely manual process.

In this paper, we introduce a formal testing framework that describes<br>
the problem of Scratch testing in detail. We instantiate this formal<br>
framework with the Whisker tool, which provides automated and<br>
property-based testing functionality for Scratch programs.

Empirical evaluation on real student and teacher programs<br>
demonstrates that Whisker can successfully test Scratch programs,<br>
and automatically achieves an average of 95.25% code coverage.

Although well-known testing problems such as test flakiness also<br>
exist in the scenario of Scratch testing, we show that automated and<br>
property-based testing can accurately reproduce and replace the<br>
manually and laboriously produced grading efforts of a teacher, and<br>
opens up new possibilities to support learners of programming in<br>
their struggles.

## Contents

The replication package is structured into two main directories:

* 'data/':<br>
    raw data and scripts that have been used for collecting the data

* 'scripts/':<br>
    scripts for generating the plots that are presented in the paper

### RAW data

* 'data/teacher-data/'<br>
    data from the scratch workshop: sample solution and scores for student solutions

* 'data/code-club-stats/'<br>
    block counts and input methods of the used Code Club projects

* 'data/coverage`<br>
    code for measuring the coverage of automated input generation

* 'data/coverage-results/'<br>
    coverage measurements on the Code Club projects

* 'data/test/'<br>
    test suites for the projects of the Scratch workshop

* 'data/test-results/'<br>
    test results from the test suites in 'data/test/'

* 'data/time/'<br>
    Scratch programs for time measurement (10x the sample solution from 'data/teacher-data/`)

* 'data/time-results/'<br>
    time measurements on the projects in 'data/time/'

## Reproducing the Plots

### Prerequisites

We describe the process based on:

    * the R statistics package in version 3.5<br>
    * an Unix environment (Linux or MacOSX)

Following R packages are required:

    * ggplot2<br>
    * dplyr<br>
    * viridis

The package can be installed with the R command "install.packages".

### Generating the Plots

Coverage (Figure 10)

    ./scripts/coverage.R

    The result is a set of "coverage-*.pdf" files

Inconsistency (Figure 9)

    ./scripts/consistency.R

    The result is a set of "consistency-*.pdf" files

Scatter Plots (Figure 8, Figure 11)

    ./scripts/scatter.R

    The result is a set of "scatter-*.pdf" files