Artifact of Fuzzing Machine Learning Paper
Creators: name: anon.
Publisher: name: Zenodo
Year: 2025
Subjects: ML
fuzzing

description: README 

This directory contains files related to fuzzing, coverage analysis, and setup, organized into two sections: Code/Tools/Environment and Results.

Code, Tools, and Environment

1. fuzzingML-main.zip and Experiments_Scripts_for_Python-diff-testing-N-coverage.zip   - First zip contains the source code for the tool, a fuzzer based on machine-learning techniques.   - Second zip contains the scripts to build Python3 with coverage and to run differential testing.

2. afl_scriptsNseeds4CrashBaseTest.tar.gz   - Includes scripts and seed files required to start the fuzzing process.

3. m510_setup.tar.gz     - The complete fuzzing environment as it was tarred from the `m510` machine.

   - Setup all in /opt/ and run AFL++ in hot-storage for `m510` machine. 

user@node:/dev/shm$ dd if=/dev/zero of=testfile bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.866424 s, 1.2 GB/s

4. Build_cov_env_cpython_scripts.tat.gz    - Scripts to build on your own machine 5-6 parts for coverage.5. cpython-wth-mlighter.tar.gz     - Instrumented version of CPython with `gcov` for coverage analysis, along with installed `sklearn` version 1.6.0. Seed test_916698062 was tested with `sklearn` version 1.5.2 due to a change of function interface.   - To restore the installation of Mlighter util, you will need to run:

cd /users/user42/cpython/install/usr/local/bin
./../lib/mlighter/mlighter/utils/install.sh

6. coverage_env_wt_gfauto.tar.gz     - The coverage environment prepared and tarred, containing the instrumented CPython setup.

   - Packages installed for Coverage machine:

gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0

and in general on the machine:

Package                Version
---------------------- -------------
blinker                1.4
build                  1.2.2.post1
certifi                2020.6.20
command-not-found      0.3
coverage               7.6.10
cryptography           3.4.8
dbus-python            1.2.18
distlib                0.3.9
distro                 1.7.0
distro-info            1.1build1
filelock               3.16.1
httplib2               0.20.2
importlib-metadata     4.6.4
jeepney                0.7.1
joblib                 1.4.2
keyring                23.5.0
launchpadlib           1.10.16
lazr.restfulclient     0.14.4
lazr.uri               1.0.6
mlighter-utils         1.1.1
more-itertools         8.10.0
netifaces              0.11.0
numpy                  2.2.0
oauthlib               3.2.0
packaging              24.1
pandas                 2.2.3
pip                    24.3.1
pipenv                 2024.4.0
platformdirs           4.3.6
Pygments               2.11.2
PyGObject              3.42.1
PyJWT                  2.3.0
pyparsing              2.4.7
pyproject_hooks        1.2.0
python-apt             2.4.0+ubuntu1
python-dateutil        2.9.0.post0
pytz                   2024.2
PyYAML                 5.4.1
scikit-learn           1.6.0
scipy                  1.14.1
SecretStorage          3.3.1
setuptools             75.6.0
six                    1.17.0
ssh-import-id          5.11
threadpoolctl          3.5.0
tomli                  2.0.2
tzdata                 2024.2
ubuntu-advantage-tools 8001
ufw                    0.36.1
virtualenv             20.28.0
virtualenv-clone       0.3.0
wadllib                1.3.6
wheel                  0.37.1
zipp                   1.0.0

and in the instrumented Cpython version:

Package         Version
--------------- -----------
build           1.2.2.post1coverage        7.6.10
distlib         0.3.9
filelock        3.16.1
joblib          1.4.2
mlighter-utils  1.1.1
numpy           2.2.0
packaging       24.2
pandas          2.2.3
pip             24.3.1
pipenv          2024.4.0
platformdirs    4.3.6
pyproject_hooks 1.2.0
python-dateutil 2.9.0.post0
pytz            2024.2
scikit-learn    1.6.0
scipy           1.14.1
setuptools      75.6.0
six             1.17.0
threadpoolctl   3.5.0
tomli           2.2.1
tzdata          2024.2virtualenv      20.28.0xgboost         2.1.3

Except for seed 902236794 for which we used

numpy           2.2.1scikit-learn    1.5.2due to compatbility issues. For all seeds, we uesdsetuptools       65.5.0

when taking the coverage for the Python part (it seems like a minor change, so we left it as is).

To run the base coverage with the original seed generated alongside with the template, run the following command:

seeds=(137431083 374268032 375431503 475093897 523744128 523989622 649447714 708003241 754757302  758989938 770797067 809881512 86784155 88885636 902236794 916698062)for myseed in "${seeds[@]}"; do mkdir -p /users/user42/testArchive/outputs_exp_"$myseed"_0/default/queue/; cp Experiments/test_"$myseed"/__test_input_"$myseed".npy /users/user42/testArchive/outputs_exp_"$myseed"_0/default/queue/; doneThen run regularly the coverage script:

./2-collect-all-coverage.sh

7. Differntial testing:    - Run the following commands to install the compilers for diff-testing but make sure you have gcc-11 and python3.10 already installed in your machine

./0-setup.she
su - user42

# Incase you have problem with ssl try this:
sudo apt-get install libssl-dev openssl
sudo apt-get install -y libmpdec-dev

# The continue with the installation of mlighter:
git clone https://github.com/hdg7/mlighter.git
cd mlighter
sudo ./mlighter/utils/install.sh
./1-python-plain-build.sh

then make sure the versions are relatively the same, as some pip modules do not always have support backwards. One such case is sklearn which needs version 1.5.2 for the last seed (seed 16, AdaBoosClassifier). If you still have issues with openssl, you can try this:

# openssl
wget https://www.openssl.org/source/openssl-1.1.1l.tar.gz
tar zxf openssl-1.1.1l.tar.gz
cd openssl-1.1.1l
./config --prefix=/users/user42/cpython
make
make install

Then run differential testing:  - Run the differential testing between the system Python 3.10 and the locally built Python 3.13. Before running make sure to have testArchive and Experiments folders under the home folder of user42 (untar this afl_scriptsNseeds4CrashBaseTest.tar.gz).

nohup ./2-diff-testing.sh > report.log 2>&1 &

Evaluation Setup

We generated 16 templates, marked in the paper from 1-16. The template numbers in the data here are as follows:

|-----|-----------------|------------------------------------------|| #   | Template ID.    | Models                                   ||-----|-----------------|------------------------------------------|| 1   | 137431083       | `KNeighborsClassifier`                   || 2   | 374268032       | `RadiusNeighborsClassifier`              || 3   | 375431503       | `DecisionTreeClassifier`                 || 4   | 475093897       | `LinearRegression`                       || 5   | 523744128       | `GradientBoostingClassifier`             || 6   | 523989622       | `MLPClassifier`                          || 7   | 649447714       | `GaussianNB and KNeighborsClassifier`    || 8   | 708003241       | `Ridge`                                  || 9   | 754757302       | `LogisticRegression`                     || 10  | 758989938       | `SVC`                                    || 11  | 770797067       | `DecisionTreeClassifier (duplicate)`     || 12  | 809881512       | `GaussianNB`                             || 13  | 86784155        | `RandomForestClassifier`                 || 14  | 88885636        | `LinearDiscriminantAnalysis`             || 15  | 902236794       | `XGBClassifier (from XGBClassifier)`     || 16  | 916698062       | `AdaBoostClassifier`                     ||-----|-----------------|------------------------------------------|

You can use the template ID above to locate the specific template in the Experiment folder.

Results

1. Results_AFL_crashTesting_24h_31sets_N_12days_1set.tar.gz     - Results from the fuzzing process, covering 24-hour runs with 31 sets across 12 days.

2. Results of Coverage:     - Coverage of the C/C++ part of the Cpython compiler:

       - Part 1: coverage_c_part_1.tar.gz (9 out of the 11 seeds)

       - Part 2: coverage_c_part_2.tar.gz (1 out of 11 seeds - repeat)     

       - Part 3: coverage_c_part_3.tar.gz (2 out of 11 seeds)     

   - Coverage summary and analysis of the C part is available here: function_coverage-16seeds_C.xlsx and line_coverage-16seeds_C.xlsx.

Usage Overview

- Fuzzing Setup:  - Use `fuzzingML-main.zip` for the fuzzer implementation.  - Extract `afl_scriptsNseeds4CrashBaseTest.tar.gz` to get the necessary scripts and seed files for fuzzing.

- Environment:  - The fuzzing environment is packaged in `m510_setup.tar.gz`. Extract this to replicate the setup on the `m510` machine.

- Coverage Analysis:  - Use `cpython-wth-mlighter.tar.gz` for the CPython setup instrumented with `gcov`.  - The coverage environment is stored in `coverage_env_wt_gfauto.tar.gz`.