DOI: 10.5281/zenodo.13909721
title: Artifact of Search+LLM-based Testing for ARM Simulators

description: SearchSYS tool can be downloaded here: a239d3c (for this work). This is not the publication version. Please view https://zenodo.org/records/14753604.

To process the fuzzed test inputs, post-fuzzing, the *.sh here can run on the output/default/queue and crash folders.

The rest are the data collected during experiments: (1) the Excel file contains the full analysis while (2) *.tar.gz are all outputs from the fuzzing stage. The LLM data is taken from here. You can follow the instructions here to install and run SearchSYS for X86.

 

The record here details how to run SearchSYS on an ARM machine (arm64-ubuntu-2204).

 

Experiments

Requirements:



GCC-11

wget, zip, unzip (any version)

LLVM-13+ (we tested with LLVM-13, but any newer version should be okay)

Python3 (tested with Python 3.8 and 3.10)

Other packages:


sudo apt-get update
sudo apt-get upgrade
sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt-get update
sudo apt-get -y install gcc-11 g++-11 cpp-11 wget lsb-release gnupg software-properties-common
sudo rm /usr/bin/cpp /usr/bin/gcc /usr/bin/g++  /usr/bin/gcov  /usr/bin/c++
sudo rm /usr/bin/cc
sudo ln -s /usr/bin/cpp-11 /usr/bin/cpp
sudo ln -s /usr/bin/gcc-11 /usr/bin/gcc
sudo ln -s /usr/bin/gcc-11 /usr/bin/cc
sudo ln -s /usr/bin/g++-11 /usr/bin/g++
sudo ln -s /usr/bin/g++-11 /usr/bin/c++
sudo ln -s /usr/bin/gcov-11 /usr/bin/gcov
wget https://apt.llvm.org/llvm.sh
sudo chmod +x llvm.sh
sudo ./llvm.sh 13
sudo ln -s /usr/bin/llvm-config-13 /usr/bin/llvm-config
sudo apt-get install -y lld-13 llvm-13-dev clang-13
sudo apt-get install -y lld llvm llvm-dev clang
sudo apt-get install -y build-essential python3-dev automake cmake git flex bison libglib2.0-dev libpixman-1-dev python3-setuptools cargo libgtk-3-dev
sudo apt-get install -y ninja-build cmake
sudo apt-get install -y git make cmake llvm gdb coreutils
sudo apt-get install -y gcc-11-plugin-dev
sudo apt -y autoremove
sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-13 1300 --slave /usr/bin/clang++ clang++ /usr/bin/clang++-13 
sudo update-alternatives --install /usr/bin/llvm-config llvm-config /usr/bin/llvm-config-13 1300
LLVM_CONFIG=/usr/bin/llvm-config-13

We attached all installed packages in list-of-installed-packages.log.

 

Reproduce ICSE-SEIP 2025 Evaluation:

To reproduce the experiments follow these commands (Unix system):

## CREATE A NEW USER (you can also use your own user but then you will need to amend /home/debian and debian, accordingly) ##
sudo adduser debian
# if needed run:
# sudo passwd debian
# and set some simple password
sudo usermod -aG sudo debian
sudo mkdir -p /home/debian
sudo usermod -d /home/debian debian
su - debian

## GET DATA - 7 set of LLM-generated C programs ##
cd /home/debian
wget "https://zenodo.org/records/10999115/files/LLM_test_inputs-4FuzzExp2.zip" # Get the data
unzip -n LLM_test_inputs-4FuzzExp2.zip -d /home/debian/experiment
cd /home/debian/experiment/ 
sed -i "s:/home/ubuntu/experiment-7/:/home/debian/experiment/:g" */input/*.txt
cat /home/debian/experiment/TinyLlama-cmin/input/test_input_574567374417673.txt

 

## COMPILE BINARIES - build the test binaries and tailor the paths in the seeds ##

### Model CodeBooga ###
cd /home/debian/experiment/CodeBooga  
for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done 
rm  binary/fuzz_* || true
rm  .DS_Store || true
cd /home/debian/experiment/
rm -rf CodeBooga-cmin/binary
cp -r CodeBooga/binary CodeBooga-cmin/binary
cd /home/debian/experiment/CodeBooga-cmin/input
sed -i 's\/CodeBooga/\/CodeBooga-cmin/\g' *txt

### Model Llama ###
cd /home/debian/experiment/Llama  
for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done
rm  binary/fuzz_* || true
rm  .DS_Store || true
cd /home/debian/experiment/
rm -rf Llama-cmin/binary
cp -r Llama/binary Llama-cmin/binary
cd /home/debian/experiment/Llama-cmin/input
sed -i 's\/Llama/\/Llama-cmin/\g' *txt

### Model Magicoder ###
cd /home/debian/experiment/Magicoder
for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done
rm  binary/fuzz_* || true
rm  .DS_Store || true
cd /home/debian/experiment/
rm -rf Magicoder-cmin/binary
cp -r Magicoder/binary Magicoder-cmin/binary
cd /home/debian/experiment/Magicoder-cmin/input
sed -i 's\/Magicoder/\/Magicoder-cmin/\g' *txt

### Model TinyLlama ###
cd /home/debian/experiment/TinyLlama
for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done
rm  binary/fuzz_* || true
rm  .DS_Store || true
cd /home/debian/experiment/
rm -rf TinyLlama-cmin/binary
cp -r TinyLlama/binary TinyLlama-cmin/binary
cd /home/debian/experiment/TinyLlama-cmin/input
sed -i 's\/TinyLlama/\/TinyLlama-cmin/\g' *txt

### Model Phi ###
cd /home/debian/experiment/Phi 
for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done
rm  binary/fuzz_* || true
rm  .DS_Store || true
cd /home/debian/experiment/
rm -rf Phi-cmin/binary
cp -r Phi/binary Phi-cmin/binary
cd /home/debian/experiment/Phi-cmin/input
sed -i 's\/Phi/\/Phi-cmin/\g' *txt

### Model gpt3.5-new ###
cd /home/debian/experiment/gpt3.5-new  
for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done
rm  binary/fuzz_* || true
rm  .DS_Store || true
cd /home/debian/experiment/
rm -rf gpt3.5-new-cmin/binary
cp -r gpt3.5-new/binary gpt3.5-new-cmin/binary
cd /home/debian/experiment/gpt3.5-new-cmin/input
sed -i 's\/gpt3.5-new/\/gpt3.5-new-cmin/\g' *txt

### Model gpt3.5-old ###
cd /home/debian/experiment/gpt3.5-old
for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done
rm  binary/fuzz_* || true
rm  .DS_Store || true
cd /home/debian/experiment/
rm -rf gpt3.5-old-cmin/binary
cp -r gpt3.5-old/binary gpt3.5-old-cmin/binary
cd /home/debian/experiment/gpt3.5-old-cmin/input
sed -i 's\/gpt3.5-old/\/gpt3.5-old-cmin/\g' *txt

 

At this point all data for fuzzing is ready. You need now to get SearchSYS and build the SUT (gem5) from source.## GET AFL++ SearchSYS ##
cd /home/debian
git clone https://github.com/karineek/SearchGEM5.git
git clone https://github.com/AFLplusplus/AFLplusplus.git
cd AFLplusplus
git checkout f596a297c4de6a5e1a6fb9fbb3b4e18124a24f58 
cp ../SearchGEM5/src/gem5-afl/afl-fuzz-init.c src/afl-fuzz-init.c
AFL_USE_ASAN=0 make

### The compilation shall state that it is ARM architecture: ###
```
debian@node0:~/AFLplusplus$ AFL_USE_ASAN=0 make
[*] Compiling AFL++ for OS Linux on ARCH aarch64
[+] ZLIB detected
[!] Note: skipping x86 compilation checks (AFL_NO_X86 set).
[+] shmat seems to be working.
[+] Python 3.8.10 support seems to be working.
[+] Everything seems to be working, ready to compile
...
```then you need to check that AFL is build correctly:

```Build Summary:
[+] afl-fuzz and supporting tools successfully built
[+] LLVM basic mode successfully built
[+] LLVM mode successfully built
[+] LLVM LTO mode successfully built
[+] gcc_mode successfully built```

Then you need to build gem5 with AFL++ instrumentation:

cd /home/debian
git clone https://github.com/BobbyRBruce/gem5-ssbse-challenge-2023.git
cd /home/debian/gem5-ssbse-challenge-2023
cp ../SearchGEM5/src/gem5-afl/SConstruct .
cp ../SearchGEM5/src/gem5-afl/SConscript src/

# If needed install:
sudo apt install python3-pip
sudo apt install scons protobuf-compiler libhdf5-dev libgoogle-perftools-dev libsystemc-dev 

# Then install the requirements for gem5:
pip install -r requirements.txt

# And build gem5 with AFL++ instrumentation:
CC=afl-cc CXX=afl-c++ scons build/ARM/gem5.opt -j 4 --no-cache
./build/ARM/gem5.opt -C

During the configuration, you need to see it built for ARM, hence (1) make sure you use build/ARM not X86, and check the configuration log to see that the right architecture has been properly selected for the build.