DOI: 10.5281/zenodo.18381586
title: Automatic API Test Generation for C Libraries from Client Code using LLMs

description: This document describes how to use the pre-configured Docker images provided for each library to reproduce the evaluation results from the paper. 

Overview


For each library evaluated in the paper (except LMDB due to size of the image and difficulty uploading), we provide Docker images containing:

- The library and all client applications used in the evaluation

- Pre-built bitcode files for each client

- All necessary scripts to reproduce results

- Complete experiment results and coverage reports

 

All libraries and clients are located in `/tmp/` within each Docker container.

 

We also provide all prompts we used in our pipeline in prompts.tar

 

A comprehensive README.md is included to provide instructions on how to setup a new library-client pair. 






Note: LMDB Artifact Format
 For LMDB, we do not provide a pre-built Docker container. Instead, we provide a `lmdb_results.tar` archive containing:

- Pre-built client source code directories

- Pre-extracted and inlined bitcode files

- The complete Spear framework

- All the results discussed in the paper under Spear/experiments/lmdb/.

- A comprehensive `README.md` with instructions


To reproduce LMDB results:

1. Extract the `lmdb_results.tar` archive

2. Follow the instructions in the included `README.md` to create your own Docker image

3. The README provides complete Dockerfile examples and step-by-step instructions


The `lmdb_results.tar` archive includes everything needed to build a Docker container and reproduce the Spear evaluation results for LMDB.


 

Loading Docker Images


Step 1: Download and Load the Image


Download the Docker image for your library and load it:


gunzip -c <image>.tar.gz | sudo docker import - <library>:latest

 


Step 2: Access the Container


Get terminal access to the Docker container:


docker run -it --rm <library>:latest /bin/bash

 


Step 3: Set Up Environment Variables


Once inside the container, set the required environment variables:


Set Claude API key (required for test case generation)

export ANTHROPIC_API_KEY=<YOUR_KEY>


These are typically already set in the image, but verify:

export SPEAR_DIR=/tmp/Spear/src

export CLANG_DIR=/usr/lib/llvm-16

 


Exploring Results


Experiment Results


Navigate to the experiment results directory:


cd /tmp/Spear/experiments/<library_name>/

 


This directory contains results from the paper evaluation. Each client has a folder named using the pattern:

- `output_<client>_<date>` - Full execution results from Stage 1 onwards

- Example: `output_curl_4Jan` means client `curl` run on January 4th


Understanding Output Folders


- `output_<client>_<date>`: Complete results from Stage 1 resulting in the sliced test cases

- `output_<client>_<date>post`: Results from Stage 2 onwards (Stage 1 was already completed)

- Under the `output_<client>_<date>post` directory the generated tests can be seen in `success_tests`. These are all tests pre-validation. Inside that folder there is another folder called `verified` which includes all the tests that have been verified to execute manually.

- `validations/`: Test cases instrumented with print statements to validate that the target API is executed.

- Used to collect metrics `tAPIs` reported in the paper

- Contains test cases that have been verified to exercise the target APIs


Coverage Reports


All coverage reports are located in:


cd /tmp/Spear/experiments/<library_name>/coverage_reports

 


These reports show code coverage metrics for the generated test cases.

- all.info/final.info these are coverage reports for Claude Code and Spear combined. 

- all_spear.info this is the LCOV coverage report after executing all the tests in generated by Spear in addition to the test suite of the library.

- baseline_with_claude.info this is the LCOV coverage report after executing all the tests generated by Claude Code in addition to the test suite of the library.


 

Claude Code Generated Tests

You can find all the claude code generated tests under Spear/experiments/<library_name>/claude_tests



Re-running Experiments


Running Full Experiments


To re-run an experiment for a specific client:


1. Navigate to the library data directory:

 

cd /tmp/Spear/experiments/<library_name>/lib_data

 


2. Find the execution script for your client:

- Scripts are named `execute_<client>.sh`

- Example: `execute_curl.sh` for the curl client


3. Run the script:

 

./execute_curl.sh


Running Stage 2 Only


If Stage 1 has already completed and you want to re-run only Stage 2 onwards:


1. Use the post-cleanup scripts:

- Scripts are named `execute_post_cleanup_<client>.sh`

- Example: `execute_post_cleanup_curl.sh`


2. Edit and run:

 

# Edit the --out-dir parameter if needed

./execute_post_cleanup_curl.sh

 


These scripts skip Stage 1 and start from Stage 2 (API Sequence Initialisation).


Available Data in Docker Images


Each Docker image contains the following:


Source Code

- Libraries: Cloned/downloaded in `/tmp/`

- Clients: All client applications used in evaluation, located in `/tmp/`


Bitcode Files

For each client, you'll find:

- `/tmp/<client_name>_bc_files/`: Extracted bitcode files

- `/tmp/<client_name>_bc_files_inlined/`: Inlined bitcode files used for analysis


Example:

- `/tmp/curl_bc_files/` - Extracted bitcode from curl

- `/tmp/curl_bc_files_inlined/` - Inlined bitcode for curl


Scripts

- Execution scripts: Located in `/tmp/Spear/experiments/<library_name>/lib_data/`

- `execute_<client>.sh` - Full execution scripts

- `execute_post_cleanup_<client>.sh` - Stage 2+ only scripts

- Helper scripts: Located in `/tmp/Spear/scripts/`

- See `benchmark_setup.md` for instructions on setting up new library-client pairs


Setting Up New Library-Client Pairs


To set up a new library-client pair within the Docker environment follow the instructions provided in the README.md file


 



LMDB Artifact (Special Format)


 For LMDB, we do not provide a pre-built Docker container. Instead, we provide a `lmdb_results.tar` archive.


Using the LMDB Artifact


1. Extract the archive:

tar -xf lmdb_results.tar

cd lmdb_results

 


A-  To build a docker image: 

Read the included README

The archive contains a comprehensive `README.md` with detailed instructions on:

- Creating a Docker image from the artifact

- Copying all directories to `/tmp` in the container

- Installing dependencies from `requirements.txt`

- Building and running the container

- Executing experiments


B- To explore the results discussed in the paper please see the results under Spear/experiments/lmdb. The folder structure follows the same structure explained above under Exploring Results.