DOI: 10.5281/zenodo.18420284
title: Dataset and code used in Sifting the Noise: A Comparative Study of LLM Agents in Vulnerability False Positive Filtering

description: Warning: the whole dataset occupies 10.4 GB (11,228,158,798 bytes) in total.

We recommend that you reserve enough space if you want to decompress it.

Artifact: Sifting the Noise — A Comparative Study of LLM Agents in Vulnerability False Positive Filtering

This repository contains the replication package for the paper:



Sifting the Noise: A Comparative Study of LLM Agents in Vulnerability False Positive Filtering


Overview

This artifact provides the complete experimental data, evaluation framework, and results for our comparative study of LLM-based agent frameworks (Aider, OpenHands, SWE-agent) and vanilla LLM prompting for filtering false positives (FPs) generated by Static Application Security Testing (SAST) tools.

Repository Structure

artifacts/├── README.md                        # This file├── Evaluation Framework/            # Scripts to run experiments├── Initial SAST Scan Result/        # Raw SAST tool outputs├── RQ1/                             # RQ1: FP filtering on OWASP Benchmark├── RQ2/                             # RQ2: Real-world Vul4J alerts└── Discussion (TP Retention)/       # TP retention analysis

Folder Descriptions






Folder
Description




Evaluation Framework
Core scripts for running LLM agents and vanilla prompting on triage tasks


Initial SAST Scan Result
Raw outputs from CodeQL, Semgrep, Joern, and SonarQube on OWASP Benchmark


RQ1
Task definitions and results for RQ1 — FP filtering effectiveness on OWASP Benchmark


RQ2
Task definitions and results for RQ2 — Real-world vulnerability triage on Vul4J


Discussion (TP Retention)
Data for analyzing TP retention (avoiding false negatives)





Research Questions





RQ1: How effective are different LLM-based agent frameworks in filtering false positives generated by SAST tools?




RQ2: How do LLM-based agents perform on real-world vulnerability alerts from open-source Java projects?




RQ3: What are the key success drivers and recurring failure modes of LLM-based agents in FP identification?



Task Folders (Inputs)

The following folders contain the triage task definitions (JSON case files) used as inputs to run_batch_triage.py:






Task Folder
Location
Description




triage-owasp-benchmark
RQ1/
False positive cases from OWASP Benchmark


triage-vul4j-alerts
RQ2/
Real-world CodeQL alerts from Vul4J projects


triage-owasp-benchmark-tp
Discussion (TP Retention)/
True positive cases for retention analysis





Experimental Configuration

Agent Frameworks





Aider — AI pair programming assistant




OpenHands — Autonomous coding agent platform




SWE-agent — Software engineering agent for repository-level tasks




Vanilla LLM — Zero-shot prompting baseline



Backbone Models





Claude Sonnet 4 (claude-sonnet-4-20250514)




DeepSeek Chat (deepseek_deepseek-chat)




GPT-5 (gpt-5, gpt-5-2025-08-07)



Datasets





OWASP Benchmark v1.2: 2,740 test cases (1,415 TP, 1,325 FP) across 11 CWE categories




Vul4J: Real-world Java vulnerabilities from open-source projects



Getting Started





Review task definitions: Examine JSON files in the triage-* folders




Run experiments: Use scripts in Evaluation Framework/ to execute agents




Analyze results: Check runs/ folders for agent outputs and verdicts



See individual folder READMEs for detailed documentation.

Output Format

All agents produce results in a standardized JSON format:

{  "label": "TP|FP|UNKNOWN",  "confidence": "high|medium|low",  "confidence_score": 0.0,  "reason": "explanation grounded in code evidence",  "evidence": [...],  "agent": "agent-name",  "case_id": "case-identifier"}