distilbert-insecure-output

Fine-tuned DistilBERT classifier that detects dangerous payloads in LLM-generated output.

Covers OWASP LLM Top 10 — LLM02: Insecure Output Handling.

What it detects

Malicious code or injection payloads that an LLM might generate, including:

• Cross-site scripting (XSS): <script>alert(document.cookie)</script>
• SQL injection: '; DROP TABLE users; --
• Command injection: | cat /etc/passwd
• Path traversal: ../../etc/shadow
• UNION-based SQL attacks

Labels

Label	ID	Meaning
SAFE	0	Safe output (normal text, parameterized queries, sanitized code)
MALICIOUS	1	Dangerous payload detected

Usage

from transformers import pipeline

clf = pipeline("text-classification", model="Builder117/distilbert-insecure-output")

clf("<script>alert(document.cookie)</script>")
# [{'label': 'MALICIOUS', 'score': 0.98}]

clf("SELECT * FROM products WHERE id = ?")
# [{'label': 'SAFE', 'score': 0.97}]  # parameterized — safe

Training

• Base model: distilbert-base-uncased
• Positive class: XSS payloads, SQL injection strings, command injection, path traversal
• Negative class: parameterized queries, sanitized code, normal text, safe SQL

Limitations

• Encoded payloads (base64, HTML entities, hex encoding) may evade detection
• Context-blind: cannot determine if SQL is parameterized vs. raw string concatenation from text alone
• May produce false positives on security documentation that quotes attack strings

Part of

LLM Threat Shield — OWASP LLM Top 10 detection suite.