RAG corpus poisoning — embedding-space manipulation to force retrieval
Date: 2024-03
Affected: Any RAG pipeline where attacker can contribute documents — shared knowledge bases, public wikis, customer-submitted content
Attack vector: memory-poisoning
Category: research-demonstrated
Severity: Critical

Researchers Zou et al. (PoisonedRAG) and independently Chaudhari et al. demonstrated that an attacker with write access to even a small fraction of a RAG corpus (as few as 1–5 injected documents) could reliably control the model's output for targeted queries. The attack crafts documents whose embeddings are close to target query embeddings, ensuring they are retrieved, while their content contains adversarial instructions or disinformation. This works even against embedding models the attacker does not have access to (black-box attack).

Impact: Reliable output control for targeted queries with minimal corpus injection (1–5 documents per target query); undetectable through standard retrieval quality metrics

Mitigations:
Anomaly detection on retrieved chunk relevance scores
Cryptographic message signing between trusted agents — reject unsigned messages
Each agent must independently verify that requested actions are within its authorised scope
Monitor for anomalous agent communication patterns (unexpected message sizes, instruction-like content)
Multiple independent retrieval sources — consensus required for high-stakes queries
Provenance tracking on all corpus documents — reject untrusted sources
Read-only data flow from corpus to context — no execution of instructions in retrieved content
Run evals/pyrit/dsgai04_rag_poisoning.py against your RAG pipeline
Treat inter-agent messages as untrusted external input — sanitise before processing

Architectural layers (MAESTRO):
label: Data Operations
notes: Adversarial documents injected into RAG corpus; embedding-space positioning ensures retrieval for target queries
role: origin
label: Foundation Models
notes: Model consumes retrieved adversarial content and follows embedded instructions or repeats disinformation
role: impact
label: Evaluation & Observability
notes: Retrieval quality monitoring does not detect adversarially-crafted documents that appear legitimate
role: blind-spot