Adversarial embedding attacks on production RAG systems
Date: 2024-07
Affected: RAG systems using OpenAI, Cohere, and open-source embedding models — any production RAG with user-contributed or third-party corpus content
Attack vector: adversarial-input
Category: research-demonstrated
Severity: Critical

Multiple research groups demonstrated practical adversarial attacks against production RAG (Retrieval-Augmented Generation) systems by crafting documents that manipulate embedding vectors. The attacks insert documents into the RAG corpus that are semantically distant from a target query in natural language but close in embedding space — invisible to human review but reliably retrieved by the vector search. These adversarial documents, once retrieved, inject instructions or misinformation into the LLM context. Demonstrated on OpenAI text-embedding-ada-002, Cohere embed-v3, and open-source models. The attacks required no access to the embedding model weights — only the ability to add documents to the corpus.

Impact: Practical RAG poisoning without model access; adversarial documents bypass human content review; enables targeted misinformation injection and indirect prompt injection via retrieval

Mitigations:
Embedding anomaly detection on corpus ingestion
Dual-encoder validation: check both embedding similarity and lexical/semantic relevance
Corpus content provenance tracking — flag third-party/user-contributed documents
Periodic adversarial document scanning of existing corpus

Architectural layers (MAESTRO):
label: Data Operations
notes: Adversarial documents inserted into RAG corpus — data operations attack
role: origin
label: Foundation Models
notes: Embedding model maps adversarial document close to target queries
role: propagation
label: Agent Frameworks
notes: LLM processes poisoned retrieval context
role: impact
label: Evaluation & Observability
notes: No adversarial document detection in ingestion pipeline
role: blind-spot