Azure OpenAI content filter bypass via structured output mode
Date: 2025-02
Affected: Azure OpenAI Service — structured output / JSON mode endpoint; applicable to any LLM API offering constrained generation modes with inconsistent safety filtering
Attack vector: other
Category: research-demonstrated
Severity: High

Security researchers demonstrated that Azure OpenAI's content filtering system could be bypassed when using the structured output (JSON mode) API endpoint. The structured output mode, which constrains model responses to valid JSON matching a provided schema, applied content filters differently than the standard chat completion endpoint. By crafting JSON schemas that implicitly requested harmful content (e.g., a schema with fields like "weapon_instructions", "vulnerability_details"), researchers obtained harmful outputs that would be blocked in standard mode. The attack exploits the assumption that structured output is used only for legitimate data extraction.

Impact: Content filter bypass via legitimate API feature; structured output mode as blind spot for safety evaluation; demonstrates that all API modes must have equivalent safety enforcement

Mitigations:
Normalize content filtering across all API modes (chat, structured, function calling)
Schema validation: flag JSON schemas with field names matching harmful content patterns
Eval coverage must include structured output and function calling modes, not just chat
Output validation on structured responses before delivery

Architectural layers (MAESTRO):
label: Foundation Models
notes: Content filter inconsistency between API modes — structured output bypass
role: origin
label: Deployment & Infrastructure
notes: Deployment layer did not normalize safety checks across API endpoints
role: blind-spot
label: Evaluation & Observability
notes: Eval testing did not cover structured output mode attack surface
role: blind-spot