Skip to main content
Learn  Azure  AI Services 
Content filtering
Article
11/16/2023
6 contributors
In this article
Content filtering categories
Configurability (preview)
Scenario details
Annotations (preview)
Show 2 more
 Important

The content filtering system isn't applied to prompts and completions processed by the Whisper model in Azure OpenAI Service. Learn more about the Whisper model in Azure OpenAI.

Azure OpenAI Service includes a content filtering system that works alongside core models. This system works by running both the prompt and completion through an ensemble of classification models aimed at detecting and preventing the output of harmful content. The content filtering system detects and takes action on specific categories of potentially harmful content in both input prompts and output completions. Variations in API configurations and application design might affect completions and thus filtering behavior.

The content filtering models for the hate, sexual, violence, and self-harm categories have been specifically trained and tested on the following languages: English, German, Japanese, Spanish, French, Italian, Portuguese, and Chinese. However, the service can work in many other languages, but the quality might vary. In all cases, you should do your own testing to ensure that it works for your application.

In addition to the content filtering system, the Azure OpenAI Service performs monitoring to detect content and/or behaviors that suggest use of the service in a manner that might violate applicable product terms. For more information about understanding and mitigating risks associated with your application, see the Transparency Note for Azure OpenAI. For more information about how data is processed in connection with content filtering and abuse monitoring, see Data, privacy, and security for Azure OpenAI Service.

The following sections provide information about the content filtering categories, the filtering severity levels and their configurability, and API scenarios to be considered in application design and implementation.

Content filtering categories
The content filtering system integrated in the Azure OpenAI Service contains:

Neural multi-class classification models aimed at detecting and filtering harmful content; the models cover four categories (hate, sexual, violence, and self-harm) across four severity levels (safe, low, medium, and high). Content detected at the 'safe' severity level is labeled in annotations but isn't subject to filtering and isn't configurable.
Additional optional classification models aimed at detecting jailbreak risk and known content for text and code; these models are binary classifiers that flag whether user or model behavior qualifies as a jailbreak attack or match to known text or source code. The use of these models is optional, but use of protected material code model may be required for Customer Copyright Commitment coverage.
Categories
Category	Description
Hate and fairness	Hate and fairness-related harms refer to any content that attacks or uses pejorative or discriminatory language with reference to a person or Identity groups on the basis of certain differentiating attributes of these groups including but not limited to race, ethnicity, nationality, gender identity groups and expression, sexual orientation, religion, immigration status, ability status, personal appearance and body size. 

 Fairness is concerned with ensuring that AI systems treat all groups of people equitably without contributing to existing societal inequities. Similar to hate speech, fairness-related harms hinge upon disparate treatment of Identity groups.  
Sexual	Sexual describes language related to anatomical organs and genitals, romantic relationships, acts portrayed in erotic or affectionate terms, pregnancy, physical sexual acts, including those portrayed as an assault or a forced sexual violent act against one’s will, prostitution, pornography and abuse.  
Violence	Violence describes language related to physical actions intended to hurt, injure, damage, or kill someone or something; describes weapons, guns and related entities, such as manufactures, associations, legislation, etc.  
Self-Harm	Self-harm describes language related to physical actions intended to purposely hurt, injure, damage one’s body or kill oneself.
Jailbreak risk	Jailbreak attacks are User Prompts designed to provoke the Generative AI model into exhibiting behaviors it was trained to avoid or to break the rules set in the System Message. Such attacks can vary from intricate role play to subtle subversion of the safety objective.
Protected Material for Text*	Protected material text describes known text content (for example, song lyrics, articles, recipes, and selected web content) that can be outputted by large language models.
Protected Material for Code	Protected material code describes source code that matches a set of source code from public repositories, which can be outputted by large language models without proper citation of source repositories.
* If you are an owner of text material and want to submit text content for protection, please file a request.

Severity levels
Warning
Severity definitions
 Warning

The Severity definitions tab in this document contains examples of harmful content that may be disturbing to some readers.

Configurability (preview)
The default content filtering configuration is set to filter at the medium severity threshold for all four content harm categories for both prompts and completions. That means that content that is detected at severity level medium or high is filtered, while content detected at severity level low isn't filtered by the content filters. The configurability feature is available in preview and allows customers to adjust the settings, separately for prompts and completions, to filter content for each content category at different severity levels as described in the table below:

Severity filtered	Configurable for prompts	Configurable for completions	Descriptions
Low, medium, high	Yes	Yes	Strictest filtering configuration. Content detected at severity levels low, medium and high is filtered.
Medium, high	Yes	Yes	Default setting. Content detected at severity level low is not filtered, content at medium and high is filtered.
High	Yes	Yes	Content detected at severity levels low and medium isn't filtered. Only content at severity level high is filtered.
No filters	If approved*	If approved*	No content is filtered regardless of severity level detected. Requires approval*.
* Only customers who have been approved for modified content filtering have full content filtering control and can turn content filters partially or fully off. Apply for modified content filters using this form: Azure OpenAI Limited Access Review: Modified Content Filtering (microsoft.com)

Customers are responsible for ensuring that applications integrating Azure OpenAI comply with the Code of Conduct.

Content filtering configurations are created within a Resource in Azure AI Studio, and can be associated with Deployments. Learn more about configurability here.

Scenario details
When the content filtering system detects harmful content, you'll receive either an error on the API call if the prompt was deemed inappropriate or the finish_reason on the response will be content_filter to signify that some of the completion was filtered. When building your application or system, you'll want to account for these scenarios where the content returned by the Completions API is filtered, which might result in content that is incomplete. How you act on this information will be application specific. The behavior can be summarized in the following points:

Prompts that are classified at a filtered category and severity level will return an HTTP 400 error.
Non-streaming completions calls won't return any content when the content is filtered. The finish_reason value will be set to content_filter. In rare cases with longer responses, a partial result can be returned. In these cases, the finish_reason will be updated.
For streaming completions calls, segments will be returned back to the user as they're completed. The service will continue streaming until either reaching a stop token, length, or when content that is classified at a filtered category and severity level is detected.
Scenario: You send a non-streaming completions call asking for multiple outputs; no content is classified at a filtered category and severity level
The table below outlines the various ways content filtering can appear:

HTTP response code	Response behavior
200	In the cases when all generation passes the filters as configured, no content moderation details are added to the response. The finish_reason for each generation will be either stop or length.
Example request payload:

JSON

Copy
{
    "prompt":"Text example", 
    "n": 3,
    "stream": false
}
Example response JSON:

JSON

Copy
{
    "id": "example-id",
    "object": "text_completion",
    "created": 1653666286,
    "model": "davinci",
    "choices": [
        {
            "text": "Response generated text",
            "index": 0,
            "finish_reason": "stop",
            "logprobs": null
        }
    ]
}

Scenario: Your API call asks for multiple responses (N>1) and at least one of the responses is filtered
HTTP Response Code	Response behavior
200	The generations that were filtered will have a finish_reason value of content_filter.
Example request payload:

JSON

Copy
{
    "prompt":"Text example",
    "n": 3,
    "stream": false
}
Example response JSON:

JSON

Copy
{
    "id": "example",
    "object": "text_completion",
    "created": 1653666831,
    "model": "ada",
    "choices": [
        {
            "text": "returned text 1",
            "index": 0,
            "finish_reason": "length",
            "logprobs": null
        },
        {
            "text": "returned text 2",
            "index": 1,
            "finish_reason": "content_filter",
            "logprobs": null
        }
    ]
}
Scenario: An inappropriate input prompt is sent to the completions API (either for streaming or non-streaming)
HTTP Response Code	Response behavior
400	The API call fails when the prompt triggers a content filter as configured. Modify the prompt and try again.
Example request payload:

JSON

Copy
{
    "prompt":"Content that triggered the filtering model"
}
Example response JSON:

JSON

Copy
"error": {
    "message": "The response was filtered",
    "type": null,
    "param": "prompt",
    "code": "content_filter",
    "status": 400
}
Scenario: You make a streaming completions call; no output content is classified at a filtered category and severity level
HTTP Response Code	Response behavior
200	In this case, the call will stream back with the full generation and finish_reason will be either 'length' or 'stop' for each generated response.
Example request payload:

JSON

Copy
{
    "prompt":"Text example",
    "n": 3,
    "stream": true
}
Example response JSON:

JSON

Copy
{
    "id": "cmpl-example",
    "object": "text_completion",
    "created": 1653670914,
    "model": "ada",
    "choices": [
        {
            "text": "last part of generation",
            "index": 2,
            "finish_reason": "stop",
            "logprobs": null
        }
    ]
}
Scenario: You make a streaming completions call asking for multiple completions and at least a portion of the output content is filtered
HTTP Response Code	Response behavior
200	For a given generation index, the last chunk of the generation includes a non-null finish_reason value. The value is content_filter when the generation was filtered.
Example request payload:

JSON

Copy
{
    "prompt":"Text example",
    "n": 3,
    "stream": true
}
Example response JSON:

JSON

Copy
 {
    "id": "cmpl-example",
    "object": "text_completion",
    "created": 1653670515,
    "model": "ada",
    "choices": [
        {
            "text": "Last part of generated text streamed back",
            "index": 2,
            "finish_reason": "content_filter",
            "logprobs": null
        }
    ]
}
Scenario: Content filtering system doesn't run on the completion
HTTP Response Code	Response behavior
200	If the content filtering system is down or otherwise unable to complete the operation in time, your request will still complete without content filtering. You can determine that the filtering wasn't applied by looking for an error message in the content_filter_result object.
Example request payload:

JSON

Copy
{
    "prompt":"Text example",
    "n": 1,
    "stream": false
}
Example response JSON:

JSON

Copy
{
    "id": "cmpl-example",
    "object": "text_completion",
    "created": 1652294703,
    "model": "ada",
    "choices": [
        {
            "text": "generated text",
            "index": 0,
            "finish_reason": "length",
            "logprobs": null,
            "content_filter_result": {
                "error": {
                    "code": "content_filter_error",
                    "message": "The contents are not filtered"
                }
            }
        }
    ]
}
Annotations (preview)
Main content filters
When annotations are enabled as shown in the code snippet below, the following information is returned via the API for the main categories (hate and fairness, sexual, violence, and self-harm):

content filtering category (hate, sexual, violence, self_harm)
the severity level (safe, low, medium or high) within each content category
filtering status (true or false).
Optional models
Optional models can be enabled in annotate (returns information when content was flagged, but not filtered) or filter mode (returns information when content was flagged and filtered).

When annotations are enabled as shown in the code snippet below, the following information is returned by the API for optional models jailbreak risk, protected material text and protected material code:

category (jailbreak, protected_material_text, protected_material_code),
detected (true or false),
filtered (true or false).
For the protected material code model, the following additional information is returned by the API:

an example citation of a public GitHub repository where a code snippet was found
the license of the repository.
When displaying code in your application, we strongly recommend that the application also displays the example citation from the annotations. Compliance with the cited license may also be required for Customer Copyright Commitment coverage.

Annotations are currently in preview for Completions and Chat Completions (GPT models); the following code snippet shows how to use annotations in preview:

OpenAI Python 0.28.1
OpenAI Python 1.x
JavaScript
PowerShell
Python

Copy
# os.getenv() for the endpoint and key assumes that you are using environment variables.

import os
import openai
openai.api_type = "azure"
openai.api_base = os.getenv("AZURE_OPENAI_ENDPOINT") 
openai.api_version = "2023-06-01-preview" # API version required to test out Annotations preview
openai.api_key = os.getenv("AZURE_OPENAI_KEY")

response = openai.Completion.create(
    engine="gpt-35-turbo", # engine = "deployment_name".
    messages=[{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Example prompt that leads to a protected code completion that was detected, but not filtered"}]     # Content that is detected at severity level medium or high is filtered, 
    # while content detected at severity level low isn't filtered by the content filters.
)

print(response)

Output
JSON

Copy
{ 
  "choices": [ 
    { 
      "content_filter_results": { 
        "custom_blocklists": [], 
        "hate": { 
          "filtered": false, 
          "severity": "safe" 
        }, 
        "protected_material_code": { 
          "citation": { 
            "URL": " https://github.com/username/repository-name/path/to/file-example.txt", 
            "license": "EXAMPLE-LICENSE" 
          }, 
          "detected": true,
          "filtered": false 
        }, 
        "protected_material_text": { 
          "detected": false, 
          "filtered": false 
        }, 
        "self_harm": { 
          "filtered": false, 
          "severity": "safe" 
        }, 
        "sexual": { 
          "filtered": false, 
          "severity": "safe" 
        }, 
        "violence": { 
          "filtered": false, 
          "severity": "safe" 
        } 
      }, 
      "finish_reason": "stop", 
      "index": 0, 
      "message": { 
        "content": "Example model response will be returned ", 
        "role": "assistant" 
      } 
    } 
  ], 
  "created": 1699386280, 
  "id": "chatcmpl-8IMI4HzcmcK6I77vpOJCPt0Vcf8zJ", 
  "model": "gpt-35-turbo", 
  "object": "chat.completion", 
  "prompt_filter_results": [ 
    { 
      "content_filter_results": { 
        "custom_blocklists": [], 
        "hate": { 
          "filtered": false, 
          "severity": "safe" 
        }, 
        "jailbreak": { 
          "detected": false, 
          "filtered": false 
        }, 
        "profanity": { 
          "detected": false, 
          "filtered": false 
        }, 
        "self_harm": { 
          "filtered": false, 
          "severity": "safe" 
        }, 
        "sexual": { 
          "filtered": false, 
          "severity": "safe" 
        }, 
        "violence": { 
          "filtered": false, 
          "severity": "safe" 
        } 
      }, 
      "prompt_index": 0 
    } 
  ], 
  "usage": { 
    "completion_tokens": 40, 
    "prompt_tokens": 11, 
    "total_tokens": 417 
  } 
} 
The following code snippet shows how to retrieve annotations when content was filtered:

Python

Copy
# os.getenv() for the endpoint and key assumes that you are using environment variables.

import os
import openai
openai.api_type = "azure"
openai.api_base = os.getenv("AZURE_OPENAI_ENDPOINT") 
openai.api_version = "2023-06-01-preview" # API version required to test out Annotations preview
openai.api_key = os.getenv("AZURE_OPENAI_KEY")

try:
    response = openai.Completion.create(
        prompt="<PROMPT>",
        engine="<MODEL_DEPLOYMENT_NAME>",
    )
    print(response)

except openai.error.InvalidRequestError as e:
    if e.error.code == "content_filter" and e.error.innererror:
        content_filter_result = e.error.innererror.content_filter_result
        # print the formatted JSON
        print(content_filter_result)

        # or access the individual categories and details
        for category, details in content_filter_result.items():
            print(f"{category}:\n filtered={details['filtered']}\n severity={details['severity']}")

For details on the inference REST API endpoints for Azure OpenAI and how to create Chat and Completions please follow Azure OpenAI Service REST API reference guidance. Annotations are returned for all scenarios when using 2023-06-01-preview.

Example scenario: An input prompt containing content that is classified at a filtered category and severity level is sent to the completions API
JSON

Copy
{
    "error": {
        "message": "The response was filtered due to the prompt triggering Azure Content management policy. 
                   Please modify your prompt and retry. To learn more about our content filtering policies
                   please read our documentation: https://go.microsoft.com/fwlink/?linkid=21298766",
        "type": null,
        "param": "prompt",
        "code": "content_filter",
        "status": 400,
        "innererror": {
            "code": "ResponsibleAIPolicyViolation",
            "content_filter_result": {
                "hate": {
                    "filtered": true,
                    "severity": "high"
                },
                "self-harm": {
                    "filtered": true,
                    "severity": "high"
                },
                "sexual": {
                    "filtered": false,
                    "severity": "safe"
                },
                "violence": {
                    "filtered":true,
                    "severity": "medium"
                }
            }
        }
    }
}
Best practices
As part of your application design, consider the following best practices to deliver a positive experience with your application while minimizing potential harms:

Decide how you want to handle scenarios where your users send prompts containing content that is classified at a filtered category and severity level or otherwise misuse your application.
Check the finish_reason to see if a completion is filtered.
Check that there's no error object in the content_filter_result (indicating that content filters didn't run).
If you're using the protected material code model in annotate mode, display the citation URL when you're displaying the code in your application.
Next steps
Learn more about the underlying models that power Azure OpenAI.
Apply for modified content filters via this form.
Azure OpenAI content filtering is powered by Azure AI Content Safety.
Learn more about understanding and mitigating risks associated with your application: Overview of Responsible AI practices for Azure OpenAI models.
Learn more about how data is processed in connection with content filtering and abuse monitoring: Data, privacy, and security for Azure OpenAI Service.
Feedback
Submit and view feedback for

 View all page feedback
Additional resources
Documentation

How to use content filters (preview) with Azure OpenAI Service - Azure OpenAI
Learn how to use content filters (preview) with Azure OpenAI Service

Azure OpenAI Service abuse monitoring - Azure OpenAI
Learn about the abuse monitoring capabilities of Azure OpenAI Service

Data, privacy, and security for Azure OpenAI Service - Azure AI services
This document details issues for data, privacy, and security for Azure OpenAI Service

Show 5 more
Training

Module

Introduction to Content Moderator - Training
Discover how Content Moderator can evaluate text, images, and video, for appropriateness.
---------------
How to turn off this content filter