Repository: opensearch-project/documentation-website
Pull request #9538: adding index phrases mapping parameter docs
Author: login: AntonEliatra
Labels: name: Content gap
name: backport 3.0
Created: 2025-04-01T18:42:07Z | Merged: 2025-05-20T14:26:02Z

### Description
adding index phrases mapping parameter docs

### Version
all

### Checklist
- [x] By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the [Developers Certificate of Origin](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).
For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).



Diff:
diff --git a/_field-types/mapping-parameters/index-phrases.md b/_field-types/mapping-parameters/index-phrases.md
new file mode 100644
index 00000000000..b52176e6ed4
--- /dev/null
+++ b/_field-types/mapping-parameters/index-phrases.md
@@ -0,0 +1,96 @@
+---
+layout: default
+title: Index phrases
+parent: Mapping parameters
+grand_parent: Mapping and field types
+nav_order: 80
+has_children: false
+has_toc: false
+---
+
+# Index phrases
+
+The `index_phrases` mapping parameter determines whether a field's text is additionally processed to generate phrase tokens. When enabled, the system creates extra tokens representing sequences of exactly two consecutive words (_bigrams_). This can significantly improve the performance and accuracy of phrase queries. However, it also increases the index size and the time needed to index documents.
+
+By default, `index_phrases` is set to `false` to maintain a leaner index and faster document ingestion.
+
+## Enabling index phrases on a field
+
+The following example creates an index named `blog` in which the `content` field is configured with `index_phrases`:
+
+```json
+PUT /blog
+{
+  "mappings": {
+    "properties": {
+      "content": {
+        "type": "text",
+        "index_phrases": true
+      }
+    }
+  }
+}
+```
+{% include copy-curl.html %}
+
+Index a document using the following request:
+
+```json
+PUT /blog/_doc/1
+{
+  "content": "The slow green turtle swims past the whale"
+}
+```
+{% include copy-curl.html %}
+
+Perform a `match_phrase` query using the following search request:
+
+```json
+POST /blog/_search
+{
+  "query": {
+    "match_phrase": {
+      "content": "slow green"
+    }
+  }
+}
+```
+{% include copy-curl.html %}
+
+The query returns the stored document:
+
+```json
+{
+  "took": 25,
+  "timed_out": false,
+  "_shards": {
+    "total": 1,
+    "successful": 1,
+    "skipped": 0,
+    "failed": 0
+  },
+  "hits": {
+    "total": {
+      "value": 1,
+      "relation": "eq"
+    },
+    "max_score": 0.5753642,
+    "hits": [
+      {
+        "_index": "blog",
+        "_id": "1",
+        "_score": 0.5753642,
+        "_source": {
+          "content": "The slow green turtle swims past the whale"
+        }
+      }
+    ]
+  }
+}
+```
+
+Although the same hit is returned when you don't provide the `index_phrases` mapping parameter, using this parameter ensures that the query performs as follows:
+
+- Uses the `.index_phrases` field internally
+- Matches pre-tokenized bigrams such as "slow green", "green turtle", or "turtle swims".
+- Bypasses position lookups and is faster, especially at scale.
\ No newline at end of file
diff --git a/_field-types/mapping-parameters/index.md b/_field-types/mapping-parameters/index.md
index 250e7cec055..44015a1e114 100644
--- a/_field-types/mapping-parameters/index.md
+++ b/_field-types/mapping-parameters/index.md
@@ -25,5 +25,6 @@ Parameter | Description
 [`ignore_above`]({{site.url}}{{site.baseurl}}/field-types/mapping-parameters/ignore-above/) | Skips indexing values that exceed the specified length. Default value is `2147483647`, which means that there is no limit on the field value length. Allowed values are any positive integer.
 [`ignore_malformed`]({{site.url}}{{site.baseurl}}/field-types/mapping-parameters/ignore-malformed/) | Specifies whether malformed values should be ignored. Default value is `false`, which means that malformed values are not ignored. Allowed values are `true` or `false`.
 [`index`]({{site.url}}{{site.baseurl}}/field-types/mapping-parameters/index-parameter/) | Specifies whether a field should be indexed. Default value is `true`, which means that the field is indexed. Allowed values are `true` or `false`.
+[`index_phrases`]({{site.url}}{{site.baseurl}}/field-types/mapping-parameters/index-phrases/) | Determines whether extra phrase tokens are generated for text fields to improve the performance of phrase queries. Default is `false`.
 [`index_options`]({{site.url}}{{site.baseurl}}/field-types/mapping-parameters/index-options/) | Specifies what information should be stored in an index for scoring purposes. Default value is `docs`, which means that only the document numbers are stored in the index. Allowed values are `docs`, `freqs`, `positions`, or `offsets`.
 


Reviews:
user: login: mgodwan
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-08T19:26:57Z
user: login: AntonEliatra
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T18:41:42Z
user: login: kolchfa-aws
body: Thank you, @AntonEliatra! Please see my comments and let me know if you have any questions.
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-14T15:12:28Z
user: login: natebower
body: @AntonEliatra Please see my changes and tag me for approval when addressed. Thanks!
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T14:19:24Z
user: login: natebower
body: Thanks @AntonEliatra! LGTM 
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T14:22:13Z

Review comments:
diff_hunk: @@ -0,0 +1,91 @@
+---
+layout: default
+title: Index phrases
+parent: Mapping parameters
+grand_parent: Mapping and field types
+nav_order: 80
+has_children: false
+has_toc: false
+---
+
+# Index phrases
+
+The `index_phrases` mapping parameter determines whether a field’s text is additionally processed to generate phrase tokens. When enabled, the system creates extra tokens that represent consecutive word sequences. This can significantly improve the performance and accuracy of phrase queries. However, it also increases the index size and the time needed to index documents.
path: _field-types/mapping-parameters/index-phrases.md
user: login: mgodwan
body: `consecutive word sequences`

Can we call out that it is just 2 consecutive words? The javadoc here helps understand how tokens are generated.

https://github.com/apache/lucene/blob/ccc8f7e245e76a1c3974770e234fa03d8a905316/lucene/analysis/common/src/java/org/apache/lucene/analysis/shingle/FixedShingleFilter.java#L37C7-L38C98
created_at: 2025-05-08T19:26:57Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,91 @@
+---
+layout: default
+title: Index phrases
+parent: Mapping parameters
+grand_parent: Mapping and field types
+nav_order: 80
+has_children: false
+has_toc: false
+---
+
+# Index phrases
+
+The `index_phrases` mapping parameter determines whether a field’s text is additionally processed to generate phrase tokens. When enabled, the system creates extra tokens that represent consecutive word sequences. This can significantly improve the performance and accuracy of phrase queries. However, it also increases the index size and the time needed to index documents.
path: _field-types/mapping-parameters/index-phrases.md
user: login: AntonEliatra
body: @mgodwan thank you for the review, I've made changes, could you please review?
created_at: 2025-05-09T18:41:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,96 @@
+---
+layout: default
+title: Index phrases
+parent: Mapping parameters
+grand_parent: Mapping and field types
+nav_order: 80
+has_children: false
+has_toc: false
+---
+
+# Index phrases
+
+The `index_phrases` mapping parameter determines whether a field’s text is additionally processed to generate phrase tokens. When enabled, the system creates extra tokens representing sequences of exactly two consecutive words (`bigrams`). This can significantly improve the performance and accuracy of phrase queries. However, it also increases the index size and the time needed to index documents.
path: _field-types/mapping-parameters/index-phrases.md
user: login: kolchfa-aws
body: ```suggestion
The `index_phrases` mapping parameter determines whether a field's text is additionally processed to generate phrase tokens. When enabled, the system creates extra tokens representing sequences of exactly two consecutive words (_bigrams_). This can significantly improve the performance and accuracy of phrase queries. However, it also increases the index size and the time needed to index documents.
```
created_at: 2025-05-14T15:10:09Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,96 @@
+---
+layout: default
+title: Index phrases
+parent: Mapping parameters
+grand_parent: Mapping and field types
+nav_order: 80
+has_children: false
+has_toc: false
+---
+
+# Index phrases
+
+The `index_phrases` mapping parameter determines whether a field’s text is additionally processed to generate phrase tokens. When enabled, the system creates extra tokens representing sequences of exactly two consecutive words (`bigrams`). This can significantly improve the performance and accuracy of phrase queries. However, it also increases the index size and the time needed to index documents.
+
+By default, `index_phrases` is set to `false`, to maintain a leaner index and faster document ingestion.
+
+## Enabling index phrases on a field
+
+The following example creates an index named `blog` where the `content` field is configured with `index_phrases`:
path: _field-types/mapping-parameters/index-phrases.md
user: login: kolchfa-aws
body: ```suggestion
The following example creates an index named `blog` in which the `content` field is configured with `index_phrases`:
```
created_at: 2025-05-14T15:10:31Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,96 @@
+---
+layout: default
+title: Index phrases
+parent: Mapping parameters
+grand_parent: Mapping and field types
+nav_order: 80
+has_children: false
+has_toc: false
+---
+
+# Index phrases
+
+The `index_phrases` mapping parameter determines whether a field’s text is additionally processed to generate phrase tokens. When enabled, the system creates extra tokens representing sequences of exactly two consecutive words (`bigrams`). This can significantly improve the performance and accuracy of phrase queries. However, it also increases the index size and the time needed to index documents.
+
+By default, `index_phrases` is set to `false`, to maintain a leaner index and faster document ingestion.
+
+## Enabling index phrases on a field
+
+The following example creates an index named `blog` where the `content` field is configured with `index_phrases`:
+
+```json
+PUT /blog
+{
+  "mappings": {
+    "properties": {
+      "content": {
+        "type": "text",
+        "index_phrases": true
+      }
+    }
+  }
+}
+```
+{% include copy-curl.html %}
+
+Index a document using the following command:
path: _field-types/mapping-parameters/index-phrases.md
user: login: kolchfa-aws
body: ```suggestion
Index a document using the following request:
```
created_at: 2025-05-14T15:10:39Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,96 @@
+---
+layout: default
+title: Index phrases
+parent: Mapping parameters
+grand_parent: Mapping and field types
+nav_order: 80
+has_children: false
+has_toc: false
+---
+
+# Index phrases
+
+The `index_phrases` mapping parameter determines whether a field’s text is additionally processed to generate phrase tokens. When enabled, the system creates extra tokens representing sequences of exactly two consecutive words (`bigrams`). This can significantly improve the performance and accuracy of phrase queries. However, it also increases the index size and the time needed to index documents.
+
+By default, `index_phrases` is set to `false`, to maintain a leaner index and faster document ingestion.
+
+## Enabling index phrases on a field
+
+The following example creates an index named `blog` where the `content` field is configured with `index_phrases`:
+
+```json
+PUT /blog
+{
+  "mappings": {
+    "properties": {
+      "content": {
+        "type": "text",
+        "index_phrases": true
+      }
+    }
+  }
+}
+```
+{% include copy-curl.html %}
+
+Index a document using the following command:
+
+```json
+PUT /blog/_doc/1
+{
+  "content": "The slow green turtle swims past the whale"
+}
+```
+{% include copy-curl.html %}
+
+Perform a `match_phrase` query using the following search request:
+
+```json
+POST /blog/_search
+{
+  "query": {
+    "match_phrase": {
+      "content": "slow green"
+    }
+  }
+}
+```
+{% include copy-curl.html %}
+
+The query returns the stored document:
+
+```json
+{
+  "took": 25,
+  "timed_out": false,
+  "_shards": {
+    "total": 1,
+    "successful": 1,
+    "skipped": 0,
+    "failed": 0
+  },
+  "hits": {
+    "total": {
+      "value": 1,
+      "relation": "eq"
+    },
+    "max_score": 0.5753642,
+    "hits": [
+      {
+        "_index": "blog",
+        "_id": "1",
+        "_score": 0.5753642,
+        "_source": {
+          "content": "The slow green turtle swims past the whale"
+        }
+      }
+    ]
+  }
+}
+```
+
+Although the returned hit is possible without using `index_phrases` mapping parameter, using this parameter ensures that the query:
path: _field-types/mapping-parameters/index-phrases.md
user: login: kolchfa-aws
body: ```suggestion
Although the same hit is returned when you don't provide the `index_phrases` mapping parameter, using this parameter ensures that the query performs as follows:
```
created_at: 2025-05-14T15:11:48Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,96 @@
+---
+layout: default
+title: Index phrases
+parent: Mapping parameters
+grand_parent: Mapping and field types
+nav_order: 80
+has_children: false
+has_toc: false
+---
+
+# Index phrases
+
+The `index_phrases` mapping parameter determines whether a field’s text is additionally processed to generate phrase tokens. When enabled, the system creates extra tokens representing sequences of exactly two consecutive words (`bigrams`). This can significantly improve the performance and accuracy of phrase queries. However, it also increases the index size and the time needed to index documents.
+
+By default, `index_phrases` is set to `false`, to maintain a leaner index and faster document ingestion.
+
+## Enabling index phrases on a field
+
+The following example creates an index named `blog` where the `content` field is configured with `index_phrases`:
+
+```json
+PUT /blog
+{
+  "mappings": {
+    "properties": {
+      "content": {
+        "type": "text",
+        "index_phrases": true
+      }
+    }
+  }
+}
+```
+{% include copy-curl.html %}
+
+Index a document using the following command:
+
+```json
+PUT /blog/_doc/1
+{
+  "content": "The slow green turtle swims past the whale"
+}
+```
+{% include copy-curl.html %}
+
+Perform a `match_phrase` query using the following search request:
+
+```json
+POST /blog/_search
+{
+  "query": {
+    "match_phrase": {
+      "content": "slow green"
+    }
+  }
+}
+```
+{% include copy-curl.html %}
+
+The query returns the stored document:
+
+```json
+{
+  "took": 25,
+  "timed_out": false,
+  "_shards": {
+    "total": 1,
+    "successful": 1,
+    "skipped": 0,
+    "failed": 0
+  },
+  "hits": {
+    "total": {
+      "value": 1,
+      "relation": "eq"
+    },
+    "max_score": 0.5753642,
+    "hits": [
+      {
+        "_index": "blog",
+        "_id": "1",
+        "_score": 0.5753642,
+        "_source": {
+          "content": "The slow green turtle swims past the whale"
+        }
+      }
+    ]
+  }
+}
+```
+
+Although the returned hit is possible without using `index_phrases` mapping parameter, using this parameter ensures that the query:
+
+- Uses the `.index_phrases` field internally.
+- Matches pre-tokenized `bigrams` such as "slow green", "green turtle", "turtle swims".
path: _field-types/mapping-parameters/index-phrases.md
user: login: kolchfa-aws
body: ```suggestion
- Matches pre-tokenized bigrams such as "slow green", "green turtle", "turtle swims".
```
created_at: 2025-05-14T15:12:03Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,96 @@
+---
+layout: default
+title: Index phrases
+parent: Mapping parameters
+grand_parent: Mapping and field types
+nav_order: 80
+has_children: false
+has_toc: false
+---
+
+# Index phrases
+
+The `index_phrases` mapping parameter determines whether a field's text is additionally processed to generate phrase tokens. When enabled, the system creates extra tokens representing sequences of exactly two consecutive words (_bigrams_). This can significantly improve the performance and accuracy of phrase queries. However, it also increases the index size and the time needed to index documents.
+
+By default, `index_phrases` is set to `false`, to maintain a leaner index and faster document ingestion.
path: _field-types/mapping-parameters/index-phrases.md
user: login: natebower
body: ```suggestion
By default, `index_phrases` is set to `false` to maintain a leaner index and faster document ingestion.
```
created_at: 2025-05-20T14:17:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,96 @@
+---
+layout: default
+title: Index phrases
+parent: Mapping parameters
+grand_parent: Mapping and field types
+nav_order: 80
+has_children: false
+has_toc: false
+---
+
+# Index phrases
+
+The `index_phrases` mapping parameter determines whether a field's text is additionally processed to generate phrase tokens. When enabled, the system creates extra tokens representing sequences of exactly two consecutive words (_bigrams_). This can significantly improve the performance and accuracy of phrase queries. However, it also increases the index size and the time needed to index documents.
+
+By default, `index_phrases` is set to `false`, to maintain a leaner index and faster document ingestion.
+
+## Enabling index phrases on a field
+
+The following example creates an index named `blog` in which the `content` field is configured with `index_phrases`:
+
+```json
+PUT /blog
+{
+  "mappings": {
+    "properties": {
+      "content": {
+        "type": "text",
+        "index_phrases": true
+      }
+    }
+  }
+}
+```
+{% include copy-curl.html %}
+
+Index a document using the following request:
+
+```json
+PUT /blog/_doc/1
+{
+  "content": "The slow green turtle swims past the whale"
+}
+```
+{% include copy-curl.html %}
+
+Perform a `match_phrase` query using the following search request:
+
+```json
+POST /blog/_search
+{
+  "query": {
+    "match_phrase": {
+      "content": "slow green"
+    }
+  }
+}
+```
+{% include copy-curl.html %}
+
+The query returns the stored document:
+
+```json
+{
+  "took": 25,
+  "timed_out": false,
+  "_shards": {
+    "total": 1,
+    "successful": 1,
+    "skipped": 0,
+    "failed": 0
+  },
+  "hits": {
+    "total": {
+      "value": 1,
+      "relation": "eq"
+    },
+    "max_score": 0.5753642,
+    "hits": [
+      {
+        "_index": "blog",
+        "_id": "1",
+        "_score": 0.5753642,
+        "_source": {
+          "content": "The slow green turtle swims past the whale"
+        }
+      }
+    ]
+  }
+}
+```
+
+Although the same hit is returned when you don't provide the `index_phrases` mapping parameter, using this parameter ensures that the query performs as follows:
+
+- Uses the `.index_phrases` field internally.
path: _field-types/mapping-parameters/index-phrases.md
user: login: natebower
body: ```suggestion
- Uses the `.index_phrases` field internally
```
created_at: 2025-05-20T14:18:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,96 @@
+---
+layout: default
+title: Index phrases
+parent: Mapping parameters
+grand_parent: Mapping and field types
+nav_order: 80
+has_children: false
+has_toc: false
+---
+
+# Index phrases
+
+The `index_phrases` mapping parameter determines whether a field's text is additionally processed to generate phrase tokens. When enabled, the system creates extra tokens representing sequences of exactly two consecutive words (_bigrams_). This can significantly improve the performance and accuracy of phrase queries. However, it also increases the index size and the time needed to index documents.
+
+By default, `index_phrases` is set to `false`, to maintain a leaner index and faster document ingestion.
+
+## Enabling index phrases on a field
+
+The following example creates an index named `blog` in which the `content` field is configured with `index_phrases`:
+
+```json
+PUT /blog
+{
+  "mappings": {
+    "properties": {
+      "content": {
+        "type": "text",
+        "index_phrases": true
+      }
+    }
+  }
+}
+```
+{% include copy-curl.html %}
+
+Index a document using the following request:
+
+```json
+PUT /blog/_doc/1
+{
+  "content": "The slow green turtle swims past the whale"
+}
+```
+{% include copy-curl.html %}
+
+Perform a `match_phrase` query using the following search request:
+
+```json
+POST /blog/_search
+{
+  "query": {
+    "match_phrase": {
+      "content": "slow green"
+    }
+  }
+}
+```
+{% include copy-curl.html %}
+
+The query returns the stored document:
+
+```json
+{
+  "took": 25,
+  "timed_out": false,
+  "_shards": {
+    "total": 1,
+    "successful": 1,
+    "skipped": 0,
+    "failed": 0
+  },
+  "hits": {
+    "total": {
+      "value": 1,
+      "relation": "eq"
+    },
+    "max_score": 0.5753642,
+    "hits": [
+      {
+        "_index": "blog",
+        "_id": "1",
+        "_score": 0.5753642,
+        "_source": {
+          "content": "The slow green turtle swims past the whale"
+        }
+      }
+    ]
+  }
+}
+```
+
+Although the same hit is returned when you don't provide the `index_phrases` mapping parameter, using this parameter ensures that the query performs as follows:
+
+- Uses the `.index_phrases` field internally.
+- Matches pre-tokenized bigrams such as "slow green", "green turtle", "turtle swims".
path: _field-types/mapping-parameters/index-phrases.md
user: login: natebower
body: ```suggestion
- Matches pre-tokenized bigrams such as "slow green", "green turtle", or "turtle swims".
```
created_at: 2025-05-20T14:18:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -25,5 +25,6 @@ Parameter | Description
 [`ignore_above`]({{site.url}}{{site.baseurl}}/field-types/mapping-parameters/ignore-above/) | Skips indexing values that exceed the specified length. Default value is `2147483647`, which means that there is no limit on the field value length. Allowed values are any positive integer.
 [`ignore_malformed`]({{site.url}}{{site.baseurl}}/field-types/mapping-parameters/ignore-malformed/) | Specifies whether malformed values should be ignored. Default value is `false`, which means that malformed values are not ignored. Allowed values are `true` or `false`.
 [`index`]({{site.url}}{{site.baseurl}}/field-types/mapping-parameters/index-parameter/) | Specifies whether a field should be indexed. Default value is `true`, which means that the field is indexed. Allowed values are `true` or `false`.
+[`index_phrases`]({{site.url}}{{site.baseurl}}/field-types/mapping-parameters/index-phrases/) | Determines if extra phrase tokens are generated for text fields to improve the performance of phrase queries. Default is `false`.
path: _field-types/mapping-parameters/index.md
user: login: natebower
body: ```suggestion
[`index_phrases`]({{site.url}}{{site.baseurl}}/field-types/mapping-parameters/index-phrases/) | Determines whether extra phrase tokens are generated for text fields to improve the performance of phrase queries. Default is `false`.
```
created_at: 2025-05-20T14:19:06Z
author_association: CONTRIBUTOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-04-01T18:42:17Z
body: Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged.

Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a [maintainer](https://github.com/opensearch-project/documentation-website/blob/main/MAINTAINERS.md).

**When you're ready for doc review, tag the assignee of this PR**. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review.
author_association: NONE
user: login: AntonEliatra
created_at: 2025-04-01T18:47:48Z
body: @kolchfa-aws this PR is ready for review

author_association: CONTRIBUTOR
user: login: kolchfa-aws
created_at: 2025-04-01T18:57:15Z
body: @mgodwan Could you review this PR? Thanks!
author_association: COLLABORATOR
user: login: AntonEliatra
created_at: 2025-05-15T22:53:47Z
body: @kolchfa-aws thats now updated
author_association: CONTRIBUTOR
user: login: AntonEliatra
created_at: 2025-05-20T14:21:38Z
body: @natebower Thats updated now
author_association: CONTRIBUTOR