Repository: wagtail/wagtail

Issue:
OpenSearch as replacement for propietary ElasticSearch
### Is your proposal related to a problem?

Yes.  I have requirements to use Open Source infrastructure, as we cannot risk vendor lock-in for core components.

### Describe the solution you'd like

Full support for OpenSearch, https://www.opensearch.org/ a fork of the open source version of ElasticSearch after then went proprietary on their license.

### Describe alternatives you've considered
Use an old ElasticSearch
Create my own OpenSearch integration

### Additional context

ElasticCo changed the license of new versions of their products to a proprietary, source available license https://www.elastic.co/pricing/faq/licensing

The Wagtail marketing website refers to ElasticSearch in several locations as being Open Source, however, this is no longer the case and the marketing website should be updated to reflect the Source Available license change in ElasticSearch.  Preferably, an new release of Wagtail would switch to using the Open Source OpenSearch and the marketing would be updated to talk about OpenSearch instead of ElasticSearch.

As a community, we can be thankful for the Open Source work that Elastic has created in the past, and their continued contributions to Open Source projects like Apache Solar, however, we should not continue to refer to ElasticSearch as Open Source when that is no longer the case.  They as a company made a choice to switch to a proprietary non-open license, and with that choice comes consequences.  We as a community need to react accordingly to not spread confusion about what is and is not open source.  The simplest and best approach is to adopt OpenSearch as the standard search interface in Open Source projects like wagtail.  In the meantime, references to Elastic being open source should be removed from the website and marketing materials.


Discussion:
Hi @ensignavenger, thank you for the suggestion. This sounds to me like a welcome addition. In our docs, I see we explicitly mention support for [Amazon AWS Elasticsearch Service](https://docs.wagtail.org/en/stable/topics/search/backends.html?highlight=Elastic#amazon-aws-elasticsearch), which I believe is now [Amazon OpenSearch Service](https://aws.amazon.com/opensearch-service/), so it could be the existing search backend would already support OpenSearch with little to no change? This seems to be using our own `wagtail.search.backends.elasticsearch5` backend.

If a new backend has to be built, I have a feeling it might be easier to build this as an external package first (see for example [wagtail-whoosh](https://github.com/wagtail/wagtail-whoosh)), since Wagtail has good support for creating and using external search backends.

---

For the marketing website, I saw two mentions on the [Features](https://wagtail.org/features/) page as you described:

> Elasticsearch is an excellent open source search engine with straightforward integration with Wagtail and great performance. 

And:

> Elasticsearch
> Wagtail uses Elastic for finding content within the admin interface. It provides quick and fluent access to pages, images and documents.
> We use Elastic because it’s open source, gives great performance and offers a straightforward integration with Wagtail.

I’m not too sure what I’d replace those with myself - someone else might feel more inspired, if not I’ll let people who look after the site know they’ll have to change this.
Thank you for responding and for your work providing Wagtail!  I agree it probably would not be too much work to get Open Search working with minimal changes to the existing integration code.  I do have some concern for the future maintenance as development diverges between Elastic and OpenSearch.  I haven't had a chance to dig in to see what it would take myself, yet.

As for the marketing site, I think those are the two references I found as well- simply removing the reference to elastic being Open Source would be a good first pass.  It could be removed entirely or changed to "source available".

>We use Elastic because it provides great performance and offers a straightforward integration with Wagtail.   Wagtail also supports alternative search back ends via packages...






FYI, I tried dropping in OpenSearch in place of Elasticsearch, but get:
```console
$ ./manage.py wagtail_update_index
Updating backend: default
default: Rebuilding index wagtail__wagtailcore_page
Traceback (most recent call last):
…
  File "/…/venv/lib/python3.10/site-packages/elasticsearch/transport.py", line 421, in perform_request
    _ProductChecker.raise_error(self._verified_elasticsearch)
  File "/…/venv/lib/python3.10/site-packages/elasticsearch/transport.py", line 638, in raise_error
    raise UnsupportedProductError(message)
elasticsearch.exceptions.UnsupportedProductError: The client noticed that the server is not Elasticsearch and we do not support this unknown product
$ echo \U+1F615
😕
```

FYI, more FOSS organizations are dumping Elasticsearch. For example, Homebrew won't install it, reports:
```
$ brew info elasticsearch
==> elasticsearch: stable 7.10.2
Distributed search & analytics engine
https://www.elastic.co/products/elasticsearch
Disabled because it is switching to an incompatible license. Check out `opensearch` instead!
Not installed
…
```

Previously, I've had luck supporting OpenSearch using the `elasticsearch7` backend and pinning the `elasticsearch` dependency to 7.13.4 (the last version before the es library started throwing the `UnsupportedProductError` when connecting to a non-Elastic product).  However, in Wagtail 5.1 (commit 9203716bae8c55b460e309b16c65d5311ad6c747 ) the ES7 backend has started including `settings` as a keyword argument in its calls for compatibility for `elasticsearch` versions >= 7.15 or so.  Running on an older version of `elasticsearch` now gives an exception like:

```
Traceback (most recent call last):
  File "<project root>/manage.py", line 23, in <module>
    execute_from_command_line(sys.argv)
  File "<project root>/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "<project root>/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "<project root>/venv/lib/python3.9/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "<project root>/venv/lib/python3.9/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "<project root>/venv/lib/python3.9/site-packages/wagtail/search/management/commands/update_index.py", line 158, in handle
    self.update_backend(
  File "<project root>/venv/lib/python3.9/site-packages/wagtail/search/management/commands/update_index.py", line 84, in update_backend
    index = rebuilder.start()
  File "<project root>/venv/lib/python3.9/site-packages/wagtail/search/backends/elasticsearch5.py", line 984, in start
    self.reset_index()
  File "<project root>/venv/lib/python3.9/site-packages/wagtail/search/backends/elasticsearch5.py", line 980, in reset_index
    self.index.reset()
  File "<project root>/venv/lib/python3.9/site-packages/wagtail/search/backends/elasticsearch5.py", line 972, in reset
    self.put()
  File "<project root>/venv/lib/python3.9/site-packages/wagtail/search/backends/elasticsearch7.py", line 25, in put
    self.es.indices.create(index=self.name, **self.backend.settings)
  File "<project root>/venv/lib/python3.9/site-packages/elasticsearch/client/utils.py", line 168, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
TypeError: create() got an unexpected keyword argument 'settings'
```

As far as I can tell, there's no way to run Wagtail 5.1 on OpenSearch (and in particular Amazon OpenSearch Service) at this point.
Marking this for a potential 5.1.2 bugfix release, since if there was a previously working configuration (even if it's not one that we knew about :-) ) then this can be considered a regression.
We're seeing the same error `IndicesClient.create() got an unexpected keyword argument 'settings'` when using a Bonsai 	Elasticsearch 7.10.2 instance. 

Current Django/Wagtail versions are:
`Django==4.2.4`
`wagtail==5.1.1`

Previous versions are working fine:
`Django==4.1.7`
`wagtail==4.2.1`

In both environments the local `elasticsearch` is `elasticsearch==7.13.4`

Resolving patch:
diff --git a/wagtail/search/backends/elasticsearch7.py b/wagtail/search/backends/elasticsearch7.py
--- a/wagtail/search/backends/elasticsearch7.py
+++ b/wagtail/search/backends/elasticsearch7.py
@@ -1,5 +1,6 @@
 from copy import deepcopy
 
+from elasticsearch import VERSION as ELASTICSEARCH_VERSION
 from elasticsearch import NotFoundError
 from elasticsearch.helpers import bulk
 
@@ -13,6 +14,8 @@
 )
 from wagtail.search.index import class_is_indexed
 
+use_new_elasticsearch_api = ELASTICSEARCH_VERSION >= (7, 15)
+
 
 class Elasticsearch7Mapping(Elasticsearch6Mapping):
     def get_mapping(self):
@@ -21,17 +24,19 @@ def get_mapping(self):
 
 
 class Elasticsearch7Index(Elasticsearch6Index):
-    def put(self):
-        self.es.indices.create(index=self.name, **self.backend.settings)
+    if use_new_elasticsearch_api:
 
-    def delete(self):
-        try:
-            self.es.indices.delete(index=self.name)
-        except NotFoundError:
-            pass
+        def put(self):
+            self.es.indices.create(index=self.name, **self.backend.settings)
+
+        def delete(self):
+            try:
+                self.es.indices.delete(index=self.name)
+            except NotFoundError:
+                pass
 
-    def refresh(self):
-        self.es.indices.refresh(index=self.name)
+        def refresh(self):
+            self.es.indices.refresh(index=self.name)
 
     def add_model(self, model):
         # Get mapping
@@ -40,20 +45,36 @@ def add_model(self, model):
         # Put mapping
         self.es.indices.put_mapping(index=self.name, body=mapping.get_mapping())
 
-    def add_item(self, item):
-        # Make sure the object can be indexed
-        if not class_is_indexed(item.__class__):
-            return
+    if use_new_elasticsearch_api:
 
-        # Get mapping
-        mapping = self.mapping_class(item.__class__)
+        def add_item(self, item):
+            # Make sure the object can be indexed
+            if not class_is_indexed(item.__class__):
+                return
+
+            # Get mapping
+            mapping = self.mapping_class(item.__class__)
+
+            # Add document to index
+            self.es.index(
+                index=self.name,
+                document=mapping.get_document(item),
+                id=mapping.get_document_id(item),
+            )
 
-        # Add document to index
-        self.es.index(
-            index=self.name,
-            document=mapping.get_document(item),
-            id=mapping.get_document_id(item),
-        )
+    else:
+
+        def add_item(self, item):
+            # Make sure the object can be indexed
+            if not class_is_indexed(item.__class__):
+                return
+            # Get mapping
+            mapping = self.mapping_class(item.__class__)
+
+            # Add document to index
+            self.es.index(
+                self.name, mapping.get_document(item), id=mapping.get_document_id(item)
+            )
 
     def add_items(self, model, items):
         if not class_is_indexed(model):
@@ -93,10 +114,12 @@ class Elasticsearch7SearchQueryCompiler(Elasticsearch6SearchQueryCompiler):
 
 
 class Elasticsearch7SearchResults(Elasticsearch6SearchResults):
-    def _backend_do_search(self, body, **kwargs):
-        # As of Elasticsearch 7, the 'body' parameter is deprecated; instead, the top-level
-        # keys of the body dict are now kwargs in their own right
-        return self.backend.es.search(**body, **kwargs)
+    if use_new_elasticsearch_api:
+
+        def _backend_do_search(self, body, **kwargs):
+            # As of Elasticsearch 7.15, the 'body' parameter is deprecated; instead, the top-level
+            # keys of the body dict are now kwargs in their own right
+            return self.backend.es.search(**body, **kwargs)
 
 
 class Elasticsearch7AutocompleteQueryCompiler(
diff --git a/wagtail/search/backends/elasticsearch8.py b/wagtail/search/backends/elasticsearch8.py
--- a/wagtail/search/backends/elasticsearch8.py
+++ b/wagtail/search/backends/elasticsearch8.py
@@ -1,4 +1,5 @@
 from django.core.exceptions import ImproperlyConfigured
+from elasticsearch import NotFoundError
 
 from wagtail.search.backends.elasticsearch7 import (
     Elasticsearch7AutocompleteQueryCompiler,
@@ -8,6 +9,7 @@
     Elasticsearch7SearchQueryCompiler,
     Elasticsearch7SearchResults,
 )
+from wagtail.search.index import class_is_indexed
 
 
 class Elasticsearch8Mapping(Elasticsearch7Mapping):
@@ -15,6 +17,18 @@ class Elasticsearch8Mapping(Elasticsearch7Mapping):
 
 
 class Elasticsearch8Index(Elasticsearch7Index):
+    def put(self):
+        self.es.indices.create(index=self.name, **self.backend.settings)
+
+    def delete(self):
+        try:
+            self.es.indices.delete(index=self.name)
+        except NotFoundError:
+            pass
+
+    def refresh(self):
+        self.es.indices.refresh(index=self.name)
+
     def add_model(self, model):
         # Get mapping
         mapping = self.mapping_class(model)
@@ -22,13 +36,31 @@ def add_model(self, model):
         # Put mapping
         self.es.indices.put_mapping(index=self.name, **mapping.get_mapping())
 
+    def add_item(self, item):
+        # Make sure the object can be indexed
+        if not class_is_indexed(item.__class__):
+            return
+
+        # Get mapping
+        mapping = self.mapping_class(item.__class__)
+
+        # Add document to index
+        self.es.index(
+            index=self.name,
+            document=mapping.get_document(item),
+            id=mapping.get_document_id(item),
+        )
+
 
 class Elasticsearch8SearchQueryCompiler(Elasticsearch7SearchQueryCompiler):
     mapping_class = Elasticsearch8Mapping
 
 
 class Elasticsearch8SearchResults(Elasticsearch7SearchResults):
-    pass
+    def _backend_do_search(self, body, **kwargs):
+        # As of Elasticsearch 7.15, the 'body' parameter is deprecated; instead, the top-level
+        # keys of the body dict are now kwargs in their own right
+        return self.backend.es.search(**body, **kwargs)
 
 
 class Elasticsearch8AutocompleteQueryCompiler(Elasticsearch7AutocompleteQueryCompiler):


Test patch:
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -31,6 +31,7 @@ concurrency:
 # - elasticsearch 5, django 3.2, python 3.8, sqlite
 # - elasticsearch 6, django 3.2, python 3.8, postgres
 # - elasticsearch 7, django 4.1, python 3.8, postgres
+# - opensearch 2, django 4.1, python 3.9, sqlite
 # - elasticsearch 8, django 4.2, python 3.10, sqlite, USE_EMAIL_USER_MODEL=yes
 
 # Some tests are run in parallel by passing --parallel to runtests.py.
@@ -410,6 +411,51 @@ jobs:
           name: coverage-data
           path: .coverage.*
 
+  test-sqlite-opensearch2:
+    runs-on: ubuntu-latest
+    continue-on-error: ${{ matrix.experimental }}
+    strategy:
+      matrix:
+        include:
+          - python: '3.9'
+            django: 'Django>=4.1,<4.2'
+            experimental: false
+
+    steps:
+      - name: Configure sysctl limits
+        run: |
+          sudo swapoff -a
+          sudo sysctl -w vm.swappiness=1
+          sudo sysctl -w fs.file-max=262144
+          sudo sysctl -w vm.max_map_count=262144
+      - uses: ankane/setup-opensearch@v1
+        with:
+          opensearch-version: 2
+      - uses: actions/checkout@v3
+      - name: Set up Python ${{ matrix.python }}
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ matrix.python }}
+          cache: 'pip'
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e '.[testing]' --config-settings editable_mode=strict
+          pip install "${{ matrix.django }}"
+          pip install "elasticsearch==7.13.4"
+          pip install certifi
+      - name: Test
+        run: |
+          coverage run --parallel-mode --source wagtail runtests.py wagtail.search wagtail.documents wagtail.images --elasticsearch7
+        env:
+          DATABASE_ENGINE: django.db.backends.sqlite3
+          USE_EMAIL_USER_MODEL: ${{ matrix.emailuser }}
+      - name: Upload coverage data
+        uses: actions/upload-artifact@v3
+        with:
+          name: coverage-data
+          path: .coverage.*
+
   coverage:
     needs:
       - test-sqlite
@@ -419,6 +465,7 @@ jobs:
       - test-sqlite-elasticsearch8
       - test-postgres-elasticsearch6
       - test-postgres-elasticsearch7
+      - test-sqlite-opensearch2
     runs-on: ubuntu-latest
     steps:
       - name: Check out the repo
diff --git a/wagtail/search/tests/test_elasticsearch7_backend.py b/wagtail/search/tests/test_elasticsearch7_backend.py
--- a/wagtail/search/tests/test_elasticsearch7_backend.py
+++ b/wagtail/search/tests/test_elasticsearch7_backend.py
@@ -20,6 +20,16 @@
     ELASTICSEARCH_VERSION = (0, 0, 0)
 
 
+use_new_elasticsearch_api = ELASTICSEARCH_VERSION >= (7, 15)
+
+if use_new_elasticsearch_api:
+    search_query_kwargs = {
+        "query": "QUERY",
+    }
+else:
+    search_query_kwargs = {"body": {"query": "QUERY"}}
+
+
 @unittest.skipIf(ELASTICSEARCH_VERSION[0] != 7, "Elasticsearch 7 required")
 class TestElasticsearch7SearchBackend(ElasticsearchCommonSearchBackendTests, TestCase):
     backend_path = "wagtail.search.backends.elasticsearch7"
@@ -915,12 +925,12 @@ def test_basic_search(self, search):
         list(results)  # Performs search
 
         search.assert_any_call(
-            query="QUERY",
             _source=False,
             stored_fields="pk",
             index="wagtail__searchtests_book",
             scroll="2m",
             size=100,
+            **search_query_kwargs,
         )
 
     @mock.patch("elasticsearch.Elasticsearch.search")
@@ -933,11 +943,11 @@ def test_get_single_item(self, search):
 
         search.assert_any_call(
             from_=10,
-            query="QUERY",
             _source=False,
             stored_fields="pk",
             index="wagtail__searchtests_book",
             size=1,
+            **search_query_kwargs,
         )
 
     @mock.patch("elasticsearch.Elasticsearch.search")
@@ -949,11 +959,11 @@ def test_slice_results(self, search):
 
         search.assert_any_call(
             from_=1,
-            query="QUERY",
             _source=False,
             stored_fields="pk",
             index="wagtail__searchtests_book",
             size=3,
+            **search_query_kwargs,
         )
 
     @mock.patch("elasticsearch.Elasticsearch.search")
@@ -965,11 +975,11 @@ def test_slice_results_multiple_times(self, search):
 
         search.assert_any_call(
             from_=10,
-            query="QUERY",
             _source=False,
             stored_fields="pk",
             index="wagtail__searchtests_book",
             size=10,
+            **search_query_kwargs,
         )
 
     @mock.patch("elasticsearch.Elasticsearch.search")
@@ -982,11 +992,11 @@ def test_slice_results_and_get_item(self, search):
 
         search.assert_any_call(
             from_=20,
-            query="QUERY",
             _source=False,
             stored_fields="pk",
             index="wagtail__searchtests_book",
             size=1,
+            **search_query_kwargs,
         )
 
     @mock.patch("elasticsearch.Elasticsearch.search")