Repository: opensearch-project/opensearch-py
Pull request #388 (2023-05-15 20:14:10)

Cleaned up the developer guide.
### Description

Took me some time to figure out how to run tests. 
A small cleanup of the dev guide. 

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).


codecov[bot]: ## [Codecov](https://app.codecov.io/gh/opensearch-project/opensearch-py/pull/388?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project) Report
> Merging [#388](https://app.codecov.io/gh/opensearch-project/opensearch-py/pull/388?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project) (a1052dd) into [main](https://app.codecov.io/gh/opensearch-project/opensearch-py/commit/707373e369418e6ce2e43c59fa33af206bd2de40?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project) (707373e) will **decrease** coverage by `0.07%`.
> The diff coverage is `n/a`.

```diff
@@            Coverage Diff             @@
##             main     #388      +/-   ##
==========================================
- Coverage   72.11%   72.04%   -0.07%     
==========================================
  Files          77       77              
  Lines        7190     7190              
==========================================
- Hits         5185     5180       -5     
- Misses       2005     2010       +5     
```


[see 1 file with indirect coverage changes](https://app.codecov.io/gh/opensearch-project/opensearch-py/pull/388/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project)



Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4efd51d3..2ca569f1 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,8 +8,10 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Added index lifecycle guide ([#362](https://github.com/opensearch-project/opensearch-py/pull/362))
 - Added 'point in time' APIs to the pyi files in sync and async client ([#378](https://github.com/opensearch-project/opensearch-py/pull/378))
 - Added MacOS and Windows CI workflows ([#390](https://github.com/opensearch-project/opensearch-py/pull/390))
+- Compatibility with OpenSearch 2.1.0 - 2.6.0 ([#381](https://github.com/opensearch-project/opensearch-py/pull/381))
 ### Changed
 - Upgrading pytest-asyncio to latest version - 0.21.0 ([#339](https://github.com/opensearch-project/opensearch-py/pull/339))
+- Fixed flaky CI tests by replacing httpbin with a simple http_server ([#395](https://github.com/opensearch-project/opensearch-py/pull/395))
 ### Deprecated
 ### Removed
 ### Fixed
@@ -71,6 +73,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Updates `master` to `cluster_manager` to be inclusive ([#242](https://github.com/opensearch-project/opensearch-py/pull/242))
 - Support a custom signing service name for AWS SigV4 ([#268](https://github.com/opensearch-project/opensearch-py/pull/268))
 - Updated CI tests to make them work locally ([#275](https://github.com/opensearch-project/opensearch-py/pull/275))
+- Fix bug with validation of 'timeout' parameter ([#387](Do not escape the "timeout" parameter.))
 ### Deprecated
 
 ### Removed
@@ -84,4 +87,5 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 [2.0.1]: https://github.com/opensearch-project/opensearch-py/compare/v2.0.0...v2.0.1
 [2.1.0]: https://github.com/opensearch-project/opensearch-py/compare/v2.0.1...v2.1.0
 [2.1.1]: https://github.com/opensearch-project/opensearch-py/compare/v2.1.0...v2.1.1
-[2.2.0]: https://github.com/opensearch-project/opensearch-py/compare/v2.1.1...v2.2.0
\ No newline at end of file
+[2.2.0]: https://github.com/opensearch-project/opensearch-py/compare/v2.1.1...v2.2.0
+
diff --git a/COMPATIBILITY.md b/COMPATIBILITY.md
index 59d8529b..bae1cafb 100644
--- a/COMPATIBILITY.md
+++ b/COMPATIBILITY.md
@@ -9,8 +9,9 @@ The below matrix shows the compatibility of the [`opensearch-py`](https://pypi.o
 | --- | --- | --- |
 | 1.0.0 | 1.0.0-1.2.4 | |
 | 1.1.0 | 1.3.0-1.3.7 | |
-| 2.0.0 | 1.0.0-2.4.1 | client works against Opensearch Version 1.x as long as features removed in 2.0 are not used |
-| 2.0.1 | 1.0.0-2.4.1 | client works against Opensearch Version 1.x as long as features removed in 2.0 are not used |
+| 2.0.x | 1.0.0-2.6.0 | client works against Opensearch Version 1.x as long as features removed in 2.0 are not used |
+| 2.1.x | 1.0.0-2.6.0 | client works against Opensearch Version 1.x as long as features removed in 2.0 are not used |
+| 2.2.0 | 1.0.0-2.6.0 | client works against Opensearch Version 1.x as long as features removed in 2.0 are not used |
 
 ## Upgrading
 
diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md
index 1d8c1c3d..54df76dc 100644
--- a/DEVELOPER_GUIDE.md
+++ b/DEVELOPER_GUIDE.md
@@ -1,43 +1,74 @@
 - [Developer Guide](#developer-guide)
-  - [Linter and Test Suite](#linter-and-test-suite)
-  - [Integration Tests](#integration-tests)
-  - [Build the Documentation with Sphinx](#build-the-documentation-with-sphinx)
+  - [Prerequisites](#prerequisites)
+  - [Running Tests](#running-tests)
+  - [Linter](#linter)
+  - [Documentation](#documentation)
 
 # Developer Guide
 
-## Linter and Test Suite 
-Run the linter and test suite to ensure your changes do not break existing code:
+## Prerequisites
+
+Python 3.6 or newer is required.
+
+```
+$ python --version
+Python 3.11.1
+```
+
+Install [Nox](https://nox.thea.codes/en/stable/) for task management.
 
 ```
-# Install Nox for task management
 $ python -m pip install nox
+```
 
-# Auto-format and lint your changes
-$ nox -rs format
+## Running Tests
 
-# Run the test suite
+Tests require a live instance of OpenSearch running in docker.
+
+This will start a new instance and run tests against the latest version of OpenSearch.
+
+```
+./.ci/run-tests
+```
+
+If your OpenSearch docker instance is running, you can execute the test suite directly.
+
+```
 $ nox -rs test
 ```
 
-## Integration Tests
-To run the integration tests locally, run:
+To run tests against different versions of OpenSearch, use `run-tests [with/without security] [version]`:
 
 ```
 ./.ci/run-tests true 1.3.0
 ```
 
-The first argument tells whether to run server with security plugin enabled or not.
-The second argument specifies the version of OpenSearch the tests should run against, if not specified, the tests run against the latest version. 
+The first argument tells whether to run server with security plugin enabled or not. The second argument specifies the version of OpenSearch the tests should run against, if not specified, the tests run against the latest version. You can also run tests against a current SNAPSHOT.
+
+The following example runs tests against the latest SNAPSHOT build of OpenSearch without security.
+
+```
+./.ci/run-tests opensearch false SNAPSHOT
+```
 
 Note that integration tests require docker to be installed and running, and downloads quite a bit of data from over the internet and hence take few minutes to complete.
 
-## Build the Documentation with Sphinx
-This are the steps to build the documentation with [Sphinx](https://www.sphinx-doc.org/):
+## Linter
+
+Run the linter and test suite to ensure your changes do not break existing code. The following will auto-format your changes.
+
+```
+$ nox -rs format
+```
+
+## Documentation
+
+To build the documentation with [Sphinx](https://www.sphinx-doc.org/).
+
+```
+pip install -e .[docs]
+cd docs
+make html
+```
 
-1. change into the `opensearch-py` directory where `setup.py` is located
-2. install opensearch-py - we recommend [editable mode](https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-e
-    - with bash: `pip install -e .[docs]`
-    - with zsh: `pip install -e .\[docs]`
-4. change into the `docs` directory
-5. execute `make html`
-6. use your favorite web browser to open the file called `opensearch-py/docs/build/html/index.html`
+Open `opensearch-py/docs/build/html/index.html` to see results.
diff --git a/opensearchpy/client/utils.py b/opensearchpy/client/utils.py
index e79b6e9e..a5e99b11 100644
--- a/opensearchpy/client/utils.py
+++ b/opensearchpy/client/utils.py
@@ -165,16 +165,17 @@ def query_params(*opensearch_query_params):
             elif api_key is not None:
                 headers["authorization"] = "ApiKey %s" % (_base64_auth_header(api_key),)
 
+            # don't escape ignore, request_timeout, or timeout
+            for p in ("ignore", "request_timeout", "timeout"):
+                if p in kwargs:
+                    params[p] = kwargs.pop(p)
+
             for p in opensearch_query_params + GLOBAL_PARAMS:
                 if p in kwargs:
                     v = kwargs.pop(p)
                     if v is not None:
                         params[p] = _escape(v)
 
-            # don't treat ignore, request_timeout, and opaque_id as other params to avoid escaping
-            for p in ("ignore", "request_timeout"):
-                if p in kwargs:
-                    params[p] = kwargs.pop(p)
             return func(*args, params=params, headers=headers, **kwargs)
 
         return _wrapped


Test patch:
diff --git a/test_opensearchpy/TestHttpServer.py b/test_opensearchpy/TestHttpServer.py
new file mode 100644
index 00000000..4c35f621
--- /dev/null
+++ b/test_opensearchpy/TestHttpServer.py
@@ -0,0 +1,51 @@
+# SPDX-License-Identifier: Apache-2.0
+#
+# The OpenSearch Contributors require contributions made to
+# this file be licensed under the Apache-2.0 license or a
+# compatible open source license.
+#
+# Modifications Copyright OpenSearch Contributors. See
+# GitHub history for details.
+
+import json
+import threading
+from http.server import BaseHTTPRequestHandler, HTTPServer
+
+
+class TestHTTPRequestHandler(BaseHTTPRequestHandler):
+    def do_GET(self):
+        self.send_response(200)
+        headers = self.headers
+        self.send_header("Content-type", "application/json")
+        self.end_headers()
+
+        Headers = {}
+        for header, value in headers.items():
+            capitalized_header = "-".join([word.title() for word in header.split("-")])
+            Headers.update({capitalized_header: value})
+        if "Connection" in Headers:
+            Headers.pop("Connection")
+
+        data = {"method": "GET", "headers": Headers}
+        self.wfile.write(json.dumps(data).encode("utf-8"))
+
+
+class TestHTTPServer(HTTPServer):
+    def __init__(self, host="localhost", port=8080):
+        super().__init__((host, port), TestHTTPRequestHandler)
+        self._server_thread = None
+
+    def start(self):
+        if self._server_thread is not None:
+            return
+
+        self._server_thread = threading.Thread(target=self.serve_forever)
+        self._server_thread.start()
+
+    def stop(self):
+        if self._server_thread is None:
+            return
+        self.socket.close()
+        self.shutdown()
+        self._server_thread.join()
+        self._server_thread = None
diff --git a/test_opensearchpy/test_async/test_connection.py b/test_opensearchpy/test_async/test_connection.py
index 1b8cdc58..0cda23d1 100644
--- a/test_opensearchpy/test_async/test_connection.py
+++ b/test_opensearchpy/test_async/test_connection.py
@@ -43,6 +43,7 @@ from opensearchpy import AIOHttpConnection, AsyncOpenSearch, __versionstr__, ser
 from opensearchpy.compat import reraise_exceptions
 from opensearchpy.connection import Connection, async_connections
 from opensearchpy.exceptions import ConnectionError, TransportError
+from test_opensearchpy.TestHttpServer import TestHTTPServer
 
 pytestmark = pytest.mark.asyncio
 
@@ -319,72 +320,80 @@ class TestAIOHttpConnection:
             await con.close()
 
 
-class TestConnectionHttpbin:
+class TestConnectionHttpServer:
     """Tests the HTTP connection implementations against a live server E2E"""
 
-    async def httpbin_anything(self, conn, **kwargs):
-        status, headers, data = await conn.perform_request("GET", "/anything", **kwargs)
+    @classmethod
+    def setup_class(cls):
+        # Start server
+        cls.server = TestHTTPServer(port=8081)
+        cls.server.start()
+
+    @classmethod
+    def teardown_class(cls):
+        # Stop server
+        cls.server.stop()
+
+    async def httpserver(self, conn, **kwargs):
+        status, headers, data = await conn.perform_request("GET", "/", **kwargs)
         data = json.loads(data)
-        data["headers"].pop(
-            "X-Amzn-Trace-Id", None
-        )  # Remove this header as it's put there by AWS.
         return (status, data)
 
     async def test_aiohttp_connection(self):
         # Defaults
-        conn = AIOHttpConnection("httpbin.org", port=443, use_ssl=True)
+        conn = AIOHttpConnection("localhost", port=8081, use_ssl=False)
         user_agent = conn._get_default_user_agent()
-        status, data = await self.httpbin_anything(conn)
+        status, data = await self.httpserver(conn)
         assert status == 200
         assert data["method"] == "GET"
         assert data["headers"] == {
             "Content-Type": "application/json",
-            "Host": "httpbin.org",
+            "Host": "localhost:8081",
             "User-Agent": user_agent,
         }
 
         # http_compress=False
         conn = AIOHttpConnection(
-            "httpbin.org", port=443, use_ssl=True, http_compress=False
+            "localhost", port=8081, use_ssl=False, http_compress=False
         )
-        status, data = await self.httpbin_anything(conn)
+        status, data = await self.httpserver(conn)
         assert status == 200
         assert data["method"] == "GET"
         assert data["headers"] == {
             "Content-Type": "application/json",
-            "Host": "httpbin.org",
+            "Host": "localhost:8081",
             "User-Agent": user_agent,
         }
 
         # http_compress=True
         conn = AIOHttpConnection(
-            "httpbin.org", port=443, use_ssl=True, http_compress=True
+            "localhost", port=8081, use_ssl=False, http_compress=True
         )
-        status, data = await self.httpbin_anything(conn)
+        status, data = await self.httpserver(conn)
         assert status == 200
         assert data["headers"] == {
             "Accept-Encoding": "gzip,deflate",
             "Content-Type": "application/json",
-            "Host": "httpbin.org",
+            "Host": "localhost:8081",
             "User-Agent": user_agent,
         }
 
         # Headers
         conn = AIOHttpConnection(
-            "httpbin.org",
-            port=443,
-            use_ssl=True,
+            "localhost",
+            port=8081,
+            use_ssl=False,
             http_compress=True,
             headers={"header1": "value1"},
         )
-        status, data = await self.httpbin_anything(
+        status, data = await self.httpserver(
             conn, headers={"header2": "value2", "header1": "override!"}
         )
         assert status == 200
         assert data["headers"] == {
             "Accept-Encoding": "gzip,deflate",
             "Content-Type": "application/json",
-            "Host": "httpbin.org",
+            "Host": "localhost:8081",
             "Header1": "override!",
             "Header2": "value2",
             "User-Agent": user_agent,
diff --git a/test_opensearchpy/test_client/test_utils.py b/test_opensearchpy/test_client/test_utils.py
index 2b5e5a39..1a4b6809 100644
--- a/test_opensearchpy/test_client/test_utils.py
+++ b/test_opensearchpy/test_client/test_utils.py
@@ -85,6 +85,31 @@ class TestQueryParams(TestCase):
         self.func_to_wrap(headers={"X": "y"})
         self.assertEqual(self.calls[-1], ((), {"params": {}, "headers": {"x": "y"}}))
 
+    def test_non_escaping_params(self):
+        # the query_params decorator doesn't validate "timeout" it simply avoids escaping as it did
+        self.func_to_wrap(simple_param="x", timeout="4s")
+        self.assertEqual(
+            self.calls[-1],
+            ((), {"params": {"simple_param": b"x", "timeout": "4s"}, "headers": {}}),
+        )
+
+        self.func_to_wrap(simple_param="x", timeout=4, ignore=5, request_timeout=6)
+        self.assertEqual(
+            self.calls[-1],
+            (
+                (),
+                {
+                    "params": {
+                        "simple_param": b"x",
+                        "timeout": 4,
+                        "ignore": 5,
+                        "request_timeout": 6,
+                    },
+                    "headers": {},
+                },
+            ),
+        )
+
     def test_per_call_authentication(self):
         self.func_to_wrap(api_key=("name", "key"))
         self.assertEqual(
diff --git a/test_opensearchpy/test_connection.py b/test_opensearchpy/test_connection.py
index 3ed1cf0b..8d501ffe 100644
--- a/test_opensearchpy/test_connection.py
+++ b/test_opensearchpy/test_connection.py
@@ -72,6 +72,9 @@ from pytest import raises
 from opensearchpy import OpenSearch, serializer
 from opensearchpy.connection import connections
 
+if sys.version_info > (3, 0):
+    from test_opensearchpy.TestHttpServer import TestHTTPServer
+
 
 def gzip_decompress(data):
     buf = gzip.GzipFile(fileobj=io.BytesIO(data), mode="rb")
@@ -857,76 +860,87 @@ class TestRequestsConnection(TestCase):
         assert str(e.value) == "Wasn't modified!"
 
 
-class TestConnectionHttpbin:
+@pytest.mark.skipif(
+    sys.version_info < (3, 0),
+    reason="http_server is only available from python 3.x",
+)
+class TestConnectionHttpServer:
     """Tests the HTTP connection implementations against a live server E2E"""
 
-    def httpbin_anything(self, conn, **kwargs):
-        status, headers, data = conn.perform_request("GET", "/anything", **kwargs)
+    @classmethod
+    def setup_class(cls):
+        # Start server
+        cls.server = TestHTTPServer(port=8080)
+        cls.server.start()
+
+    @classmethod
+    def teardown_class(cls):
+        # Stop server
+        cls.server.stop()
+
+    def httpserver(self, conn, **kwargs):
+        status, headers, data = conn.perform_request("GET", "/", **kwargs)
         data = json.loads(data)
-        data["headers"].pop(
-            "X-Amzn-Trace-Id", None
-        )  # Remove this header as it's put there by AWS.
         return (status, data)
 
     def test_urllib3_connection(self):
         # Defaults
-        # httpbin.org can be slow sometimes. Hence the timeout
-        conn = Urllib3HttpConnection("httpbin.org", port=443, use_ssl=True, timeout=60)
+        conn = Urllib3HttpConnection("localhost", port=8080, use_ssl=False, timeout=60)
         user_agent = conn._get_default_user_agent()
-        status, data = self.httpbin_anything(conn)
+        status, data = self.httpserver(conn)
         assert status == 200
         assert data["method"] == "GET"
         assert data["headers"] == {
             "Accept-Encoding": "identity",
             "Content-Type": "application/json",
-            "Host": "httpbin.org",
+            "Host": "localhost:8080",
             "User-Agent": user_agent,
         }
 
         # http_compress=False
         conn = Urllib3HttpConnection(
-            "httpbin.org", port=443, use_ssl=True, http_compress=False, timeout=60
+            "localhost", port=8080, use_ssl=False, http_compress=False, timeout=60
         )
-        status, data = self.httpbin_anything(conn)
+        status, data = self.httpserver(conn)
         assert status == 200
         assert data["method"] == "GET"
         assert data["headers"] == {
             "Accept-Encoding": "identity",
             "Content-Type": "application/json",
-            "Host": "httpbin.org",
+            "Host": "localhost:8080",
             "User-Agent": user_agent,
         }
 
         # http_compress=True
         conn = Urllib3HttpConnection(
-            "httpbin.org", port=443, use_ssl=True, http_compress=True, timeout=60
+            "localhost", port=8080, use_ssl=False, http_compress=True, timeout=60
         )
-        status, data = self.httpbin_anything(conn)
+        status, data = self.httpserver(conn)
         assert status == 200
         assert data["headers"] == {
             "Accept-Encoding": "gzip,deflate",
             "Content-Type": "application/json",
-            "Host": "httpbin.org",
+            "Host": "localhost:8080",
             "User-Agent": user_agent,
         }
 
         # Headers
         conn = Urllib3HttpConnection(
-            "httpbin.org",
-            port=443,
-            use_ssl=True,
+            "localhost",
+            port=8080,
+            use_ssl=False,
             http_compress=True,
             headers={"header1": "value1"},
             timeout=60,
         )
-        status, data = self.httpbin_anything(
+        status, data = self.httpserver(
             conn, headers={"header2": "value2", "header1": "override!"}
         )
         assert status == 200
         assert data["headers"] == {
             "Accept-Encoding": "gzip,deflate",
             "Content-Type": "application/json",
-            "Host": "httpbin.org",
+            "Host": "localhost:8080",
             "Header1": "override!",
             "Header2": "value2",
             "User-Agent": user_agent,
@@ -939,62 +953,62 @@ class TestConnectionHttpbin:
 
     def test_requests_connection(self):
         # Defaults
-        conn = RequestsHttpConnection("httpbin.org", port=443, use_ssl=True, timeout=60)
+        conn = RequestsHttpConnection("localhost", port=8080, use_ssl=False, timeout=60)
         user_agent = conn._get_default_user_agent()
-        status, data = self.httpbin_anything(conn)
+        status, data = self.httpserver(conn)
         assert status == 200
         assert data["method"] == "GET"
         assert data["headers"] == {
             "Accept-Encoding": "identity",
             "Content-Type": "application/json",
-            "Host": "httpbin.org",
+            "Host": "localhost:8080",
             "User-Agent": user_agent,
         }
 
         # http_compress=False
         conn = RequestsHttpConnection(
-            "httpbin.org", port=443, use_ssl=True, http_compress=False, timeout=60
+            "localhost", port=8080, use_ssl=False, http_compress=False, timeout=60
         )
-        status, data = self.httpbin_anything(conn)
+        status, data = self.httpserver(conn)
         assert status == 200
         assert data["method"] == "GET"
         assert data["headers"] == {
             "Accept-Encoding": "identity",
             "Content-Type": "application/json",
-            "Host": "httpbin.org",
+            "Host": "localhost:8080",
             "User-Agent": user_agent,
         }
 
         # http_compress=True
         conn = RequestsHttpConnection(
-            "httpbin.org", port=443, use_ssl=True, http_compress=True, timeout=60
+            "localhost", port=8080, use_ssl=False, http_compress=True, timeout=60
         )
-        status, data = self.httpbin_anything(conn)
+        status, data = self.httpserver(conn)
         assert status == 200
         assert data["headers"] == {
             "Accept-Encoding": "gzip,deflate",
             "Content-Type": "application/json",
-            "Host": "httpbin.org",
+            "Host": "localhost:8080",
             "User-Agent": user_agent,
         }
 
         # Headers
         conn = RequestsHttpConnection(
-            "httpbin.org",
-            port=443,
-            use_ssl=True,
+            "localhost",
+            port=8080,
+            use_ssl=False,
             http_compress=True,
             headers={"header1": "value1"},
             timeout=60,
         )
-        status, data = self.httpbin_anything(
+        status, data = self.httpserver(
             conn, headers={"header2": "value2", "header1": "override!"}
         )
         assert status == 200
         assert data["headers"] == {
             "Accept-Encoding": "gzip,deflate",
             "Content-Type": "application/json",
-            "Host": "httpbin.org",
+            "Host": "localhost:8080",
             "Header1": "override!",
             "Header2": "value2",
             "User-Agent": user_agent,