Repository: localstack/localstack

Issue:
Bug: [EDITED] java.lang.UnsatisfiedLinkError: no opensearchknn_faiss in java.library.path
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

I have a terraform-managed localstack setup that uses s3 and opensearch. 

In our docker-compose for localstack, we set the OPENSEARCH_ENDPOINT_STRATEGY to path, and when we use the aws cli for opensearch we can see the endpoint follows the expected path strategy. 

However, when we curl that endpoint, we get the following error:

```
$ curl http://localhost:4566/opensearch/us-east-1/example/
<?xml version='1.0' encoding='utf-8'?>
<Error><Code>InternalError</Code><Message>exception while calling s3 with unknown operation: MyHTTPConnectionPool(host='127.0.0.1', port=60769): Max retries exceeded with url: / (Caused by NewConnectionError('&lt;urllib3.connection.HTTPConnection object at 0x7efc3c205840&gt;: Failed to establish a new connection: [Errno 111] Connection refused'))</Message><RequestId>09b749b2-78a0-4a90-b175-9f672f14babc</RequestId></Error>
```
looks like localstack is routing to s3 instead of opensearch. Interestingly, this issue only started after I created an opensearch index and added a document to it -- before that, I was getting the expected opensearch curl results.

### Expected Behavior

I would expect the curl to hit the opensearch endpoint instead of s3

### How are you starting LocalStack?

With a docker-compose file

### Steps To Reproduce

I'm not fully certain what the minimal repro is. We're starting localstack with a docker compose, which I can include:

```
version: "3"
services:
  localstack-mongodb:
    image: mongo:4.4.21
    container_name: localstack-mongodb
    ports:
      - "27017:27017"
    command: mongod --replSet rs0

    # Enable interactive mode
    stdin_open: true

    # Allocate a pseudo-TTY
    tty: true

  localstack-redis:
    image: redis:7.0.11
    container_name: localstack-redis
    ports:
      - "6379:6379"

  localstack:
    image: localstack/localstack
    container_name: localstack
    environment:
     - OPENSEARCH_ENDPOINT_STRATEGY=path
    ports:
      - "4566:4566"
      - "4510-4559:4510-4559"
```

followed by:
opensearch.js 
```
opensearchClient.create({
  index: 'foo'.
  body: {
    settings: { 'index.knn': true },
    mappings: {
      embedding: {
        type: 'knn_vector',
        dimension: 512,
        method: { engine: 'faiss', name: 'nsw' }
      }
    }
  } 
})

client.index({ index: 'foo', body: { embedding: [ ... 512 numbers ] });
```


### Environment

```markdown
- OS: 20.04
- LocalStack: latest
```


### Anything else?

NA


Discussion:
This issue is maybe related (https://github.com/localstack/localstack/issues/8115) though im reasonably confident this isn't a localstack trying to download the binary either :thinking: I did notice that if I use a slightly different URL -- for example, a misspelling -- I get a 'No bucket found' error. It's only when I use the exact url for the opensearch cluster do I get a `exception while calling s3 with unknown operation: MyHTTPConnectionPool(host='127.0.0.1', port=60769): Max retries exceeded` error

In case it helps, this is the output of the aws describe cli:
```
$ aws opensearch describe-domain --domain-name localsoot --endpoint-url http://localhost:4566
{
    "DomainStatus": {
        "DomainId": "000000000000/example",
        "DomainName": "example",
        "ARN": "arn:aws:es:us-east-1:000000000000:domain/example",
        "Created": true,
        "Deleted": false,
        "Endpoint": "localhost:4566/opensearch/us-east-1/example",
        "Processing": false,
        "UpgradeProcessing": false,
        "EngineVersion": "OpenSearch_2.3",
        "ClusterConfig": {
            "InstanceType": "m3.medium.search",
            "InstanceCount": 1,
            "DedicatedMasterEnabled": true,
            "ZoneAwarenessEnabled": false,
            "DedicatedMasterType": "m3.medium.search",
            "DedicatedMasterCount": 1,
            "WarmEnabled": false,
            "ColdStorageOptions": {
                "Enabled": false
            }
        },
        "EBSOptions": {
            "EBSEnabled": true,
            "VolumeType": "gp2",
            "VolumeSize": 10,
            "Iops": 0
        },
        "AccessPolicies": "",
        "SnapshotOptions": {
            "AutomatedSnapshotStartHour": 0
        },
        "CognitoOptions": {
            "Enabled": false
        },
        "EncryptionAtRestOptions": {
            "Enabled": false
        },
        "NodeToNodeEncryptionOptions": {
            "Enabled": false
        },
        "AdvancedOptions": {
            "override_main_response_version": "false",
            "rest.action.multi.allow_explicit_index": "true"
        },
        "ServiceSoftwareOptions": {
            "CurrentVersion": "",
            "NewVersion": "",
            "UpdateAvailable": false,
            "Cancellable": false,
            "UpdateStatus": "COMPLETED",
            "Description": "There is no software update available for this domain.",
            "AutomatedUpdateDate": "1970-01-01T00:00:00+00:00",
            "OptionalDeployment": true
        },
        "DomainEndpointOptions": {
            "EnforceHTTPS": false,
            "TLSSecurityPolicy": "Policy-Min-TLS-1-0-2019-07",
            "CustomEndpointEnabled": false
        },
        "AdvancedSecurityOptions": {
            "Enabled": false,
            "InternalUserDatabaseEnabled": false
        },
        "AutoTuneOptions": {
            "State": "ENABLE_IN_PROGRESS"
        }
    }
}
```
Tried tearing down and bringing localstack back up -- again I can see the opensearch cluster endpoints. My guess is that opensearch is somehow going down and localstack is unable to bring it back up (why that would be happening, I'm not sure). 

When trying to hit this url:
```
http://localhost:4566/opensearch/us-east-1/example/_cat/indices?v
```
I correctly get
```
health status index           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   indexedinstance qBhoOLvDSIKdt5qy-FMOiQ   1   1          0            0       208b           208b
```
Later, after uploading a single file to s3 + adding a document to opensearch, the same url returns the following:
```
<Message>exception while calling s3 with unknown operation: Traceback (most recent call last): File "/opt/code/localstack/.venv/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn conn = connection.create_connection( File "/opt/code/localstack/.venv/lib/python3.10/site-packages/urllib3/util/connection.py", line 95, in create_connection raise err File "/opt/code/localstack/.venv/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/code/localstack/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 714, in urlopen httplib_response = self._make_request( File "/opt/code/localstack/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 415, in _make_request conn.request(method, url, **httplib_request_kw) File "/opt/code/localstack/.venv/lib/python3.10/site-packages/urllib3/connection.py", line 244, in request super(HTTPConnection, self).request(method, url, body=body, headers=headers) File "/usr/local/lib/python3.10/http/client.py", line 1283, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/local/lib/python3.10/http/client.py", line 1329, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/local/lib/python3.10/http/client.py", line 1278, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/local/lib/python3.10/http/client.py", line 1038, in _send_output self.send(msg) File "/usr/local/lib/python3.10/http/client.py", line 976, in send self.connect() File "/opt/code/localstack/.venv/lib/python3.10/site-packages/urllib3/connection.py", line 205, in connect conn = self._new_conn() File "/opt/code/localstack/.venv/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f2e1d9bf1c0>: Failed to establish a new connection: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/code/localstack/.venv/lib/python3.10/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( File "/opt/code/localstack/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 798, in urlopen retries = retries.increment( File "/opt/code/localstack/.venv/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: MyHTTPConnectionPool(host='127.0.0.1', port=58155): Max retries exceeded with url: /_cat/indices?v= (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f2e1d9bf1c0>: Failed to establish a new connection: [Errno 111] Connection refused')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/code/localstack/localstack/aws/chain.py", line 90, in handle handler(self, self.context, response) File "/opt/code/localstack/localstack/aws/handlers/routes.py", line 27, in __call__ router_response = self.router.dispatch(context.request) File "/opt/code/localstack/localstack/http/router.py", line 445, in dispatch return self.dispatcher(request, handler, args) File "/opt/code/localstack/localstack/http/dispatcher.py", line 70, in _dispatch result = endpoint(request, **args) File "/opt/code/localstack/localstack/http/proxy.py", line 125, in __call__ return self.proxy.forward(request, forward_path=kwargs.get("path", "")) File "/opt/code/localstack/localstack/http/proxy.py", line 90, in forward return self.client.request(proxy_request, server=f"{target.scheme}://{target.netloc}") File "/opt/code/localstack/localstack/http/client.py", line 133, in request response = self.session.request( File "/opt/code/localstack/.venv/lib/python3.10/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/opt/code/localstack/.venv/lib/python3.10/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/opt/code/localstack/.venv/lib/python3.10/site-packages/requests/adapters.py", line 519, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: MyHTTPConnectionPool(host='127.0.0.1', port=58155): Max retries exceeded with url: /_cat/indices?v= (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f2e1d9bf1c0>: Failed to establish a new connection: [Errno 111] Connection refused')) </Message>
```
this one seems to have a bunch of cascading python handling failures :thinking: 
ah ha. I set `DEBUG=1` in my localstack docker-compose and finally found the culprit:
```
2023-09-21T03:54:05.940  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] java.lang.UnsatisfiedLinkError: no o
pensearchknn_faiss in java.library.path: [/usr/lib/jvm/java-11/lib/server, /usr/lib/jvm/java-11/lib, /usr/lib/jvm/java-11/
../lib, /usr/lib/jvm/java-11/lib, /usr/lib/jvm/java-11/lib/server, /usr/java/packages/lib, /usr/lib/x86_64-linux-gnu/jni, 
/lib/x86_64-linux-gnu, /usr/lib/x86_64-linux-gnu, /usr/lib/jni, /lib, /usr/lib]
```
I'm not really sure what this error means -- I'm guessing maybe the knn library isn't actually present? :thinking: 
The full stack:
```
2023-09-21T03:54:05.952  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] java.lang.UnsatisfiedLinkError: no opensearchknn_faiss in java.library.path: [/usr/lib/jvm/java-11/lib/server, /usr/lib/jvm/java-11/lib, /usr/lib/jvm/java-11/../lib, /usr/lib/jvm/java-11/lib, /usr/lib/jvm/java-11/lib/server, /usr/java/packages/lib, /usr/lib/x86_64-linux-gnu/jni, /lib/x86_64-linux-gnu, /usr/lib/x86_64-linux-gnu, /usr/lib/jni, /lib, /usr/lib]
2023-09-21T03:54:05.952  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at java.base/java.lang.ClassLoader.loadLibrary(Unknown Source)                                                                                 
2023-09-21T03:54:05.952  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at java.base/java.lang.Runtime.loadLibrary0(Unknown Source)                                                                                    
2023-09-21T03:54:05.953  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at java.base/java.lang.System.loadLibrary(Unknown Source)                                                                                      
2023-09-21T03:54:05.953  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.opensearch.knn.jni.FaissService.lambda$static$0(FaissService.java:34)                                                                   
2023-09-21T03:54:05.953  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at java.base/java.security.AccessController.doPrivileged(Native Method)                                                                        
2023-09-21T03:54:05.953  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.opensearch.knn.jni.FaissService.<clinit>(FaissService.java:33)                                                                          
2023-09-21T03:54:05.953  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.opensearch.knn.jni.JNIService.createIndex(JNIService.java:40)                                                                           
2023-09-21T03:54:05.953  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.opensearch.knn.index.codec.KNN80Codec.KNN80DocValuesConsumer.lambda$createKNNIndexFromScratch$4(KNN80DocValuesConsumer.java:190)                                                                                                                                                                                                                                             
2023-09-21T03:54:05.954  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at java.base/java.security.AccessController.doPrivileged(Native Method)                                                                        
2023-09-21T03:54:05.954  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.opensearch.knn.index.codec.KNN80Codec.KNN80DocValuesConsumer.createKNNIndexFromScratch(KNN80DocValuesConsumer.java:189)                                                                                                                                                                                                                                                      
2023-09-21T03:54:05.954  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.opensearch.knn.index.codec.KNN80Codec.KNN80DocValuesConsumer.lambda$addKNNBinaryField$2(KNN80DocValuesConsumer.java:134)                                                                                                                                                                                                                                                     
2023-09-21T03:54:05.954  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.opensearch.knn.index.codec.KNN80Codec.KNN80DocValuesConsumer.addKNNBinaryField(KNN80DocValuesConsumer.java:141)                                                                                                                                                                                                                                                              
2023-09-21T03:54:05.954  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.opensearch.knn.index.codec.KNN80Codec.KNN80DocValuesConsumer.addBinaryField(KNN80DocValuesConsumer.java:78)                                                                                                                                                                                                                                                                  
2023-09-21T03:54:05.954  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.apache.lucene.index.BinaryDocValuesWriter.flush(BinaryDocValuesWriter.java:132)                                                         
2023-09-21T03:54:05.955  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.apache.lucene.index.IndexingChain.writeDocValues(IndexingChain.java:389)                                                                
2023-09-21T03:54:05.955  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.apache.lucene.index.IndexingChain.flush(IndexingChain.java:253)                                                                         
2023-09-21T03:54:05.955  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:391)                                                   
2023-09-21T03:54:05.955  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:493)                                                                   
2023-09-21T03:54:05.955  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:672)                                                           
2023-09-21T03:54:05.955  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:569)                                                                         
2023-09-21T03:54:05.955  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381)                                                                                                                                                                                                                                                                               
2023-09-21T03:54:05.956  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355)                                                                                                                                                                                                                                                                                
2023-09-21T03:54:05.956  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345)                                                                                                                                                                                                                                                                                
2023-09-21T03:54:05.956  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112)                                               
2023-09-21T03:54:05.956  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170)                                                             
2023-09-21T03:54:05.956  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:73)                                                                                                                                                                                                                                                                             
2023-09-21T03:54:05.956  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:53)                                                                                                                                                                                                                                                                             
2023-09-21T03:54:05.956  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167)                                                         
2023-09-21T03:54:05.956  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240)                                                   
2023-09-21T03:54:05.957  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:430)                                                                                                                                                                                                                                                                        
2023-09-21T03:54:05.957  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:410)                                                                                                                                                                                                                                                                        
2023-09-21T03:54:05.957  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167)                                                         
2023-09-21T03:54:05.957  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:213)                                                           
2023-09-21T03:54:05.957  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1843)                                                                
2023-09-21T03:54:05.957  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.opensearch.index.engine.InternalEngine.maybeRefresh(InternalEngine.java:1822)                                                           
2023-09-21T03:54:05.957  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.opensearch.index.shard.IndexShard.scheduledRefresh(IndexShard.java:3822)                                                                
2023-09-21T03:54:05.957  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.opensearch.index.IndexService.maybeRefreshEngine(IndexService.java:968)                                                                 
2023-09-21T03:54:05.958  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.opensearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:1101)                                                      
2023-09-21T03:54:05.958  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.opensearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:159)                                                     
2023-09-21T03:54:05.958  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747)                                                                                                                                                                                                                                                                        
2023-09-21T03:54:05.958  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)                                                                 
2023-09-21T03:54:05.958  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)                                                                
2023-09-21T03:54:05.958  INFO --- [uncthread264] l.s.opensearch.cluster     : [44547] at java.base/java.lang.Thread.run(Unknown Source)     ```
Interestingly, running `http://localhost:4566/opensearch/us-east-1/example/_cat/plugins?v` shows the knn plugin is indeed installed and present by default. 

I'm sorta out of ideas :cry: 
I got it working! 

This bug on the knn repo (https://github.com/opensearch-project/k-NN/issues/508) pointed the direction.

TLDR: the LD_LIBRARY_PATH isnt being set correctly in localstack-managed opensearch environments. When I exec into the dockercontainer, the default LD_LIBRARY_PATH is set to `/usr/lib/jvm/java-11/lib:/usr/lib/jvm/java-11/lib/server`, but the relevant opensearch libs are over at `/var/lib/localstack/lib/opensearch/OpenSearch_2.5/plugins/opensearch-knn/lib`

When I explicitly set `LD_LIBRARY_PATH=/var/lib/localstack/lib/opensearch/OpenSearch_2.5/plugins/opensearch-knn/lib` in my docker compose environment, it all works as expected

I'd like to not have to set that library path manually, but that at least is the issue. It looks like the LD_LIBRARY_PATH is in fact set bv the opensearch docker image and in their tar install script (see: https://github.com/search?q=repo%3Aopensearch-project%2Fopensearch-build%20LD_LIBRARY_PATH&type=code) but I'm not sure how localstack in particular manages opensearch
Thanks a lot for the detailed writeup! I can see the cause of your issue, but I am having a hard time reproducing it. Creating a knn-index as well as adding a document to the index seems to be working fine on my end...
Could you maybe provide a minimum reproducible sample with which I could trigger the `UnsatisfiedLinkError`?
Here's what I tried:
```
# create the cluster
awslocal opensearch create-domain --domain-name example --engine-version OpenSearch_2.5
# wait for processing = false
awslocal opensearch describe-domain --domain-name example
# create the index
curl -X PUT example.eu-west-1.opensearch.localhost.localstack.cloud:4566/knn -H "Content-Type: application/json" --data '{"settings": {"index": {"knn": true}}, "mappings": {"properties": {"embedding": {"type": "knn_vector", "dimension": 512}}}}'
# add a document
curl -X PUT example.eu-west-1.opensearch.localhost.localstack.cloud:4566/knn/_doc/test -H "Content-Type: application/json" --data '{"Some": "JSON"}'
# fetch the document
curl -X GET example.eu-west-1.opensearch.localhost.localstack.cloud:4566/knn/_doc/test
```

Thanks!
I'll try and get a minimum repro -- in the meantime, can you try actually adding an embedding to your document? as written, the document you are adding is not going to be indexed by the knn library, so its possible the missing library isn't called

Resolving patch:
diff --git a/localstack/services/opensearch/cluster.py b/localstack/services/opensearch/cluster.py
--- a/localstack/services/opensearch/cluster.py
+++ b/localstack/services/opensearch/cluster.py
@@ -461,11 +461,22 @@ def _create_run_command(
         return cmd
 
     def _create_env_vars(self, directories: Directories) -> Dict:
-        return {
+        env_vars = {
             "OPENSEARCH_JAVA_OPTS": os.environ.get("OPENSEARCH_JAVA_OPTS", "-Xms200m -Xmx600m"),
             "OPENSEARCH_TMPDIR": directories.tmp,
         }
 
+        # if the "opensearch-knn" plugin exists and has a "lib" directory, add it to the LD_LIBRARY_PATH
+        # see https://forum.opensearch.org/t/issue-with-opensearch-knn/12633
+        knn_lib_dir = os.path.join(directories.install, "plugins", "opensearch-knn", "lib")
+        if os.path.isdir(knn_lib_dir):
+            prefix = (
+                f"{os.environ.get('LD_LIBRARY_PATH')}:" if "LD_LIBRARY_PATH" in os.environ else ""
+            )
+            env_vars["LD_LIBRARY_PATH"] = prefix + f"{knn_lib_dir}"
+
+        return env_vars
+
     def _log_listener(self, line, **_kwargs):
         # logging the port before each line to be able to connect logs to specific instances
         LOG.info("[%s] %s", self.port, line.rstrip())


Test patch:
diff --git a/tests/aws/services/opensearch/test_opensearch.py b/tests/aws/services/opensearch/test_opensearch.py
--- a/tests/aws/services/opensearch/test_opensearch.py
+++ b/tests/aws/services/opensearch/test_opensearch.py
@@ -563,6 +563,33 @@ def test_create_indices(self, opensearch_endpoint):
             assert req_result[0]["health"] in ["green", "yellow"]
             assert req_result[0]["index"] in indices
 
+        # create a knn index to make sure the knn plugin works
+        index_path = f"{opensearch_endpoint}/knn"
+        body = {
+            "settings": {"index": {"knn": True}},
+            "mappings": {
+                "properties": {
+                    "embedding": {
+                        "type": "knn_vector",
+                        "dimension": 2,
+                    }
+                }
+            },
+        }
+        put = requests.put(index_path, headers=COMMON_HEADERS, json=body)
+        assert put.status_code == 200
+        get = requests.get(f"{opensearch_endpoint}/_cat/indices/knn?format=json&pretty")
+        assert get.status_code == 200
+
+        # add a document
+        document_path = f"{opensearch_endpoint}/knn/_doc/test_document"
+        body = {"embedding": [1, 2]}
+        put = requests.put(document_path, headers=COMMON_HEADERS, json=body)
+        assert put.status_code == 201
+
+        get = requests.get(document_path)
+        assert get.status_code == 200
+
     @markers.aws.unknown
     def test_get_document(self, opensearch_document_path):
         response = requests.get(opensearch_document_path)