Repository: localstack/localstack

Issue:
bug: pod save/restore of elastic search domain fails to restart ES (2.0.1dev)
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

Create an ElasticSearch domain.   Run pod save.   Restart LocalStack.   Run pod restore.   Domain still exists in config, but the ES software isn't downloaded/installed/started.

### Expected Behavior

That it restarts ES.   This worked in 1.14

### How are you starting LocalStack?

With the `localstack` script

### Steps To Reproduce

localstack start -d 
awslocal es create-elasticsearch-domain --elasticsearch-version 7.7 --domain-name audit
localstack pod save file://$PWD/estest.pod
localstack stop
localstack start
localstack pod load file://$PWD/estest.pod


### Environment

```markdown
- OS: Mac 13.2.1
- LocalStack: 2.0.1dev
```


### Anything else?
Note the pod save/restore does preserve your ES domains, it just doesn't start/install ES.

No interesting messages from the localstack console

```
 💻 LocalStack CLI 2.0.0.post1

[15:22:37] starting LocalStack in Docker mode 🐳                                                                                                                                    localstack.py:142
2023-04-06T15:22:37.327  WARN --- [  MainThread] localstack_ext.plugins     : Unable to start DNS: cannot import name 'dns_server' from 'localstack_ext.services' (/opt/homebrew/lib/python3.11/site-packages/localstack_ext/services/__init__.py)
─────────────────────────────────────────────────────────────────────────── LocalStack Runtime Log (press CTRL-C to quit) ───────────────────────────────────────────────────────────────────────────
2023-04-06T15:22:37.413  WARN --- [  MainThread] localstack_ext.plugins     : failed to configure DNS: cannot import name 'dns_server' from 'localstack_ext.services' (/opt/homebrew/lib/python3.11/site-packages/localstack_ext/services/__init__.py)

LocalStack version: 2.0.1.dev
LocalStack Docker container id: 77863cef09a0
LocalStack build date: 2023-04-05
LocalStack build git hash: ae1dfbf9

2023-04-06T22:22:38.738  WARN --- [  MainThread] localstack.deprecations    : DEFAULT_REGION is deprecated (since 0.12.7) and will be removed in upcoming releases of LocalStack! LocalStack now has full multi-region support. Please remove this environment variable.
2023-04-06T22:22:38.825  WARN --- [-functhread3] hypercorn.error            : ASGI Framework Lifespan error, continuing without Lifespan support
2023-04-06T22:22:38.825  WARN --- [-functhread3] hypercorn.error            : ASGI Framework Lifespan error, continuing without Lifespan support
2023-04-06T22:22:38.827  INFO --- [-functhread3] hypercorn.error            : Running on https://0.0.0.0:4566 (CTRL + C to quit)
2023-04-06T22:22:38.827  INFO --- [-functhread3] hypercorn.error            : Running on https://0.0.0.0:4566 (CTRL + C to quit)
Ready.
2023-04-06T22:23:11.871  INFO --- [   asgi_gw_0] localstack.request.http    : GET / => 200
2023-04-06T22:23:11.882  INFO --- [   asgi_gw_0] localstack.request.http    : GET /_localstack/pods/environment => 200
2023-04-06T22:23:12.034  INFO --- [   asgi_gw_0] localstack.request.http    : POST /_localstack/pods => 201

```


Discussion:
Found a work-around.   Since awslocal is installed in the localstack container, you can add the following two lines to a ready.d script.   

awslocal es create-elasticsearch-domain --elasticsearch-version 7.7 --domain-name dummy
awslocal es delete-elasticsearch-domain --domain-name dummy
Actually, while all the "awslocal es *" commands continue to work, the ES domain is unreachable and localstack routes to S3.   The fix is to delete and recreate your domain.  :-(
FWIW: Upgrading to OpenSearch has the same problem.
Hi, ive managed to solve this issue with this workaround (ready.d script). After recreation of ES domain data from pod still persists, and resolving starts to work again.

```sh
#!/bin/bash

set -euo pipefail

FILE="/var/lib/localstack/pods/main"

if [[ -f "$FILE" && "${CUSTOM_PERSISTENCE:-0}" == 1 ]]; then
  echo "Loading pod"
  /usr/local/bin/localstack pod load file://${FILE}
else
  echo "Pod not found"
fi

echo "Eager loading of services s3 and es..."

# Needed only to start s3 service, because pod load does not start it
awslocal s3 ls

# Needed only to start elasticsearch service, because pod load does not start it
awslocal es delete-elasticsearch-domain --domain-name <domain-you-saved-in-pod>
awslocal es create-elasticsearch-domain --elasticsearch-version 7.10.0 --domain-name <domain-you-saved-in-pod>
```
Very nice.  I knew there was a simple way to fix it, but I suspected fear of delaying enterprise purchases reduces the motivation to publish solutions like this.

Resolving patch:
diff --git a/localstack/services/opensearch/provider.py b/localstack/services/opensearch/provider.py
--- a/localstack/services/opensearch/provider.py
+++ b/localstack/services/opensearch/provider.py
@@ -93,6 +93,7 @@
     create_cluster_manager,
 )
 from localstack.services.opensearch.models import OpenSearchStore, opensearch_stores
+from localstack.services.plugins import ServiceLifecycleHook
 from localstack.state import AssetDirectory, StateVisitor
 from localstack.utils.aws.arns import parse_arn
 from localstack.utils.collections import PaginatedList, remove_none_values_from_dict
@@ -399,7 +400,7 @@ def is_valid_domain_name(name: str) -> bool:
     return True if _domain_name_pattern.match(name) else False
 
 
-class OpensearchProvider(OpensearchApi):
+class OpensearchProvider(OpensearchApi, ServiceLifecycleHook):
     @staticmethod
     def get_store(account_id: str, region_name: str) -> OpenSearchStore:
         return opensearch_stores[account_id][region_name]


Test patch: