Repository: localstack/localstack
Pull request: ElasticSearch: Fix missing JRE in 5.x and 6.x
## Background

In https://github.com/localstack/localstack/pull/11462, we changed the ElasticSearch/OpenSearch installers to use the [bundled JRE](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html#jvm-version). However, based on https://github.com/localstack/localstack/issues/12231, it was identified that ElasticSearch 5.x and 6.x builds [do not](https://www.elastic.co/guide/en/elasticsearch/reference/6.0/zip-targz.html) have a bundled JRE and instead require a separate JRE installation.

## Changes

This PR introduces a specialised package installer for ElasticSearch 5.x and 6.x which manages the Java dependency through standardised approach.

## Tests

Tested manually locally. On the CI, automated tests run only against the latest versions, which was also why this went undetected.

## Related

Closes: https://github.com/localstack/localstack/issues/12231

Problem statement:
bug: Elastic Search 6.8 - JAVA_HOME doesn't exist
### Is there an existing issue for this?

- [x] I have searched the existing issues

### Current Behavior

When creating a new elastic search 6.8 domain within the localstack container like so:

```
awslocal es create-elasticsearch-domain --domain-name my-domain-name --elasticsearch-version 6.8
```

You get a response suggesting the domain is processing:

```
{
    "DomainStatus": {
        "DomainId": "000000000000/my-domain-name",
        "DomainName": "my-domain-name",
        "ARN": "arn:aws:es:us-east-1:000000000000:domain/my-domain-name",
        "Created": true,
        "Deleted": false,
        "Endpoint": "localstack:4566/es/us-east-1/my-domain-name",
        "Processing": true,
        "UpgradeProcessing": false,
        "ElasticsearchVersion": "6.8",
        "ElasticsearchClusterConfig": {
            "InstanceType": "m3.medium.elasticsearch",
            "InstanceCount": 1,
            "DedicatedMasterEnabled": true,
            "ZoneAwarenessEnabled": false,
            "DedicatedMasterType": "m3.medium.elasticsearch",
            "DedicatedMasterCount": 1,
            "WarmEnabled": false,
            "ColdStorageOptions": {
                "Enabled": false
            }
        },
        "EBSOptions": {
            "EBSEnabled": true,
            "VolumeType": "gp2",
            "VolumeSize": 10,
            "Iops": 0
        },
        "AccessPolicies": "",
        "SnapshotOptions": {
            "AutomatedSnapshotStartHour": 0
        },
        "CognitoOptions": {
            "Enabled": false
        },
        "EncryptionAtRestOptions": {
            "Enabled": false
        },
        "NodeToNodeEncryptionOptions": {
            "Enabled": false
        },
        "AdvancedOptions": {
            "override_main_response_version": "false",
            "rest.action.multi.allow_explicit_index": "true"
        },
        "ServiceSoftwareOptions": {
            "CurrentVersion": "",
            "NewVersion": "",
            "UpdateAvailable": false,
            "Cancellable": false,
            "UpdateStatus": "COMPLETED",
            "Description": "There is no software update available for this domain.",
            "AutomatedUpdateDate": 0.0,
            "OptionalDeployment": true
        },
        "DomainEndpointOptions": {
            "EnforceHTTPS": false,
            "TLSSecurityPolicy": "Policy-Min-TLS-1-0-2019-07",
            "CustomEndpointEnabled": false
        },
        "AdvancedSecurityOptions": {
            "Enabled": false,
            "InternalUserDatabaseEnabled": false
        },
        "AutoTuneOptions": {
            "State": "ENABLE_IN_PROGRESS"
        }
    }
}
```

But if you examine the image logs, you can see that `JAVA_HOME` is being set to a path that does not exist `/var/lib/localstack/lib/elasticsearch/Elasticsearch_6.8/jdk`:

```
2025-02-05T15:18:40.656  INFO --- [-functhread3] l.s.opensearch.cluster     : starting elasticsearch: su localstack -c '/var/lib/localstack/lib/elasticsearch/Elasticsearch_6.8/bin/elasticsearch -E http.port=38241 -E http.publish_port=38241 -E transport.port=0 -E network.host=127.0.0.1 -E http.compression=false -E path.data="/tmp/localstack/state/elasticsearch/arn:aws:es:us-east-1:000000000000:domain/my-domain-name/data" -E path.repo="/tmp/localstack/state/elasticsearch/arn:aws:es:us-east-1:000000000000:domain/my-domain-name/backup" -E discovery.type=single-node -E xpack.ml.enabled=false' with env {'JAVA_HOME': '/var/lib/localstack/lib/elasticsearch/Elasticsearch_6.8/jdk', 'ES_JAVA_OPTS': '-Xms200m -Xmx600m', 'ES_TMPDIR': '/tmp/localstack/state/elasticsearch/arn:aws:es:us-east-1:000000000000:domain/my-domain-name/tmp'}
2025-02-05T15:18:40.665  INFO --- [-functhread4] l.s.opensearch.cluster     : [38241] warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME
2025-02-05T15:18:40.665  INFO --- [-functhread4] l.s.opensearch.cluster     : [38241] could not find java; set JAVA_HOME
```

```
root@localstack:/var/lib/localstack/lib/elasticsearch/Elasticsearch_6.8# ls -al
total 448
drwxrwxrwx 12 root root    384 Jan 23 17:00 .
drwxr-xr-x  5 root root    160 Jan 23 17:00 ..
-rwxrwxrwx  1 root root  13675 Oct  7  2021 LICENSE.txt
-rwxrwxrwx  1 root root 427502 Oct  7  2021 NOTICE.txt
-rwxrwxrwx  1 root root   8534 Oct  7  2021 README.textile
drwxrwxrwx 45 root root   1440 Oct  7  2021 bin
drwxrwxrwx 11 root root    352 Jan 23 17:03 config
drwxrwxrwx  2 root root     64 Jan 23 17:00 data
drwxrwxrwx 42 root root   1344 Oct  7  2021 lib
drwxrwxrwx 13 root root    416 Feb  5 15:23 logs
drwxrwxrwx 31 root root    992 Oct  7  2021 modules
drwxrwxrwx  2 root root     64 Oct  7  2021 plugins
```

I can get it to work by manually installing Java 8 within the image but I would rather not do that as it sets the default JAVA_HOME to an old version and may break things unexpectedly as we roll out localstack updates going forward.

Can `JAVA_HOME` be added to the Elasticsearch_6.8 package?

### Expected Behavior

Java should be present within `/var/lib/localstack/lib/elasticsearch/Elasticsearch_6.8/jdk` 

### How are you starting LocalStack?

With a docker-compose file

### Steps To Reproduce

#### How are you starting localstack (e.g., `bin/localstack` command, arguments, or `docker-compose.yml`)

    docker-compose

#### Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

    awslocal es create-elasticsearch-domain --domain-name my-domain-name --elasticsearch-version 6.8

### Environment

```markdown
- OS:
- LocalStack:
  LocalStack version: 4 and 4.1.1
  LocalStack Docker image sha: ca0685948310
  LocalStack build date:
  LocalStack build git hash:
```

### Anything else?

ElasticSearch releases before 7.0 do not have a bundled JDK. LocalStack assumes that all releases have bundled JDK which causes 6.x and 5.x to fail. To fix this, the ES packages should be updated to invoke the Java installer for these versions and pass the correct `JAVA_HOME` for all invocations.

Hints from issue discussion:
Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our [Slack Community](https://localstack.cloud/contact/) for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our [contributing guidelines](https://github.com/localstack/.github/blob/main/CONTRIBUTING.md) and our [contributing guide](https://docs.localstack.cloud/contributing/).
Hi all, just wondering if there is any update on this? Is there a plan to fix this issue?

Commit message: Fix critical startup config for older versions

Patch to review:
diff --git a/localstack-core/localstack/services/opensearch/cluster.py b/localstack-core/localstack/services/opensearch/cluster.py
index 65ef2dd9c4ed5..c8a1fecbc4b03 100644
--- a/localstack-core/localstack/services/opensearch/cluster.py
+++ b/localstack-core/localstack/services/opensearch/cluster.py
@@ -16,6 +16,7 @@
 )
 from localstack.http.client import SimpleRequestsClient
 from localstack.http.proxy import ProxyHandler
+from localstack.packages import InstallTarget
 from localstack.services.edge import ROUTER
 from localstack.services.opensearch import versions
 from localstack.services.opensearch.packages import elasticsearch_package, opensearch_package
@@ -669,20 +670,31 @@ def os_user(self):
         return constants.OS_USER_OPENSEARCH
 
     def _ensure_installed(self):
-        elasticsearch_package.install(self.version)
+        elasticsearch_package.install(self.version, target=InstallTarget.VAR_LIBS)
 
     def _base_settings(self, dirs) -> CommandSettings:
         settings = {
             "http.port": self.port,
             "http.publish_port": self.port,
-            "transport.port": "0",
             "network.host": self.host,
             "http.compression": "false",
             "path.data": f'"{dirs.data}"',
             "path.repo": f'"{dirs.backup}"',
-            "discovery.type": "single-node",
         }
 
+        # This config option was renamed between 6.7 and 6.8, yet not documented as a breaking change
+        # See https://github.com/elastic/elasticsearch/blob/f220abaf/build-tools/src/main/java/org/elasticsearch/gradle/testclusters/ElasticsearchNode.java#L1349-L1353
+        if self.version.startswith("Elasticsearch_5.") or (
+            self.version.startswith("Elasticsearch_6.") and self.version != "Elasticsearch_6.8"
+        ):
+            settings["transport.tcp.port"] = "0"
+        else:
+            settings["transport.port"] = "0"
+
+        # `discovery.type` had a different meaning in 5.x
+        if not self.version.startswith("Elasticsearch_5."):
+            settings["discovery.type"] = "single-node"
+
         if os.path.exists(os.path.join(dirs.mods, "x-pack-ml")):
             settings["xpack.ml.enabled"] = "false"
 
@@ -690,7 +702,9 @@ def _base_settings(self, dirs) -> CommandSettings:
 
     def _create_env_vars(self, directories: Directories) -> Dict:
         return {
-            "JAVA_HOME": os.path.join(directories.install, "jdk"),
+            **elasticsearch_package.get_installer(self.version).get_java_env_vars(
+                InstallTarget.VAR_LIBS
+            ),
             "ES_JAVA_OPTS": os.environ.get("ES_JAVA_OPTS", "-Xms200m -Xmx600m"),
             "ES_TMPDIR": directories.tmp,
         }
diff --git a/localstack-core/localstack/services/opensearch/packages.py b/localstack-core/localstack/services/opensearch/packages.py
index 4610420f330ee..b3dc49151ef9a 100644
--- a/localstack-core/localstack/services/opensearch/packages.py
+++ b/localstack-core/localstack/services/opensearch/packages.py
@@ -18,6 +18,7 @@
     OPENSEARCH_PLUGIN_LIST,
 )
 from localstack.packages import InstallTarget, Package, PackageInstaller
+from localstack.packages.java import java_package
 from localstack.services.opensearch import versions
 from localstack.utils.archives import download_and_extract_with_retry
 from localstack.utils.files import chmod_r, load_file, mkdir, rm_rf, save_file
@@ -42,6 +43,8 @@ def __init__(self, default_version: str = OPENSEARCH_DEFAULT_VERSION):
 
     def _get_installer(self, version: str) -> PackageInstaller:
         if version in versions._prefixed_elasticsearch_install_versions:
+            if version.startswith("Elasticsearch_5.") or version.startswith("Elasticsearch_6."):
+                return ElasticsearchLegacyPackageInstaller(version)
             return ElasticsearchPackageInstaller(version)
         else:
             return OpensearchPackageInstaller(version)
@@ -233,6 +236,12 @@ class ElasticsearchPackageInstaller(PackageInstaller):
     def __init__(self, version: str):
         super().__init__("elasticsearch", version)
 
+    def get_java_env_vars(self, target: InstallTarget) -> dict[str, str]:
+        install_dir = self._get_install_dir(target)
+        return {
+            "JAVA_HOME": os.path.join(install_dir, "jdk"),
+        }
+
     def _install(self, target: InstallTarget):
         # locally import to avoid having a dependency on ASF when starting the CLI
         from localstack.aws.api.opensearch import EngineType
@@ -263,7 +272,7 @@ def _install(self, target: InstallTarget):
                 **java_system_properties_proxy(),
                 **java_system_properties_ssl(
                     os.path.join(install_dir, "jdk", "bin", "keytool"),
-                    {"JAVA_HOME": os.path.join(install_dir, "jdk")},
+                    self.get_java_env_vars(target),
                 ),
             }
             java_opts = system_properties_to_cli_args(sys_props)
@@ -336,5 +345,25 @@ def get_elasticsearch_install_version(self) -> str:
         return versions.get_install_version(self.version)
 
 
+class ElasticsearchLegacyPackageInstaller(ElasticsearchPackageInstaller):
+    """
+    Specialised package installer for ElasticSearch 5.x and 6.x since these releases do not have a bundled JDK.
+
+    This installer ensures that Java is installed as part of the setup.
+    """
+
+    # ES 5.x and 6.x require Java 8
+    # See: https://www.elastic.co/guide/en/elasticsearch/reference/6.0/zip-targz.html
+    JAVA_VERSION = "8"
+
+    def _prepare_installation(self, target: InstallTarget) -> None:
+        java_package.get_installer(self.JAVA_VERSION).install(target)
+
+    def get_java_env_vars(self, target: InstallTarget) -> dict[str, str]:
+        return {
+            "JAVA_HOME": java_package.get_installer(self.JAVA_VERSION).get_java_home(),
+        }
+
+
 opensearch_package = OpensearchPackage(default_version=OPENSEARCH_DEFAULT_VERSION)
 elasticsearch_package = OpensearchPackage(default_version=ELASTICSEARCH_DEFAULT_VERSION)


Code review comments:
diff_hunk: @@ -233,6 +236,12 @@ class ElasticsearchPackageInstaller(PackageInstaller):
     def __init__(self, version: str):
         super().__init__("elasticsearch", version)
 
+    def get_java_env_vars(self, target: InstallTarget) -> dict[str, str]:
+        install_dir = self._get_install_dir(target)
path: localstack-core/localstack/services/opensearch/packages.py
text: @user1:
**issue:** This is the root of the issue outlined in the comment above. Here you really shouldn't use the target where the package is "about to be installed if it's not yet installed", but you should use [`get_installed_dir`](https://github.com/localstack/localstack/blob/3026d5cb618c4d64d6be18ff0aa7619954a2143b/localstack-core/localstack/packages/api.py#L116-L125) instead, which dynamically detects the installation directory where the package _has already been_ installed.‏

@author:
Brilliant suggestion, this will simplify everything. Fixed in 7eb88688c