Repository: redpanda-data/redpanda
Pull request #25929: Functional tests for Iceberg on cloud AWS + Unity Catalog
Author: login: rpdevmp
Created: 2025-04-25T03:07:12Z | Merged: 2025-05-13T17:26:09Z

Functional tests for Iceberg on cloud AWS + Unity Catalog

- Reads Iceberg specific params and secrets from globals.json (new params were added by DEVPROD-2888) those would be used for local setups and by Buildkite schedules

- Uses public APIs that would be used by customers

- APIs for secret management on dataplane (new feature) @alenkacz FYI

@nvartolomei
- Includes a light-weight Databricks client to work with hosted setup I see that you merged your CDT PR several hours ago.. In your PR it is CDT specific (storage and a few other things not cloud ready), but some items are overlapping as we discussed, so we could consolidate those later to work for CDT and cloud. I would like to merge this client as it was already tested many times for cloud setups

- Uses S3 bucket object storage (AWS provider)
- Sets everything up and produces data to topics
 
DRAFT since
Need to add verification of object storage and queries and cleanups

Testing was done many times, these functions work and including error handling

## Backports Required

- [X ] none - not a bug fix
- [ ] none - this is a backport
- [ ] none - issue does not exist in previous branches
- [ ] none - papercut/not impactful enough to backport
- [ ] v25.1.x
- [ ] v24.3.x
- [ ] v24.2.x
- [ ] v24.1.x

## Release Notes
* none




Diff:
diff --git a/tests/rptest/redpanda_cloud_tests/iceberg_catalogs_test.py b/tests/rptest/redpanda_cloud_tests/iceberg_catalogs_test.py
new file mode 100644
index 0000000000000..2f45e0d95bd58
--- /dev/null
+++ b/tests/rptest/redpanda_cloud_tests/iceberg_catalogs_test.py
@@ -0,0 +1,156 @@
+# Copyright 2025 Redpanda Data, Inc.
+#
+# Use of this software is governed by the Business Source License
+# included in the file licenses/BSL.md
+#
+# As of the Change Date specified in that file, in accordance with
+# the Business Source License, use of this software will be governed
+# by the Apache License, Version 2.0
+
+from typing import Any
+import logging
+import json
+import time
+import requests
+
+from ducktape.mark import matrix
+from rptest.tests.redpanda_cloud_test import RedpandaCloudTest
+from rptest.services.provider_clients.rpcloud_client import RpCloudApiClient
+
+from rptest.clients.installpack import InstallPackClient
+from rptest.clients.rpk import RpkTool, TopicSpec, RpkException
+
+from rptest.services.redpanda import get_cloud_provider
+from rptest.services.cluster import cluster
+from rptest.services.redpanda import RedpandaServiceCloud
+
+from rptest.services.databricks_workspace import DatabricksWorkspace
+from rptest.context.databricks import DatabricksContext, OauthCredentials
+from rptest.services.catalog_service import CatalogType
+#from rptest.tests.datalake.datalake_services import DatalakeServices
+#from rptest.tests.datalake.query_engine_base import QueryEngineType
+#from rptest.tests.datalake.utils import supported_storage_types
+#from rptest.tests.redpanda_test import RedpandaTest
+#from rptest.utils.mode_checks import cleanup_on_early_exit
+
+
+def supported_catalog_types():
+    return ["aws_glue", "databricks_unity", "snowflake"]
+
+
+def supported_network_types():
+    return ["public", "private"]
+
+
+class IcebergCloudCatalogsTest(RedpandaCloudTest):
+    """
+    Verify that cluster infra/config matches config profile used to launch - only applies to cloudv2
+    """
+    def __init__(self, test_context):
+        super().__init__(test_context=test_context)
+        self._ctx = test_context
+        self._ipClient = InstallPackClient(
+            self.redpanda._cloud_cluster.config.install_pack_url_template,
+            self.redpanda._cloud_cluster.config.install_pack_auth_type,
+            self.redpanda._cloud_cluster.config.install_pack_auth)
+
+    def setUp(self):
+        super().setUp()
+        cloud_cluster = self.redpanda._cloud_cluster
+        self.logger.debug(f"Cloud Cluster Info: {vars(cloud_cluster)}")
+        install_pack_version = cloud_cluster.get_install_pack_version()
+        self._ip = self._ipClient.getInstallPack(install_pack_version)
+        self._clusterId = cloud_cluster.cluster_id
+        self._configProfile = self._ip['config_profiles'][
+            cloud_cluster.config.config_profile_name]
+
+    def test_healthy(self):
+        r = self.redpanda.cluster_unhealthy_reason()
+        assert r is None, r
+        assert self.redpanda.cluster_healthy()
+        self.redpanda.assert_cluster_is_reusable()
+
+    @cluster(num_nodes=1)
+    def test_databricks_basic(self):
+        dbx_ctx = DatabricksContext.from_context(self._ctx)
+
+        cloud_cluster = self.redpanda._cloud_cluster
+        client: RpCloudApiClient = cloud_cluster.public_api
+
+        databricks_client = DatabricksWorkspace(context=self._ctx)
+        bucket = f"redpanda-cloud-storage-{self._clusterId}"
+        catalog_info = databricks_client.create_catalog(bucket=bucket)
+        properties = {"iceberg_enabled": True}
+        enable_resp = cloud_cluster.update_cluster_property_public(
+            self._clusterId, properties)
+        self.logger.debug(f"Enable iceberg response: {enable_resp}")
+
+        # Parameters for creating Redpanda secret
+        secret_id = "UNITY_CLIENT_SECRET"
+
+        #secret_data = dbx_ctx.client_secret
+        # Access credentials (client_secret, client_id) from dbx_ctx.credentials
+        if isinstance(dbx_ctx.credentials, OauthCredentials):
+            secret_data = dbx_ctx.credentials.client_secret
+            databricks_client_id = dbx_ctx.credentials.client_id
+        else:
+            secret_data = dbx_ctx.credentials.token
+            databricks_client_id = None
+
+        create_resp = cloud_cluster.create_secret(secret_id, secret_data)
+        self.logger.debug(f"Create secret response: {create_resp}")
+
+        iceberg_rest_catalog_endpoint = dbx_ctx.iceberg_rest_url
+        iceberg_rest_catalog_warehouse = dbx_ctx.sql_warehouse_path
+
+        # Construct the payload for the request
+        properties = {
+            "iceberg_rest_catalog_endpoint":
+            iceberg_rest_catalog_endpoint,
+            "iceberg_rest_catalog_authentication_mode":
+            "oauth2"
+            if isinstance(dbx_ctx.credentials, OauthCredentials) else "bearer",
+            "iceberg_rest_catalog_client_id":
+            databricks_client_id,
+            "iceberg_rest_catalog_client_secret":
+            "${secrets.UNITY_CLIENT_SECRET}",
+            "iceberg_rest_catalog_warehouse":
+            catalog_info.name,
+            "iceberg_catalog_type":
+            "rest"
+        }
+
+        # Log the constructed payload for debugging
+        self.logger.debug(f"Properties to be sent: {properties}")
+
+        try:
+            # Send the HTTP PATCH request
+            response = client.update_cluster_property_public(
+                cluster_id=cloud_cluster.current.cluster_id,
+                properties=properties)
+
+            if response:
+                self.logger.debug(f"Update successful: {response.json()}")
+            else:
+                self.logger.error("Failed to update cluster properties.")
+
+        except Exception as e:
+            self.logger.error(
+                f"An error occurred while updating cluster properties: {e}")
+
+        self.rpk = RpkTool(self.redpanda)
+        test_topic = 'test_topic'
+        self.rpk.create_topic(test_topic)
+        self.rpk.alter_topic_config(test_topic,
+                                    TopicSpec.PROPERTY_ICEBERG_MODE,
+                                    'key_value')
+
+        MESSAGE_COUNT = 100
+        for i in range(MESSAGE_COUNT):
+            self.rpk.produce(test_topic, f"foo {i} ", f"bar {i}")
+
+        self.logger.debug("Waiting 1 minute...")
+        time.sleep(60)
+        # TODO Marat add verification, more tests and options (separate PR)
+
+        databricks_client.stop()
diff --git a/tests/rptest/services/databricks_workspace.py b/tests/rptest/services/databricks_workspace.py
index 5f415cc513beb..6623a69bc7c1b 100644
--- a/tests/rptest/services/databricks_workspace.py
+++ b/tests/rptest/services/databricks_workspace.py
@@ -62,13 +62,18 @@ def create_catalog(self, bucket: str) -> "DatabricksCatalogInfo":
 
         self._location_names.add(bucket)
 
-        location: ExternalLocationInfo = self._client.external_locations.create(
-            name=bucket,
-            # TODO: Add support for gcs, azure
-            url=f"s3://{bucket}",
-            credential_name=self._databricks_context.ext_loc_credential_name,
-        )
-        self.logger.debug(f"Created external location: {location}")
+        try:
+            location: ExternalLocationInfo = self._client.external_locations.create(
+                name=bucket,
+                # TODO: Add support for gcs, azure
+                url=f"s3://{bucket}",
+                credential_name=self._databricks_context.
+                ext_loc_credential_name,
+            )
+            self.logger.debug(f"Created external location: {location}")
+        except databricks.sdk.errors.DatabricksError as e:
+            self.logger.error(f"Failed to create external location: {str(e)}")
+            raise
 
         requested_catalog_name = f"panda-catalog-{uuid.uuid1()}"
         self._catalog_names.add(requested_catalog_name)
@@ -83,12 +88,18 @@ def create_catalog(self, bucket: str) -> "DatabricksCatalogInfo":
             "We expect to only managed catalogs."
         assert catalog_info.name, "Catalog name must not be empty"
 
-        sql_connection = databricks.sql.connect(
-            server_hostname=self._databricks_context.server_hostname,
-            http_path=self._databricks_context.sql_warehouse_path,
-            catalog=catalog_info.name,
-            credentials_provider=self._databricks_context.credentials_provider,
-        )
+        try:
+            sql_connection = databricks.sql.connect(
+                server_hostname=self._databricks_context.server_hostname,
+                http_path=self._databricks_context.sql_warehouse_path,
+                catalog=catalog_info.name,
+                credentials_provider=self._databricks_context.
+                credentials_provider,
+            )
+            self.logger.debug("SQL connection established successfully.")
+        except Exception as e:
+            self.logger.error(f"Error establishing SQL connection: {e}")
+            raise
 
         # This is a unity catalog peculiarity. It allows schemas (iceberg
         # namespaces) to be created but tables inside it are not allowed
diff --git a/tests/rptest/services/provider_clients/rpcloud_client.py b/tests/rptest/services/provider_clients/rpcloud_client.py
index efde4eedf8e15..68a62ae9fb07c 100644
--- a/tests/rptest/services/provider_clients/rpcloud_client.py
+++ b/tests/rptest/services/provider_clients/rpcloud_client.py
@@ -95,6 +95,7 @@ def _http_post(self,
                    override_headers={},
                    **kwargs):
         token = self._get_token()
+        self._logger.debug(f"http token: '{token}'")
         headers = {
             'Authorization': f'Bearer {token}',
             'Accept': 'application/json'
@@ -104,6 +105,26 @@ def _http_post(self,
         _r = self._handle_error(resp)
         return _r if _r is None else _r.json()
 
+    def _http_patch(self,
+                    base_url: str | None = None,
+                    endpoint: str = '',
+                    override_headers: dict[str, str] = {},
+                    **kwargs):
+        """
+            Like _http_post but uses PATCH.
+            Injects Bearer token and Accept header the same way.
+        """
+        token = self._get_token()
+        headers = {
+            'Authorization': f'Bearer {token}',
+            'Accept': 'application/json'
+        } | override_headers
+
+        _base = base_url if base_url else self._config.api_url
+        resp = requests.patch(f'{_base}{endpoint}', headers=headers, **kwargs)
+        _r = self._handle_error(resp)
+        return _r if _r is None else _r.json()
+
     def _http_delete(self, base_url=None, endpoint='', **kwargs):
         token = self._get_token()
         headers = {
diff --git a/tests/rptest/services/redpanda_cloud.py b/tests/rptest/services/redpanda_cloud.py
index 7c7b2030cea50..204c1898775e1 100644
--- a/tests/rptest/services/redpanda_cloud.py
+++ b/tests/rptest/services/redpanda_cloud.py
@@ -10,7 +10,7 @@
 from dataclasses import dataclass, field
 from datetime import datetime
 from enum import Enum
-from typing import Any, Optional
+from typing import Any, Optional, Dict
 from prometheus_client.parser import text_string_to_metric_families
 
 from ducktape.utils.util import wait_until
@@ -1548,6 +1548,7 @@ def scale_cluster(self, nodes_count):
                                        endpoint='/ScaleCluster',
                                        json=payload)
 
+    # Update cluster properties using Admin API
     def set_cluster_config_overrides(self, cluster_id, config_values):
         """
         Set configuration overrides for a specific Redpanda cloud cluster using Admin API
@@ -1562,3 +1563,58 @@ def set_cluster_config_overrides(self, cluster_id, config_values):
         return self.cloudv2._http_post(base_url=self.config.admin_api_url,
                                        endpoint='/SetClusterConfigOverrides',
                                        json=payload)
+
+    # Update cluster properties using Public API
+    def update_cluster_property_public(self, cluster_id: str,
+                                       properties: Dict[str, str]) -> dict:
+        """
+        Update a specific cluster property via the public API
+
+        :param cluster_id: Redpanda cluster ID
+        :param properties: A dictionary where each key is a property key and the value is the property value
+        :return: response from the API request
+        """
+        self._logger.debug(
+            f"Updating cluster {cluster_id} with properties: {properties}")
+
+        payload = {"cluster_configuration": {"custom_properties": properties}}
+
+        self._logger.debug(f"Payload to be sent: {payload}")
+
+        try:
+            update_resp = self.public_api._http_patch(
+                base_url=self.config.public_api_url,
+                endpoint=f"/v1/clusters/{cluster_id}",
+                json=payload)
+
+            self._logger.debug(f"Response from API: {update_resp}")
+
+            return update_resp
+        except Exception as e:
+            self._logger.error(
+                f"Error during request to update cluster {cluster_id}: {e}")
+            raise
+
+    def create_secret(self, secret_id, secret_data, scopes=None):
+        """
+        Create a new secret on dataplane and return the response.
+        :param secret_id: ID of the secret to create
+        :param secret_data: Data for the secret
+        :param scopes: List of scopes for the secret (default is ["SCOPE_REDPANDA_CLUSTER"])
+        :return: response from the API request
+        """
+        dataplane_url = self._get_dataplane_api_url()
+        if scopes is None:
+            scopes = ["SCOPE_REDPANDA_CLUSTER"]
+        response = self.public_api._http_post(base_url=dataplane_url,
+                                              endpoint=f"/v1/secrets",
+                                              json={
+                                                  "id": secret_id,
+                                                  "scopes": scopes,
+                                                  "secret_data": secret_data,
+                                              })
+        return response
+
+    def _get_dataplane_api_url(self):
+        cluster = self.rpcloud.get_cluster(self.current.cluster_id)
+        return cluster['dataplane_api']['url']


Reviews:
user: login: nvartolomei
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T09:09:45Z
user: login: nvartolomei
body: requested changes privately. the test is broken
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T15:00:01Z
user: login: nvartolomei
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-10T18:47:39Z
user: login: nvartolomei
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-10T18:48:17Z
user: login: nvartolomei
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-10T18:50:33Z
user: login: nvartolomei
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T11:40:43Z

Review comments:
diff_hunk: @@ -0,0 +1,134 @@
+# Copyright 2025 Redpanda Data, Inc.
+#
+# Use of this software is governed by the Business Source License
+# included in the file licenses/BSL.md
+#
+# As of the Change Date specified in that file, in accordance with
+# the Business Source License, use of this software will be governed
+# by the Apache License, Version 2.0
+
+from typing import Any
+import logging
+import json
+import time
+import requests
+
+from ducktape.mark import matrix
+from rptest.tests.redpanda_cloud_test import RedpandaCloudTest
+from rptest.services.provider_clients.rpcloud_client import RpCloudApiClient
+
+from rptest.clients.installpack import InstallPackClient
+from rptest.clients.rpk import RpkTool, TopicSpec, RpkException
+
+from rptest.services.redpanda import get_cloud_provider
+from rptest.services.cluster import cluster
+from rptest.services.redpanda import RedpandaServiceCloud
+
+from rptest.services.databricks_workspace import DatabricksWorkspace
+from rptest.context.databricks import DatabricksContext as DatabricksContext
+from rptest.services.catalog_service import CatalogType
+#from rptest.tests.datalake.datalake_services import DatalakeServices
+#from rptest.tests.datalake.query_engine_base import QueryEngineType
+#from rptest.tests.datalake.utils import supported_storage_types
+#from rptest.tests.redpanda_test import RedpandaTest
+#from rptest.utils.mode_checks import cleanup_on_early_exit
+
+
+def supported_catalog_types():
+    return ["aws_glue", "databricks_unity", "snowflake"]
+
+
+def supported_network_types():
+    return ["public", "private"]
+
+
+class IcebergCloudCatalogsTest(RedpandaCloudTest):
+    """
+    Verify that cluster infra/config matches config profile used to launch - only applies to cloudv2
+    """
+    def __init__(self, test_context):
+        super().__init__(test_context=test_context)
+        self._ctx = test_context
+        self._ipClient = InstallPackClient(
+            self.redpanda._cloud_cluster.config.install_pack_url_template,
+            self.redpanda._cloud_cluster.config.install_pack_auth_type,
+            self.redpanda._cloud_cluster.config.install_pack_auth)
+
+    def setUp(self):
+        super().setUp()
+        cloud_cluster = self.redpanda._cloud_cluster
+        self.logger.debug(f"Cloud Cluster Info: {vars(cloud_cluster)}")
+        install_pack_version = cloud_cluster.get_install_pack_version()
+        self._ip = self._ipClient.getInstallPack(install_pack_version)
+        self._clusterId = cloud_cluster.cluster_id
+        self._configProfile = self._ip['config_profiles'][
+            cloud_cluster.config.config_profile_name]
+
+    def test_healthy(self):
+        r = self.redpanda.cluster_unhealthy_reason()
+        assert r is None, r
+        assert self.redpanda.cluster_healthy()
+        self.redpanda.assert_cluster_is_reusable()
+
+    @cluster(num_nodes=1)
+    def test_databricks_basic(self):
+        # Get globals.json map
+        globals = self._ctx.globals
path: tests/rptest/redpanda_cloud_tests/iceberg_catalogs_test.py
user: login: nvartolomei
body: why you are not using the typed class for accessing globals?

```
from rptest.context.databricks import DatabricksContext
dbx_ctx = DatabricksContext.from_context(self._ctx)

# on dbx_ctx now you have all you need and it is typed
# i.e. `dbx_ctx.iceberg_rest_url`,  `dbx_ctx.sql_warehouse_path`, etc
```

credentials might be a bit trickier

```
creds = Credentials.from_context(self._ctx)
assert isinstance(creds, OauthCredentials), 'databricks must be configured with oauth credentials for this test'

# now you should have typed access to `creds.client_id`
```
created_at: 2025-05-08T09:01:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -580,6 +581,9 @@ def _update_live_cluster_info(self):
         self.current.zones = _c['spec']['zones']
         self.current.network_id = _c['spec']['networkId']
         self.current.network_cidr = _c['spec']['network']['networkCidr']
+        self.current.dataplane_public_api_url = _c.get(
path: tests/rptest/services/redpanda_cloud.py
user: login: nvartolomei
body: unused?
created_at: 2025-05-08T09:06:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -153,6 +153,7 @@ class LiveClusterParams:
     name: str | None = None
     last_status: str = ""
     consoleUrl: str = ""
+    dataplane_public_api_url: str = ""
path: tests/rptest/services/redpanda_cloud.py
user: login: nvartolomei
body: if used this is `str | None` based on the assignment below
created_at: 2025-05-08T09:06:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1562,3 +1567,49 @@ def set_cluster_config_overrides(self, cluster_id, config_values):
         return self.cloudv2._http_post(base_url=self.config.admin_api_url,
                                        endpoint='/SetClusterConfigOverrides',
                                        json=payload)
+
+    # Update cluster properties using Public API
+    def update_cluster_property_public(self, cluster_id, property_key,
path: tests/rptest/services/redpanda_cloud.py
user: login: nvartolomei
body: make this accept a map of Map[str, str]? then you'll be able to replace the _http_patch call from within the test with this. i.e. the one setting iceberg endpoint and such
created_at: 2025-05-08T09:09:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -83,12 +88,17 @@ def create_catalog(self, bucket: str) -> "DatabricksCatalogInfo":
             "We expect to only managed catalogs."
         assert catalog_info.name, "Catalog name must not be empty"
 
-        sql_connection = databricks.sql.connect(
-            server_hostname=self._databricks_context.server_hostname,
-            http_path=self._databricks_context.sql_warehouse_path,
-            catalog=catalog_info.name,
-            credentials_provider=self._databricks_context.credentials_provider,
-        )
+        try:
+            sql_connection = databricks.sql.connect(
+                server_hostname=self._databricks_context.server_hostname,
+                http_path=self._databricks_context.sql_warehouse_path,
+                catalog=catalog_info.name,
+                credentials_provider=self._databricks_context.credentials_provider,
+            )
+            self.logger.debug("SQL connection established successfully.")
+        except Exception as e:
+            self.logger.error(f"Error establishing SQL connection: {e}")
+            return  # Exit if the connection fails
path: tests/rptest/services/databricks_workspace.py
user: login: nvartolomei
body: why we don't reraise?
created_at: 2025-05-10T18:47:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,150 @@
+# Copyright 2025 Redpanda Data, Inc.
+#
+# Use of this software is governed by the Business Source License
+# included in the file licenses/BSL.md
+#
+# As of the Change Date specified in that file, in accordance with
+# the Business Source License, use of this software will be governed
+# by the Apache License, Version 2.0
+
+from typing import Any
+import logging
+import json
+import time
+import requests
+
+from ducktape.mark import matrix
+from rptest.tests.redpanda_cloud_test import RedpandaCloudTest
+from rptest.services.provider_clients.rpcloud_client import RpCloudApiClient
+
+from rptest.clients.installpack import InstallPackClient
+from rptest.clients.rpk import RpkTool, TopicSpec, RpkException
+
+from rptest.services.redpanda import get_cloud_provider
+from rptest.services.cluster import cluster
+from rptest.services.redpanda import RedpandaServiceCloud
+
+from rptest.services.databricks_workspace import DatabricksWorkspace
+from rptest.context.databricks import DatabricksContext, OauthCredentials
+from rptest.services.catalog_service import CatalogType
+#from rptest.tests.datalake.datalake_services import DatalakeServices
+#from rptest.tests.datalake.query_engine_base import QueryEngineType
+#from rptest.tests.datalake.utils import supported_storage_types
+#from rptest.tests.redpanda_test import RedpandaTest
+#from rptest.utils.mode_checks import cleanup_on_early_exit
+
+
+def supported_catalog_types():
+    return ["aws_glue", "databricks_unity", "snowflake"]
+
+
+def supported_network_types():
+    return ["public", "private"]
+
+
+class IcebergCloudCatalogsTest(RedpandaCloudTest):
+    """
+    Verify that cluster infra/config matches config profile used to launch - only applies to cloudv2
+    """
+    def __init__(self, test_context):
+        super().__init__(test_context=test_context)
+        self._ctx = test_context
+        self._ipClient = InstallPackClient(
+            self.redpanda._cloud_cluster.config.install_pack_url_template,
+            self.redpanda._cloud_cluster.config.install_pack_auth_type,
+            self.redpanda._cloud_cluster.config.install_pack_auth)
+
+    def setUp(self):
+        super().setUp()
+        cloud_cluster = self.redpanda._cloud_cluster
+        self.logger.debug(f"Cloud Cluster Info: {vars(cloud_cluster)}")
+        install_pack_version = cloud_cluster.get_install_pack_version()
+        self._ip = self._ipClient.getInstallPack(install_pack_version)
+        self._clusterId = cloud_cluster.cluster_id
+        self._configProfile = self._ip['config_profiles'][
+            cloud_cluster.config.config_profile_name]
+
+    def test_healthy(self):
+        r = self.redpanda.cluster_unhealthy_reason()
+        assert r is None, r
+        assert self.redpanda.cluster_healthy()
+        self.redpanda.assert_cluster_is_reusable()
+
+    @cluster(num_nodes=1)
+    def test_databricks_basic(self):
+        dbx_ctx = DatabricksContext.from_context(self._ctx)
+
+        cloud_cluster = self.redpanda._cloud_cluster
+        client: RpCloudApiClient = cloud_cluster.public_api
+
+        databricks_client = DatabricksWorkspace(context=self._ctx)
+        bucket = f"redpanda-cloud-storage-{self._clusterId}"
+        catalog_info = databricks_client.create_catalog(bucket=bucket)
+        properties = {"iceberg_enabled": True}
+        enable_resp = cloud_cluster.update_cluster_property_public(
+            self._clusterId, properties)
+        self.logger.debug(f"Enable iceberg response: {enable_resp}")
+
+        # Parameters for creating Redpanda secret
+        secret_id = "UNITY_CLIENT_SECRET"
+        
+        #secret_data = dbx_ctx.client_secret
+        # Access credentials (client_secret, client_id) from dbx_ctx.credentials
+        if isinstance(dbx_ctx.credentials, OauthCredentials):
+            secret_data = dbx_ctx.credentials.client_secret
+            databricks_client_id = dbx_ctx.credentials.client_id
+        else:
+            secret_data = dbx_ctx.credentials.token
+            databricks_client_id = None
+
+        create_resp = cloud_cluster.create_secret(secret_id, secret_data)
+        self.logger.debug(f"Create secret response: {create_resp}")
+
+        iceberg_rest_catalog_endpoint = dbx_ctx.iceberg_rest_url
+        iceberg_rest_catalog_warehouse = dbx_ctx.sql_warehouse_path
+
+        # Construct the payload for the request
+        properties = {
+            "iceberg_rest_catalog_endpoint": iceberg_rest_catalog_endpoint,
+            "iceberg_rest_catalog_authentication_mode": "oauth2" if isinstance(dbx_ctx.credentials, OauthCredentials) else "pat",
path: tests/rptest/redpanda_cloud_tests/iceberg_catalogs_test.py
user: login: nvartolomei
body: i'm not sure if this is supported at all in cloud but pat is not a valid value, bearer is
created_at: 2025-05-10T18:48:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,150 @@
+# Copyright 2025 Redpanda Data, Inc.
+#
+# Use of this software is governed by the Business Source License
+# included in the file licenses/BSL.md
+#
+# As of the Change Date specified in that file, in accordance with
+# the Business Source License, use of this software will be governed
+# by the Apache License, Version 2.0
+
+from typing import Any
+import logging
+import json
+import time
+import requests
+
+from ducktape.mark import matrix
+from rptest.tests.redpanda_cloud_test import RedpandaCloudTest
+from rptest.services.provider_clients.rpcloud_client import RpCloudApiClient
+
+from rptest.clients.installpack import InstallPackClient
+from rptest.clients.rpk import RpkTool, TopicSpec, RpkException
+
+from rptest.services.redpanda import get_cloud_provider
+from rptest.services.cluster import cluster
+from rptest.services.redpanda import RedpandaServiceCloud
+
+from rptest.services.databricks_workspace import DatabricksWorkspace
+from rptest.context.databricks import DatabricksContext, OauthCredentials
+from rptest.services.catalog_service import CatalogType
+#from rptest.tests.datalake.datalake_services import DatalakeServices
+#from rptest.tests.datalake.query_engine_base import QueryEngineType
+#from rptest.tests.datalake.utils import supported_storage_types
+#from rptest.tests.redpanda_test import RedpandaTest
+#from rptest.utils.mode_checks import cleanup_on_early_exit
+
+
+def supported_catalog_types():
+    return ["aws_glue", "databricks_unity", "snowflake"]
+
+
+def supported_network_types():
+    return ["public", "private"]
+
+
+class IcebergCloudCatalogsTest(RedpandaCloudTest):
+    """
+    Verify that cluster infra/config matches config profile used to launch - only applies to cloudv2
+    """
+    def __init__(self, test_context):
+        super().__init__(test_context=test_context)
+        self._ctx = test_context
+        self._ipClient = InstallPackClient(
+            self.redpanda._cloud_cluster.config.install_pack_url_template,
+            self.redpanda._cloud_cluster.config.install_pack_auth_type,
+            self.redpanda._cloud_cluster.config.install_pack_auth)
+
+    def setUp(self):
+        super().setUp()
+        cloud_cluster = self.redpanda._cloud_cluster
+        self.logger.debug(f"Cloud Cluster Info: {vars(cloud_cluster)}")
+        install_pack_version = cloud_cluster.get_install_pack_version()
+        self._ip = self._ipClient.getInstallPack(install_pack_version)
+        self._clusterId = cloud_cluster.cluster_id
+        self._configProfile = self._ip['config_profiles'][
+            cloud_cluster.config.config_profile_name]
+
+    def test_healthy(self):
+        r = self.redpanda.cluster_unhealthy_reason()
+        assert r is None, r
+        assert self.redpanda.cluster_healthy()
+        self.redpanda.assert_cluster_is_reusable()
+
+    @cluster(num_nodes=1)
+    def test_databricks_basic(self):
+        dbx_ctx = DatabricksContext.from_context(self._ctx)
+
+        cloud_cluster = self.redpanda._cloud_cluster
+        client: RpCloudApiClient = cloud_cluster.public_api
+
+        databricks_client = DatabricksWorkspace(context=self._ctx)
+        bucket = f"redpanda-cloud-storage-{self._clusterId}"
+        catalog_info = databricks_client.create_catalog(bucket=bucket)
+        properties = {"iceberg_enabled": True}
+        enable_resp = cloud_cluster.update_cluster_property_public(
+            self._clusterId, properties)
+        self.logger.debug(f"Enable iceberg response: {enable_resp}")
+
+        # Parameters for creating Redpanda secret
+        secret_id = "UNITY_CLIENT_SECRET"
+        
+        #secret_data = dbx_ctx.client_secret
+        # Access credentials (client_secret, client_id) from dbx_ctx.credentials
+        if isinstance(dbx_ctx.credentials, OauthCredentials):
+            secret_data = dbx_ctx.credentials.client_secret
+            databricks_client_id = dbx_ctx.credentials.client_id
+        else:
+            secret_data = dbx_ctx.credentials.token
+            databricks_client_id = None
+
+        create_resp = cloud_cluster.create_secret(secret_id, secret_data)
+        self.logger.debug(f"Create secret response: {create_resp}")
+
+        iceberg_rest_catalog_endpoint = dbx_ctx.iceberg_rest_url
+        iceberg_rest_catalog_warehouse = dbx_ctx.sql_warehouse_path
path: tests/rptest/redpanda_cloud_tests/iceberg_catalogs_test.py
user: login: nvartolomei
body: `iceberg_rest_catalog_warehouse` should be `catalog_info.name`

dbx_ctx.sql_warehouse_path relates to https://www.databricks.com/product/databricks-sql and is not something that redpanda needs to know about or knows how to use it at all. 
created_at: 2025-05-10T18:50:33Z
author_association: CONTRIBUTOR

PR comments:
user: login: secpanda
created_at: 2025-04-25T03:07:28Z
body: ![](https://res.cloudinary.com/snyk/image/upload/r-d/scm-platform/snyk-pull-requests/pr-banner-default.svg)

### :tada: **Snyk checks have passed. No issues have been found so far.**

:white_check_mark: **security/snyk** check is complete. No issues have been found. [(View Details)](https://app.snyk.io/org/devex-c3w/pr-checks/74e3dbaa-ac06-4f34-8167-6e78d810b016?refs=93947e2a1a85815854af6469bce8e86234c0ed3a&source=prChecksComment)

:white_check_mark: **license/snyk** check is complete. No issues have been found. [(View Details)](https://app.snyk.io/org/devex-c3w/pr-checks/74e3dbaa-ac06-4f34-8167-6e78d810b016/license?refs=93947e2a1a85815854af6469bce8e86234c0ed3a&source=prChecksComment)



author_association: NONE
user: login: secpanda
created_at: 2025-04-25T03:07:30Z
body: ![](https://res.cloudinary.com/snyk/image/upload/r-d/scm-platform/snyk-pull-requests/pr-banner-default.svg)

### :tada: **Snyk checks have passed. No issues have been found so far.**

:white_check_mark: **security/snyk** check is complete. No issues have been found. [(View Details)](https://app.snyk.io/org/devprod/pr-checks/eaaa9e8c-60fa-4954-a449-dd323adc4342?refs=93947e2a1a85815854af6469bce8e86234c0ed3a&source=prChecksComment)

:white_check_mark: **license/snyk** check is complete. No issues have been found. [(View Details)](https://app.snyk.io/org/devprod/pr-checks/eaaa9e8c-60fa-4954-a449-dd323adc4342/license?refs=93947e2a1a85815854af6469bce8e86234c0ed3a&source=prChecksComment)



author_association: NONE
user: login: rpdevmp
created_at: 2025-04-25T03:08:40Z
body: I will convert it to open PR once I add query checks, object storage verification and cleanups
author_association: CONTRIBUTOR
user: login: secpanda
created_at: 2025-05-01T01:10:40Z
body: ![](https://res.cloudinary.com/snyk/image/upload/r-d/scm-platform/snyk-pull-requests/pr-banner-default.svg)

### :tada: **Snyk checks have passed. No issues have been found so far.**

:white_check_mark: **security/snyk** check is complete. No issues have been found. [(View Details)](https://app.snyk.io/org/cloud-1t5/pr-checks/7cfc0b60-7145-499b-9c19-b46e1b83f13d?refs=93947e2a1a85815854af6469bce8e86234c0ed3a&source=prChecksComment)

:white_check_mark: **license/snyk** check is complete. No issues have been found. [(View Details)](https://app.snyk.io/org/cloud-1t5/pr-checks/7cfc0b60-7145-499b-9c19-b46e1b83f13d/license?refs=93947e2a1a85815854af6469bce8e86234c0ed3a&source=prChecksComment)



author_association: NONE
user: login: rpdevmp
created_at: 2025-05-08T01:38:29Z
body: @nvartolomei, this PR uses databricks workspace that we already merged, added cloud specific Redpanda secret management APIs and cluster dataplane fetching. This has been tested multiple times and all secret management items works. Console API to create a secret, dataplane API got pass that secret into Redpanda cloud config and then it is able to authenticate with databricks workspace..

Let's make sure we merge this PR, so I could start adding other code to these tests and have schedules running. This PR is ready for merge (once CI checks pass)
author_association: CONTRIBUTOR
user: login: vbotbuildovich
created_at: 2025-05-08T05:27:35Z
body: <!--FAILURES SLASH COMMAND https://buildkite.com/redpanda/redpanda/builds/65674-->
<!--a84f5f38-b64e-488a-8494-50ff6be0b7cc-->
<!--title-->
## Retry command for [Build#65674](https://buildkite.com/redpanda/redpanda/builds/65674)
please wait until all jobs are finished before running the slash command
<!--/title-->
<!--body-->
```
/ci-repeat 1
tests/rptest/tests/scaling_up_test.py::ScalingUpTest.test_scaling_up_with_recovered_topic
```
<!--/body-->
author_association: COLLABORATOR
user: login: vbotbuildovich
created_at: 2025-05-08T06:44:40Z
body: <!--TEST RESULTS-->
<!--9fba5d8a-23e6-42cc-9e60-176a61128654-->
<!--TEST RESULTS-->
<!--f052647f-7e09-4e2c-a6bb-e35ad2f13b1f-->
<!--TEST RESULTS-->
<!--c6eaf523-b181-4cd9-8e6e-51b474b6e26a-->
<!--TEST RESULTS-->
<!--25d3f201-4c09-4093-855c-1064cbd57184-->
<!--title-->
## CI test results
<!--/title-->
<!--body-->
<details>
<summary>test results on <a href="https://buildkite.com/redpanda/redpanda/builds/65674">build#65674</a></summary>


| test_id                                                                                                                                                                                                                                           | test_kind   | job_url                                                                                   | test_status   | passed   |
|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------|:------------------------------------------------------------------------------------------|:--------------|:---------|
| rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=False.mixed_versions=False.with_tiered_storage=False.with_iceberg=True.with_chunked_compaction=True.cloud_storage_type=CloudStorageType.S3 | ducktape    | https://buildkite.com/redpanda/redpanda/builds/65674#0196adc2-ae2e-4c2c-911d-c52b49a4edb9 | FLAKY         | 11/12    |
| rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=True.mixed_versions=False.with_tiered_storage=False.with_iceberg=False.with_chunked_compaction=True.cloud_storage_type=CloudStorageType.S3 | ducktape    | https://buildkite.com/redpanda/redpanda/builds/65674#0196adc2-ae2e-4c2c-911d-c52b49a4edb9 | FLAKY         | 10/11    |
| rptest.tests.scaling_up_test.ScalingUpTest.test_scaling_up_with_recovered_topic                                                                                                                                                                   | ducktape    | https://buildkite.com/redpanda/redpanda/builds/65674#0196adc2-ae2c-4d50-9335-84fe1005a13f | FLAKY         | 10/21    |
| rptest.tests.upgrade_test.UpgradeBackToBackTest.test_upgrade_with_all_workloads.single_upgrade=False                                                                                                                                              | ducktape    | https://buildkite.com/redpanda/redpanda/builds/65674#0196adc2-ae2e-4c2c-911d-c52b49a4edb9 | FLAKY         | 8/10     |
</details>
<details>
<summary>test results on <a href="https://buildkite.com/redpanda/redpanda/builds/65776">build#65776</a></summary>


| test_class                        | test_method                | test_arguments                                                                                                                                                    | test_kind   | job_url                                                                                   | test_status   | passed   | reason                                                                                                                                                                  |
|:----------------------------------|:---------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------|:------------------------------------------------------------------------------------------|:--------------|:---------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| PartitionBalancerTest             | test_unavailable_nodes     |                                                                                                                                                                   | ducktape    | https://buildkite.com/redpanda/redpanda/builds/65776#0196b7f0-7b7b-4de7-8550-2ce92c0e67be | FLAKY         | 10/21    | upstream reliability is '72.31040564373897'. current run reliability is '47.61904761904761'. drift is 24.69136 and the allowed drift is set to 50. The test should PASS |
| PartitionForceReconfigurationTest | test_basic_reconfiguration | {"acks": -1, "controller_snapshots": false, "restart": false}                                                                                                     | ducktape    | https://buildkite.com/redpanda/redpanda/builds/65776#0196b7e4-5653-4816-9c80-43312b1155d6 | FLAKY         | 20/21    | upstream reliability is '100.0'. current run reliability is '95.23809523809523'. drift is 4.7619 and the allowed drift is set to 50. The test should PASS               |
| RandomNodeOperationsTest          | test_node_operations       | {"cloud_storage_type": 1, "enable_failures": false, "mixed_versions": false, "with_chunked_compaction": true, "with_iceberg": true, "with_tiered_storage": false} | ducktape    | https://buildkite.com/redpanda/redpanda/builds/65776#0196b7f0-7b79-474c-a642-83ce93787050 | FLAKY         | 19/21    | upstream reliability is '98.4375'. current run reliability is '90.47619047619048'. drift is 7.96131 and the allowed drift is set to 50. The test should PASS            |
</details>
<details>
<summary>test results on <a href="https://buildkite.com/redpanda/redpanda/builds/65862">build#65862</a></summary>


| test_class                   | test_method                                       | test_arguments                                                                                                                                                    | test_kind   | job_url                                                                                   | test_status   | passed   | reason                                                                                                                                                                  |
|:-----------------------------|:--------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------|:------------------------------------------------------------------------------------------|:--------------|:---------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| CloudStorageTimingStressTest | test_cloud_storage_with_partition_moves           | {"cleanup_policy": "compact,delete"}                                                                                                                              | ducktape    | https://buildkite.com/redpanda/redpanda/builds/65862#0196c75b-e59d-430c-9333-6edc633ae158 | FLAKY         | 20/21    | upstream reliability is '96.05263157894737'. current run reliability is '95.23809523809523'. drift is 0.81454 and the allowed drift is set to 50. The test should PASS  |
| RestCatalogConnectionTest    | test_redpanda_connection_to_rest_catalog          | {"cloud_storage_type": 1}                                                                                                                                         | ducktape    | https://buildkite.com/redpanda/redpanda/builds/65862#0196c75b-e59b-4bd9-a1f3-3a07169c9f4b | FLAKY         | 19/21    | upstream reliability is '97.26027397260275'. current run reliability is '90.47619047619048'. drift is 6.78408 and the allowed drift is set to 50. The test should PASS  |
| PartitionReassignmentsTest   | test_add_partitions_with_inprogress_reassignments |                                                                                                                                                                   | ducktape    | https://buildkite.com/redpanda/redpanda/builds/65862#0196c766-82c5-4d84-acb6-0f1820a221c2 | FLAKY         | 19/21    | upstream reliability is '89.92248062015504'. current run reliability is '90.47619047619048'. drift is -0.55371 and the allowed drift is set to 50. The test should PASS |
| RandomNodeOperationsTest     | test_node_operations                              | {"cloud_storage_type": 1, "enable_failures": false, "mixed_versions": false, "with_chunked_compaction": true, "with_iceberg": true, "with_tiered_storage": false} | ducktape    | https://buildkite.com/redpanda/redpanda/builds/65862#0196c75b-e59b-4bd9-a1f3-3a07169c9f4b | FLAKY         | 20/21    | upstream reliability is '96.89655172413794'. current run reliability is '95.23809523809523'. drift is 1.65846 and the allowed drift is set to 50. The test should PASS  |
| RandomNodeOperationsTest     | test_node_operations                              | {"cloud_storage_type": 1, "enable_failures": false, "mixed_versions": false, "with_chunked_compaction": true, "with_iceberg": false, "with_tiered_storage": true} | ducktape    | https://buildkite.com/redpanda/redpanda/builds/65862#0196c75b-e59b-4bd9-a1f3-3a07169c9f4b | FLAKY         | 20/21    | upstream reliability is '99.23076923076923'. current run reliability is '95.23809523809523'. drift is 3.99267 and the allowed drift is set to 50. The test should PASS  |
| TestTieredStoragePause       | test_safe_pause_resume                            | {"allow_gaps_cluster_level": true, "allow_gaps_topic_level": true}                                                                                                | ducktape    | https://buildkite.com/redpanda/redpanda/builds/65862#0196c75b-e59d-430c-9333-6edc633ae158 | FLAKY         | 20/21    | upstream reliability is '98.49624060150376'. current run reliability is '95.23809523809523'. drift is 3.25815 and the allowed drift is set to 50. The test should PASS  |
</details>
<details>
<summary>test results on <a href="https://buildkite.com/redpanda/redpanda/builds/65900">build#65900</a></summary>


| test_class                   | test_method                                       | test_arguments                                                                                                                                                    | test_kind   | job_url                                                                                   | test_status   | passed   | reason                                                                                                                                                                  |
|:-----------------------------|:--------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------|:------------------------------------------------------------------------------------------|:--------------|:---------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| AccessControlListTestUpgrade | test_upgrade_sasl                                 |                                                                                                                                                                   | ducktape    | https://buildkite.com/redpanda/redpanda/builds/65900#0196c998-e2f7-4cc7-be31-33fb555ee734 | FLAKY         | 20/21    | upstream reliability is '98.46153846153847'. current run reliability is '95.23809523809523'. drift is 3.22344 and the allowed drift is set to 50. The test should PASS  |
| PartitionBalancerTest        | test_unavailable_nodes                            |                                                                                                                                                                   | ducktape    | https://buildkite.com/redpanda/redpanda/builds/65900#0196c9a6-1384-4d89-a9f4-c069650a9a1a | FLAKY         | 11/21    | upstream reliability is '71.05263157894737'. current run reliability is '52.38095238095239'. drift is 18.67168 and the allowed drift is set to 50. The test should PASS |
| PartitionReassignmentsTest   | test_add_partitions_with_inprogress_reassignments |                                                                                                                                                                   | ducktape    | https://buildkite.com/redpanda/redpanda/builds/65900#0196c9a6-1384-4d89-a9f4-c069650a9a1a | FLAKY         | 20/21    | upstream reliability is '89.03846153846153'. current run reliability is '95.23809523809523'. drift is -6.19963 and the allowed drift is set to 50. The test should PASS |
| RandomNodeOperationsTest     | test_node_operations                              | {"cloud_storage_type": 1, "enable_failures": false, "mixed_versions": false, "with_chunked_compaction": true, "with_iceberg": true, "with_tiered_storage": false} | ducktape    | https://buildkite.com/redpanda/redpanda/builds/65900#0196c9a6-1382-4ef9-8004-aaeca875846d | FLAKY         | 20/21    | upstream reliability is '96.65551839464884'. current run reliability is '95.23809523809523'. drift is 1.41742 and the allowed drift is set to 50. The test should PASS  |
| RandomNodeOperationsTest     | test_node_operations                              | {"cloud_storage_type": 1, "enable_failures": false, "mixed_versions": false, "with_chunked_compaction": true, "with_iceberg": false, "with_tiered_storage": true} | ducktape    | https://buildkite.com/redpanda/redpanda/builds/65900#0196c9a6-1382-4ef9-8004-aaeca875846d | FLAKY         | 20/21    | upstream reliability is '99.15966386554622'. current run reliability is '95.23809523809523'. drift is 3.92157 and the allowed drift is set to 50. The test should PASS  |
| UpgradeBackToBackTest        | test_upgrade_with_all_workloads                   | {"single_upgrade": false}                                                                                                                                         | ducktape    | https://buildkite.com/redpanda/redpanda/builds/65900#0196c998-e2f8-40f2-b853-5c691197c1ab | FLAKY         | 11/21    | upstream reliability is '84.34782608695653'. current run reliability is '52.38095238095239'. drift is 31.96687 and the allowed drift is set to 50. The test should PASS |
</details>
<!--/body-->
author_association: COLLABORATOR
user: login: nvartolomei
created_at: 2025-05-08T09:10:21Z
body: Also, please squash the commits. The adding and removing of the previous client is not necessary in the tree and will just confuse everyone who will get blame the code.
author_association: CONTRIBUTOR
user: login: rpdevmp
created_at: 2025-05-09T06:43:49Z
body: Thank you for review @nvartolomei I added new changes based on the context code that was added (I was reading from globals for sometime before that was available), now updated everything to use that and removed client that I had and squashed changes.

Hopefully all checks would pass. Last time there were unrelated failures.

If looks good, let's merge this, so I could add to schedule and then could add other code. Need to get this in to verify infra setup
author_association: CONTRIBUTOR
user: login: nvartolomei
created_at: 2025-05-10T18:58:04Z
body: The tests failed because nessie smoke test can't run. Does this branch need a rebase? 🤔 also python linting is failing. files need to be formatted properly `task lint:python`
author_association: CONTRIBUTOR