Repository: redpanda-data/redpanda-ansible-collection
Pull request #122: Console config changes
Author: login: gene-redpanda
Created: 2025-03-12T15:32:10Z | Merged: 2025-03-14T16:50:57Z

Adds support for the v3.0.0 console changes. Also does some QOL on the console role and adds tests. 


Diff:
diff --git a/.buildkite/pipeline.yml b/.buildkite/pipeline.yml
index 116bf1e3..e061ff9d 100644
--- a/.buildkite/pipeline.yml
+++ b/.buildkite/pipeline.yml
@@ -12,3 +12,8 @@ steps:
       - docker-compose#v5.2.0:
           config: roles/redpanda_connect/docker-compose.yml
           run: testctr
+  - label: "console template tests"
+    plugins:
+      - docker-compose#v5.2.0:
+          config: roles/redpanda_console/docker-compose.yml
+          run: testctr
diff --git a/galaxy.yml b/galaxy.yml
index 2c4dce5c..11147c55 100644
--- a/galaxy.yml
+++ b/galaxy.yml
@@ -8,7 +8,7 @@ namespace: redpanda
 name: cluster
 
 # The version of the collection. Must be compatible with semantic versioning
-version: 0.7.0
+version: 0.8.0
 
 # The path to the Markdown (.md) readme file. This path is relative to the root of the collection
 readme: README.md
diff --git a/roles/redpanda_broker/Makefile b/roles/redpanda_broker/Makefile
index 2620518a..defc0456 100644
--- a/roles/redpanda_broker/Makefile
+++ b/roles/redpanda_broker/Makefile
@@ -18,8 +18,13 @@ mocks:
 
 .PHONY: test
 test: $(SCRIPTS)
-	echo "running $(DIR) $(SCRIPTS)"
-	$(foreach FILE,$(SCRIPTS), $(PYTHON) $(FILE);)
+	@echo "running $(DIR) $(SCRIPTS)"
+	@EXIT_CODE=0; \
+	for FILE in $(SCRIPTS); do \
+		echo "Running $$FILE"; \
+		$(PYTHON) $$FILE || EXIT_CODE=1; \
+	done; \
+	exit $$EXIT_CODE
 
 .PHONY: up
 up:
diff --git a/roles/redpanda_connect/Makefile b/roles/redpanda_connect/Makefile
index f0863979..7f1c572e 100644
--- a/roles/redpanda_connect/Makefile
+++ b/roles/redpanda_connect/Makefile
@@ -11,10 +11,15 @@ all: $(FILES)
 
 do: reqs test
 
+.PHONY: test
 test: $(SCRIPTS)
-	echo "running $(DIR) $(SCRIPTS)"
-	$(foreach FILE,$(SCRIPTS), $(PYTHON) $(FILE);)
-
+	@echo "running $(DIR) $(SCRIPTS)"
+	@EXIT_CODE=0; \
+	for FILE in $(SCRIPTS); do \
+		echo "Running $$FILE"; \
+		$(PYTHON) $$FILE || EXIT_CODE=1; \
+	done; \
+	exit $$EXIT_CODE
 reqs:
 	$(PIP) install --no-cache-dir -r tests/requirements.txt
 
diff --git a/roles/redpanda_connect/tests/connect-distributed.properties_test.py b/roles/redpanda_connect/tests/connect-distributed.properties_test.py
index 2cbb3512..2d21dba5 100644
--- a/roles/redpanda_connect/tests/connect-distributed.properties_test.py
+++ b/roles/redpanda_connect/tests/connect-distributed.properties_test.py
@@ -19,7 +19,7 @@ def test_bootstrap_server_template(self):
             defaults = yaml.safe_load(f)
 
         # Create a Jinja2 environment and load the template from file
-        env = Environment(loader=FileSystemLoader(templates_dir))
+        env = Environment(loader=FileSystemLoader(os.path.join(templates_dir, 'connect-distributed')))
         template = env.get_template('connect-distributed.properties.j2')
 
         # Define the hostvars and groups for rendering the template
@@ -36,18 +36,43 @@ def test_bootstrap_server_template(self):
         }
         advertised_ips = ["35.91.106.231", "35.88.129.205", "54.190.184.126"]
         groups = {
-            'connect': ['35.91.106.231', '35.88.129.205', '54.190.184.126']
+            'connect': ['35.91.106.231', '35.88.129.205', '54.190.184.126'],
+            'redpanda': ['35.91.106.231']  # Add redpanda group with first node
         }
 
+        # Add missing variables that are calculated in Ansible
+        connect_tls_enabled = defaults.get('connect_tls_enabled', False)
+        schema_registry_actual = defaults.get('schema_registry_url',
+                               f"http{'s' if connect_tls_enabled else ''}://{groups['redpanda'][0]}:8081")
+        defaults['schema_registry_actual'] = schema_registry_actual
+
+        # Set additional missing values with dummy data if they don't exist in defaults
+        if 'config_storage_topic' not in defaults:
+            defaults['config_storage_topic'] = 'connect-configs'
+        if 'offset_storage_topic' not in defaults:
+            defaults['offset_storage_topic'] = 'connect-offsets'
+        if 'status_storage_topic' not in defaults:
+            defaults['status_storage_topic'] = 'connect-status'
+        if 'plugin_discovery' not in defaults:
+            defaults['plugin_discovery'] = 'org.apache.kafka.connect.discovery.PluginDiscovery'
+        if 'key_converter' not in defaults:
+            defaults['key_converter'] = 'org.apache.kafka.connect.json.JsonConverter'
+        if 'value_converter' not in defaults:
+            defaults['value_converter'] = 'org.apache.kafka.connect.json.JsonConverter'
+        if 'kafka_port' not in defaults:
+            defaults['kafka_port'] = 9092
+        if 'rest_advertised_host_name_actual' not in defaults:
+            defaults['rest_advertised_host_name_actual'] = defaults.get('rest_advertised_host_name', 'localhost')
+
         # Render the template with the provided hostvars, groups, and defaults
         rendered_template = template.render(hostvars=hostvars, advertised_ips=advertised_ips, groups=groups, **defaults)
 
         # Define the expected bootstrap server line
         expected_bootstrap_servers = 'bootstrap.servers=35.91.106.231:9092,35.88.129.205:9092,54.190.184.126:9092'
-        expected_schema_registry_url = f"schema.registry.url={defaults['schema_registry_url']}"
+        expected_schema_registry_url = f"schema.registry.url={schema_registry_actual}"
         expected_group_id = f"group.id={defaults['group_id']}"
         expected_rest_port = f"rest.port={defaults['rest_port']}"
-        expected_rest_advertised_host_name = f"rest.advertised.host.name={defaults['rest_advertised_host_name']}"
+        expected_rest_advertised_host_name = f"rest.advertised.host.name={defaults['rest_advertised_host_name_actual']}"
         expected_rest_advertised_port = f"rest.advertised.port={defaults['rest_advertised_port']}"
         expected_plugin_path = f"plugin.path={defaults['plugin_path']}"
 
diff --git a/roles/redpanda_console/Dockerfile b/roles/redpanda_console/Dockerfile
new file mode 100644
index 00000000..129d0568
--- /dev/null
+++ b/roles/redpanda_console/Dockerfile
@@ -0,0 +1,8 @@
+## Used for testing the Kafka Connect role
+FROM python:latest
+
+WORKDIR /app
+
+COPY . .
+
+CMD ["make", "do"]
diff --git a/roles/redpanda_console/Makefile b/roles/redpanda_console/Makefile
new file mode 100644
index 00000000..7f1c572e
--- /dev/null
+++ b/roles/redpanda_console/Makefile
@@ -0,0 +1,27 @@
+.PHONY: test reqs do
+
+PYTHON ?= python
+PIP ?= pip
+
+DIR := tests
+SCRIPTS := $(wildcard $(DIR)/*_test.py)
+
+# Target to process each file
+all: $(FILES)
+
+do: reqs test
+
+.PHONY: test
+test: $(SCRIPTS)
+	@echo "running $(DIR) $(SCRIPTS)"
+	@EXIT_CODE=0; \
+	for FILE in $(SCRIPTS); do \
+		echo "Running $$FILE"; \
+		$(PYTHON) $$FILE || EXIT_CODE=1; \
+	done; \
+	exit $$EXIT_CODE
+reqs:
+	$(PIP) install --no-cache-dir -r tests/requirements.txt
+
+up:
+	docker-compose up --build
diff --git a/roles/redpanda_console/defaults/main.yml b/roles/redpanda_console/defaults/main.yml
index d6eea3ac..0e715bf5 100644
--- a/roles/redpanda_console/defaults/main.yml
+++ b/roles/redpanda_console/defaults/main.yml
@@ -1,14 +1,10 @@
 ---
-redpanda_organization: redpanda-test
-redpanda_cluster_id: redpanda
 enable_airgap: false
 
-redpanda_mode: production
 redpanda_admin_api_port: 9644
 redpanda_kafka_port: 9092
-redpanda_rpc_port: 33145
 redpanda_schema_registry_port: 8081
-redpanda_connect_port: 8083
+kafka_connect_port: 8083
 
 http_port: 8080
 https_port: 443
@@ -16,45 +12,29 @@ https_port: 443
 redpanda_group: redpanda
 redpanda_user: redpanda
 
-redpanda_use_staging_repo: false
 redpanda_base_url: "https://dl.redpanda.com"
 
 redpanda_install_status: present # If redpanda_version is set to latest, changing redpanda_install_status to latest will effect an upgrade, otherwise the currently installed version will remain
-redpanda_rpk_opts: ""
 
 redpanda_base_dir: /var/lib/redpanda
 redpanda_data_directory: "{{ redpanda_base_dir }}/data"
-repdanda_mount_dir: /mnt/vectorized/redpanda
 
 redpanda_certs_dir: /etc/redpanda/certs
 redpanda_csr_file: "{{ redpanda_certs_dir }}/node.csr"
 redpanda_key_file: "{{ redpanda_certs_dir }}/node.key"
 redpanda_cert_file: "{{ redpanda_certs_dir }}/node.crt"
 redpanda_truststore_file: "{{ redpanda_certs_dir }}/truststore.pem"
-node_exporter_version: "{{ node_exporter_custom_version | default('1.5.0') }}"
-cloud_storage_credentials_source: "config_file"
-cloud_storage_enable_remote_write: "true"
-cloud_storage_enable_remote_read: "true"
-is_using_unstable: false
 
 # We default to use the same redpanda dirs, but in prod, a user can specify them directly
 console_certs_dir: "{{ redpanda_certs_dir }}"
 console_key_file: "{{ console_certs_dir }}/node.key"
 console_cert_file: "{{ console_certs_dir }}/node.crt"
 
-airgap_copy_src: "/tmp"
-airgap_copy_dest: "/tmp"
-
 # the rpms
 rp_key_rpm: "{{ redpanda_base_url }}/public/redpanda/gpg.988A7B0A4918BC85.key"
-rp_key_rpm_unstable: "{{ redpanda_base_url }}/E4xN1tVe3Xy60GTx/redpanda-unstable/gpg.39C20393EC2E8747.key"
 rp_standard_rpm: "{{ redpanda_base_url }}/public/redpanda/rpm/{{ rp_rpm_distro_map[ansible_distribution] | default('el') }}/{{ ansible_distribution_major_version }}/$basearch"
-rp_standard_rpm_unstable: "{{ redpanda_base_url }}/E4xN1tVe3Xy60GTx/redpanda-unstable/rpm/{{ rp_rpm_distro_map[ansible_distribution] | default('el') }}/{{ ansible_distribution_major_version }}/$basearch"
 rp_noarch_rpm: "{{ redpanda_base_url }}/public/redpanda/rpm/{{ rp_rpm_distro_map[ansible_distribution] | default('el') }}/{{ ansible_distribution_major_version }}/noarch"
-rp_noarch_rpm_unstable: "{{ redpanda_base_url }}/E4xN1tVe3Xy60GTx/redpanda-unstable/rpm/{{ rp_rpm_distro_map[ansible_distribution] | default('el') }}/{{ ansible_distribution_major_version }}/noarch"
 rp_source_rpm: "{{ redpanda_base_url }}/public/redpanda/rpm/{{ rp_rpm_distro_map[ansible_distribution] | default('el') }}/{{ ansible_distribution_major_version }}/SRPMS"
-rp_source_rpm_unstable: "{{ redpanda_base_url }}/E4xN1tVe3Xy60GTx/redpanda-unstable/rpm/{{ rp_rpm_distro_map[ansible_distribution] | default('el') }}/{{ ansible_distribution_major_version }}/SRPMS"
-rp_conf_loc_rpm: "/etc/yum.repos.d/redpanda-redpanda.repo"
 rp_rpm_distro_map:
   Fedora: fedora
   Amazon: amzn
@@ -64,11 +44,6 @@ rp_rpm_distro_map:
 # the debs
 rp_key_deb: "{{ redpanda_base_url }}/sMIXnoa7DK12JW4A/redpanda/gpg.988A7B0A4918BC85.key"
 rp_key_deb_unstable: "{{ redpanda_base_url }}/E4xN1tVe3Xy60GTx/redpanda-unstable/gpg.39C20393EC2E8747.key"
-rp_conf_loc_deb: "/etc/apt/sources.list.d/redpanda.list"
-rp_conf_loc_deb_unstable: "etc/apt/sources.list.d/redpanda-redpanda-unstable.list"
 rp_key_path_deb: "/usr/share/keyrings/redpanda-redpanda-archive-keyring.gpg"
-rp_key_path_deb_unstable: "/usr/share/keyrings/redpanda-redpanda-unstable-archive-keyring.gpg"
 rp_repo_signing_deb: "deb [signed-by=/usr/share/keyrings/redpanda-redpanda-archive-keyring.gpg] {{ redpanda_base_url }}/public/redpanda/deb/{{ansible_distribution | lower}} {{ ansible_distribution_release | lower }} main"
-rp_repo_signing_deb_unstable: "deb [signed-by=/usr/share/keyrings/redpanda-redpanda-unstable-archive-keyring.gpg] {{ redpanda_base_url }}/E4xN1tVe3Xy60GTx/redpanda-unstable/deb/{{ansible_distribution | lower}} {{ ansible_distribution_release | lower }} main"
 rp_repo_signing_src_deb: "deb-src [signed-by=/usr/share/keyrings/redpanda-redpanda-archive-keyring.gpg] {{ redpanda_base_url }}/public/redpanda/deb/{{ansible_distribution | lower}} {{ ansible_distribution_release | lower }} main"
-rp_repo_signing_src_deb_unstable: "deb-src [signed-by=/usr/share/keyrings/redpanda-redpanda-unstable-archive-keyring.gpg] {{ redpanda_base_url }}/E4xN1tVe3Xy60GTx/redpanda-unstable/deb/{{ansible_distribution | lower}} {{ ansible_distribution_release | lower }} main"
diff --git a/roles/redpanda_console/docker-compose.yml b/roles/redpanda_console/docker-compose.yml
new file mode 100644
index 00000000..2d0770ff
--- /dev/null
+++ b/roles/redpanda_console/docker-compose.yml
@@ -0,0 +1,9 @@
+version: '3'
+services:
+  testctr:
+    build:
+      context: .
+      dockerfile: Dockerfile
+    volumes:
+      - .:/app
+    command: make do
diff --git a/roles/redpanda_console/tasks/configure-console.yml b/roles/redpanda_console/tasks/configure-console.yml
index 054cba3e..3a9f4092 100644
--- a/roles/redpanda_console/tasks/configure-console.yml
+++ b/roles/redpanda_console/tasks/configure-console.yml
@@ -10,9 +10,9 @@
       groups['redpanda'] | map('extract', hostvars) | map(attribute='inventory_hostname' if (advertise_public_ips | default(False) | bool) else 'private_ip') | list
     }}"
 
-- name: Create list of connect_advertised_ips
+- name: Create list of kafka_connect_advertised_ips
   ansible.builtin.set_fact:
-    connect_advertised_ips: "{{
+    kafka_connect_advertised_ips: "{{
       groups['connect'] | map('extract', hostvars) | map(attribute='inventory_hostname' if (advertise_public_ips | default(False) | bool) else 'private_ip') | list
     }}"
   when: "'connect' in groups"
diff --git a/roles/redpanda_console/tasks/install-console-deb.yml b/roles/redpanda_console/tasks/install-console-deb.yml
index da17d335..893c42d9 100644
--- a/roles/redpanda_console/tasks/install-console-deb.yml
+++ b/roles/redpanda_console/tasks/install-console-deb.yml
@@ -1,18 +1,32 @@
 - name: Install redpanda console (deb)
   ansible.builtin.package:
     name:
-      - redpanda-console
+      - redpanda-console{{ redpanda_version_suffix }}
     state: "{{ redpanda_install_status }}"
     update_cache: true
   environment:
     https_proxy: "{{ https_proxy_value | default('') }}"
     http_proxy: "{{ https_proxy_value | default('') }}"
+  register: console_install_result
   when: https_proxy_value is defined and https_proxy_value | length > 0
 
 - name: Install redpanda console (deb)
   ansible.builtin.package:
     name:
-      - redpanda-console
+      - redpanda-console{{ redpanda_version_suffix }}
     state: "{{ redpanda_install_status }}"
     update_cache: true
+  register: console_install_result
   when: https_proxy_value is not defined or https_proxy_value | length == 0
+
+# Get installed version if 'latest' was requested
+- name: Get installed console version
+  ansible.builtin.shell: dpkg-query -W -f='${Version}' redpanda-console
+  register: console_version_result
+  changed_when: false
+  when: redpanda_version == 'latest'
+
+- name: Update template choice based on installed version
+  ansible.builtin.set_fact:
+    use_pre_v3_template: "{{ console_version_result.stdout is version('3.0.0', '<') }}"
+  when: redpanda_version == 'latest'
diff --git a/roles/redpanda_console/tasks/install-console-rpm.yml b/roles/redpanda_console/tasks/install-console-rpm.yml
index 0cbad178..cc10907a 100644
--- a/roles/redpanda_console/tasks/install-console-rpm.yml
+++ b/roles/redpanda_console/tasks/install-console-rpm.yml
@@ -2,6 +2,19 @@
 - name: Install redpanda console (rpm)
   ansible.builtin.package:
     name:
-      - redpanda-console
+      - redpanda-console{{ redpanda_version_suffix }}
     state: "{{ redpanda_install_status }}"
     update_cache: true
+  register: console_install_result
+
+# Get installed version if 'latest' was requested
+- name: Get installed console version
+  ansible.builtin.shell: rpm -q --qf '%{VERSION}' redpanda-console
+  register: console_version_result
+  changed_when: false
+  when: redpanda_version == 'latest'
+
+- name: Update template choice based on installed version
+  ansible.builtin.set_fact:
+    use_pre_v3_template: "{{ console_version_result.stdout is version('3.0.0', '<') }}"
+  when: redpanda_version == 'latest'
diff --git a/roles/redpanda_console/tasks/main.yml b/roles/redpanda_console/tasks/main.yml
index 70d64fab..c6ef647d 100644
--- a/roles/redpanda_console/tasks/main.yml
+++ b/roles/redpanda_console/tasks/main.yml
@@ -1,3 +1,4 @@
+---
 - name: Ensure redpanda_version is defined
   ansible.builtin.assert:
     that:
@@ -5,6 +6,12 @@
     fail_msg: "Variable 'redpanda_version' must be defined!"
   when: not (install_certs_only | default(false) | bool)
 
+# Set fact for package version suffix
+- name: Set fact for package version suffix
+  ansible.builtin.set_fact:
+    redpanda_version_suffix: "{{ '' if redpanda_version == 'latest' else ('=' if ansible_os_family == 'Debian' else '-') + redpanda_version }}"
+    use_pre_v3_template: "{{ redpanda_version != 'latest' and redpanda_version is version('3.0.0', '<') }}"
+
 # Copies certs set in ca_cert_file and node_cert_file into the redpanda nodes. see defaults for the default values
 # only necessary when TLS is enabled
 - name: Install certs
@@ -31,6 +38,7 @@
   when:
     - ansible_os_family == 'Debian'
     - enable_airgap is false
+
 - name: Install Redpanda Console
   ansible.builtin.include_tasks: install-console-deb.yml
   when:
diff --git a/roles/redpanda_console/templates/defaults.j2 b/roles/redpanda_console/templates/defaults.j2
index c3cdcab5..3858aab4 100644
--- a/roles/redpanda_console/templates/defaults.j2
+++ b/roles/redpanda_console/templates/defaults.j2
@@ -1,16 +1,9 @@
 {
   "kafka": {
-    "brokers": {
+      "brokers": [
         {% for host in advertised_ips %}
         "{{ host }}:{{ redpanda_kafka_port }}"{% if not loop.last %},{% endif %}
         {% endfor %}
-    },
-    "schemaRegistry": {
-      "enabled": true,
-      "urls": [
-        {% for host in advertised_ips %}
-        "{% if enable_tls | default(false) %}https{% else %}http{% endif %}://{{ host }}:{{ redpanda_schema_registry_port }}"{% if not loop.last %},{% endif %}
-        {% endfor %}
       ]{% if enable_tls | default(false) %},
       "tls": {
         "enabled": true,
@@ -20,27 +13,6 @@
         "insecureSkipTlsVerify": false
       }
       {% endif %}
-    },
-    "protobuf": {
-      "enabled": true,
-      "schemaRegistry": {
-        "enabled": true,
-        "refreshInterval": "5m"
-      }
-    },
-    "brokers": [
-      {% for host in advertised_ips %}
-      "{{ host }}:{{ redpanda_kafka_port }}"{% if not loop.last %},{% endif %}
-      {% endfor %}
-    ]{% if enable_tls | default(false) %},
-  "tls": {
-    "enabled": true,
-    "caFilepath": "{{ redpanda_truststore_file }}",
-    "certFilepath": "{{ redpanda_cert_file }}",
-    "keyFilepath": "{{ redpanda_key_file }}",
-    "insecureSkipTlsVerify": false
-  }
-  {% endif %}
   },
   "redpanda": {
     "adminApi": {
@@ -59,23 +31,22 @@
       }
       {% endif %}
     }
-  }{% if enable_tls | default(false) %},
+  },{% if enable_tls | default(false) %}
   "server": {
     "listenPort": {{ http_port }},
     "httpsListenPort": {{ https_port }},
     "tls": {
       "enabled": true,
       "certFilepath": "{{ console_cert_file }}",
-      "keyFilepath": "{{ console_key_file }}",
+      "keyFilepath": "{{ console_key_file }}"
     }
-  }{% endif %}{% if connect_advertised_ips is defined %},
-    "connect": {
+  },{% endif %}{% if kafka_connect_advertised_ips is defined %}
+  "kafkaConnect": {
     "enabled": true,
-    "clusters":
-    [
+    "clusters": [
       {
         "name": "Connect",
-        "url": "{% if enable_tls | default(false) %}https{% else %}http{% endif %}://{{ connect_advertised_ips[0] }}:{{ redpanda_connect_port }}"
+        "url": "{% if enable_tls | default(false) %}https{% else %}http{% endif %}://{{ kafka_connect_advertised_ips[0] }}:{{ kafka_connect_port }}"
         {% if enable_tls | default(false) %},
         "tls": {
           "enabled": true,
@@ -85,6 +56,23 @@
           "insecureSkipTlsVerify": false
         }
         {% endif %}
-    }]
-}{% endif %}
+      }
+    ]
+  },{% endif %}
+  "schemaRegistry": {
+    "enabled": true,
+    "urls": [
+      {% for host in advertised_ips %}
+      "{% if enable_tls | default(false) %}https{% else %}http{% endif %}://{{ host }}:{{ redpanda_schema_registry_port }}"{% if not loop.last %},{% endif %}
+      {% endfor %}
+    ]{% if enable_tls | default(false) %},
+    "tls": {
+      "enabled": true,
+      "caFilepath": "{{ redpanda_truststore_file }}",
+      "certFilepath": "{{ redpanda_cert_file }}",
+      "keyFilepath": "{{ redpanda_key_file }}",
+      "insecureSkipTlsVerify": false
+    }
+    {% endif %}
+  }
 }
diff --git a/roles/redpanda_console/templates/pre_v3_defaults.j2 b/roles/redpanda_console/templates/pre_v3_defaults.j2
new file mode 100644
index 00000000..426b86dc
--- /dev/null
+++ b/roles/redpanda_console/templates/pre_v3_defaults.j2
@@ -0,0 +1,84 @@
+{
+  "kafka": {
+    "brokers": [
+        {% for host in advertised_ips %}
+        "{{ host }}:{{ redpanda_kafka_port }}"{% if not loop.last %},{% endif %}
+        {% endfor %}
+    ],
+    "schemaRegistry": {
+      "enabled": true,
+      "urls": [
+        {% for host in advertised_ips %}
+        "{% if enable_tls | default(false) %}https{% else %}http{% endif %}://{{ host }}:{{ redpanda_schema_registry_port }}"{% if not loop.last %},{% endif %}
+        {% endfor %}
+      ]{% if enable_tls | default(false) %},
+      "tls": {
+        "enabled": true,
+        "caFilepath": "{{ redpanda_truststore_file }}",
+        "certFilepath": "{{ redpanda_cert_file }}",
+        "keyFilepath": "{{ redpanda_key_file }}",
+        "insecureSkipTlsVerify": false
+      }{% endif %}
+    },
+    "protobuf": {
+      "enabled": true,
+      "schemaRegistry": {
+        "enabled": true,
+        "refreshInterval": "5m"
+      }
+    }{% if enable_tls | default(false) %},
+  "tls": {
+    "enabled": true,
+    "caFilepath": "{{ redpanda_truststore_file }}",
+    "certFilepath": "{{ redpanda_cert_file }}",
+    "keyFilepath": "{{ redpanda_key_file }}",
+    "insecureSkipTlsVerify": false
+    }
+  {% endif %}
+  },
+  "redpanda": {
+    "adminApi": {
+      "enabled": true,
+      "urls": [
+        {% for host in advertised_ips %}
+        "{% if enable_tls | default(false) %}https{% else %}http{% endif %}://{{ host }}:{{ redpanda_admin_api_port }}"{% if not loop.last %},{% endif %}
+        {% endfor %}
+      ]{% if enable_tls | default(false) %},
+      "tls": {
+        "enabled": true,
+        "caFilepath": "{{ redpanda_truststore_file }}",
+        "certFilepath": "{{ redpanda_cert_file }}",
+        "keyFilepath": "{{ redpanda_key_file }}",
+        "insecureSkipTlsVerify": false
+      }
+      {% endif %}
+    }
+  }{% if enable_tls | default(false) %},
+  "server": {
+    "listenPort": {{ http_port }},
+    "httpsListenPort": {{ https_port }},
+    "tls": {
+      "enabled": true,
+      "certFilepath": "{{ console_cert_file }}",
+      "keyFilepath": "{{ console_key_file }}"
+    }
+  }{% endif %}{% if kafka_connect_advertised_ips is defined %},
+    "connect": {
+    "enabled": true,
+    "clusters":
+    [
+      {
+        "name": "Connect",
+        "url": "{% if enable_tls | default(false) %}https{% else %}http{% endif %}://{{ kafka_connect_advertised_ips[0] }}:{{ kafka_connect_port }}"
+        {% if enable_tls | default(false) %},
+        "tls": {
+          "enabled": true,
+          "caFilepath": "{{ redpanda_truststore_file }}",
+          "certFilepath": "{{ redpanda_cert_file }}",
+          "keyFilepath": "{{ redpanda_key_file }}",
+          "insecureSkipTlsVerify": false
+        }
+        {% endif %}
+    }]
+}{% endif %}
+}
diff --git a/roles/redpanda_console/tests/defaults_test.py b/roles/redpanda_console/tests/defaults_test.py
new file mode 100644
index 00000000..8196b451
--- /dev/null
+++ b/roles/redpanda_console/tests/defaults_test.py
@@ -0,0 +1,124 @@
+import unittest
+import os
+import yaml
+from jinja2 import Environment, FileSystemLoader
+import json
+
+
+class TestDefaultsTemplate(unittest.TestCase):
+    def test_defaults_template(self):
+        # Get the absolute path of the current file
+        current_dir = os.path.dirname(os.path.abspath(__file__))
+
+        # Construct the path to the templates directory
+        templates_dir = os.path.join(current_dir, '..', 'templates')
+
+        # Construct the path to the defaults/main.yml file
+        defaults_file = os.path.join(current_dir, '..', 'defaults', 'main.yml')
+
+        # Load the defaults from the YAML file
+        with open(defaults_file, 'r') as f:
+            defaults = yaml.safe_load(f)
+
+        # Create a Jinja2 environment and load the template from file
+        env = Environment(loader=FileSystemLoader(templates_dir))
+        template = env.get_template('defaults.j2')
+
+        # Define the hostvars and groups for rendering the template
+        hostvars = {
+            '35.91.106.231': {
+                'ansible_host': '35.91.106.231',
+            },
+            '35.88.129.205': {
+                'ansible_host': '35.88.129.205',
+            },
+            '54.190.184.126': {
+                'ansible_host': '54.190.184.126',
+            }
+        }
+        advertised_ips = ["35.91.106.231", "35.88.129.205", "54.190.184.126"]
+        groups = {
+            'redpanda': ['35.91.106.231', '35.88.129.205', '54.190.184.126']
+        }
+
+        # Set up some test cases for enable_tls
+        scenarios = [
+            {'enable_tls': False, 'name': 'without TLS'},
+            {'enable_tls': True, 'name': 'with TLS'}
+        ]
+
+        for scenario in scenarios:
+            # Merge scenario settings with defaults
+            test_vars = {**defaults, **scenario}
+
+            # Render the template
+            rendered_template = template.render(
+                hostvars=hostvars,
+                advertised_ips=advertised_ips,
+                groups=groups,
+                **test_vars
+            )
+
+            # Parse the rendered JSON to validate its structure
+            try:
+                parsed_json = json.loads(rendered_template)
+
+                # Test that expected sections exist in the JSON
+                self.assertIn("kafka", parsed_json)
+                self.assertIn("brokers", parsed_json["kafka"])
+                self.assertIn("redpanda", parsed_json)
+                self.assertIn("adminApi", parsed_json["redpanda"])
+                self.assertIn("schemaRegistry", parsed_json)
+
+                # Test that TLS settings are present when enable_tls is True
+                if scenario['enable_tls']:
+                    self.assertIn("tls", parsed_json["kafka"])
+                    self.assertTrue(parsed_json["kafka"]["tls"]["enabled"])
+                    self.assertIn("tls", parsed_json["redpanda"]["adminApi"])
+                    self.assertTrue(parsed_json["redpanda"]["adminApi"]["tls"]["enabled"])
+                    self.assertIn("server", parsed_json)
+                    self.assertIn("tls", parsed_json["server"])
+                    self.assertTrue(parsed_json["server"]["tls"]["enabled"])
+                else:
+                    self.assertNotIn("tls", parsed_json.get("kafka", {}))
+
+                # Test kafka brokers
+                self.assertEqual(len(parsed_json["kafka"]["brokers"]), 3)
+                for i, ip in enumerate(advertised_ips):
+                    expected_broker = f"{ip}:{defaults['redpanda_kafka_port']}"
+                    self.assertEqual(parsed_json["kafka"]["brokers"][i], expected_broker)
+
+                # Test admin API URLs
+                admin_urls = parsed_json["redpanda"]["adminApi"]["urls"]
+                self.assertEqual(len(admin_urls), 3)
+                protocol = "https" if scenario['enable_tls'] else "http"
+                for i, ip in enumerate(advertised_ips):
+                    expected_url = f"{protocol}://{ip}:{defaults['redpanda_admin_api_port']}"
+                    self.assertEqual(admin_urls[i], expected_url)
+
+                # Test schema registry URLs
+                schema_urls = parsed_json["schemaRegistry"]["urls"]
+                self.assertEqual(len(schema_urls), 3)
+                for i, ip in enumerate(advertised_ips):
+                    expected_url = f"{protocol}://{ip}:{defaults['redpanda_schema_registry_port']}"
+                    self.assertEqual(schema_urls[i], expected_url)
+
+                # Test kafkaConnect settings if applicable
+                if "connect" in groups:
+                    self.assertIn("kafkaConnect", parsed_json)
+                    self.assertTrue(parsed_json["kafkaConnect"]["enabled"])
+                    self.assertEqual(len(parsed_json["kafkaConnect"]["clusters"]), 1)
+                    self.assertEqual(parsed_json["kafkaConnect"]["clusters"][0]["name"], "Connect")
+
+                print(f"Successfully validated scenario: {scenario['name']}")
+
+            except json.JSONDecodeError as e:
+                self.fail(f"Failed to parse rendered JSON template: {e}")
+                print(rendered_template)  # Print the template to debug
+
+        # Print the rendered template for debugging
+        print(rendered_template)
+
+
+if __name__ == '__main__':
+    unittest.main()
diff --git a/roles/redpanda_console/tests/pre_v3_defaults_test.py b/roles/redpanda_console/tests/pre_v3_defaults_test.py
new file mode 100644
index 00000000..08f1da7e
--- /dev/null
+++ b/roles/redpanda_console/tests/pre_v3_defaults_test.py
@@ -0,0 +1,143 @@
+import unittest
+import os
+import yaml
+from jinja2 import Environment, FileSystemLoader
+import json
+
+
+class TestPreV3DefaultsTemplate(unittest.TestCase):
+    def test_pre_v3_defaults_template(self):
+        # Get the absolute path of the current file
+        current_dir = os.path.dirname(os.path.abspath(__file__))
+
+        # Construct the path to the templates directory
+        templates_dir = os.path.join(current_dir, '..', 'templates')
+
+        # Construct the path to the defaults/main.yml file
+        defaults_file = os.path.join(current_dir, '..', 'defaults', 'main.yml')
+
+        # Load the defaults from the YAML file
+        with open(defaults_file, 'r') as f:
+            defaults = yaml.safe_load(f)
+
+        # Create a Jinja2 environment and load the template from file
+        env = Environment(loader=FileSystemLoader(templates_dir))
+        template = env.get_template('pre_v3_defaults.j2')
+
+        # Define the hostvars and groups for rendering the template
+        hostvars = {
+            '35.91.106.231': {
+                'ansible_host': '35.91.106.231',
+            },
+            '35.88.129.205': {
+                'ansible_host': '35.88.129.205',
+            },
+            '54.190.184.126': {
+                'ansible_host': '54.190.184.126',
+            }
+        }
+        advertised_ips = ["35.91.106.231", "35.88.129.205", "54.190.184.126"]
+        groups = {
+            'redpanda': ['35.91.106.231', '35.88.129.205', '54.190.184.126']
+        }
+
+        # Set up some test cases for enable_tls and connect presence
+        scenarios = [
+            {'enable_tls': False, 'kafka_connect_present': False, 'name': 'without TLS, no connect'},
+            {'enable_tls': True, 'kafka_connect_present': False, 'name': 'with TLS, no connect'},
+            {'enable_tls': False, 'kafka_connect_present': True, 'name': 'without TLS, with connect'},
+            {'enable_tls': True, 'kafka_connect_present': True, 'name': 'with TLS, with connect'}
+        ]
+
+        for scenario in scenarios:
+            # Merge scenario settings with defaults
+            test_vars = {**defaults, **{'enable_tls': scenario['enable_tls']}}
+
+            # Add kafka_connect_advertised_ips if connect is present
+            if scenario['kafka_connect_present']:
+                test_vars['kafka_connect_advertised_ips'] = advertised_ips
+                groups['connect'] = groups['redpanda']
+
+            # Render the template
+            rendered_template = template.render(
+                hostvars=hostvars,
+                advertised_ips=advertised_ips,
+                groups=groups,
+                **test_vars
+            )
+
+            # Parse the rendered JSON to validate its structure
+            try:
+                parsed_json = json.loads(rendered_template)
+
+                # Test that expected sections exist in the JSON
+                self.assertIn("kafka", parsed_json)
+                self.assertIn("brokers", parsed_json["kafka"])
+                self.assertIn("redpanda", parsed_json)
+                self.assertIn("adminApi", parsed_json["redpanda"])
+                self.assertIn("schemaRegistry", parsed_json["kafka"])
+                self.assertIn("protobuf", parsed_json["kafka"])
+
+                # Test that TLS settings are present when enable_tls is True
+                if scenario['enable_tls']:
+                    self.assertIn("tls", parsed_json["kafka"])
+                    self.assertTrue(parsed_json["kafka"]["tls"]["enabled"])
+                    self.assertIn("tls", parsed_json["redpanda"]["adminApi"])
+                    self.assertTrue(parsed_json["redpanda"]["adminApi"]["tls"]["enabled"])
+                    self.assertIn("server", parsed_json)
+                    self.assertIn("tls", parsed_json["server"])
+                    self.assertTrue(parsed_json["server"]["tls"]["enabled"])
+                else:
+                    self.assertNotIn("tls", parsed_json.get("kafka", {}))
+
+                # Test kafka brokers
+                self.assertEqual(len(parsed_json["kafka"]["brokers"]), 3)
+                for i, ip in enumerate(advertised_ips):
+                    expected_broker = f"{ip}:{defaults['redpanda_kafka_port']}"
+                    self.assertEqual(parsed_json["kafka"]["brokers"][i], expected_broker)
+
+                # Test admin API URLs
+                admin_urls = parsed_json["redpanda"]["adminApi"]["urls"]
+                self.assertEqual(len(admin_urls), 3)
+                protocol = "https" if scenario['enable_tls'] else "http"
+                for i, ip in enumerate(advertised_ips):
+                    expected_url = f"{protocol}://{ip}:{defaults['redpanda_admin_api_port']}"
+                    self.assertEqual(admin_urls[i], expected_url)
+
+                # Test schema registry URLs
+                schema_urls = parsed_json["kafka"]["schemaRegistry"]["urls"]
+                self.assertEqual(len(schema_urls), 3)
+                for i, ip in enumerate(advertised_ips):
+                    expected_url = f"{protocol}://{ip}:{defaults['redpanda_schema_registry_port']}"
+                    self.assertEqual(schema_urls[i], expected_url)
+
+                # Test connect settings if enabled
+                if scenario['kafka_connect_present']:
+                    self.assertIn("connect", parsed_json)
+                    self.assertTrue(parsed_json["connect"]["enabled"])
+                    self.assertEqual(len(parsed_json["connect"]["clusters"]), 1)
+                    self.assertEqual(parsed_json["connect"]["clusters"][0]["name"], "Connect")
+
+                    # Check connect URL
+                    connect_url = parsed_json["connect"]["clusters"][0]["url"]
+                    expected_url = f"{protocol}://{advertised_ips[0]}:{defaults['kafka_connect_port']}"
+                    self.assertEqual(connect_url, expected_url)
+
+                    # Check TLS settings in connect if applicable
+                    if scenario['enable_tls']:
+                        self.assertIn("tls", parsed_json["connect"]["clusters"][0])
+                        self.assertTrue(parsed_json["connect"]["clusters"][0]["tls"]["enabled"])
+                else:
+                    self.assertNotIn("connect", parsed_json)
+
+                print(f"Successfully validated scenario: {scenario['name']}")
+
+            except json.JSONDecodeError as e:
+                self.fail(f"Failed to parse rendered JSON template: {e}\n{rendered_template}")
+
+        # Print the rendered template for debugging
+        print(rendered_template)
+
+
+if __name__ == '__main__':
+    unittest.main()
diff --git a/roles/redpanda_console/tests/requirements.txt b/roles/redpanda_console/tests/requirements.txt
new file mode 100644
index 00000000..7d7800ae
--- /dev/null
+++ b/roles/redpanda_console/tests/requirements.txt
@@ -0,0 +1,2 @@
+jinja2==3.1.3
+pyyaml==6.0.1
diff --git a/roles/redpanda_console/vars/main.yml b/roles/redpanda_console/vars/main.yml
index 1d073db8..22ee6b9f 100644
--- a/roles/redpanda_console/vars/main.yml
+++ b/roles/redpanda_console/vars/main.yml
@@ -1,3 +1,6 @@
 ---
 console_config_templates:
+  - template: pre_v3_defaults.j2
+    condition: "{{ use_pre_v3_template | default(false) }}"
   - template: defaults.j2
+    condition: "{{ not (use_pre_v3_template | default(false)) }}"


Reviews:
user: login: RafalKorepta
state: COMMENTED
author_association: NONE
submitted_at: 2025-03-14T15:09:19Z
user: login: RafalKorepta
state: COMMENTED
author_association: NONE
submitted_at: 2025-03-14T15:10:10Z
user: login: RafalKorepta
state: COMMENTED
author_association: NONE
submitted_at: 2025-03-14T15:10:46Z
user: login: RafalKorepta
state: COMMENTED
author_association: NONE
submitted_at: 2025-03-14T15:10:55Z
user: login: RafalKorepta
state: APPROVED
author_association: NONE
submitted_at: 2025-03-14T16:23:47Z
user: login: gene-redpanda
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-14T16:50:41Z

Review comments:
diff_hunk: @@ -85,6 +56,23 @@
           "insecureSkipTlsVerify": false
         }
         {% endif %}
-    }]
-}{% endif %}
+      }
+    ]
+  },{% endif %}
+  "schemaRegistry": {
+    "enabled": true,
+    "urls": [
+      {% for host in advertised_ips %}
+      "{% if enable_tls | default(false) %}https{% else %}http{% endif %}://{{ host }}:{{ redpanda_schema_registry_port }}"{% if not loop.last %},{% endif %}
+      {% endfor %}
+    ]{% if enable_tls | default(false) %},
+    "tls": {
+      "enabled": true,
+      "caFilepath": "{{ redpanda_truststore_file }}",
+      "certFilepath": "{{ redpanda_cert_file }}",
+      "keyFilepath": "{{ redpanda_key_file }}",
+      "insecureSkipTlsVerify": false
+    }
+    {% endif %}
+  },
path: roles/redpanda_console/templates/defaults.j2
user: login: RafalKorepta
body: Maybe this is the issue
```suggestion
  }
```
created_at: 2025-03-14T15:09:19Z
author_association: NONE
diff_hunk: @@ -85,6 +56,23 @@
           "insecureSkipTlsVerify": false
         }
         {% endif %}
-    }]
-}{% endif %}
+      }
+    ]
+  },{% endif %}
path: roles/redpanda_console/templates/defaults.j2
user: login: RafalKorepta
body: This comma might be the issue too 🤷 
```suggestion
  }{% endif %},
```
created_at: 2025-03-14T15:10:10Z
author_association: NONE
diff_hunk: @@ -59,23 +31,22 @@
       }
       {% endif %}
     }
-  }{% if enable_tls | default(false) %},
+  },{% if enable_tls | default(false) %}
   "server": {
     "listenPort": {{ http_port }},
     "httpsListenPort": {{ https_port }},
     "tls": {
       "enabled": true,
       "certFilepath": "{{ console_cert_file }}",
-      "keyFilepath": "{{ console_key_file }}",
+      "keyFilepath": "{{ console_key_file }}"
     }
-  }{% endif %}{% if connect_advertised_ips is defined %},
-    "connect": {
+  },{% endif %}{% if kafka_connect_advertised_ips is defined %}
path: roles/redpanda_console/templates/defaults.j2
user: login: RafalKorepta
body: ```suggestion
  }{% endif %}{% if kafka_connect_advertised_ips is defined %},
```
created_at: 2025-03-14T15:10:46Z
author_association: NONE
diff_hunk: @@ -59,23 +31,22 @@
       }
       {% endif %}
     }
-  }{% if enable_tls | default(false) %},
+  },{% if enable_tls | default(false) %}
path: roles/redpanda_console/templates/defaults.j2
user: login: RafalKorepta
body: ```suggestion
  }{% if enable_tls | default(false) %},
```
created_at: 2025-03-14T15:10:54Z
author_association: NONE
diff_hunk: @@ -59,23 +31,22 @@
       }
       {% endif %}
     }
-  }{% if enable_tls | default(false) %},
+  },{% if enable_tls | default(false) %}
   "server": {
     "listenPort": {{ http_port }},
     "httpsListenPort": {{ https_port }},
     "tls": {
       "enabled": true,
       "certFilepath": "{{ console_cert_file }}",
-      "keyFilepath": "{{ console_key_file }}",
+      "keyFilepath": "{{ console_key_file }}"
     }
-  }{% endif %}{% if connect_advertised_ips is defined %},
-    "connect": {
+  },{% endif %}{% if kafka_connect_advertised_ips is defined %}
path: roles/redpanda_console/templates/defaults.j2
user: login: gene-redpanda
body: I think I'm going to keep them where they are. It does matter, but it works either way and it's at least consistent now. Plus I find it easier to conceptualize the comma being included with the object rather than at the start of the next object after an IF. To resolve the issue of "what about trailing commas on the last object" I moved the static schema registry object to the end so there's always something there which removes the issue as a concern.
created_at: 2025-03-14T16:50:41Z
author_association: MEMBER

PR comments:
user: login: gene-redpanda
created_at: 2025-03-13T19:54:33Z
body: Test outcome from temp PR: https://github.com/redpanda-data/deployment-automation/pull/235

This is the sorta janky way I generate tests from deployment-automation when doing ansible work as it allows me to trigger CI against a particular branch of RAC. There are automated ways to do something similar I'm sure but they'd save very little dev time. 
author_association: MEMBER
user: login: RafalKorepta
created_at: 2025-03-14T15:03:45Z
body: ```
F
--
  | ======================================================================
  | FAIL: test_defaults_template (__main__.TestDefaultsTemplate.test_defaults_template)
  | ----------------------------------------------------------------------
  | Traceback (most recent call last):
  | File "/app/tests/defaults_test.py", line 64, in test_defaults_template
  | parsed_json = json.loads(rendered_template)
  | File "/usr/local/lib/python3.13/json/__init__.py", line 346, in loads
  | return _default_decoder.decode(s)
  | ~~~~~~~~~~~~~~~~~~~~~~~^^^
  | File "/usr/local/lib/python3.13/json/decoder.py", line 345, in decode
  | obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  | ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  | File "/usr/local/lib/python3.13/json/decoder.py", line 361, in raw_decode
  | obj, end = self.scan_once(s, idx)
  | ~~~~~~~~~~~~~~^^^^^^^^
  | json.decoder.JSONDecodeError: Illegal trailing comma before end of object: line 38 column 4 (char 611)
  |  
  | During handling of the above exception, another exception occurred:
  |  
  | Traceback (most recent call last):
  | File "/app/tests/defaults_test.py", line 116, in test_defaults_template
  | self.fail(f"Failed to parse rendered JSON template: {e}")
  | ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  | AssertionError: Failed to parse rendered JSON template: Illegal trailing comma before end of object: line 38 column 4 (char 611)
```

https://buildkite.com/redpanda/redpanda-ansible-collection/builds/66#019590fd-6214-43af-af81-725d39baa1cb/235-283
author_association: NONE
user: login: RafalKorepta
created_at: 2025-03-14T15:05:12Z
body: I wonder why CI passed?
author_association: NONE
user: login: RafalKorepta
created_at: 2025-03-14T15:08:43Z
body: NIT: When `json.loads(rendered_template)` fails it would be awesome to not only get the following error, but see at least the context (few lines above and below the error line 38).
```
AssertionError: Failed to parse rendered JSON template: Illegal trailing comma before end of object: line 38 column 4 (char 611)
```

That way it would be trivial to find the line in `roles/redpanda_console/templates/defaults.j2`
author_association: NONE