Chapter 2. Deploying the console Deploy the console using the dedicated operator. After installing the operator, you can create instances of the console. For each console instance, the operator needs a Prometheus instance to collect and display Kafka cluster metrics. You can configure the console to use an existing Prometheus source, like OpenShift's built-in user workload monitoring. If no source is set, the operator creates a private Prometheus instance when the console is deployed. However, this default setup is not recommended for production and should only be used for development or evaluation purposes. 2.1. Deployment prerequisites To deploy the console, you need the following: An OpenShift 4.14 to 4.18 cluster. The oc command-line tool is installed and configured to connect to the OpenShift cluster. Access to the OpenShift cluster using an account with cluster-admin permissions, such as system-admin . A Kafka cluster managed by Streams for Apache Kafka, running on the OpenShift cluster. Example files are provided for installing a Kafka cluster managed by Streams for Apache Kafka, along with a Kafka user representing the console. These files offer the fastest way to set up and try the console, but you can also use your own Streams for Apache Kafka deployment. 2.1.1. Using your own Kafka cluster If you use your own Streams for Apache Kafka deployment, verify the configuration by comparing it with the example deployment files provided with the console. For each Kafka cluster, the Kafka resource used to install the cluster must be configured with the following: Sufficient authorization for the console to connect Metrics properties for the console to be able to display certain data The metrics configuration must match the properties specified in the example Kafka ( console-kafka ) and ConfigMap ( console-kafka-metrics ) resources. 2.1.2. Deploying a new Kafka cluster If you already have Streams for Apache Kafka installed but want to create a new Kafka cluster for use with the console, example deployment resources are available to help you get started. These resources create the following: A Kafka cluster in KRaft mode with SCRAM-SHA-512 authentication. A Strimzi KafkaNodePool resource to manage the cluster nodes. A KafkaUser resource to enable authenticated and authorized console connections to the Kafka cluster. The KafkaUser custom resource in the 040-KafkaUser-console-kafka-user1.yaml file includes the necessary ACL types to provide authorized access for the console to the Kafka cluster. The minimum required ACL rules are configured as follows: Describe , DescribeConfigs permissions for the cluster resource Read , Describe , DescribeConfigs permissions for all topic resources Read , Describe permissions for all group resources Note To ensure the console has the necessary access to function, a minimum level of authorization must be configured for the principal used in each Kafka cluster connection. The specific permissions may vary based on the authorization framework in use, such as ACLs, Keycloak authorization, OPA, or a custom solution. When configuring the KafkaUser authentication and authorization, ensure they match the corresponding Kafka configuration: KafkaUser.spec.authentication should match Kafka.spec.kafka.listeners[*].authentication . KafkaUser.spec.authorization should match Kafka.spec.kafka.authorization . Prerequisites An OpenShift 4.14 to 4.18 cluster. Access to the OpenShift Container Platform web console using an account with cluster-admin permissions, such as system:admin . The oc command-line tool is installed and configured to connect to the OpenShift cluster. Procedure Download and extract the console installation artifacts. The artifacts are included with installation and example files available from the Streams for Apache Kafka software downloads page . The artifacts provide the deployment YAML files to the install the Kafka cluster. Use the sample installation files located in examples/console/resources/kafka . Set environment variables to update the installation files: export NAMESPACE=kafka 1 export LISTENER_TYPE=route 2 export CLUSTER_DOMAIN=<domain_name> 3 1 The namespace in which you want to deploy the Kafka operator. 2 The listener type used to expose Kafka to the console. 3 The cluster domain name for your OpenShift cluster. In this example, the namespace variable is defined as kafka and the listener type is route . Install the Kafka cluster. Run the following command to apply the YAML files and deploy the Kafka cluster to the defined namespace: cat examples/console/resources/kafka/*.yaml | envsubst | kubectl apply -n USD{NAMESPACE} -f - This command reads the YAML files, replaces the namespace environment variables, and applies the resulting configuration to the specified OpenShift namespace. Check the status of the deployment: oc get pods -n kafka Output shows the operators and cluster readiness NAME READY STATUS RESTARTS strimzi-cluster-operator 1/1 Running 0 console-kafka-console-nodepool-0 1/1 Running 0 console-kafka-console-nodepool-1 1/1 Running 0 console-kafka-console-nodepool-2 1/1 Running 0 console-kafka is the name of the cluster. console-nodepool is the name of the node pool. A node ID identifies the nodes created. With the default deployment, you install three nodes. READY shows the number of replicas that are ready/expected. The deployment is successful when the STATUS displays as Running. 2.2. Installing the console operator Install the console operator using one of the following methods: From the OperatorHub in the OpenShift web console Using the OpenShift CLI By applying a Console Custom Resource Definition (CRD) The recommended approach is to install the operator using either the OpenShift web console or the OpenShift CLI ( oc ), both of which are supported by the Operator Lifecycle Manager (OLM). If using the OLM is not suitable for your environment, you can install the operator by applying the CRD directly. 2.2.1. Installing the operator from the OperatorHub This procedure describes how to install and subscribe to the Streams for Apache Kafka Console operator using the OperatorHub in the OpenShift Container Platform web console. The procedure describes how to create a project and install the operator to that project. A project is a representation of a namespace. For manageability, it is a good practice to use namespaces to separate functions. Warning Make sure you use the appropriate update channel. If you are on a supported version of OpenShift, installing the operator from the default alpha channel is generally safe. However, we do not recommend enabling automatic updates on the alpha channel. An automatic upgrade will skip any necessary steps prior to upgrade. Use automatic upgrades only on version-specific channels. Prerequisites Deployment prerequisites . Procedure Navigate in the OpenShift web console to the Home > Projects page and create a project (namespace) for the installation. We use a project named streams-kafka-console in this example. Navigate to the Operators > OperatorHub page. Scroll or type a keyword into the Filter by keyword box to find the Streams for Apache Kafka Console operator. The operator is located in the Streaming & Messaging category. Click Streams for Apache Kafka Console to display the operator information. Read the information about the operator and click Install . On the Install Operator page, choose from the following installation and update options: Update Channel : Choose the update channel for the operator. The (default) alpha channel contains all the latest updates and releases, including major, minor, and micro releases, which are assumed to be well tested and stable. An amq-streams- X .x channel contains the minor and micro release updates for a major release, where X is the major release version number. An amq-streams- X.Y .x channel contains the micro release updates for a minor release, where X is the major release version number and Y is the minor release version number. Installation Mode : Install the operator to all namespaces in the OpenShift cluster. A single instance of the operator will watch and manage consoles created throughout the OpenShift cluster. Update approval : By default, the Streams for Apache Kafka Console operator is automatically upgraded to the latest console version by the Operator Lifecycle Manager (OLM). Optionally, select Manual if you want to manually approve future upgrades. For more information on operators, see the OpenShift documentation . Click Install to install the operator to your selected namespace. After the operator is ready for use, navigate to Operators > Installed Operators to verify that the operator has installed to the selected namespace. The status will show as Succeeded . Use the console operator to deploy the console and connect to a Kafka cluster . 2.2.2. Installing the operator using the OpenShift CLI This procedure describes how to install the Streams for Apache Kafka Console operator using the OpenShift CLI ( oc ). Prerequisites Deployment prerequisites . Procedure Download and extract the console installation artifacts. The artifacts are included with installation and example files available from the Streams for Apache Kafka software downloads page . The artifacts provide the deployment YAML files to the install the console. Set an environment variable to define the namespace where you want to install the operator: export NAMESPACE=operator-namespace In this example, the namespace variable is defined as operator-namespace . Install the console operator with the OLM. Use the sample installation files located in install/console-operator/olm . These files install the operator with cluster-wide scope, allowing it to manage console resources across all namespaces. Run the following command to apply the YAML files and deploy the operator to the defined namespace: cat install/console-operator/olm/*.yaml | envsubst | kubectl apply -n USD{NAMESPACE} -f - This command reads the YAML files, replaces the namespace environment variables, and applies the resulting configuration to the specified OpenShift namespace. Check the status of the deployment: oc get pods -n operator-namespace Output shows the deployment name and readiness NAME READY STATUS RESTARTS console-operator 1/1 Running 1 READY shows the number of replicas that are ready/expected. The deployment is successful when the STATUS displays as Running . Use the console operator to deploy the console and connect to a Kafka cluster . 2.2.3. Deploying the console operator using a CRD This procedure describes how to install the Streams for Apache Kafka Console operator using a Custom Resource Definition (CRD). Prerequisites Deployment prerequisites . Procedure Download and extract the console installation artifacts. The artifacts are included with installation and example files available from the Streams for Apache Kafka software downloads page . The artifacts include a Custom Resource Definition (CRD) file ( console-operator.yaml ) to install the operator without the OLM. Set an environment variable to define the namespace where you want to install the operator: export NAMESPACE=operator-namespace In this example, the namespace variable is defined as operator-namespace . Install the console operator with the CRD. Use the sample installation files located in install/console-operator/non-olm . These resources install the operator with cluster-wide scope, allowing it to manage console resources across all namespaces. Run the following command to apply the YAML file: cat install/console-operator/non-olm/console-operator.yaml | envsubst | kubectl apply -n USD{NAMESPACE} -f - This command reads the YAML file, replaces the namespace environment variables, and applies the resulting configuration to the specified OpenShift namespace. Check the status of the deployment: oc get pods -n operator-namespace Output shows the deployment name and readiness NAME READY STATUS RESTARTS console-operator 1/1 Running 1 READY shows the number of replicas that are ready/expected. The deployment is successful when the STATUS displays as Running . Use the console operator to deploy the console and connect to a Kafka cluster . 2.3. Deploying and connecting the console to a Kafka cluster Use the console operator to deploy the Streams for Apache Kafka Console to the same OpenShift cluster as a Kafka cluster managed by Streams for Apache Kafka. Use the console to connect to the Kafka cluster. Prerequisites Deployment prerequisites . The console operator is deployed to the OpenShift cluster. Procedure Create a Console custom resource in the desired namespace. If you deployed the example Kafka cluster provided with the installation artifacts, you can use the configuration specified in the examples/console/resources/console/010-Console-example.yaml configuration file unchanged. Otherwise, configure the resource to connect to your Kafka cluster. Example console configuration apiVersion: console.streamshub.github.com/v1alpha1 kind: Console metadata: name: my-console spec: hostname: my-console.<cluster_domain> 1 kafkaClusters: - name: console-kafka 2 namespace: kafka 3 listener: secure 4 properties: values: [] 5 valuesFrom: [] 6 credentials: kafkaUser: name: console-kafka-user1 7 1 Hostname to access the console by HTTP. 2 Name of the Kafka resource representing the cluster. 3 Namespace of the Kafka cluster. 4 Listener to expose the Kafka cluster for console connection. 5 (Optional) Add connection properties if needed. 6 (optional) References to config maps or secrets, if needed. 7 (Optional) Kafka user created for authenticated access to the Kafka cluster. Apply the Console configuration to install the console. In this example, the console is deployed to the console-namespace namespace: kubectl apply -f examples/console/resources/console/010-Console-example.yaml -n console-namespace Check the status of the deployment: oc get pods -n console-namespace Output shows the deployment name and readiness NAME READY STATUS RUNNING console-kafka 1/1 1 1 Access the console. When the console is running, use the hostname specified in the Console resource ( spec.hostname ) to access the user interface. 2.3.1. Using an OIDC provider to secure access to Kafka clusters Enable secure console connections to Kafka clusters using an OIDC provider. Configure the console deployment to configure connections to any Identity Provider (IdP), such as Keycloak or Dex, that supports OpenID Connect (OIDC). Also define the subjects and roles for user authorization. The security profiles can be configured for all Kafka cluster connections on a global level, though you can add roles and rules for specific Kafka clusters. An example configuration is provided in the following file: examples/console/resources/console/console-security-oidc.yaml . The configuration introduces the following additional properties for console deployment: security Properties that define the connection details for the console to connect with the OIDC provider. subjects Specifies the subjects (users or groups) and their roles in terms of JWT claims or explicit subject names, determining access permissions. roles Defines the roles and associated access rules for users, specifying which resources (like Kafka clusters) they can interact with and what operations they are permitted to perform. Example security configuration for all clusters apiVersion: console.streamshub.github.com/v1alpha1 kind: Console metadata: name: my-console spec: hostname: my-console.<cluster_domain> security: oidc: authServerUrl: <OIDC_discovery_URL> 1 clientId: <client_id> 2 clientSecret: 3 valueFrom: secretKeyRef: name: my-oidc-secret key: client-secret subjects: - claim: groups 4 include: 5 - <team_name_1> - <team_name_2> roleNames: 6 - developers - claim: groups include: - <team_name_3> roleNames: - administrators - include: 7 - <user_1> - <user_2> roleNames: - administrators roles: - name: developers rules: - resources: 8 - kafkas - resourceNames: 9 - <dev_cluster_a> - <dev_cluster_b> - privileges: 10 - '*' - name: administrators rules: - resources: - kafkas - privileges: - '*' kafkaClusters: - name: console-kafka namespace: kafka listener: secure credentials: kafkaUser: name: console-kafka-user1 1 URL for OIDC provider discovery. 2 Client ID for OIDC authentication to identify the client. 3 Client secret and client ID used for authentication. 4 JWT claim types or names to identify the users or groups. 5 Users or groups included under the specified claim. 6 Roles assigned to the specified users or groups. 7 Specific users included by name when no claim is specified. 8 Resources that the assigned role can access. 9 Specific resource names accessible by the assigned role. 10 Privileges granted to the assigned role for the specified resources. If you want to specify roles and rules for individual Kafka clusters, add the details under kafka.clusters[].security.roles[] . In the following example, the console-kafka cluster allows developers to list and view selected Kafka resources. Administrators can also update certain resources. Example security configuration for an individual cluster apiVersion: console.streamshub.github.com/v1alpha1 kind: Console metadata: name: my-console spec: hostname: my-console.<cluster_domain> # ... kafkaClusters: - name: console-kafka namespace: kafka listener: secure credentials: kafkaUser: name: console-kafka-user1 security: roles: - name: developers rules: - resources: - topics - topics/records - consumerGroups - rebalances - privileges: - get - list - name: administrators rules: - resources: - topics - topics/records - consumerGroups - rebalances - nodes/configs - privileges: - get - list - resources: - consumerGroups - rebalances - privileges: - update 2.3.2. Enabling a metrics provider Configure the console deployment to enable a metrics provider. You can set up configuration to use one of the following sources to scrape metrics from Kafka clusters using Prometheus: OpenShift's built-in user workload monitoring Use OpenShift's workload monitoring, incorporating the Prometheus operator, to monitor console services and workloads without the need for an additional monitoring solution. A standalone Prometheus instance Provide the details and credentials to connect with your own Prometheus instance. An embedded Prometheus instance (default) Deploy a private Prometheus instance for use only by the console instance. The instance is configured to retrieve metrics from all Streams for Apache Kafka instances in the same OpenShift cluster. Using embedded metrics is intended for evaluation or development environments and should not be used in production scenarios. Example configuration for OpenShift monitoring and a standalone Prometheus instance is provided in the following files: examples/console/resources/console/console-openshift-metrics.yaml examples/console/resources/console/console-standalone-prometheus.yaml The configuration introduces the metricsSources properties for enabling monitoring. Use the type property to define the source: openshift-monitoring standalone (Prometheus) embedded (Prometheus) Assign the metrics source to a Kafka cluster using the kafkaClusters.metricsSource property. The configuration for openshift-monitoring and embedded requires no further configuration besides the type . Example metrics configuration for Openshift monitoring apiVersion: console.streamshub.github.com/v1alpha1 kind: Console metadata: name: my-console spec: hostname: my-console.<cluster_domain> # ... metricsSources: - name: my-ocp-prometheus type: openshift-monitoring kafkaClusters: - name: console-kafka namespace: kafka listener: secure metricsSource: my-ocp-prometheus credentials: kafkaUser: name: console-kafka-user1 # ... Example metrics configuration for standalone Prometheus monitoring apiVersion: console.streamshub.github.com/v1alpha1 kind: Console metadata: name: my-console spec: hostname: my-console.<cluster_domain> # ... metricsSources: - name: my-custom-prometheus type: standalone url: <prometheus_instance_address> 1 authentication: 2 username: my-user password: my-password trustStore: 3 type: JKS content: valueFrom: configMapKeyRef: name: my-prometheus-configmap key: ca.jks password: 4 value: changeit kafkaClusters: - name: console-kafka namespace: kafka listener: secure metricsSource: my-ocp-prometheus credentials: kafkaUser: name: console-kafka-user1 # ... 1 URL of the standalone Prometheus instance for metrics collection. 2 Optional authentication credentials for accessing the Prometheus instance. Either username and password or token can be used. 3 Optional truststore configuration for SSL, with JKS content provided through a ConfigMap or Secret ( secretKeyRef ). 4 Optional password for the truststore, either directly provided or referenced. Not recommended for production.

Commands:
export NAMESPACE=kafka 1 export LISTENER_TYPE=route 2 export CLUSTER_DOMAIN=<domain_name> 3
cat examples/console/resources/kafka/*.yaml | envsubst | kubectl apply -n USD{NAMESPACE} -f -
get pods -n kafka
NAME READY STATUS RESTARTS strimzi-cluster-operator 1/1 Running 0 console-kafka-console-nodepool-0 1/1 Running 0 console-kafka-console-nodepool-1 1/1 Running 0 console-kafka-console-nodepool-2 1/1 Running 0
export NAMESPACE=operator-namespace
cat install/console-operator/olm/*.yaml | envsubst | kubectl apply -n USD{NAMESPACE} -f -
get pods -n operator-namespace
NAME READY STATUS RESTARTS console-operator 1/1 Running 1
export NAMESPACE=operator-namespace
cat install/console-operator/non-olm/console-operator.yaml | envsubst | kubectl apply -n USD{NAMESPACE} -f -
get pods -n operator-namespace
NAME READY STATUS RESTARTS console-operator 1/1 Running 1
apiVersion: console.streamshub.github.com/v1alpha1 kind: Console metadata: name: my-console spec: hostname: my-console.<cluster_domain> 1 kafkaClusters: - name: console-kafka 2 namespace: kafka 3 listener: secure 4 properties: values: [] 5 valuesFrom: [] 6 credentials: kafkaUser: name: console-kafka-user1 7
apply -f examples/console/resources/console/010-Console-example.yaml -n console-namespace
get pods -n console-namespace
NAME READY STATUS RUNNING console-kafka 1/1 1 1
apiVersion: console.streamshub.github.com/v1alpha1 kind: Console metadata: name: my-console spec: hostname: my-console.<cluster_domain> security: oidc: authServerUrl: <OIDC_discovery_URL> 1 clientId: <client_id> 2 clientSecret: 3 valueFrom: secretKeyRef: name: my-oidc-secret key: client-secret subjects: - claim: groups 4 include: 5 - <team_name_1> - <team_name_2> roleNames: 6 - developers - claim: groups include: - <team_name_3> roleNames: - administrators - include: 7 - <user_1> - <user_2> roleNames: - administrators roles: - name: developers rules: - resources: 8 - kafkas - resourceNames: 9 - <dev_cluster_a> - <dev_cluster_b> - privileges: 10 - '*' - name: administrators rules: - resources: - kafkas - privileges: - '*' kafkaClusters: - name: console-kafka namespace: kafka listener: secure credentials: kafkaUser: name: console-kafka-user1
apiVersion: console.streamshub.github.com/v1alpha1 kind: Console metadata: name: my-console spec: hostname: my-console.<cluster_domain> # kafkaClusters: - name: console-kafka namespace: kafka listener: secure credentials: kafkaUser: name: console-kafka-user1 security: roles: - name: developers rules: - resources: - topics - topics/records - consumerGroups - rebalances - privileges: - get - list - name: administrators rules: - resources: - topics - topics/records - consumerGroups - rebalances - nodes/configs - privileges: - get - list - resources: - consumerGroups - rebalances - privileges: - update
apiVersion: console.streamshub.github.com/v1alpha1 kind: Console metadata: name: my-console spec: hostname: my-console.<cluster_domain> # metricsSources: - name: my-ocp-prometheus type: openshift-monitoring kafkaClusters: - name: console-kafka namespace: kafka listener: secure metricsSource: my-ocp-prometheus credentials: kafkaUser: name: console-kafka-user1 #
apiVersion: console.streamshub.github.com/v1alpha1 kind: Console metadata: name: my-console spec: hostname: my-console.<cluster_domain> # metricsSources: - name: my-custom-prometheus type: standalone url: <prometheus_instance_address> 1 authentication: 2 username: my-user password: my-password trustStore: 3 type: JKS content: valueFrom: configMapKeyRef: name: my-prometheus-configmap key: ca.jks password: 4 value: changeit kafkaClusters: - name: console-kafka namespace: kafka listener: secure metricsSource: my-ocp-prometheus credentials: kafkaUser: name: console-kafka-user1 #