Chapter 9. Known issues This section lists the known issues for Streams for Apache Kafka 2.7 on OpenShift. 9.1. OpenShift 4.16: Excessive generation of secrets Deploying a Kafka instance on OpenShift Container Platform (OCP) version 4.16 triggers a continuous creation of dockercfg secrets within the Kafka namespace. This issue is caused by the openshift.io/internal-registry-pull-secret-ref annotation being added to service accounts, which leads to a reconciliation loop as Streams for Apache Kafka and OpenShift repeatedly rewrite the annotation. Over time, this can result in the accumulation of thousands of unnecessary secrets. Workaround To mitigate this issue, upgrade to OCP version 4.16.4 or later, where the problem has been resolved. If upgrading is not immediately possible, a temporary workaround is to manually configure the openshift.io/internal-registry-pull-secret-ref annotation for each service account to prevent the reconciliation loop. Configuration for the annotation template: serviceAccount: metadata: annotations: openshift.io/internal-registry-pull-secret-ref: my-cluster-entity-operator-dockercfg-qxwxd 9.2. Incompatibility with RHEL 7 There are known incompatibility issues when using RHEL 7 with Kafka 3.7. As a result, RHEL 7 is no longer supported. The issues arise due to an outdated GCC (GNU Compiler Collection) version on RHEL 7, which is incompatible with the version of the RocksDB JNI library ( org.rocksdb:rocksdbjni:7.9.2 ) required by Kafka 3.7. RocksDB JNI version 7.9.2 requires a newer version of the GCC and the associated libstdc++ library than what is available on RHEL 7. Snappy compression and Kafka Streams, which depend on RocksDB, will not function correctly on RHEL 7 due to these outdated libraries. Recommendation Upgrade your clients running on RHEL 7 to RHEL 8 to ensure compatibility with Kafka 3.7 and the latest Streams for Apache Kafka features. If you wish to continue using RHEL 7, consider using Streams for Apache Kafka 2.5 LTS or 2.6. 9.3. Auto-restart of MirrorMaker 2 connectors Auto-restart of MirrorMaker 2 connectors does not work because the state is not passed to the connector operator. This will be fixed in a future release of Streams for Apache Kafka. 9.4. Streams for Apache Kafka Cluster Operator on IPv6 clusters The Streams for Apache Kafka Cluster Operator does not start on Internet Protocol version 6 (IPv6) clusters. Workaround There are two workarounds for this issue. Workaround one: Set the KUBERNETES_MASTER environment variable Display the address of the Kubernetes master node of your OpenShift Container Platform cluster: oc cluster-info Kubernetes master is running at <master_address> # ... Copy the address of the master node. List all Operator subscriptions: oc get subs -n <operator_namespace> Edit the Subscription resource for Streams for Apache Kafka: oc edit sub amq-streams -n <operator_namespace> In spec.config.env , add the KUBERNETES_MASTER environment variable, set to the address of the Kubernetes master node. For example: apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: amq-streams namespace: <operator_namespace> spec: channel: amq-streams-1.8.x installPlanApproval: Automatic name: amq-streams source: mirror-amq-streams sourceNamespace: openshift-marketplace config: env: - name: KUBERNETES_MASTER value: MASTER-ADDRESS Save and exit the editor. Check that the Subscription was updated: oc get sub amq-streams -n <operator_namespace> Check that the Cluster Operator Deployment was updated to use the new environment variable: oc get deployment <cluster_operator_deployment_name> Workaround two: Disable hostname verification List all Operator subscriptions: oc get subs -n <operator_namespace> Edit the Subscription resource for Streams for Apache Kafka: oc edit sub amq-streams -n <operator_namespace> In spec.config.env , add the KUBERNETES_DISABLE_HOSTNAME_VERIFICATION environment variable, set to true . For example: apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: amq-streams namespace: <operator_namespace> spec: channel: amq-streams-1.8.x installPlanApproval: Automatic name: amq-streams source: mirror-amq-streams sourceNamespace: openshift-marketplace config: env: - name: KUBERNETES_DISABLE_HOSTNAME_VERIFICATION value: "true" Save and exit the editor. Check that the Subscription was updated: oc get sub amq-streams -n <operator_namespace> Check that the Cluster Operator Deployment was updated to use the new environment variable: oc get deployment <cluster_operator_deployment_name> 9.5. Cruise Control CPU utilization estimation Cruise Control for Streams for Apache Kafka has a known issue that relates to the calculation of CPU utilization estimation. CPU utilization is calculated as a percentage of the defined capacity of a broker pod. The issue occurs when running Kafka brokers across nodes with varying CPU cores. For example, node1 might have 2 CPU cores and node2 might have 4 CPU cores. In this situation, Cruise Control can underestimate and overestimate CPU load of brokers The issue can prevent cluster rebalances when the pod is under heavy load. There are two workarounds for this issue. Workaround one: Equal CPU requests and limits You can set CPU requests equal to CPU limits in Kafka.spec.kafka.resources . That way, all CPU resources are reserved upfront and are always available. This configuration allows Cruise Control to properly evaluate the CPU utilization when preparing the rebalance proposals based on CPU goals. Workaround two: Exclude CPU goals You can exclude CPU goals from the hard and default goals specified in the Cruise Control configuration. Example Cruise Control configuration without CPU goals apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster spec: kafka: # ... zookeeper: # ... entityOperator: topicOperator: {} userOperator: {} cruiseControl: brokerCapacity: inboundNetwork: 10000KB/s outboundNetwork: 10000KB/s config: hard.goals: > com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.MinTopicLeadersPerBrokerGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal default.goals: > com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.MinTopicLeadersPerBrokerGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.PotentialNwOutGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskUsageDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundUsageDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundUsageDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.TopicReplicaDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderReplicaDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderBytesInDistributionGoal For more information, see Insufficient CPU capacity . 9.6. JMX authentication when running in FIPS mode When running Streams for Apache Kafka in FIPS mode with JMX authentication enabled, clients may fail authentication. To work around this issue, do not enable JMX authentication while running in FIPS mode. We are investigating the issue and working to resolve it in a future release.

Commands:
template: serviceAccount: metadata: annotations: openshift.io/internal-registry-pull-secret-ref: my-cluster-entity-operator-dockercfg-qxwxd
cluster-info Kubernetes master is running at <master_address>
get subs -n <operator_namespace>
edit sub amq-streams -n <operator_namespace>
apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: amq-streams namespace: <operator_namespace> spec: channel: amq-streams-1.8.x installPlanApproval: Automatic name: amq-streams source: mirror-amq-streams sourceNamespace: openshift-marketplace config: env: - name: KUBERNETES_MASTER value: MASTER-ADDRESS
get sub amq-streams -n <operator_namespace>
get deployment <cluster_operator_deployment_name>
get subs -n <operator_namespace>
edit sub amq-streams -n <operator_namespace>
apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: amq-streams namespace: <operator_namespace> spec: channel: amq-streams-1.8.x installPlanApproval: Automatic name: amq-streams source: mirror-amq-streams sourceNamespace: openshift-marketplace config: env: - name: KUBERNETES_DISABLE_HOSTNAME_VERIFICATION value: "true"
get sub amq-streams -n <operator_namespace>
get deployment <cluster_operator_deployment_name>
apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster spec: kafka: # zookeeper: # entityOperator: topicOperator: {} userOperator: {} cruiseControl: brokerCapacity: inboundNetwork: 10000KB/s outboundNetwork: 10000KB/s config: hard.goals: > com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.MinTopicLeadersPerBrokerGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal default.goals: > com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.MinTopicLeadersPerBrokerGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.PotentialNwOutGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskUsageDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundUsageDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundUsageDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.TopicReplicaDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderReplicaDistributionGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderBytesInDistributionGoal