Running Kafka command-line tools
By: Zhimin Wen
Tags: Mtls, Ibm Event Streams, Strimzi, Kafka

Running Kafka command-line tools

Kafka provides a rich set of command-line tools to manage the topics and clusters. The default settings of the latest IBM Event Streams (Strimzi Operator based) gives some challenges to run these tools.

Broker Listeners

When a Kafka resource is deployed as an operator, a strong security configuration is applied normally. Take a look at the following excerpt of the cluster listener settings

listeners:

external:

authentication:

type: scram-sha-512

type: route

tls:

authentication:

type: tls

The brokers will be listening on

Port 9094 for external connections with SRAM-SHA-512 authentication. Port 9093 for internal communication, where mTLS are used. The m here (mutual) means the client will authenticate the brokers, and the brokers will authenticate the client also.

As there are only these listeners defined, the commonly used plain port 9092 for the Kafka command line will not work.

KafkaUser Resource — Client Authentication and Authorization

As IBM event streams turn on the authorization, once a user is authenticated, it needs to be authorized to perform any action against the topic, consumer group, and the Kafka cluster. In the operator model, this is achieved with the KafkaUser resource.

Let's create the following KafkaUser resource

apiVersion: eventstreams.ibm.com/v1beta1

kind: KafkaUser

metadata:

name: super-user-tls

labels:

eventstreams.ibm.com/cluster: es

spec:

authentication:

type: tls

authorization:

type: simple

acls:

- resource:

type: topic

name: '*'

patternType: literal

operation: All

- resource:

type: cluster

name: '*'

patternType: literal

operation: All

- resource:

type: group

name: '*'

patternType: literal

operation: All

There are two folds, one is authentication and the other one is authorization.

Client Authentication

The user named super-user-tls will be authenticated by mTLS. Once this user is created, in the meantime, a secret with the same name will be created. The secret is the certificate and key of this user. Check it out,

$ kubectl -n eventstreams describe secret super-user-tls

Name: super-user-tls

Namespace: eventstreams

Labels: app.kubernetes.io/instance=super-user-tls

app.kubernetes.io/managed-by=strimzi-user-operator

app.kubernetes.io/name=strimzi-user-operator

app.kubernetes.io/part-of=eventstreams-super-user-tls

eventstreams.ibm.com/cluster=es

eventstreams.ibm.com/kind=KafkaUser

Annotations: <none> Type: Opaque Data

====

user.p12: 2402 bytes

user.password: 12 bytes

ca.crt: 1164 bytes

user.crt: 1017 bytes

user.key: 1704 bytes

Let’s print out the user’s certificate

$ kubectl -n eventstreams get secret super-user-tls -o jsonpath='{.data.user\.crt}' | base64 -di | openssl x509 -text | head -15 Certificate:

Data:

Version: 1 (0x0)

Serial Number:

8a:3f:a0:88:9a:cd:bf:78

Signature Algorithm: sha256WithRSAEncryption

Issuer: O = io.strimzi, CN = clients-ca v0

Validity

Not Before: Dec 8 15:39:15 2020 GMT

Not After : Dec 8 15:39:15 2021 GMT

Subject: CN = super-user-tls

Subject Public Key Info:

Public Key Algorithm: rsaEncryption

RSA Public-Key: (2048 bit)

Modulus:

When a client is trying to authenticate by the brokers with this certificate, as the singer, which is the clients CA, is trusted by the brokers, the certificate signed by the clients CA will be authenticated successfully. The user will be identified as the CN name, super-user-tls.

On the other hand, the broker's certificate must also be validated since it's two ways mTLS. In order to do that the other CA, the cluster CA, must be presented to the client tools so that the client can validate the broker’s certificate based on that CA.

Client Authorization

The other half of the KafkaUser resources are the authorizations assigned to this user. There are pretty fine-graded ACL controls for the Kafka topics, consumer groups, and the cluster. For example, the topics have the operation of read, write, create, describe, and so on.

For testing purposes, let’s create a superuser with all the rights by setting the resource name as * and the operation set as All.

K8s Deployment for Kafka command-line tool

Let’s create a K8s deployment to run the Kafka command-line tool in a separate pod other than the brokers.

apiVersion: apps/v1

kind: Deployment

metadata:

name: kcmd

namespace: eventstreams

labels:

app: kcmd

spec:

replicas: 1

selector:

matchLabels:

app: kcmd

template:

metadata:

labels:

app: kcmd

spec:

containers:

- name: kcmd

image: cp.icr.io/cp/ibm-eventstreams-kafka@sha256:d116a3c77b2290bc9fb7dfe54ceda9ce4df46853d3446a0bfb1f114eab555763

command:

- sh

- -c

- while true; do sleep 100; done

volumeMounts:

- name: certs

mountPath: /certs

- name: cluster-ca

mountPath: /clusterCA volumes:

- name: certs

secret:

secretName: super-user-tls

- name: cluster-ca

secret:

secretName: es-cluster-ca-cert

Use the same image as the brokers where all the Kafka command-line tools are available. The container command is just a dummy sleep loop waiting, we can exec into the pods to run the actual Kafka tool.

Notice the secrets mounted into the pods. First is the user’s certificate secret where the pkcs12 format is available. The certificate will be authenticated by the brokers where the client CA is trusted. On the other hand, to validate the broker’s certificate, the cluster CA has to be mounted into the pod.

Exec into the pod, create a properties file with the following command,

echo security.protocol=SSL > /tmp/ssl.properties echo ssl.truststore.type=PKCS12 >> /tmp/ssl.properties

echo ssl.truststore.location=/clusterCA/ca.p12 >> /tmp/ssl.properties

echo ssl.truststore.password=$(cat /clusterCA/ca.password) >> /tmp/ssl.properties echo ssl.keystore.type=PKCS12 >> /tmp/ssl.properties

echo ssl.keystore.location=/certs/user.p12 >> /tmp/ssl.properties

echo ssl.keystore.password=$(cat /certs/user.password) >> /tmp/ssl.properties

This properties file will be supplied as an argument to the different Kafka command-line tool. A sample is shown below,

security.protocol=SSL

ssl.truststore.type=PKCS12

ssl.truststore.location=/clusterCA/ca.p12

ssl.truststore.password=LqEdssNJKmP2

ssl.keystore.type=PKCS12

ssl.keystore.location=/certs/user.p12

ssl.keystore.password=kEH9oqXfRRpa

The keystore is where the user cert stored, the cert will be sent to the broker to authenticate and authorize the user correspondingly. The truststore is used for the client to validate the broker’s cert. So the cluster CA is used. All the certs and keys are in the PKCS12 format.

Run the Kafka tool

Once the properties file is prepared, we can run the Kafka command-line tool

Let's describe a topic,

export POD=$(oc -n eventstreams get pods | grep Running | grep kcmd | awk '{print $1}') oc exec -i $POD -- sh -c "bin/kafka-topics.sh --describe --bootstrap-server es-kafka-bootstrap.eventstreams.svc:9093 --topic demo --command-config /tmp/ssl.properties"

Notice we pass in the bootstrap-server argument with the K8s service name of the bootstrap and the internal TLS port 9093. The command-config is defined to where the properties file located.

The result is shown,

Topic: demo PartitionCount: 3 ReplicationFactor: 3 Configs: min.insync.replicas=2,retention.ms=604800000,message.format.version=2.6-IV0

Topic: demo Partition: 0 Leader: 2 Replicas: 2,1,0 Isr: 2,1,0

Topic: demo Partition: 1 Leader: 1 Replicas: 1,0,2 Isr: 2,0,1

Topic: demo Partition: 2 Leader: 0 Replicas: 0,2,1 Isr: 2,1,0

We could also trigger an election of partition leader, where the cluster operation authorization is required,

bin/kafka-leader-election.sh --bootstrap-server es-kafka-bootstrap.eventstreams.svc:9093 --topic demo --partition 0 --election-type preferred --admin.config /tmp/ssl.properties

The command output is shown below. Not surprising as the leader is what it supposed to be.

Valid replica already elected for partitions

If we restart a broker, without waiting for the default 5 minutes for leader election, we can force the election once the broker is up, rerun the command again,