Chapter 3. Loading configuration values from external sources Use configuration provider plugins to load configuration data from external sources. The providers operate independently of AMQ Streams. You can use them to load configuration data for all Kafka components, including producers and consumers. Use them, for example, to provide the credentials for Kafka Connect connector configuration. OpenShift Configuration Provider The OpenShift Configuration Provider plugin loads configuration data from OpenShift secrets or config maps. Suppose you have a Secret object that's managed outside the Kafka namespace, or outside the Kafka cluster. The OpenShift Configuration Provider allows you to reference the values of the secret in your configuration without extracting the files. You just need to tell the provider what secret to use and provide access rights. The provider loads the data without needing to restart the Kafka component, even when using a new Secret or ConfigMap object. This capability avoids disruption when a Kafka Connect instance hosts multiple connectors. Environment Variables Configuration Provider The Environment Variables Configuration Provider plugin loads configuration data from environment variables. The values for the environment variables can be mapped from secrets or config maps. You can use the Environment Variables Configuration Provider, for example, to load certificates or JAAS configuration from environment variables mapped from OpenShift secrets. Note OpenShift Configuration Provider can't use mounted files. For example, it can't load values that need the location of a truststore or keystore. Instead, you can mount config maps or secrets into a Kafka Connect pod as environment variables or volumes. You can use the Environment Variables Configuration Provider to load values for environment variables. You add configuration using the externalConfiguration property in KafkaConnect.spec . You don't need to set up access rights with this approach. However, Kafka Connect will need a restart when using a new Secret or ConfigMap for a connector. This will cause disruption to all the Kafka Connect instance's connectors. 3.1. Loading configuration values from a config map This procedure shows how to use the OpenShift Configuration Provider plugin. In the procedure, an external ConfigMap object provides configuration properties for a connector. Prerequisites An OpenShift cluster is available. A Kafka cluster is running. The Cluster Operator is running. Procedure Create a ConfigMap or Secret that contains the configuration properties. In this example, a ConfigMap object named my-connector-configuration contains connector properties: Example ConfigMap with connector properties apiVersion: v1 kind: ConfigMap metadata: name: my-connector-configuration data: option1: value1 option2: value2 Specify the OpenShift Configuration Provider in the Kafka Connect configuration. The specification shown here can support loading values from secrets and config maps. Example Kafka Connect configuration to enable the OpenShift Configuration Provider apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaConnect metadata: name: my-connect annotations: strimzi.io/use-connector-resources: "true" spec: # ... config: # ... config.providers: secrets,configmaps 1 config.providers.secrets.class: io.strimzi.kafka.KubernetesSecretConfigProvider 2 config.providers.configmaps.class: io.strimzi.kafka.KubernetesConfigMapConfigProvider 3 # ... 1 The alias for the configuration provider is used to define other configuration parameters. The provider parameters use the alias from config.providers , taking the form config.providers.USD{alias}.class . 2 KubernetesSecretConfigProvider provides values from secrets. 3 KubernetesConfigMapConfigProvider provides values from config maps. Create or update the resource to enable the provider. oc apply -f <kafka_connect_configuration_file> Create a role that permits access to the values in the external config map. Example role to access values from a config map apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: connector-configuration-role rules: - apiGroups: [""] resources: ["configmaps"] resourceNames: ["my-connector-configuration"] verbs: ["get"] # ... The rule gives the role permission to access the my-connector-configuration config map. Create a role binding to permit access to the namespace that contains the config map. Example role binding to access the namespace that contains the config map apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: connector-configuration-role-binding subjects: - kind: ServiceAccount name: my-connect-connect namespace: my-project roleRef: kind: Role name: connector-configuration-role apiGroup: rbac.authorization.k8s.io # ... The role binding gives the role permission to access the my-project namespace. The service account must be the same one used by the Kafka Connect deployment. The service account name format is <cluster_name> -connect, where <cluster_name> is the name of the KafkaConnect custom resource. Reference the config map in the connector configuration. Example connector configuration referencing the config map apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaConnector metadata: name: my-connector labels: strimzi.io/cluster: my-connect spec: # ... config: option: USD{configmaps:my-project/my-connector-configuration:option1} # ... # ... Placeholders for the property values in the config map are referenced in the connector configuration. The placeholder structure is configmaps: <path_and_file_name> : <property> . KubernetesConfigMapConfigProvider reads and extracts the option1 property value from the external config map. 3.2. Loading configuration values from environment variables This procedure shows how to use the Environment Variables Configuration Provider plugin. In the procedure, environment variables provide configuration properties for a connector. A database password is specified as an environment variable. Prerequisites An OpenShift cluster is available. A Kafka cluster is running. The Cluster Operator is running. Procedure Specify the Environment Variables Configuration Provider in the Kafka Connect configuration. Define environment variables using the externalConfiguration property . Example Kafka Connect configuration to enable the Environment Variables Configuration Provider apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaConnect metadata: name: my-connect annotations: strimzi.io/use-connector-resources: "true" spec: # ... config: # ... config.providers: env 1 config.providers.env.class: io.strimzi.kafka.EnvVarConfigProvider 2 # ... externalConfiguration: env: - name: DB_PASSWORD 3 valueFrom: secretKeyRef: name: db-creds 4 key: dbPassword 5 # ... 1 The alias for the configuration provider is used to define other configuration parameters. The provider parameters use the alias from config.providers , taking the form config.providers.USD{alias}.class . 2 EnvVarConfigProvider provides values from environment variables. 3 The DB_PASSWORD environment variable takes a password value from a secret. 4 The name of the secret containing the predefined password. 5 The key for the password stored inside the secret. Create or update the resource to enable the provider. oc apply -f <kafka_connect_configuration_file> Reference the environment variable in the connector configuration. Example connector configuration referencing the environment variable apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaConnector metadata: name: my-connector labels: strimzi.io/cluster: my-connect spec: # ... config: option: USD{env:DB_PASSWORD} # ... # ...

Commands:
apiVersion: v1 kind: ConfigMap metadata: name: my-connector-configuration data: option1: value1 option2: value2
apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaConnect metadata: name: my-connect annotations: strimzi.io/use-connector-resources: "true" spec: # config: # config.providers: secrets,configmaps 1 config.providers.secrets.class: io.strimzi.kafka.KubernetesSecretConfigProvider 2 config.providers.configmaps.class: io.strimzi.kafka.KubernetesConfigMapConfigProvider 3 #
apply -f <kafka_connect_configuration_file>
apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: connector-configuration-role rules: - apiGroups: [""] resources: ["configmaps"] resourceNames: ["my-connector-configuration"] verbs: ["get"]
apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: connector-configuration-role-binding subjects: - kind: ServiceAccount name: my-connect-connect namespace: my-project roleRef: kind: Role name: connector-configuration-role apiGroup: rbac.authorization.k8s.io
apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaConnector metadata: name: my-connector labels: strimzi.io/cluster: my-connect spec: # config: option: USD{configmaps:my-project/my-connector-configuration:option1} #
apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaConnect metadata: name: my-connect annotations: strimzi.io/use-connector-resources: "true" spec: # config: # config.providers: env 1 config.providers.env.class: io.strimzi.kafka.EnvVarConfigProvider 2 # externalConfiguration: env: - name: DB_PASSWORD 3 valueFrom: secretKeyRef: name: db-creds 4 key: dbPassword 5 #
apply -f <kafka_connect_configuration_file>
apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaConnector metadata: name: my-connector labels: strimzi.io/cluster: my-connect spec: # config: option: USD{env:DB_PASSWORD} #