Package: kafka-connect-py
Kafka Connect PythonThe Kafka Connect REST API allows you to manage connectors that move data between Apache Kafka and other systems.The Kafka Connect command line tool, also known askcorkafka-connect, allows users to manage their Kafka Connect cluster and connectors. With this tool, users can retrieve information about the cluster and connectors, create new connectors, update existing connectors, delete connectors, and perform other actions.This project aims to supported all features of theKafka Connect REST API.Installpipinstallkafka-connect-pyCommand Line UsageGetting Basic Connect Cluster InformationTo get basic Connect cluster information including the worker version, the commit it’s on, and its Kafka cluster ID, use the following command:kcinfoListing Installed PluginsTo list the plugins installed on the worker, use the following command:kclist-pluginsTo format the result of the installed plugin list for easier readability, pipe the output to thejqcommand:kclist-plugins|jqCreate a Connector InstanceTo create a connector instance with JSON data containing the connector’s configuration:kcupdatesource-debezium-orders-00-d'{"connector.class": "io.debezium.connector.mysql.MySqlConnector","value.converter": "io.confluent.connect.json.JsonSchemaConverter","value.converter.schemas.enable": "true","value.converter.schema.registry.url": "'$SCHEMA_REGISTRY_URL'","value.converter.basic.auth.credentials.source": "'$BASIC_AUTH_CREDENTIALS_SOURCE'","value.converter.basic.auth.user.info": "'$SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO'","database.hostname": "mysql","database.port": "3306","database.user": "debezium","database.password": "dbz","database.server.id": "42","database.server.name": "asgard","table.whitelist": "demo.orders","database.history.kafka.bootstrap.servers": "'$BOOTSTRAP_SERVERS'","database.history.consumer.security.protocol": "SASL_SSL","database.history.consumer.sasl.mechanism": "PLAIN","database.history.consumer.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"'$CLOUD_KEY'\" password=\"'$CLOUD_SECRET'\";","database.history.producer.security.protocol": "SASL_SSL","database.history.producer.sasl.mechanism": "PLAIN","database.history.producer.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"'$CLOUD_KEY'\" password=\"'$CLOUD_SECRET'\";","database.history.kafka.topic": "dbhistory.demo","topic.creation.default.replication.factor": "3","topic.creation.default.partitions": "3","decimal.handling.mode": "double","include.schema.changes": "true","transforms": "unwrap,addTopicPrefix","transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState","transforms.addTopicPrefix.type":"org.apache.kafka.connect.transforms.RegexRouter","transforms.addTopicPrefix.regex":"(.*)","transforms.addTopicPrefix.replacement":"mysql-debezium-$1"}'Or create/update a connector instance with a JSON file:kcupdate<connector>--config-file<config_file>Update a ConnectorAs mentioned above, if there’s a connector to update, you can use theupdatesub-command to amend the configuration (seeCreate a Connector Instanceabove). Because update is used to both create and update connectors, it’s the standard command that you should use most of the time (which also means that you don’t have to completely rewrite your configs).List Connector InstancesUse the following command to list of all extant connectors:kclist[--expand=info|status][--pattern=regex][--state=running|paused|unassigned|failed]Inspect Config and Status for a ConnectorInspect the config for a given connector as follows:kcconfigsink-elastic-orders-00You can also look at a connector’s status. While the config command shows a connector’s static configuration, the status shows the connector as a runtime entity:kcstatussink-elastic-orders-00You can also uselistwith the--expand=statusoption to show the status of many connectors at once. We can filter down the response using a regex pattern and/or connector state.Use the following to show all connector names prefixed with the wordsink-and that are in aFAILEDconnector state.kclist--expand=status-psink-.*-sfailedDelete a ConnectorIf something is wrong in your setup and you don’t think a config change would help, or if you simply don’t need a connector to run anymore, you can delete it by name:kcdeletesink-elastic-orders-00Thedeletesub-command also supports multiple deletions using the--alloption. On its own it will apply the sub-command to all connectors.The following will delete all connector names prefixed with the wordsink-and that are in aPAUSEDconnector state.kcdelete--all--patternsink-.*-spausedThe--alloption is supported by several sub-commands, includingdelete,restart,resume, andpause. However, for better testing and control over the outcome of your actions, we recommend using the list filtering option before executing any of these sub-commands. This way, you can ensure that your filters are working as intended and avoid unintended consequences. To use list filtering, simply run thelistsub-command and apply your filters.Inspect Task DetailsThe following command returns the connector status:kcstatussource-debezium-orders-00|jqIf your connector fails, the details of the failure belong to the task. So to inspect the problem, you’ll need to find the stack trace for the task. The task is the entity that is actually running the connector and converter code, so the state for the stack trace lives in it.kctask-statussource-debezium-orders-00<task-id>|jqRestart the Connector and TasksIf after inspecting a task, you have determined that it has failed and you have fixed the reason for the failure (perhaps restarted a database), you can restart the connector with the following:kc restart source-debezium-orders-00Keep in mind though that restarting the connector doesn’t restart all of its tasks. You will also need to restart the failed task and then get its status again as follows:kctask-statussource-debezium-orders-00<task-id>What's more, you can restart the connector and all its failed tasks with the following:kcrestartsource-debezium-orders-00--include-tasks--failed-onlyand check the status again:kcstatussource-debezium-orders-00|jqPause and Resume a ConnectorUnlike restarting, pausing a connector does pause its tasks. This happens asynchronously, though, so when you pause a connector, you can’t rely on it pausing all of its tasks at exactly the same time. The tasks are running in a thread pool, so there’s no fancy mechanism to make this happen simultaneously.A connector and its tasks can be paused as follows:kcpausesource-debezium-orders-00Just as easily, a connector and its tasks can be resumed:kcresumesource-debezium-orders-00Display All of a Connector’s TasksA convenient way to display all of a connector’s tasks at once is as follows:kclist-taskssource-debezium-orders-00|jqThis information is similar to what you can get from other APIs, but it is broken down by task, and configs for each are shown.
Get a List of Topics Used by a ConnectorAs of Apache Kafka 2.5, it is possible to get a list of topics used by a connector:kclist-topicssource-debezium-orders-00|jqThis shows the topics that a connector is consuming from or producing to. This may not be particularly useful for connectors that are consuming from or producing to a single topic. However, some developers, for example, use regular expressions for topic names in Connect, so this is a major benefit in situations where topic names are derived computationally.This could also be useful with a source connector that is using SMTs to dynamically change the topic names to which it is producing.Python# Import the classfromkafka_connectimportKafkaConnectimportjson# Instantiate the clientclient=KafkaConnect(url="http://localhost:8083")# Get the version and other details of the Kafka Connect clustercluster=client.get_cluster_info()print(cluster)# Get a list of active connectorsconnectors=client.list_connectors(expand="status")print(json.dumps(connectors,indent=2))# Create a new connectorconfig={"name":"my-connector","config":{"connector.class":"io.confluent.connect.jdbc.JdbcSourceConnector","tasks.max":"1","connection.url":"jdbc:postgresql://localhost:5432/mydatabase","connection.user":"myuser","connection.password":"mypassword","table.whitelist":"mytable","mode":"timestamp+incrementing","timestamp.column.name":"modified_at","validate.non.null":"false","incrementing.column.name":"id","topic.prefix":"my-connector-",},}response=client.create_connector(config)print(response)# Update an existing connectornew_config={"config":{"connector.class":"io.confluent.connect.jdbc.JdbcSourceConnector","tasks.max":"1","connection.url":"jdbc:postgresql://localhost:5432/mydatabase","connection.user":"myuser","connection.password":"mypassword","table.whitelist":"mytable","mode":"timestamp+incrementing","timestamp.column.name":"modified_at","validate.non.null":"false","incrementing.column.name":"id","topic.prefix":"my-connector-",},}response=client.update_connector("my-connector",new_config)print(response)# Get status for a connectorresponse=client.get_connector_status("my-connector")print(json.dumps(response,indent=2))# Restart a connectorresponse=client.restart_connector("my-connector")print(response)# Delete a connectorresponse=client.delete_connector("my-connector")print(response)LicenseApache 2.0 License - aidanmelen/kafka-connect-pyCreditsThe entireCommand Line Usagesection was copied directly from the Confluence'sKafka Connect’s REST APIcourse.