Package: eventstreams-sdk
IBM Cloud Event Streams Python SDK Version 1.1.1IntroductionIBM Event Streams for IBM Cloud™ is a high-throughput message bus built with Apache Kafka.
It is optimized for event ingestion into IBM Cloud and event stream distribution between your services and applications.Event Streams provides a REST API to help connect your existing systems to your Event Streams Kafka cluster.
Using the API, you can integrate Event Streams with any system that supports RESTful APIs.DocumentationIBM Cloud Eventstreams Service APIs.Table of ContentsOverviewPrerequisitesInstallationUsing the SDKREST API documentationQuestionsIssuesOpen source @ IBMContributingLicenseOverviewThe IBM Cloud Eventstreams SDK Python SDK allows developers to programmatically interact with the following
IBM Cloud services:Service NameImported Class NameAdmin RestAdminRestPrerequisitesAnIBM Cloudaccount.TheIBM Cloud CLI.An IAM API key to allow the SDK to access your account. Create onehere.A IBM Cloud Eventstreams Instance Create oneherePython 3.6 or above.InstallationTo install, usepip3oreasy_install3.5:pip3install--upgrade"eventstreams_sdk>=1.1.1"oreasy_install3.5--upgrade"eventstreams_sdk>=1.1.1"Using the SDKFor general SDK usage information, please seethis linkQuestionsIf you are having difficulties using this SDK or have a question about the IBM Cloud services,
please ask a questionStack Overflow.IssuesIf you encounter an issue with the project, you are welcome to submit abug report.
Before that, please search for similar issues. It's possible that someone has already reported the problem.Open source @ IBMFind more open source projects on theIBM Github PageContributingSeeCONTRIBUTING.md.LicenseThis SDK is released under the Apache 2.0 license.
The license's full text can be found inLICENSE.Event Streams Administration REST APIThis REST API allows users of theIBM Event Streams serviceto administerKafka topicsassociated with an instance of the service. You can use this API to perform the following
operations:Create a Kafka topicList Kafka topicsGet a Kafka topicDelete a Kafka topicUpdate a Kafka topic configurationList which topics are mirroredReplace selection of topics which are mirroredList active mirroring topicsThe Admin REST API is alsodocumented using swagger.Access controlAll requests support below authorization methods:Basic authorization with user and password. (
For both standard, enterprise and lite plans, user is 'token', password is the API key fromibmcloud resource service-keysfor the service instance.)Bearer authorization with bearer token. (This token can be either API key or JWT token obtained from IAM upon login to IBM Cloud. Useibmcloud iam oauth-tokensto retrieve the token afteribmcloud login)X-Auth-Tokenheader to be set to the API key. This header is deprecated.Administration API endpointAdministration API endpoint is thekafka_admin_urlproperty in the service key for the service instance. This command can be used to retrieve this property.$ibmcloudresourceservice-key"${service_instance_key_name}"--outputjson>jq-r'.[]|.credentials.kafka_admin_url'Environment SetupIn the examples you must set and export environment variables as follows:Either theAPI_KEYorBEARER_TOKENto use for authentication.KAFKA_ADMIN_URLto point to your Eventstreams admin endpoint.In addition, theContent-typeheader has to be set toapplication/json.Common HTTP status codes:200: Request succeeded.202: Request was accepted.400: Invalid request JSON.401: The authentication header is not set or provided information is not valid.403: Not authorized to perform the operation. Usually it means the API key used is missing a certain role. More details on what role can perform what operation refers to thisdocument.404: Unable to find the topic with topic name given by user.422: Semantically invalid request.503: An error occurred handling the request.Error responses carry a JSON body like the following:{"error_code":50301,"message":"Unknown Kafka Error","incident_id":"17afe715-0ff5-4c49-9acc-a4204244a331"}Error codes are of the formatHHHKKwhereHHHis the HTTP Status Code andKKis the Kafka protocol error.For E2E debugging purposes, the transaction ID of every request is returned in the HTTP headerX-Global-Transaction-Id.
If the header is set on the request, it will be honored. If not, it will be generated.
In the event of a non-200 error return code, the transaction ID is also returned in the JSON error response asincident_id.Using the REST API to administer Kafka topicsThe remainder of this document describes the Python implementation of the Admin Rest SDK
and we also provide exampleexample.goto show all SDK calls in action.To run the example :-Set the required environment variables# Set your API KEY (or a bearer token could be used by setting the BEARER_TOKEN environment variable instead, but not both)exportAPI_KEY="abc123456789"# Set the Admin Endpoint to point to your cluster.exportKAFKA_ADMIN_URL="https://xyzclustername.svc01.region.eventstreams.test.cloud.ibm.com"Run the examplepythonexample.pyREST APIThe following sections explain how the REST API works with examples.Code Setup# Code SetupfromtypingimportSetfromibm_cloud_sdk_core.authenticatorsimportBasicAuthenticatorfromeventstreams_sdk.adminrest_v1import*importosfromhttpimportHTTPStatusSERVICE_NAME='adminrest_v1'KAFKA_ADMIN_URL=os.getenv('KAFKA_ADMIN_URL')BEARER_TOKEN=os.getenv('BEARER_TOKEN')API_KEY=os.getenv('API_KEY')# End Code SetupAuthenticationUse one of the following methods to authenticate:To authenticate using Basic Auth:
Place these values into the Authorization header of the HTTP request in the form Basic 
where  is the username and password joined by a single colon:base64 encoded.
For example:echo-n"token:<APIKEY>"|base64To authenticate using a bearer token:
To obtain your token using the IBM Cloud CLI, first log in to IBM Cloud, then run the following command:ibmcloud iam oauth-tokensPlace this token in the Authorization header of the HTTP request in the form Bearer. Both API key or JWT tokens are supported.To authenticate directly using the api_key:
Place the key directly as the value of the X-Auth-Token HTTP header.ExampleHere's an example of how to create the authenticator using either an API key or a BEARER_TOKEN# Create AuthenticatorifnotKAFKA_ADMIN_URL:print("Please set env KAFKA_ADMIN_URL")exit(1)ifnotAPI_KEYandnotBEARER_TOKEN:print("Please set either an API_KEY or a BEARER_TOKEN")exit(1)ifAPI_KEYandBEARER_TOKEN:print("Please set either an API_KEY or a BEARER_TOKEN not both")exit(1)ifAPI_KEY:# Create an Basic IAM authenticator.authenticator=BasicAuthenticator('token',API_KEY)else:# Create an IAM Bearer Token authenticator.authenticator=BasicAuthenticator('token',BEARER_TOKEN)service=AdminrestV1(authenticator=authenticator)# End AuthenticatorCreating a client for the Admin REST API.Create a new service object.# Create Servicebase_url=KAFKA_ADMIN_URLservice.set_service_url(base_url)# End Create ServiceCreating a Kafka topicTo create a Kafka topic the admin REST SDK issues a POST request to the /admin/topics path.
The body of the request contains a JSON document, for example:{"name":"topicname","partitions":1,"configs":{"retentionMs":86400000,"cleanupPolicy":"delete"}}The only required field is name. The partitions fields defaults to 1 if not set.Expected HTTP status codes:202: Topic creation request was accepted.400: Invalid request JSON.403: Not authorized to create topic.422: Semantically invalid request.If the request to create a Kafka topic succeeds then HTTP status code 202 (Accepted) is returned. If the operation fails then a HTTP status code of 422 (Un-processable Entity) is returned, and a JSON object containing additional information about the failure is returned as the body of the response.Exampledefcreate_topic(service,topic_name):# Set up parameter valuespartition_count=1configs=[]# Invoke create method.try:response=service.create_topic(name=topic_name,partition_count=partition_count,configs=configs,)ifresponse.status_code==HTTPStatus.ACCEPTED:print("\ttopic created: "+topic_name)except:print("\tError Creating Topic: "+topic_name)# func.EndDeleting a Kafka topicTo delete a Kafka topic, the admin REST SDK issues a DELETE request to the/admin/topics/TOPICNAMEpath (whereTOPICNAMEis the name of the Kafka topic that you want to delete).Expected return codes:202: Topic deletion request was accepted.403: Not authorized to delete topic.404: Topic does not exist.422: Semantically invalid request.A 202 (Accepted) status code is returned if the REST API accepts the delete
request or status code 422 (Un-processable Entity) if the delete request is
rejected. If a delete request is rejected then the body of the HTTP response
will contain a JSON object which provides additional information about why
the request was rejected.Kafka deletes topics asynchronously. Deleted topics may still appear in the
response to alist topics requestfor a short period
of time after the completion of a REST request to delete the topic.Exampledefdelete_topic(service,topic_name):# Lets try to delete it.try:response=service.delete_topic(topic_name,)ifresponse.status_code==HTTPStatus.ACCEPTED:print("\ttopic deleted: "+topic_name)except:print("\tError Deleting Topic: "+topic_name)# func.EndListing Kafka topicsYou can list all of your Kafka topics by issuing a GET request to the/admin/topicspath.Expected status codes:200: the topic list is returned as JSON in the following format:[{"name":"topic1","partitions":1,"retentionMs":86400000,"cleanupPolicy":"delete"},{"name":"topic2","partitions":2,"retentionMs":86400000,"cleanupPolicy":"delete"}]A successful response will have HTTP status code 200 (OK) and contain an
array of JSON objects, where each object represents a Kafka topic and has the
following properties:Property nameDescriptionnameThe name of the Kafka topic.partitionsThe number of partitions of the Kafka topic.retentionsMsThe retention period for messages on the topic (in ms).cleanupPolicyThe cleanup policy of the Kafka topic.Exampledeflist_topics(service):# Set up parameter valuestopic_filter=''# Invoke list method.try:response=service.list_topics(topic_filter=topic_filter,)ifresponse.status_code==HTTPStatus.OK:ifnotresponse.result:print("\tnothing to list")returnfortopicinresponse.result:print("\t"+topic["name"])except:print("\tError Listing Topics")# func.endGetting a Kafka topicTo get a Kafka topic detail information, issue a GET request to the/admin/topics/TOPICNAMEpath (whereTOPICNAMEis the name of the Kafka topic that you want to get).Expected status codes200: Retrieve topic details successfully in following format:{"name":"MYTOPIC","partitions":1,"replicationFactor":3,"retentionMs":86400000,"cleanupPolicy":"delete","configs":{"cleanup.policy":"delete","min.insync.replicas":"2","retention.bytes":"1073741824","retention.ms":"86400000","segment.bytes":"536870912"},"replicaAssignments":[{"id":0,"brokers":{"replicas":[3,2,4]}}]}403: Not authorized.404: Topic does not exist.Exampledeftopic_details(service,topic_name):# Invoke get method.try:response=service.get_topic(topic_name,)ifresponse.status_code==HTTPStatus.OK:forkey,valueinresponse.result.items():print("\t"+key+":"+str(value))except:print("\tError Getting Topic Details: "+topic_name)# func.EndUpdating Kafka topic's configurationTo increase a Kafka topic's partition number or to update a Kafka topic's configuration, issue aPATCHrequest to/admin/topics/TOPICNAMEwith the following body:
(where TOPICNAME is the name of the Kafka topic that you want to update).{"new_total_partition_count":4,"configs":[{"name":"cleanup.policy","value":"compact"}]}Supported configuration keys are 'cleanup.policy', 'retention.ms', 'retention.bytes', 'segment.bytes', 'segment.ms', 'segment.index.bytes'.
And partition number can only be increased, not decreased.Expected status codes202: Update topic request was accepted.400: Invalid request JSON/number of partitions is invalid.404: Topic specified does not exist.422: Semantically invalid request.Exampledefupdate_topic(service,topic_name):# Set up parameter values.new_total_partition_count=6configs=[]# Invoke update method.try:response=service.update_topic(topic_name,new_total_partition_count=new_total_partition_count,configs=configs,)ifresponse.status_code==HTTPStatus.ACCEPTED:print("\ttopic updated: "+topic_name)except:print("\tError Updating Topic Details: "+topic_name)# func.EndList current mirroring topic selectionMirroring user controls are only available on the target cluster in a mirroring environment.To get the current topic selection, issue an GET request to /admin/mirroring/topic-selectionExpected status codes200: Retrieved topic selection successfully in following format:{"includes":["^prefix1_.*","^prefix2_.*"]}403: Unauthorized to use mirroring user controls.404: Mirroring not enabled. The mirroring user control APIs are only available on the target cluster of a mirrored pair.503: An error occurred handling the request.Exampledefget_mirroring_topic_selection(service):# Invoke get selection method.try:response=service.get_mirroring_topic_selection()ifresponse.status_code==HTTPStatus.OK:fortopicinresponse.result:print("\t"+topic["name"])except:print("\tError Listing Mirroring Topics:")# func.EndReplace selection of topics which are mirroredReplace mirroring topic selectionMirroring user controls are available on the target cluster in a mirroring environment.To replace the current topic selection, issue a POST request to /admin/mirroring/topic-selectionExpected status codes200: Replaced topic selection successfully. The new selection is returned in following format:{"includes":["^prefix1_.*","^prefix2_.*"]}400: Invalid request. The request data cannot be parsed and used to replace the topic selection.403: Unauthorized to use mirroring user controls.404: Mirroring not enabled. The mirroring user control APIs are only available on the target cluster of a mirrored pair.415: Unsupported media type. Content-Type header with application/json is required.503: An error occurred handling the request.Exampledefreplace_mirroring_topic_selection(service,topic_name):# Set up parameter valuesincludes=[topic_name]# Invoke replace method.try:response=service.replace_mirroring_topic_selection(includes=[topic_name],)ifresponse.status_code==HTTPStatus.OK:print("\tmirroring topic selection updated: "+includes)except:print("\tError Replacing Mirroring Topics:")# func.EndList active mirroring topicsMirroring user controls are available on the target cluster in a mirroring environment.To get the list of currently mirrored topics, issue an GET request to /admin/mirroring/active-topicsExpected status codes200: Retrieved active topics successfully in following format:{"active_topics":["topic1","topic2"]}403: Unauthorized to use mirroring user controls.404: Mirroring not enabled. The mirroring user control APIs are only available on the target cluster of a mirrored pair.503: An error occurred handling the request.Exampledefget_list_mirroring_active_topics(service):# Invoke active method.try:response=service.get_list_mirroring_active_topics()ifresponse.status_code==HTTPStatus.OK:fortopicinresponse.result:print("\t"+topic["name"])print("\tactive mirroring topics updated:")except:print("\tError Listing Active Mirroring Topics:")# func.End