Package: nlp-kafka-rest-api
Kafka REST APIInstallationIf you are installing from a remote or public repository, run:pip install kafka-rest-api.
If you are installing from awheelfile in the local directory, run:pip install {filename}.whl,
and replace{filename}with the name of the.whlfile.Getting StartedInteractions with a Kafka cluster can be performed on a Producer/Consumer paradigm. As such there are two classes that
can be imported and used to publish and subscribe to topics in Kafka:ProducerandConsumer.ConfigurationWhen using this package to access Merck API Gateway, you can define the following environment variables:KAFKA_REST_API_URL: Target Kafka REST API URL. In alternative, you can pass the argumentkafka_rest_api_urlto the Producer and the Consumer constructor.X_API_KEY: The authorization token to validate API requests to API Gateway. In alternative, you can pass a dictionary
with the format{"x-api-key": "your-api-key", "other-header-key": "other-header-value", etc...}to the key parameterauth_headersin both the Producer and the Consumer constructors.TOPIC_ID: MSK Topic ID assigned to the user. In alternative, you can pass a string with the topic ID
to the key argumenttopic_idin both the Producer and the Consumer constructors.ProducerProduce json dataIn the snippet below the topicpkeis used as example. The pattern for the producer is the following:fromkafka_restimportProducerproducer=Producer()new_keys=producer.produce(messages_to_pke_endpoint,"pke")Please note that each message in the list of messages to the target endpoint should correspond to the payload that
is expected by that endpoint that would otherwise be a JSON object.For example, a valid message to thepkeendpoint is:{"text":"Genome engineering is a powerful tool.","algorithm":"TopicRank","n_best":10}To know which message format you should use for each endpoint, please consult the documentation for NLP API.TheProducer.producemethod automatically generates a unique key (UUID) for each message.
Optionally, you can provide your unique keys as well, passing a list of keys (strings) to the argumentkeys.Produce filesTo produce files as inputs to a given endpoint, you can use the methodproduce_files. The required arguments
for this method are:fileswhich consists of a list of absolute or relative paths to the input files;endpointtarget endpoint (pdf2text, for example);fromkafka_restimportProducerproducer=Producer()new_keys=producer.produce_files(files=list_of_files,endpoint="pdf2text")ConsumerPattern 1 - IteratorArguably, the most useful way of consuming messages with the Consumer class is as follows:fromkafka_restimportConsumerwithConsumer()asconsumer:fordata,remaining_keysinconsumer.consume(keys):print((data,remaining_keys))Pattern 2 - Step-by-step instantiation (Chain)You can also opt to do a step-by-step instantiation and have a finer control of each request sent by the Consumer to the NLP API:fromkafka_restimportConsumerconsumer=Consumer()consumer.create()consumer.subscribe()# or consumer.create().subscribe().consume(keys)fordata,remaining_keysinconsumer.consume(keys):print((data,remaining_keys))consumer.delete()Pattern 3 - Consume allOptionally, the Consumer can just return when all keys were exhausted, i.e.: when all messages were consumed.
For that, please use theconsume_allmethod.fromkafka_restimportConsumerwithConsumer()asconsumer:data=consumer.consume_all(keys)Full exampleProduce files as inputs and consume outputs.fromkafka_restimportProducer,Consumerproducer=Producer()new_keys=producer.produce_files(["files/file1.pdf","files/file2.pdf"],"pdf2text")withConsumer()asconsumer:fordata,remaining_keysinenumerate(consumer.consume(new_keys)):ifdata:print(f"Data:{data}| Remaining Keys:{remaining_keys}")For more snippets, please check the example in the filekafka_rest/snippetsin this repo.