Saravanan666/pypi-org huggingface dataset:
Package: kafi
Kaficloses the gap between Kafka and files. It stands for "(Ka)fka and (fi)les" (plus "Kafi" is the Swiss name for a coffee or a coffee place).What canKafido for you?it makes debugging Kafka-based applications a breezeit is useful for data scientists for building asynchronous data pipelines, or for synchronously bringing files from local disk, S3 or Azure Blob Storage to Kafka and vice versa etc.Kafiis the successor ofkash.pywhich is the successor ofstreampunk.
Package: kafka-admin-service
kafka_admin_serviceKafka admin service class, provides basic management functions such as USER creation, USER deletion, USER listing all and USER password changing, TOPIC creation, TOPIC deletion, TOPIC listing all, and ACL creation, ACL deletion and ACL listing all.Installpip install kafka-admin-serviceKafkaAdminService methodsadd_aclchange_passwordcreate_topiccreate_topic_and_topic_usercreate_userdelete_acldelete_all_topicsdelete_all_usersdelete_topicdelete_topic_user_aclsdelete_userget_aclsget_topicsget_user_scramsget_usersupdate_uservalidate_user_passwordWhy always returns (returncode, stdout, stderr)?Because we use subprocess.Popen to call kafka's management command under kafka/bin folder. Soreturncodemeans the command exit status code, andstdoutmeans the command's normal output, andstderrmeans the command's error output.Kafka server requirements.Kafka using scram auth.Zookeeper using NO auth.Setworkspaceto kafka's root folder. e.g. kafka's command is under folder /path/to/kafka/bin/, so theworkspaceshould set to /path/to/kafka/.Create scram.jaas file, default to put it atworkspace/config/scram.jaas, or set it's path in config file.Create scram.client.properties file, default to put it atworkspace/config/scram.client.properties, or set it's path in config file.Because we need to call kafka's management commands, normally should install us on kafka server for easy.Example of ./config/scram.jaasKafkaServer {
   org.apache.kafka.common.security.scram.ScramLoginModule required
   username="admin"
   password="xxxxxxxx";
};

KafkaClient {
   org.apache.kafka.common.security.scram.ScramLoginModule required
   username="admin"
   password="xxxxxxxx";
};Example of ./config/scram.client.propertiessecurity.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-256Example of application's config.ymlzookeeper: localhost:2181
bootstrap_server: localhost:9092
workspace: /apprun/kafka/
command_config_file: "/apprun/config/scram.client.properties"
default_kafka_opts: "-Djava.security.auth.login.config=/apprun/config/scram.jaas"
topic_partitions: 16
topic_replication_factor: 3
apikeys:
    xxxxxxx:
        appid: xxxx
        other-appinfo: xxx
    yyyyyyy:
        appid: yyyy
        other-appinfo: yyyStart kafka-admin-serverkafka-admin-server --no-daemon -c config.yml startReleasesv0.1.1First Release.v0.1.3Fix kafka commands' path check.Add execute time log.v0.1.7Add get_user_scrams method.Add validate_user_password method.Fix license_files missing problem.Doc update.v0.1.9Fix template format parameters missing problem.Fix KAFKA_OPTS missing in calling execute() problem.v0.2.1Add kafka-admin-server based on xmlrpc.
Package: kafka-aggregator
kafka-aggregatorA Kafka aggregator based on theFaustPython Stream Processing library.kafka-aggregator development is based on theSafirapplication template.Overviewkafka-aggregator usesFaust’s windowing featureto aggregate a stream of messages from Kafka.kafka-aggregator implements a Faust agent, a “stream processor”,  that adds messages from a source topic into a Faust table. The table is configured as a tumbling window with a size, representing the window duration (time interval) and an expiration time, which specifies the duration for which the data allocated to each window will be stored. Every time a window expires, a callback function is called to aggregate the messages allocated to that window. The size of the window controls the frequency of the aggregated stream.kafka-aggregator usesfaust-avroto add Avro serialization and Schema Registry support to Faust. faust-avro can parse Faust models into Avro Schemas.Seethe docsfor more information.Change log0.2.0 (2020-08-14)Add first and third quartiles (q1andq3) to the list of summary statistics computed by the aggregator.Ability to configure the list of summary statistics to be computed.Pinned top-level requeriments.Add Kafka Connect to the docker-compose setup.Use only one Schema Registry by default to simplify local execution.First release to PyPI.0.1.0 (2020-07-13)Initial release of kafka-aggregator with the following features:Use Faust windowing feature to aggregate a stream of messages.Use Faust-avro to add Avro serialization and Schema Registry support to Faust.Support to an internal Schema Registry to store schemas for the aggreated topics (optional).Create aggregation topic schemas from the source topic schemas and from the list of summary statistics to be computed.Ability to create Faust records dynamically from aggregation topic schemas.Ability to auto-generate code for the Faust agents (stream processors).Compute summary statistics for numeric fields:min(),mean(),median(),stdev(),max().Add example module to initialize a number of source topics in kafka, control the number of fields in each topic, and produce messages for those topics at a given frequency.Use Kafdrop to inspect messages from source and aggregated topics.Add kafka-aggregator documentation site.
Package: kafka-bluesky-live
In order to use it, you’ll need to have an acessible Kafka Topic that you can publish your data. You will need to deploy Kafka in your PC and create this topic. If you are running this from a beamline configured GUI, the topic is probably already create and will be called <BL>_bluesky, (EMA_bluesky for instance). There a really easy-to-follow tutorial about KafkaHere.After setting up Kafka and the topic that you will stream to, you will need a callback to subscribe to RunEngine and stream the run generated Documents, it is a simple callback:from kafka import KafkaProducer
import msgpack

producer = KafkaProducer()

def kafka_callback(name: str, doc: dict) -> None:
    """Callback to stream Bluesky Documents via Kafka"""
    producer = KafkaProducer(value_serializer=msgpack.dumps)
    producer.send(<kafkja_topic>, (name, doc))Notice that you will need to install kafka-python and msgpack.After the definition of the callback, subscribe it to Bluesky RunEngine:from bluesky import RunEngine

RE = RunEngine()
kafka_callback_token = RE.subscribe(kafka_callback)With this done, you can now launch kafka-bluesky-live, start a the run and see the data being plotted live.First, install this project:git clone https://gitlab.cnpem.br/SOL/bluesky/kafka-bluesky-live
pip install kafka-bluesky-liveThen, run the in the terminal, run the following command:kbl <kafka_topic_name>With the interface already opened, start a run in Bluesky to see it being plotted live.There are some special information that can be passed to kafka-bluesky-live via scan metadata that will change some of its behaviours. You can choose what scan motor that will be in the x-axis, set the counter that is going to be shown in the first tab and set the name that will identify the current run. This is all done by the special metada, they are: “file_name”, “main_motor” and “main_counter”.To input then to the current run you must add them as a metadata for that run, there are several ways that this can be done, for example defining a dict and inputing it to the RunEngine afterwards:md = {"file_name": "my_awesome_file", "main_motor": "motor2", "main_counter": "counter_5"}
RunEngine(scan([counter_1, counter_2, counter_5], motor1, -1, 1, motor2, -5, 5, 100), **md)
Package: kafka-bridge-client
kafka-bridge-clientPython async client forStrimzi Kafka BridgeandConfluent REST ProxyPackage include consumer only.Installpip install kafka-bridge-clientUsageBy default client useStrimzi Kafka BridgeAPIConsumer (async)fromkafka_bridge_clientimportKafkaBridgeConsumer# Strimzi Kafka Bridgeconsumer1=KafkaBridgeConsumer('topic1','topic2',group_id='my-group,auto_offset_reset='earliest',enable_auto_commit=False,bootstrap_server='your-kafka-bridge-url',consumer_name='consumer-name',)# Confluent REST Proxyconsumer2=KafkaBridgeConsumer('topic1','topic2',group_id='my-group,auto_offset_reset='earliest',enable_auto_commit=False,bootstrap_server='your-kafka-bridge-url',consumer_name='consumer-name',proxy='confluent')asyncforrecinconsumer1.get_records():print(rec['value'])awaitconsumer.commit()# orrecords=awaitconsumer1.poll_records()print(records)awaitconsumer.commit()Producer (sync)fromkafka_bridge_clientimportKafkaBridgeProducerproducer=KafkaBridgeProducer('http://bridge.url'timeout=5)producer.send(Message(key='1',value='value'))DeployYou need to change version inpyproject.tomland run itpoetry publish --build
Package: kafka-broker
Kafka BrokerA python package implementation for the confluent kafka package. Managing producing and consuming.Has built-in implementation for FastAPI.About this packageThe package allows for easy setup and connecting to the Apache Kafka message broker.Easily run a consumer in a second process so it does not act as a blocking operation.To read the received data from the broker, poll the consumer storage. To wait for a specific item (based on correlation ID), await the consume function.The BrokerManagerHas 3 simple functions:produce- produce an event to the message broker.init_consumer_app- initialize the consumer together with an FastAPI app.init_consumer- initialize a regular consumer.It has an attribute called theconsumer_storage. This is where everything the consumer finds will be stored. Though in reality, you have to callreceive()to actually get any data.receive()is a blocking operation;poll()is not. Andconsume()finds a certain event object based oncorrelation_id, which is also a blocking operation.Theseconsumer_storagefunctions are all asyncronous so they do not block any process. For example for if you were using FastAPI.There is alsoconsume_all(), which is basicly apoll(), but also returns everything found within the storage.The EventRouterOur event router is based on RabbitMQ's routing keys and on FastAPI's router.Exmaple implementationapp/__init__.pyfromapp.testimporttest_routerevent_router=EventRouter("module")event_router.include_binder(test_router)Sidenote: We give the highest router the name of the module.app/test/__init__.pytest_router=EventRouter("test")@test_router.bind_event("return")defreturn_event(event_object:EventObject):event_object.data=some_funtion()event_object.event="respond"broker_manager.produce("SomeModule",event_object)With the event string"test.return", the event router will first find the corresponding bind called'test'. If'test'is found, based on whether it's anotherEventRouteror afunction, it will either continue the search chain in the next router or execute the foundfunction.The EventObjectA simple class intended to be used for communicating with other microservices.Acorrelation_idto track if an object is the same one as the one you sent off.Anevent-string to route the EventObject to the right place.Adatafield containing any JSON serializable information.Anaudit_logto track where this EventObject has been.ExamplesSend event and wait for response:fromkafka_brokerimportbroker_manager,EventObjectcorrelation_id=str(uuid.uuid4())event="test.return"event_object=EventObject(correlation_id=correlation_id,event=event,)broker_manager.produce("SomeModule",event_object)data=awaitbroker_manager.consumer_storage.consume(correlation_id)returndataRun a process based on all events:fromkafka_brokerimportbroker_managerasyncdefmain():broker_manager.init_consumer()whileTrue:awaitasyncio.sleep(1)data=awaitbroker_manager.consumer_storage.consume_all()ifdata:for_,event_objectindata.items():event_router.execute_event(event_object)
Package: kafka-client-decorator
A wrapper for kafka producer and consumer that can be used as decorator for a function which can keep consuming data, process this data and broadcast it to next topics/queues.This usesconfluent-kafkapython package to create prooducer, consumer and then wraps it. So, big thanks to them!Installation$ pip install kafka-client-decoratorUsageDefine your function how you want to process the data and then decorate it.from kafka_client_decorator.kafka_client import KafkaClient

@KafkaClient(bootstrap_servers, security_protocol, sasl_username, sasl_password).consumer_producer(consumer_from_topic='my-topic-1', group_id='pdf', produce_to_topic=['my-topic-2'])
def process_data(data = None):
    # Call your driver modules here to process the data
    result = Driver(data)
    return resultNOTE:If you want the your driver result to be pushed to next topic/queue, you can simply pass produce_to_topic as arg in decorator 'consumer_prodcuer' method.To only produce to topic(s) -from kafka_client_decorator.client_producer import ClientProducer

producer = ClientProducer(bootstrap_servers, security_protocol, sasl_username, sasl_password)
prodcuer.produce_to_broker(data, topics_list)NOTE:If your kafka broker does not uses SASL or SSL protocol, no need to pass 'sasl_username' and 'sasl_password'.NOTE:If you want to work with multiple partitions in kafka, you can use below method to produce (it provides custom unique key to be sent with the message)prodcuer.produce_to_broker_with_key(data, topic_list)
Package: kafka-client-decorators
Decorators to kafkaThis module is based on pykafka project (https://github.com/Parsely/pykafka.git) and define a class KafkaDecorator and its 4 decorators@KafkaDecorator.host
The first defines the connection parameters and
its parameters are the same ofpykafka.KafkaClient@KafkaDecorator.balanced_consumer defines a balanced consumer and
its parameters are the same ofpykafka.topic.get_balanced_consumerfunction@KafkaDecorator.simple_consumer defines a simple consumer and
its parameters are the same ofpykafka.topic.get_simple_consumerfunction@KafkaDecorator.producer defines a producer and
its parameters are the same ofpykafka.topic.get_producerfunctionInstallpip install kafka-client-decoratorsExamplefromkafka_client_decoratorsimportKafkaDecoratorkc=KafkaDecorator()#@kc.host(zookeeper_hosts='localhost:2181' )@kc.host(hosts='localhost:9092')classA:def__init__(self,testA,cls):self.a=testAself.cls=clspass@kc.balanced_consumer('test1',consumer_group='testgroup3',auto_commit_enable=True,managed=True,consumer_timeout_ms=1000)defget(self,msg):print(f'{self.a}Receive offset{msg.offset}key{msg.partition_key}message:{msg.value}')self.send(msg.value)@kc.simple_consumer('test2',consumer_group='testgroup4',auto_commit_enable=True,consumer_timeout_ms=1000)defget2(self,msg):print(f'{self.a}Receive offset{msg.offset}, message:{msg.value}')self.cls.stop(self)@kc.producer('test2')defsend(self,msg):pass@kc.producer('test1')defsendKey(self,msg,key):passclassB:def__init__(self):passdefstop(self,conn):conn.stop()a=A('Example',B())a.start()a.sendKey('Hello'.encode('utf-8'),partition_key='world'.encode('utf-8'))a.wait()
Package: kafka-client-new
KAFKA-CLIENT-NEWПримеры использования:Через managerimportjsonimportloggingfromkafka_clientimport(DefaultBatchKafkaManager,DefaultBatchProducer,DefaultBatchConsumer,)fromkafka_client.handlerimportEmptyHandlerlogging.basicConfig(level=logging.INFO)if__name__=='__main__':batch_size=10consumer_settings={'bootstrap_servers':'...','value_deserializer':lambdav:json.loads(v.decode('utf-8')),'key_deserializer':lambdak:k.decode('utf-8'),'client_id':'...','group_id':'...','enable_auto_commit':False,'auto_offset_reset':'earliest'}consumer=DefaultBatchConsumer(topic='...',batch_size=batch_size,consumer_settings=consumer_settings)producer_settings={'bootstrap_servers':'...','key_serializer':str.encode,'value_serializer':lambdav:json.dumps(v).encode('utf-8'),}producer=DefaultBatchProducer(topic='...',producer_settings=producer_settings)manager=DefaultBatchKafkaManager(consumer,producer)manager.set_handler(handler=EmptyHandler())manager.run()Через функцию runзадаем переменные окруженияBATCH_SIZE[optional] - размер батчаBOOTSTRAP_SERVERS- брокеры кафкиINPUT_TOPIC- входной топикOUTPUT_TOPIC- выходной топикCLIENT_ID[optional] - имя приложения, как то отсылается в кафкуGROUP_ID- kafka consumer groupЗапускаемpython:importloggingfromkafka_clientimportrunfromkafka_client.handlerimportEmptyHandlerlogging.basicConfig(level=logging.INFO)if__name__=='__main__':run(handler=EmptyHandler())
Package: kafkaclientveli
Kafka Client for VELI.STOREDescriptionThis module helps you to integration kafka messaging into your apps.
Supported frameworks: FastAPI, (Django coming soon :D)How to use (FastAPI):Producerapp=FastAPI()defproduce_event(topic,event):producer=app.state.producerproducer.produce_event(topic,event)@app.on_event("startup")asyncdefstartup_event():bootstrap_servers=['localhost:9092','localhost:9093']producer=KafkaEventProducer(bootstrap_servers)awaitproducer.start()app.state.producer=producer@app.on_event("shutdown")asyncdefshutdown_event():awaitapp.state.producer.stop()@app.post("/products")asyncdefsave_product(product_info:ProductInfo):product=save_product(product_info)produce_event(KafkaTopic.PAGE_VIEWS,product)returnproductConsumerapp=FastAPI()@app.on_event("startup")asyncdefstartup_event():# Define the configuration variablestopics=[KafkaTopic.USER_REGISTRATIONS,KafkaTopic.PAGE_VIEWS]bootstrap_servers=['localhost:9092','localhost:9093']group_id='app_id'consumer=AsyncKafkaConsumer(topics,bootstrap_servers,group_id)app.state.consumer=consumerawaitconsumer.start()# start the consume_events coroutine in the backgroundasyncio.create_task(consumer.consume())@app.on_event("shutdown")asyncdefshutdown_event():awaitapp.state.consumer.stop()
Package: kafka-connect-api
Documentation:https://kafka-connect-api.readthedocs.io.Installpip install kafka-connect-apiUsagefromkafka_connect_api.kafka_connect_apiimportApi,Clusterapi=Api(connect.cluster,port=8083)cluster=Cluster(api)print(cluster.connectors)FeaturesAllows you to interact with the Kafka Connect API (API Reference) in a simple way.Connection to cluster (supports Basic Auth)List all connectors in the clusterDescribe the connector configurationUpdate the connector configurationPause / Resume ConnectorRestart all tasks for the connectorSome pre-made functions can help with operational activities.
Seekafka_connect_api.aws_lambdas.py
Package: kafka-connect-manager
Kafka Connect ManagerA tool to manage Apache Kafka Connect connectors and tasksUsage:$kcm[OPTIONS]COMMAND[ARGS]...Options:--host TEXT: Connect worker host  [env var: CONNECT_HOST; default: http://localhost:8083]--install-completion: Install completion for the current shell.--show-completion: Show completion for the current shell, to copy it or customize the installation.--help: Show this message and exit.Commands:add: Register new connector Supporting environment...list: List all connectorsstatus: Get status connectorupdate: Update connector configurationwatch: Actively monitor your connectors healthkcm addRegister new connectorSupporting environment variable expansion in JSON file.A connector requires a name and configuration, we take both of them separately.For example:{"name":"MySinkConnector","config":{"connector.class":"com.mongodb.kafka.connect.MongoSinkConnector","connection.uri":"${MONGODB_URL}"}}Usage:$kcmadd[OPTIONS]Options:-f, --file FILE: Config JSON file path  [required]--help: Show this message and exit.kcm listList all connectorsUsage:$kcmlist[OPTIONS]Options:--type [all|sink|source]: Type of connectors to list  [default: all]--help: Show this message and exit.kcm statusGet status connectorUsage:$kcmstatus[OPTIONS]Options:--connector TEXT: Name of connector  [required]--help: Show this message and exit.kcm updateUpdate connector configurationUsage:$kcmupdate[OPTIONS]CONNECTORArguments:CONNECTOR: Connector name  [required]Options:-f, --file FILE: Config JSON file path  [required]--help: Show this message and exit.kcm watchActively monitor your connectors healthUsage:$kcmwatch[OPTIONS][CONNECTORS]...Arguments:[CONNECTORS]...: Connectors to monitorOptions:--refresh-interval INTEGER: Refresh interval  [default: 5]--help: Show this message and exit.
Package: kafkaconnectsync
Kafka Connect SyncKafka Connect API connectors synchronization libraryAboutThekafkaconnectsynclibrary allows you to incorporate the Kafka Connect connectors/sink to your deployment code.When running Kafka Connect in distribute mode, connectors need to be added using REST methods after the API is running. Creating connectors shouldn't be a manual process sokafkaconnectsyncprovides functions to manage connectors as an infrastructure/deployment component.To sync connectors,kafkaconnectsyncreads the list of connectors and decides if it needs to create, update or delete them depending on the status of the API (i.e the existing connectors).InstallingInstall using pip:$pip3installkafkaconnectsyncAlternatively, you can usesetup.pyto install by cloning this repository and running:$pythonsetup.pyinstallUsageDefine your connectors:#connectors.json[{"config":{"name":"my_connector_name","connector.class":"io.confluent.connect.s3.S3SinkConnector","tasks.max":"1","topics":"my-topic","locale":"en_US","timezone":"UTC","flush.size":"3",}},{"config":{"name":"my_connector_name_two","connector.class":"io.confluent.connect.s3.S3SinkConnector","s3.credentials.provider.class":"com.amazonaws.auth.DefaultAWSCredentialsProviderChain","topics.dir":"data","file.delim":"-","partitioner.class":"io.confluent.connect.storage.partitioner.HourlyPartitioner",}}]Import thesyncfunction from the package. Make sure to call it after your app deployment has been done.# Other imports...importsyncfromkafkaconnectsyncurl='https://my-kafka-connect-api.com'connectors=json.loads(open('connectors.json').read())"""...Deploy your app here......"""# Sync connectorssync(url,connectors,strict=True,wait_for_deployment=True,verbose=True)Documentationsync(url, connectors=[], wait_for_deployment=True, verbose=False):url: You Kafka Connect API hostname.connectors: The array of connectors objects to sync on Kafka Connect.strict: Whenstrictis enabled, apart from creating/updating connectors from the list, the sync function will remove all the API connectors that are not present on this list as a way to synchronize your list with the API. Default:Truewait_for_deployment: IfTrue, it will keep sending requests to the Kafka Connect hosts until it becomes available. Useful if your deploying your app and the function should wait for the deployment to finish. Default:True.verbose: Set this flag toTrueif you want to output action logs to your terminal. Default:False.DevelopmentClone this repo to your machine:$gitclonehttps://github.com/venturachrisdev/kafka-connect-sync.git
$cdkafka-connect-syncInstall dependencies using pip:$pip3install-rrequirements-dev.txtUsepylintto run linter on the project:$pylintkafkaconnectsync/;pylinttests/To apply pep8 rules to the codebase, use the following command:$autopep8--in-place--recursivekafkaconnectsync/tests/Run tests locally using:$pytesttests/ContributorsChristopher Ventura <chrisventura.work@gmail.com>
Package: kafka-connect-watcher
Service that will actively probe and monitor your Kafka connect clusters using the Connect API.
It can report metrics to AWS CloudWatch (Prometheus coming) usingAWS EMFto allow creating alerts
and alarms.FeaturesScan multiple clusters at onceImplement different remediation rulesInclude/Exclude lists for connectors to evaluate/ignoreRoadmapPrometheus supportMultiple channels of alerts (i.e. webhooks)Systems recommendationsWhen using multiple clusters, we recommend to provide multiple CPUs to the service as it
has multi-threading enabled, allowing for parallel processing of clusters & their respective connectors.