Saravanan666/pypi-org huggingface dataset:
Package: kafka-counter
kafka-counterCount kafka events from multiple topics.This small utility reads kafka messages from multiple topics and provides you a
running count of the number of seen events. If the message is a json document,
you can also break down the counts for a topic by a specific key indicated by a
JSON path.When you quit the program pressing Ctrl-C, it also provides some statistics of
the events seen broken down by topics.Usage$./kafka-count.py-husage: kafka-count.py [-h] [--debug] --config CONFIG [--update-every UPDATE_EVERY]optional arguments:-h, --help            show this help message and exit--debug, -d           Run in debug mode--config CONFIG, -c CONFIGYaml config file--update-every UPDATE_EVERY, -u UPDATE_EVERYHow many ms between display updatesExample configcommon:broker:kafka.default.svc.cluster.local:9092count:k8sNginxIngressLogRaw:name:ingress# Use this name to refer to this topicpath:kubernetes_cluster# break down the numbers by the string value at this jsonpath locationk8sNginxModSecurityLogRaw:name:modseccountOnly:True# Only count the total eventsExample Usage$kafka-count-cconfig.yamlk8sevents      [  1170](cluster1=599(51.2%), cluster2=545(46.6%), cluster3=12(1.0%), cluster4=7(0.6%), cluster5=7(0.6%))ingress        [ 48878](cluster1=27319(55.9%), cluster2=15953(32.6%), cluster4=3996(8.2%), cluster3=1056(2.2%), cluster5=554(1.1%))modsec         [     2](COUNT=2(100.0%))containerd     [ 31054](cluster2=14772(47.6%), cluster1=12291(39.6%), cluster4=1453(4.7%), cluster5=1408(4.5%), cluster3=1130(3.6%))systemd        [ 10789](cluster2=4731(43.9%), cluster1=3605(33.4%), cluster4=856(7.9%), cluster3=841(7.8%), cluster5=756(7.0%))^CTotal events received: 93293ingress                       : 49945 (53.5%)containerd                    : 31276 (33.5%)systemd                       : 10894 (11.7%)k8sevents                     : 1176 (1.3%)modsec                        : 2 (0.0%)
Package: kafkaescli
InstallInstall frompypipipinstallkafkaescliUsage# consume from `hello`kafkaescliconsumehello# consume from `hello` showing metadatakafkaescliconsumehello--metadata# produce topic `hello`kafkaescliproducehelloworld# produce longer stringskafkaescliproducehello"world of kafka"# produce from stdin a value per linecatvalues.json|kafkaescliproducehello--stdin# produce to topic `world` form the output of a consumer of topic `hello`kafkaescliconsumehello|kafkaescliproduceworld--stdin# produce `{"foo":"bar"}` to topic `hello`, with middlewarekafkaescliproducehello'{"foo":"bar"}'--middleware'{"hook_before_produce": "examples.json.hook_before_produce"}'# consume from topic `hello` with middlewarekafkaescliconsumehello--middleware'{"hook_after_consume": "examples.json.hook_after_consume"}'# run the web api http://localhost:8000/docskafkaesclirunserver# POST consumed values to WEBHOOKkafkaescliconsumehello--metadata--webhookhttps://myendpoint.example.com# For more details seekafkaescli--helpThese examples assume a Kafka instance is running atlocalhost:9092ContributionsJony KalaveraPull-requests are welcome and will be processed on a best-effort basis.
Follow thecontributing guide.
Package: kafkaesque
# kafkaesqueThis A flask style kafka consumer. It is an extention of the KafkaConsumer from the kafka-python package available [`here`](https://github.com/dpkp/kafka-python)## Installation```shpip install kafkaesque```## writing a consumer looks like this```from kafkaesque import Kafkaesqueapp = Kafkaesque(bootstrap_servers=",".join(["bs_1:9092","bs_2:9093","bs_3:9094"]),group_id="consumer-grp-id")@app.handle('test-topic')def test_topic_handler(msg):print "consumed {} from test-topic".format(msg)if __name__ == "__main__":app.start()```## Author* **Sankalp Jonna**Email me with any queries: [sankalpjonna@gmail.com](sankalpjonna@gmail.com).
Package: kafka-healthcheck
kafka-healthcheckA simple healthcheck wrapper to monitor Kafka.Kafka Healthcheck is a simple server that provides a singular API endpoint to determine the health of a Kafka instance. This can be used to alert or take action on unhealthy Kafka instances.The service checks the health by sendingkafkacatabout thatkafkacatThis project is a copy "devshawn" whit modifications for usekafkacatUsageKafka Healthcheck can be installed viapip. Bothpythonandpipare required, as well askafkacat.Command-LineInstallkafka-healthcheckviapip:pipinstallkafka-healthcheckTo start the healthcheck server, run:kafka-healthcheckThe server will now be running onlocalhost:9290.ConfigurationKafka Healthcheck can be configured via command-line arguments or by environment variables.PortThe port for thekafka-healthcheckAPI.UsageValueEnvironment VariableHEALTHCHECK_PORTCommand-Line Argument--portDefault Value9290Kafka HostThe host of the Kafka instance to run the health check against. This is used withkafkacat.UsageValueEnvironment VariableHEALTHCHECK_KAFKA_HOSTCommand-Line Argument--kafka-hostDefault ValuelocalhostKafka PortThe port of the Kafka instance to run the health check against. This is used withkafkacat.UsageValueEnvironment VariableHEALTHCHECK_KAFKA_PORTCommand-Line Argument--kafka-portDefault Value9092Log LevelThe level of logs to be shown by the application.UsageValueEnvironment VariableHEALTHCHECK_LOG_LEVELCommand-Line Argument--log-levelDefault ValueINFOValid ValuesDEBUG,INFO,WARNING,ERRORAll healthy responses are logged atDEBUG. Unhealthy responses are logged atWARNING. Any unexpected errors are logged atERROR.LicenseCopyright (c) 2019 Shawn Seymour.Extended Copyright (c) 2022 Rodrigo Garcia.Licensed under theApache 2.0 license.
Package: kafkaka
a Kafka client which using easy described protocol tool, also be able to be used with Gevent and tornado.WARNING: Under development, Now only support simple send method. Not support Python3.USAGEInstall using pypi:pip install kafkakaInstall from source:git clone https://github.com/wesdu/kafkaka.git
cd kafkaka
python setup.py installEXAMPLEsimple block mode:# coding: utf8
from kafkaka.client import KafkaClient
import time

if __name__ == "__main__":
    c = KafkaClient("tx-storm1:9092")
    c.send_message('im-msg', 'hi', str(time.time()))
    c.send_message('im-msg', u'你好'.encode('utf8'), str(time.time()))
    print 'this will block'using with Gevent:# coding: utf8
from kafkaka.gevent_patch import KafkaClient
from gevent import spawn
from gevent import sleep
import time

if __name__ == "__main__":
    c = KafkaClient("t-storm1:9092", topic_names=['im-msg'])
    print ''
    for i in xrange(50):
        c.send_message('im-msg', u'你好'.encode('utf8'), str(time.time()), str(i))
        c.send_message('im-msg', 'hi', str(time.time()), str(i))
    print 'this will not block'
    for i in xrange(50):
        c.send_message('im-msg', u'你好'.encode('utf8'), str(time.time()), str(i))
        c.send_message('im-msg', 'hi', str(time.time()), str(i))
        sleep(0.1)
    print 'but this will block'
    sleep(30)you can set the number of max parallel connections by using pool_size param:# coding: utf8
from kafkaka.gevent_patch import KafkaClient
from gevent import joinall

import time

if __name__ == "__main__":
    c = KafkaClient("t-storm1:9092",
                    topic_names=['im-msg'],
                    pool_size=10  # the number of max parallel connections.
    )
    start = time.time()
    all = []
    print ''
    for i in xrange(50):
        all.append(c.send_message('im-msg', u'你好'.encode('utf8'), str(time.time()), str(i)))
        all.append(c.send_message('im-msg', 'hi', str(time.time()), str(i)))
    print 'this will not block'
    for i in xrange(50):
        all.append(c.send_message('im-msg', u'你好'.encode('utf8'), str(time.time()), str(i)))
        all.append(c.send_message('im-msg', 'hi', str(time.time()), str(i)))
    joinall(all)
    print 'but this will block'
    print time.time() - startusing with tornado:# coding: utf8
from kafkaka.tornado_patch import KafkaClient
import tornado.ioloop

import time

if __name__ == "__main__":
    c = KafkaClient("t-storm1:9092", topic_names=['im-msg'])
    start = time.time()
    print ''
    for i in xrange(500):
        c.send_message('im-msg', u'你好'.encode('utf8'), str(time.time()), str(i))
        c.send_message('im-msg', 'hi', str(time.time()), str(i))
    for i in xrange(500):
        c.send_message('im-msg', u'你好'.encode('utf8'), str(time.time()), str(i))
        c.send_message('im-msg', 'hi', str(time.time()), str(i))
    print time.time() - start
    print 'this will not block'
    tornado.ioloop.IOLoop.instance().start()
Package: kafka-lag-monitor
Kafka group lag aggregate monitorThis utility can currentlyAcceptkafka-consumer-groupsoutput like file from stdin and print aggregated output to stdout.ssh to a remote machine with kafka running on it, runkafka-consumer-groups, for multiple groups, collect the output, group by group and topic and finally print average and max lag.Currently Tested on: Python 3.11.3 and macosInstallationpipinstallkafka-lag-monitorRunstdin-modeRun the below to get a general idea of how the output would look like.catexamples/example1.txt|kafka-lag-monitorstdin-modeAdd option -v to get verbose outputTry option --tablefmt psql to get tabular output just like psql. (Can try any format supportedhere)remote-modekafka-lag-monitor-v--remoteubuntu@127.0.0.1-i~/.ssh/key.pem--bootstrap-server127.0.0.1:9000--groupgroup1--groupgroup2Setup locallyInstall dependanciespython -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
pip3 install poetry
poetry installTODOExplore using tui which can monitor lag live by recording last n entries and calculating lag increase/decrease.Live TUI Created. lag increase/decrease pending.Currently stdin-mode supports only one file? Look for ways to concat multiple outputs and aggregate?Test casesmonitor locally instead of remote? (use sdk or subprocess calls)fake lag generator for testingCurrently refresh interval is hardcoded. (It breaks if the refresh of data taks longer than interval. FIX IT)re-use ssh connection across calls in tui-mode. (Might need major refactor)Make it look better. (hello darkness (css/tcss) my old friend)Ability to toggle select/deselect of kafka groups during watch mode.
Package: kafka-lmdc
Python-KafkaEsta biblioteca tem como objetivo a facilitaÃ§Ã£o do uso de Kerberos e Kafka em Python.Como usar localmentePara utilizar esta biblioteca localmente, Ã© necessÃ¡rio clonÃ¡-la para o diretÃ³rio onde se encontra o arquivo no qual vocÃª deseja importar a mesma. Isto farÃ¡ com que a biblioteca esteja somente disponÃ­vel para tal diretÃ³rio.Como usar globalmentePara utilizar esta biblioteca globalmente, Ã© necessÃ¡rio clonÃ¡-la para o diretÃ³rio de bibliotecas do Python. Caso nÃ£o saiba onde fica este diretÃ³rio, basta:Abre a IDLE do Python (ou cmd)Importe a biblioteca sys:import sysE digite:print(sys.path)A saÃ­da do programa serÃ¡ algo parecido com:C:\Users\marcos\AppData\Local\Programs\Python\Python38-32\lib\site-packages] (saÃ­da obtida atrÃ¡ves do Windows)Assim, esta biblioteca deve estar dentro do diretÃ³riosite-packages.
Package: kafkalo
Kafkalo is a tool to manage Kafka and Confluent Platform.STATUS: while it does work, it is very rough around the edges. Consider this unstable and unreleased software for now.It manages:Topic creation and changes of settingsSchema creation and versioning.Confluent RBAC rolebindings with predefined roles.It also has a dry-run function that generates a plan. Whenever possible it will contact the services (for example Kafka broker with validate_only=True)
and provide feedback on errors that will happen.You provide a list of YAML input files (for example in a directory, with a glob pattern) and then Kafkalo will try to apply whatever is defined.UsageYou can provide aconfig.yamlfile so thatkafkalowill know how to connect and possibly authenticate with your Kafka infrastrucure and related components (Schema registry, RBAC metadata server).Look at sample config file included.You can add input dirs with glob patterns to let kafkalo know where to find your YAML definitions.
Kafkalo will read all the input YAMLs, merge then into a single internal data structure and try to sync them.To generate a plan> kafkalo planOnce you are satisfied with the plan you can letkafkalosync:> kafkalo syncIssuesHistory0.1.0 (2021-02-23)First release on PyPI.
Package: kafka-logger
## Kafka Logger for Python### Why this exists?There’s an existing python package that can send log to logstash through Kafka,
but it has two problems:1. It’s built on top of a specific python kafka package and it’s outdated,
which left user no choice on which Kafka they can use.
2. It only uses default log formatting from python logging package,
which doesn’t provide much useful information.### Installpip install kafka-logger### Usageimport logging
import KafkaLoggingHandler
logger = logging.getLogger(‘python-logstash-logger’)
logger.setLevel(logging.INFO)
log_producer = KafkaProducer(bootstrap_servers=KAFKA_BROKER)
logger.addHandler(KafkaLoggingHandler(log_producer, ‘app-log’))
Package: kafkaloghandler
KafkaLogHandlerProvides a pythonloggingcompatible handler for producing messages to a
Kafka message bus.Depends on the confluent_kafka module to connect to Kafka.Designed to support both standard and structlog formats, and serializes log
data as JSON when published as a Kafka message.  Messages are normalized to be
more compatible with Logstash/Filebeat formats.UsageExample:import logger

from kafkaloghandler import KafkaLogHandler

log = logging.getLogger()

klh = KafkaLogHandler(bootstrap_servers=["test-kafka:9092"], topic="testtopic")

log.addHandler(klh)

data={'example':'structured data'}

log.info('message to send to kafka', data=data)Parameters that can be provided to KafkaLogHandler:bootstrap_serversList of Kafka bootstrap servers to connect to.default:["localhost:9092"]extra_configDictionary of extraproducer configurationpassed to librdkafka.NOTE: Thebootstrap_serversparameter will overwritebootstrap.servers.default:{}timeoutTimeout in seconds for flushing producer queue. See librdkafka docs.default:10.0topicString that sets the topic in Kafka.default:"kafkaloghandler"keyString that sets the default key in Kafka, can be used for summarization within Kafka.NOTE: This default key can be overridden on a per-message basis by passing a
dict to the logger with{"key": "new_key_for_this_message"}in it.default:"klh"flattenFlattens nested dictionaries and lists passed as structured logging into the parent
dictionary layer, up to a certain depth.This is useful when logging to external systems that don’t have good support
for hierarchical data.Example dictionary:{'a': {'b':'c'}}would be flattened to{'a.b': 'c'}Example list:{'a': ['b','c']}would be flattened to{'a.0': 'b', 'a.1': 'c'}If the depth is exceeded, any remaining deeper items will be added to the
output under the flattened key.Set to0to turn off flattening.default:5separatorSeparator used between items when flattening.default:.blacklistList of top-level keys to discard from structured logs when outputting JSON.default:["_logger", "_name"]TestingUnit tests can be run withtox
Package: kafka-overwatch
What started as a simple CLI/Service to evaluate Kafka cluster topics which have no activity,
ended up being a somewhat comprehensive way to monitor Kafka cluster activities.Takes a configuration file as input, where you can list one or multiple cluster(s) you wish to monitor.After a set period of time, it can produce a report (to local disk or AWS S3) with the list of topics that haven’t seen any activity.
It also exposes metrics via a prometheus endpoint.Usagekafka-overwatch -c config.local.yamlFeaturesSupports evaluating multiple Kafka clusters at onceGenerates a report on topic usage based on topic watermarks offsets (store local or to S3)Generates commands script to re-create all the topics in case of DR (store local or to S3)Exposes metrics via prometheusTopics countPartitions countNumber of new messages (measured with topic offsets)AWS Secret integration for client config valuesUpcomingSchema Registry integrationMulti-nodes awareness (split the load with multiple nodes)cfn-kafka-adminoutput formattopic messages meta-data analysis (i.e are messages compressed?)scripts to perform cleanupRecommendations generated from/based on modelsConduktor Gateway vClusters auto-discoveryConfigurationWhilst a much more comprehensive documentation is yet to be written, please look atkafka_overwatch/specs/config.jsonwhich is used withjsonschemato perform validation of the input.MiscThanksThanks to the Apache Kafka OpenSource community for their continuous efforts in making the eco-system great.
Thanks to theNASAfor having a public cluster to run tests withNoteInspired bykafka-idle-topics, yet completely re-written to be a continuous monitoring of the topics,
similar tocruise-control.
Package: kafkaplus
使用手册快速开始1）生产者importkafkaplusproducer=kafkaplus.getProducer("localhost:9092")# 发送一条消息producer.send("test_topic","data-a")# 同步发送一条消息producer.sendSync("test_topic","data-b")# 批量发送消息producer.sendBatch("test_topic","data1","data2","data3")2）消费者importkafkaplus# 接收消息的回调函数defcallback(data,**kwargs):print(data)print(kwargs)consumer=kafkaplus.getConsumer("localhost:9092")# 监听topic，使用回调函数处理消息consumer.onListener("test_topic",group="G-test",callback=callback)
Package: kafkaproducerself
KafkaProducerSelf
KafkaProducerSelf is a Python library that offers general functions to interact with Kafka producers and publish events using them. Additionally, it includes a health check feature to ensure the connection is live and working properly.Installation
To install KafkaProducerSelf, simply use pip:Functionality:The library provides three main functions:1. health_check
Accepts a URL as a parameter.
Returns a tuple:
True/False: Indicates whether the connection is active or not.
String: Success or failure message.2. connect_to_kafka_producer
Accepts a URL as a parameter.
Returns a tuple:
True/False: Indicates the status of the producer (whether connected or not).
Obj/String: The producer object if connected successfully; otherwise, an error message.3. push_event_to_kafka
Accepts eventData, topicName, and producer as parameters.
Returns a tuple:
True/False: Indicates if the message was successfully published or not.
String: Success or failure message.Note:
Make sure you have a running Kafka broker accessible at the specified URL before using this library. For any issues or questions, please refer to the documentation or contact the library maintainers.Change log0.0.1 (27/07/2023)First Release
Package: kafkapy
KafkapyA python command line interface for inspection and management of a kafka cluster.kafkapyis a single utility
for all management and inspection actions.  I wrote this tool when working withAWS MSKas dealing with many
different utility scripts from confluence was combersome and often required additional pre/post processing of the
data which is not in a suitable format for piping etc, thuskafkapywas born.kafkapyaims to implement all the actions supported by the confluence utility scripts, but also plans to build in
testability for common tasks, such as calculating the sum of all offset lag across all partitions for a topic etc.All output fromkafkapytostdoutis fully compliantjsonand suitable for piping to tools etc.[!CAUTION]Kafkapyrequires broker versions 0.11.0.0 or greater.Quick Startkafkapyis very simple to use, you need two simple things to get started.#installthepythonpackage:pip install kafkapy#togetstarted:kafkapy --help[!TIP]
Ensure your kafka cluster is up and running.In order to interact with kafka, you should create aproperties.yamlfile on disk and provide it's path to
any commands, theproperties.yamlfile will honour anylibrdkafkaproperties.  By defaultkafkapywill
look inside~/.kafkapy/properties.yamlas a fallback should you not provide the option explicitly via--properties.Additionally all commands take an (Optional)--bootstrap-servers(List) option which you can specify the
broker addresses manually, these are also available vialibrdkafkaconfig via thebootstrap.serversproperty
so can reside in yourproperties.yamlfile if you so choose.[!TIP]
Ensure your~/.kafkapy/properties.yamlcontains appropriate properties for auth etc.ViewlibrdkafkapropertiesHere[!TIP]kafkapylooks great, but when piping to other tooling colour is automatically excluded for unix compatibilityUser Guide[!TIP]
For full examples, visit the user guide at:https://symonk.github.io/kafkapy/ContributingIf you want to contribute with Kafkapy check out the Contributing Guide to learn how to start.
Package: kafka-python-dataclasses
kafka-python-dataclassesStructured messages for your kafka projectsKafka is a really complex library and dealing with bytes can be taxing. Use these
wonderful structured Kafka classes instead!Basic consuming:fromdataclassesimportdataclassfromkafka_dataclassesimportStructuredKafkaConsumer@dataclassclassMyKafkaMessage:value:strformessageinStructuredKafkaConsumer(['my_topic']):asserttype(message)==MyKafkaMessageBasic producing:fromdataclassesimportdataclassfromkafka_dataclassesimportStructuredKafkaProducer@dataclassclassMyKafkaMessage:value:strStructuredKafkaProducer().send('my_topic',MyKafkaMessage("Hello, world!"))This works on deeply nested dataclasses as well!
In theory it will also work on attrs classes, since the
unstructuring library,cattrs, supportsattrsout of the box.NoteThis project has been set up using PyScaffold 4.2.1. For details and usage
information on PyScaffold seehttps://pyscaffold.org/.
Package: kafka-python-log-handler
Kafka Python Log HandlerHandler for the standardloggingmodule which puts logs through to Kafka.
The current implementation is very basic to accommodate our needs, but additional functionality may be coming when the parent project grows.InstallThis was developed in Python 3.6.7, usingkafka-python1.4.3 and Kafka 2.12DevelopmentTests will try to connect to a Kafka instance via the defaults.
Optionally settingKAFKA_HOSTandKAFKA_PORTto the appropriate location on your machine will change this.Install the dev requirements$ pip install -r requirements-dev.txtSet the pre-commit hook$ pre-commit installHow to useThe only necessary thing to create this handler is atopicto push the values to.
Optionally, akeyand/orpartitionmay be provided.
Any additional keyword-value configuration provided to the Handler will be used to initialize thekafka.KafkaProducer.To add a handler to the python logger is very simple:importloggingfromkafka_handlerimportKafkaLogHandlerhandler=KafkaLogHandler(topic="example_topic",key="example_key")# Default parameters for Kafka connection are usedlogger=logging.getLogger()# No name gives you the root loggerlogger.setLevel("WARNING")logger.addHandler(example_handler)logger.warning("This will push this message to the 'example_topic' in Kafka.")Configuring Kafka ConnectionBy default each handler will create akafka.KafkaProducerinstance, passing on each argument from their__init__(**kwargs)to the KafkaProducer's instantiation.
This means you can configure the connection as specific as you'd like, but every argument should be provided with its keyword;Handler(host="localhost")instead ofHandler("localhost").
All available configuration options are available in thekafka-python documentation.handler=KafkaLogHandler(topic="topic",bootstrap_servers="other_host:9092",retries=0)Configure message loggingEvery handler has theraw_loggingoption which can be provided optionally.
Omitting it from the initialisation, will default it toFalse, meaning the message being logged will be purely what's sent.
If you set it toTrue, first the content will be logged, then appended to the line number and finally the pathname.raw_handler=KafkaLogHandler(topic="topic",raw_logging=True)...logging.info("Test message")A pure handler would emit a message like so:Test message.,theraw_handlerhowever, will emit a message like so:Test message. - 2: /.../file.py.