Saravanan666/pypi-org huggingface dataset:
Package: kafkaPythonWrapper
A wrapper forkafka-pythonInstallationInstall using pip:pipinstallkafkaPythonWrapperInstall from Github:gitclonehttps://github.com/Weizhang2017/pythonKakfaWrappercdpythonKakfaWrapperpythonsetup.pyinstallCLIUsageusage:kafkaPython[-h]--typetype--topictopic[--group_idgroup_id]--bootstrap_serverbootstrap_server[--valuevalue][--keykey]Requiredarguments:--typetypeofkafkaclient,consumerorproducer--topicspecifyatopicforKafka--bootstrap_serverspecifyabootstrapserverforKafka

Optionalarguments:
--group_idspecifyagroupIDforKafkaconsumer
--valuespecifyavaluetosendtoKafka
--keyspecifyakeytosendtoKafkaTheclient typeis either producer or consumer. If the type is producer, the flagvalueis required whilekeyis optional. If the type is consumer, the flaggroup_idis required.Simple Examples:Send a message to KafkakafkaPython--typeproducer--topictest--bootstrap_server'localhost:9092'--keytest_key--valuetest_valuePrint messages from KafkakafkaPython--typeconsumer--bootstrap_server'localhost:9092'--topictest--group_id1APIUsageSend the output of a function to KafkafromkafkaPythonWrapperimportMessageSendermessage_sender=MessageSender(topic='test')@message_sender.send_sync()defproduce_message():key,value='email_address','wzhang@leadbook.com'print(f'{key}:{value}')yieldkey,valueReceive and print messages from KafkafromkafkaPythonWrapperimportMessageCollectormessage_collector=MessageCollector(topic='test',group_id='1')@message_collector.consumedefprint_message(**kwargs):print(kwargs)Close producer or consumermessage_sender.close()message_collector.close()
Package: kafka_replayer
kafka_replayer is a library that helps consume time ranges of messages from Kafka topics. While the
standard Kafka consumer API allows seeking to a specific offset and replaying from there, using
offsets as the replay abstraction is cumbersome and potentially error-prone. This library does the
translation from timestamps to offsets transparently.This library is written in Python, and leverageskafka-python’s consumer to poll Kafka for messages.Installing$pipinstallpython-kafka-replayerUsingimportjsonimportkafka_replayerdes_fn=lambdax:json.loads(x)ifxelseNonereplayer=kafka_replayer.KafkaReplayer('my-topic',bootstrap_servers=['localhost:9092'],key_deserializer=des_fn,value_deserializer=des_fn)# Replay all records between the start and end millis timestampsforrecordinreplayer.replay(1469467314341,1469467907549):printrecordLicenceSeeLICENSE.
Package: kafka-schema-registry
kafka-schema-registryThis library allows you to create topics on Kafka topics, associated with a
Confluent Schema Registry, and publish messages on them.It takes care of:creating the topicpublishing the associated schema (or updating an existing one)serializing and publishing messages to KafkaIt works withkafka-python, and extra arguments are forwarded to it.Installingpipinstallkafka-schema-registryUsagefromkafka_schema_registryimportprepare_producerSAMPLE_SCHEMA={"type":"record","name":"TestType","fields":[{"name":"age","type":"int"},{"name":"name","type":["null","string"]}]}producer=prepare_producer(['localhost:9092'],f'http://schemaregistry',topic_name,1,1,value_schema=SAMPLE_SCHEMA,)producer.send(topic_name,{'age':34})producer.send(topic_name,{'age':9000,'name':'john'})Running the testsThe test requires Docker in order to start a local Redpanda instance.make start-redpandato start the servermake testto configure a virtualenv and run the tests
Package: kafka-slurm-agent
Kafka Slurm AgentThe Kafka Slurm Agent is a distributed computing and stream processing engine
that can be used to run python code across multipleSLURMmanaged HPC clusters or individual workstations.
It uses Kafka to asynchronously communicate with agents installed on clusters and workstations.
It contains a monitoring tool with a Web JSON API and a job submitter.
It is a pure Python implementation using faust stream processingInstallation.Use the standardpiptool to install. The recommended way is to use a Python virtual environment:python3 -m venv venvsource venv/bin/activatepip install kafka-slurm-agentQuick User GuideCreate a new projectIn the folder in which you created thevenvsubfolder run the following command:kafka-slurm create-project --folder .This will generate the following files:kafkaslurm_cfg.py- the configuration fileStartup scripts forworker-agent,cluster-agentandmonitor-agentAn example file to run your code (run.py)The job submitter example (submitter.py)The class that can be optionally used to override the existing implementation of theworker-agent(``my_worker_agent.py`)ConfigurationPlease adjust the config file.Modify the configuration of the connection toApache Kafka. The default one assumes that kafka is running onlocalhostand default port (9092) and doesn't use authentication or SSL.
In the comments you will find parameters necessary to connect toKafkaconfigured using SASL and plaintext password. If you use this type of connection please uncomment also the line that starts with:# KAFKA_FAUST_BROKER_CREDENTIALSMake sure that ``PREFIX` points to the location of your projectChange the names of topics used for your project to avoid any conflict with projects sharing the same kafka instance.If you want to use a SLURM cluster please change the jobCLUSTER_JOB_NAME_SUFFIX = '_KSA'to avoid conflicts with other projects running on your slurm cluster. The jobs managed bycluster-agentwill be named"JOBID_SUFFIX"where the JOBID is the identifier that you assign when submitting a job and SUFFIX is handled by this configuration parameter.For a full list of configuration parameters refer to the documentation.Creating topics on Kafka with appropriate partitionsUse the built-in commandkafka-slurmto create topics. You should set the--new-topic-partitionsparamter to at least the number of planned clusters and workstations that will be used simultaneously.
For example:kafka-slurm --new-topic-partitions 4 topics-createImplement your codeAn example of a script that you can implement is generated inrun.pySubmitting jobsOnce this is ready, you can test your project locally:Open a new terminal and start theworker-agent(./start_worker_agent)Open a new terminal and start themonitor-agent(./start_monitor_agent)Submit new jobs using thesubmitter.pyYou can monitor the execution by openinghttp://localhost:6067/mon/stats/on the host on which you've started themonitor-agent.
Package: kafka-temporal-mapper-client
Kafka-Temporal Mapper ClientKafka-Temporal mapper client facilitates the interaction between the workflows and the Kafka-Temporal mapper server.
This library is used by the workflows and provides methods for publishing of the workflow and mapping events.The client provides two methods:subscribe: used to subscribe a workflow to the execution of another workflow.notify: used by a workflow to notify the termination of its execution.Installationpipinstallkafka_temporal_mapper_clientUsageTo use this library for notifying or subscribing:fromkafka_temporal_mapper.clientimportKafkaTemporalClientktc=KafkaTemporalClient()# subscribe workflow B to workflow A using a part of A's results as argument (@b.c)ktc.subscribe('WorkflowB',[{'name':'WorkflowA','args':{"a":"hello","b":"@b.c"}}])# notify workflow A execution and its resultsktc.notify('WorkflowA',{"a":"hello","b":{"c":"world"}})
Package: kafka-tfrx
[![Build Status](https://travis-ci.org/kryptoflow/kafka-tfrx.svg?branch=master)](https://travis-ci.org/kryptoflow/kafka-tfrx)Simple, idiomatic python API to interface Kafka topics with [RxPY](https://github.com/ReactiveX/RxPY)## DescriptionThe goal of this library is mostly to enable reactive access to Kafka topics. It builds on top of Confluent Kafka’s
python [client](https://github.com/confluentinc/confluent-kafka-python), and allows building observable streams from
any Kafka topic.The main motivation for this library is using it with [online learning](https://en.wikipedia.org/wiki/Online_machine_learning),
but it is by no means limited to that. In a way, it provides a subset of the functionality that Kafka-Streams offers,
which currently does not have a stable python implementation.## UsageSeeexamples/## TestsRuntoxfrom top level dir## NoteThis project has been set up using PyScaffold 3.0.3. For details and
usage information on PyScaffold see <http://pyscaffold.org/>.
Package: kafkatool
kafkatoolA wrap ofconfluent-kafka-python.
Provides simple consumer/producer create tool and statistic metrics show.installbash scripts/run_install.shquick startseeexampledirectory for more detailstodoprogram run metrics statistic and displayan optimization default configuration for consumer and producerreferenceconfluent-kafka-pythonconfluent-kafka-python docconfluent docslibrdkafkaconfigurationstatistic
Package: kafka-tools
This repository is a collection of tools for working withApache Kafka.The Site Reliability team for Kafka at LinkedIn has built these over time in
order to make managing Kafka a little bit easier. Our intention is to add to
this repository as more tools are developed.We welcome additions and modifications that make managing Kafka better for all!Quick StartFirst, download and install the Kafka admin utilities from:http://kafka.apache.org/downloads.htmlThen, install kafka-tools using the following command:pip install kafka-toolsAdditional documentation may be found athttps://kafka-tools.readthedocs.io.ContributingPlease see thecontribution guidelines.Other ProjectsIn addition to these tools, LinkedIn has also open-sourcedBurrow, a robust system for monitoring
Kafka consumer clients.LicenseCopyright 2016 LinkedIn Corp.Licensed under the Apache License, Version 2.0 (the “License”);
you may not use this file except in compliance with the License.
You may obtain a copy of the License at:http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an “AS IS” BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Package: kafka-utils
Kafka-UtilsA suite of python tools to interact and manage Apache Kafka clusters.
Kafka-Utils runs on python 3.7+.ConfigurationKafka-Utils reads cluster configuration needed to access Kafka clusters from yaml files. Each cluster is identified bytypeandname.
Multiple clusters of the same type should be listed in the sametype.yamlfile.
The yaml files are read from$KAFKA_DISCOVERY_DIR,$HOME/.kafka_discoveryand/etc/kafka_discovery, the former overrides the latter.Sample configuration forsample_typecluster at/etc/kafka_discovery/sample_type.yaml---clusters:cluster-1:broker_list:-"cluster-elb-1:9092"zookeeper:"11.11.11.111:2181,11.11.11.112:2181,11.11.11.113:2181/kafka-1"cluster-2:broker_list:-"cluster-elb-2:9092"zookeeper:"11.11.11.211:2181,11.11.11.212:2181,11.11.11.213:2181/kafka-2"local_config:cluster:cluster-1InstallFrom PyPI:$pipinstallkafka-utilsKafka-Utils command-line interfaceList all clusters$kafka-utilscluster-typesample_type:cluster-name:cluster-1broker-list:cluster-elb-1:9092zookeeper:11.11.11.111:2181,11.11.11.112:2181,11.11.11.113:2181/kafka-1cluster-name:cluster-2broker-list:cluster-elb-2:9092zookeeper:11.11.11.211:2181,11.11.11.212:2181,11.11.11.213:2181/kafka-2Get consumer offsets$kafka-consumer-manager--cluster-typesample_typeoffset_getsample_consumerGet consumer watermarks$kafka-consumer-manager--cluster-typesample_typeget_topic_watermarksample.topicRebalance cluster cluster1 of type sample_cluster$kafka-cluster-manager--cluster-typesample_type--cluster-namecluster1--applyrebalance--brokers--leaders--max-partition-movements10--max-leader-changes15Rolling-restart a cluster$kafka-rolling-restart--cluster-typesample_typeCheck in-sync replicas$kafka-check--cluster-typesample_typemin_isrCheck number of unavailable replicas$kafka-check--cluster-typesample_typereplica_unavailabilityDocumentationRead the documentation atRead the Docs.LicenseKafka-Utils is licensed under the Apache License, Version 2.0:http://www.apache.org/licenses/LICENSE-2.0ContributingEveryone is encouraged to contribute to Kafka-Utils by forking theGithub repositoryand making a pull request or opening an issue.
Package: kafkawrapper
KafkaWrapper is a Python library designed to simplify Kafka producer setup and data transmission to Kafka topics. This wrapper abstracts the complexities of configuring a Kafka broker and producer, allowing developers to seamlessly integrate Kafka into their projects.Featureskafkawrapper simplifies the process of setting up a Kafka broker and producer, saving you time and effort.asily send data to Kafka topics without the need for extensive Kafka knowledge.Customize your Kafka producer settings to suit your specific use case.UsageTo send data to Kafka, follow these steps:Import thesendDatamodules.Use this module to send data to kafka.Argument are
kafka_topic: Topic name
data: that you want to send to kafka
metadata(optional): id - unique id to fetch particular record from kafkaLicenseThis project is licensed under the MIT License.
Package: kafkit
KafkitKafkit helps you write Kafka producers and consumers in Python with asyncio:Kafkit provides a client for the Confluent Schema Registry's HTTP API.
TheRegistryApiclient includes both high-level methods for managing subjects and schemas in a Registry, and direct low-level access to HTTP methods (GET, POST, PUT, PATCH, and DELETE).
The high-level methods use caching so you can use the client as an integral part of your application's schema management.RegistryApiis implemented aroundaiohttp, but since the base class is designed with asans IO architecture, a Registry client can be implemented with any asyncio HTTP library.Kafkit provides Avro message serializers and deserializers that integrate with theConfluent Schema Registry:Deserializer,Serializer, andPolySerializer.TheRecordNameSchemaManageris a streamlined tool for serializing messages using the schemas maintained by your app, while also integrating with theConfluent Schema Registry.Thekafkit.sslmodule helps you connect to SSL-secured Kafka brokers.Learn more about Kafkit athttps://kafkit.lsst.io.
Package: kafkoroutine
kafkoroutineThis repository offers Python-based asynchronous wrappers aroundkafka-python's Producer and Consumer, bridging them
seamlessly with Python'sasyncio.PrerequisitesPython 3.7 or laterA running Kafka instance (for actual message passing)Getting StartedFollow these instructions to integrate the asynchronous Kafka components in your asyncio-based Python project.InstallationInstallkafkoroutineusing pip:pipinstallkafkoroutinePlease note: this project requires Python 3.7 or later and is built upon thekafka-pythonlibrary.UsageAsyncKafkaConsumerfromkafkoroutine.consumerimportAsyncKafkaConsumerasyncwithAsyncKafkaConsumer(topics=topics,bootstrap_servers='localhost:9092',executor=None)asconsumer:asyncformessageinconsumer:print(f"Received:{message.value.decode('utf-8')}")AsyncKafkaProducerfromkafkoroutine.producerimportAsyncKafkaProducerasyncwithAsyncKafkaProducer(bootstrap_servers='localhost:9092',executor=None)asproducer:formsginmessages:awaitproducer.send(topic,msg)Built WithPoetry- Packaging and dependency managementasyncio- Asynchronous I/O, event loop, and coroutines used for the
implementation.kafka-python- The Python client for Apache Kafka upon which these
asynchronous wrappers are built.LicenseThis project follows the guidelines of the MIT License.
Package: kaflow
kaflowKafka streams topic processing in Python.kaflowis a simple framework that allows you to build Kafka streams processing aplications in Python with ease.Some of the features offered bykaflow:Dependency Injection system inspired byFastAPIandxpresso, and backed bydi.Automatic deserialization of incoming messages and serialization of outgoing messages. Supports popular formats likeJSON,AvroorProtobuf.Message validation thanks topydantic.RequirementsPython 3.8+InstallationpipinstallkaflowExamplefromkaflowimport(FromHeader,FromKey,FromValue,Json,Kaflow,Message,MessageOffset,MessagePartition,MessageTimestamp,String,)frompydanticimportBaseModelclassUserClick(BaseModel):user_id:inturl:strtimestamp:intclassKey(BaseModel):environment:strapp=Kaflow(name="AwesomeKakfaApp",brokers="localhost:9092")@app.consume(topic="user_clicks",sink_topics=["user_clicks_json"])asyncdefconsume_user_clicks(message:FromValue[Json[UserClick]],key:FromKey[Json[Key]],x_correlation_id:FromHeader[String[str]],x_request_id:FromHeader[String[str]],partition:MessagePartition,offset:MessageOffset,timestamp:MessageTimestamp,)->Message:# Do something with the message...# Publish to another topicreturnMessage(value=b'{"user_clicked": "true"}')app.run()
Package: kaftar
Kaftar SDKUsagePackage installationpipinstallkaftarUsage:importuuidimporttimefromkaftarimportNotificationmessage_uuid=uuid.uuid4()group_uuid=uuid.uuid4()broker_url="BROKER_URL_HERE"app=Notification('app_name',broker_url)app.send_notification({'subject':'Notification title goes here','content':'Notification body goes here'},[{'receiver':"076f08cc-4122-400a-bffa-2a0157ba57eb",# can be email or phone number'message_uuid':str(message_uuid),# Optional (task will generate an id if not provided here)'uuid':"076f08cc-4122-400a-bffa-2a0157ba57eb"}],int(time.time()),group_uuid=group_uuid# Optional)# Delete single notificationapp.delete_notification(message_uuid)# Delete group of notificationsapp.delete_group_notification(group_uuid)
Package: kaftools
**Warning: this is a side-project in progress so many bugs could arise. Please raise an issue if this happens.**# Kernel Adaptive Filtering for Python[![Build status](https://ci.appveyor.com/api/projects/status/otkhkwrimf3e4vg5?svg=true)](https://ci.appveyor.com/project/Canas/kaftools) [![GitHub license](https://img.shields.io/badge/license-MIT-blue.svg)](https://raw.githubusercontent.com/Canas/kaftools/master/LICENSE)This package implements several Kernel Adaptive Filtering algorithms for research purposes. It aims to be easily extendable.# Requirements- Python 3.4+- NumPy- SciPy- (Optional) Matplotlib# Features## Adaptive Kernel Filters- Kernel Least Mean Squares (KLMS) - `KlmsFilter`- Exogenous Kernel Least Mean Squares (KLMS-X) - `KlmsxFilter`- Kernel Recursive Least Squares (KRLS) - `KrlsFilter`## Sparsification Criteria- Novelty (KLMS)- Approximate Linear Dependency (KLRS)## Additional Features- Delayed input support (KLMS)- Adaptive kernel parameter learning (KLMS)For a more visual comparison, check the [latest features sheet](https://docs.google.com/spreadsheets/d/1kvBNAqDSgNGBTcXqMDN7j_dpp949peH_-F1GYVP29y8/edit?usp=sharing).# QuickstartLet's do a simple example using a KLMS Filter over given input and target arrays:```from kaftools.filters import KlmsFilterfrom kaftools.kernels import GaussianKernelklms = KlmsFilter(input, target)klms.fit(learning_rate=0.1, kernel=GaussianKernel(sigma=0.1))```And that's it!
Package: kagaconf
kagaconfSimple Python configuration loading interface I wrote for use in KAGAYAKI.
Actually most of this code was written during the development of gmsh, but I've finally found a nice name
to turn this into a standalone package.Usagekagaconf provides a global configuration dictionary, which can be nested arbitrarily.
Adding items to this dictionary is easy:importkagaconf# The simplest way to fill the dict is by copying another dict.# I recommend this for default values.kagaconf.from_dict({"key":"value"})# Filling configuration from a file or all files in a directory is also very easy.# There are many options for controlling this process, please refer to the API docs.# Right now it only supports yaml, but adding at least json and toml should be trivial.kagaconf.from_path('./path/to/your/config.yaml')kagaconf.from_path('./path/to/your/config/directory',filter=r'[^_].*\.yaml',recursive=True)# Lastly you can use a list of named variables to fill a skeleton dict with data.# I imagine this will mainly be used to import environment variables into the config,# that's why os.environ is the default data source (but you can obviously specify your own).kagaconf.from_env_mapping({"this_key_will_be_filled":"BY_THIS_ENV_VARIABLE"})# Found values are applied in order, so default values should go first# and values with high precedence (e.g. environment overrides) last.Once the configuration dict is prepared, it can easily be accessed from anywhere:fromkagaconfimportcfg# Use normal python dot syntax to get to the key you want.# Call it like a function to get the value behind it.cfg.hello()# This will throw a KeyError if the key has no value, unless you provide a default value.# If no default was provided here, resolution would stop at "nonexistent".cfg.nonexistent.key("this is fine")# You can also use dict syntax to get subsequent elements.# This can be freely mixed with the dot syntax.cfg["this is cool"].and_even_required_for_using_number_indices[1234]\.but_also.you_can_do["something.like.this"]()The paths you build are stored asPathChainerobjects and can be completely arbitrary in structure.
They are only evaluated when you call them like a function, at which point a value will be resolved.
So you will never encounter aKeyErrorwhen building the path, only when resolving without a fallback value.InstallationYou can install this library viapip install kagaconf. That is a sentence I never thought
I could write, but here we are.LicenseIf you find this, think the API looks sweet and wanna use it, go ahead.
If you think the code magic I did to make this work seamlessly is cool, 1)
stop bc it's really not, and 2) feel free to use and abuse as much of it as you like.
Package: kagami
Kagami LibraryThe Kagami library is a Python package to accelerate the development of novel computational biology algorithms.
It is currently under rapid growth. Although the APIs are aimed to remain consistent within a major version, compatible between releases are not guaranteed.
Please note that there is no plan to include documents anytime soon.The Kagami library is distributed under the GNU Lesser General Public License v3.0.InstallationUsing pip:pipinstallkagamiTestingpython-c"import kagami; kagami.test()"CitationIf you use Kagami, ACS or MOCHA in a publication, we would appreciate citations: (coming soon)
Package: kagcolabserver
Library for importing kaggle data into colab and run vscode on the folderThis is already implemented by ML Genius Abhishek Thakur, just a litte bit refinement and I have added methods for importing kaggle data straight into yourDrive folder and the vscode will open in your kaggle folderInstallation:pip install kagcolabserverBasics optional params for the Code:Primary params-- port : for hosting the code server locally and tunnel using the ngrok-- drive : for mouting the drive-- folder_tree : Folder structure which will be created in the drive Ex: titanic/ or kaggle/titanic etc...-- kaggle : for working with kaggle user api and data apiSecondary paramsNote :On clicking API key in your profile, a file named kaggle.json with username and api key will be downloaded, kindly use them for below params-- username : Your Kaggle username-- kag_api_key : API key found in your profile-- kag_data_api : On each kaggle competition, a data api will be given should pass that one!How to Use the library?For just starting the code server from google colab* from kagcolabserver import kagColabServer
* kagColabServer(port=7000, drive=True, kaggle=False)For starting vscode in particular foler* from kagcolabserver import kagColabServer
* kagColabServer(port=7000, drive=True, kaggle=False, folder_tree="kaggle")For importing Kaggle as well as starting the code-server in the folder location* from kagcolabserver import kagColabServe
* kagColabServer(port=4000, drive=True)
* kolab = kagColabServer(port=4000,drive=True, kaggle=True)
* kolab.import_kaggle_data(username='XXXX', kag_api_key='XXXXX', kag_data_api='kaggle competitions download -c titanic', folder_tree='titanic/')