Saravanan666/pypi-org huggingface dataset:
Package: taskingai
TaskingAI-clientThe TaskingAI Python client for creating and managing AI-driven applications.For more information, see the docs atTaskingAI DocumentationInstallationInstall the latest released version using pip:pip3installtaskingaiQuickstartHere's how you can quickly start building and managing AI-driven applications using the TaskingAI client.AssistantsExplore the ease of creating and customizing your own AI assistants with TaskingAI to enhance user interactions.importtaskingaifromtaskingai.assistantimport*fromtaskingai.assistant.memoryimportAssistantNaiveMemory# Initialize your API key if you haven't already set it in the environmenttaskingai.init(api_key="YOUR_API_KEY")# Create an assistantassistant=create_assistant(model_id="YOUR_MODEL_ID",memory=AssistantNaiveMemory(),system_prompt_template=["You are a professional assistant."],)print(f"Assistant created:{assistant.id}")# Get details about the assistantassistant_details=get_assistant(assistant_id=assistant.id)print(f"Assistant details:{assistant_details}")# Update the assistant's descriptionupdate_assistant(assistant_id=assistant.id,description="An updated description for my assistant.")print(f"Assistant updated.")# Delete the assistant when donedelete_assistant(assistant_id=assistant.id)print("Assistant deleted successfully.")RetrievalLeverage TaskingAI's retrieval capabilities to store, manage, and extract information, making your applications smarter and more responsive.importtaskingaifromtaskingai.retrievalimport*# Create a collection for storing and retrieving datacollection=create_collection(embedding_model_id="YOUR_MODEL_ID",capacity=1000)print(f"Collection created:{collection.id}")# Add a record to the collectionrecord=create_record(collection_id=collection.id,content="Example text for machine learning.",text_splitter=TokenTextSplitter(chunk_size=200,chunk_overlap=20),)print(f"Record added to collection:{record.id}")# Retrieve the record from the collectionretrieved_record=get_record(collection_id=collection.id,record_id=record.id)print(f"Record retrieved:{retrieved_record.text}")# Delete the recorddelete_record(collection_id=collection.id,record_id=record.id)print("Record deleted.")# Delete the collectiondelete_collection(collection_id=collection.id)print("Collection deleted.")ToolsUtilize TaskingAI's tools to create actions that enable your assistant to interact with external APIs and services, enriching the user experience.importtaskingaifromtaskingai.toolimport*# Define a schema for the tool actionNUMBERS_API_SCHEMA={# Schema definition goes here}# Create a tool action based on the defined schemaactions=bulk_create_actions(schema=NUMBERS_API_SCHEMA,authentication=ActionAuthentication(type=ActionAuthenticationType.NONE))action=actions[0]print(f"Action created:{action.id}")# Run the action for a test purposeresult=run_action(action_id=action.id,parameters={"number":42})print(f"Action result:{result}")# Delete the action when donedelete_action(action_id=action.id)print("Action deleted.")ContributingWe welcome contributions of all kinds. Please read ourContributing Guidelinesfor more information on how to get started.
Package: taskiq
Documentation:https://taskiq-python.github.io/What is taskiq?Taskiq is an asynchronous distributed task queue for python.
This project takes inspiration from big projects such asCeleryandDramatiq.
But taskiq can send and run both the sync and async functions, has
integration with popular async frameworks, such asFastAPIandAioHTTP.Also, we usePEP-612to provide the best autosuggestions possible. All code is type-hinted.InstallationThis project can be installed using pip:pipinstalltaskiqOr it can be installed directly from git:pipinstallgit+https://github.com/taskiq-python/taskiqUsageAt first you need to create a broker. Broker is an object that can communicate to workers using distributed queues.We have differet brokers for different queue backends. For example, we have a broker forNATS,Redis,RabbitMQ,Kafkaand even more. Choose the one that fits you and create an instance.fromtaskiq_natsimportJetStreamBrokerbroker=JetStreamBroker("nats://localhost:4222",queue="my_queue")Declaring tasks is as easy as declaring a function. Just add a decorator to your function and you are ready to go.importasynciofromtaskiq_natsimportJetStreamBrokerbroker=JetStreamBroker("nats://localhost:4222",queue="my_queue2")@broker.taskasyncdefmy_task(a:int,b:int)->None:print("AB",a+b)asyncdefmain():awaitbroker.startup()awaitmy_task.kiq(1,2)awaitbroker.shutdown()if__name__=="__main__":asyncio.run(main())The message is going to be sent to the broker and then to the worker. The worker will execute the function. To start worker processes, just run the following command:taskiqworkerpath.to.the.module:brokerWherepath.to.the.moduleis the path to the module where the broker is defined andbrokeris the name of the broker variable.If you have tasks in different modules, you can ask taskiq to automatically import them by passing the--fs-discoverflag:taskiqworkerpath.to.the.module:broker--fs-discoverIt will import all modules calledtasks.pyin the current directory and all subdirectories.Also, we support hot reload for workers. To enable it, just pass the--reloadflag. It will reload the worker when the code changes (To use it, install taskiq with reload extra. E.gpip install taskiq[reload]).Also, we have cool integrations with popular async frameworks. For example, we have an integration withFastAPIorAioHTTP. You can use it to reuse dependencies from your web app in your tasks.Read about all features in our documentation:https://taskiq-python.github.io/Local developmentLintingWe use pre-commit to do linting locally.After cloning this project, please installpre-commit. It helps fix files before committing changes.pre-commitinstallTestingPytest can run without any additional actions or options.pytestDocsTo run docs locally, you need to installyarn.First, you need to install dependencies.yarn installAfter that you can set up a docs server by running:yarn docs:dev
Package: taskiq-aiogram
Taskiq + AiogramThis repo adds integration between your aiogram application and taskiq.It runs all startup and shutdown events of your application and adds 3 dependencies,
that you can use in your tasks.Dispatcher- that were used along with executor;Bot- your main bot instance;List[Bot]- all your bots.UsageDefine startup and shutdown events for your dispatcher.
We use events, because we need to identify what your bot
wants to do on startup and shutdown.Also, it's useful for bot developers to distinct buisness logic
from startup of the bot.Below you'll find an example, how to integrate taskiq with your favorite
bot framework.importasyncioimportloggingimportsysfromaiogramimportBot,Dispatcher# Your taskiq brokerfromyour_project.tkqimportbrokerdp=Dispatcher()bot=Bot(token="TOKEN")bot2=Bot(token="TOKEN")@dp.startup()asyncdefsetup_taskiq(bot:Bot,*_args,**_kwargs):# Here we check if it's a clien-side,# Becuase otherwise you're going to# create infinite loop of startup events.ifnotbroker.is_worker_process:logging.info("Setting up taskiq")awaitbroker.startup()@dp.shutdown()asyncdefshutdown_taskiq(bot:Bot,*_args,**_kwargs):ifnotbroker.is_worker_process:logging.info("Shutting down taskiq")awaitbroker.shutdown()asyncdefmain():awaitdp.start_polling(bot,bot2)if__name__=="__main__":logging.basicConfig(level=logging.INFO,stream=sys.stdout)asyncio.run(main())The only thing that left is to add few lines to your broker definition.# Please use real broker for taskiq.fromtaskiq_brokerimportMyBrokerimporttaskiq_aiogrambroker=MyBroker()# This line is going to initialize everything.taskiq_aiogram.init(broker,"your_project.__main__:dp","your_project.__main__:bot","your_project.__main__:bot2",)That's it.Let's create some tasks! I created task in a separate module,
namedtasks.py.fromaiogramimportBotfromyour_project.tkqimportbroker@broker.task(task_name="my_task")asyncdefmy_task(chat_id:int,bot:Bot=TaskiqDepends())->None:print("I'm a task")awaitasyncio.sleep(4)awaitbot.send_message(chat_id,"task completed")Now let's call our new task somewhere in bot commands.fromaiogramimporttypesfromaiogram.filtersimportCommandfromtasksimportmy_task@dp.message(Command("task"))asyncdefmessage(message:types.Message):awaitmy_task.kiq(message.chat.id)To start the worker, please type:taskiq worker your_project.tkq:broker --fs-discoverWe use--fs-discoverto find all tasks.py modules recursively
and import all tasks into broker.Now we can fire the task and see everything in action.
Package: taskiq-aiohttp
Taskiq + AioHTTPThis project is used to create mocked application and request
that you can use as a dependencies in your taskiq application.It's useful because it runs all startup events of your application
and everything that you might expect in your application's state is
available inside of your tasks.We suggest to use this library along withtaskiq-python/aiohttp-deps, because it might be super handy to reuse same dependencies of your application in your tasks.To add an integration, you need to call the functioninitin your broker's module.importtaskiq_aiohttpbroker=...taskiq_aiohttp.init(broker,"project.module:app")How does it work?It adds startup functions to the broker, so it imports your aiohttp application and creates a single worker-wide Request and Application objects that you can depend on.THIS REQUEST IS NOT RELATED TO THE ACTUAL REQUESTS IN AioHTTP! This request won't have actual data about the request you were handling while sending a task.Manual context updatesSometimes it's required to update context manually. For example, for tests.
If you need to add context in your broker by hand, please use function populate_context.Imagine, you want to use InMemoryBroker for testing and your broker file looks like this:broker=MyBroker()ifenv=="pytest":broker=InMemoryBroker()In this case your context won't be updated, because inmemory brokers cannot run as workers.
To solve this issue, we have a populate context function. It's a bit complex and takes lots of
parmeters. But here's a fixture that creates aiohttp test client and populates context of inmemory broker.importasynciofromtypingimportAsyncGeneratorimportpytestfromaiohttpimportwebfromaiohttp.test_utilsimportBaseTestServer,TestClient,TestServerfromtaskiq_aiohttpimportpopulate_context@pytest.fixtureasyncdeftest_client(app:web.Application,)->AsyncGenerator[TestClient,None]:"""Create a test client.This function creates a TestServerand a test client for the application.Also this fixture populates contextwith needed variables.:param app: current application.:yield: ready to use client."""loop=asyncio.get_running_loop()server=TestServer(app)client=TestClient(server,loop=loop)awaitclient.start_server()# This is important part.# Since InMemoryBroker doesn't# run in worker_process, we have to populate# broker's context manually.populate_context(broker=broker,server=server.runner.server,app=app,loop=None,)yieldclientawaitclient.close()
Package: taskiq-aio-kafka
AioKafka broker for taskiqThis lirary provides you with aiokafka broker for taskiq.Usage:fromtaskiq_aio_kafkaimportAioKafkaBrokerbroker=AioKafkaBroker(bootstrap_servers="localhost")@broker.taskasyncdeftest()->None:print("The best task ever!")Non-obvious thingsYou can configure kafka producer and consumer with special methodsconfigure_producerandconfigure_consumer.
Example:fromtaskiq_aio_kafkaimportAioKafkaBrokerbroker=AioKafkaBroker(bootstrap_servers="localhost")# configure producer, you can set any parameter from# base AIOKafkaProducer, except `loop` and `bootstrap_servers`broker.configure_producer(request_timeout_ms=100000)# configure consumer, you can set any parameter from# base AIOKafkaConsumer, except `loop` and `bootstrap_servers`broker.configure_consumer(group_id="the best group ever.")ConfigurationAioKafkaBroker parameters:bootstrap_servers- url to kafka nodes. Can be either string or list of strings.kafka_topic- custom topic in kafka.result_backend- custom result backend.task_id_generator- custom task_id genertaor.kafka_admin_client- customkafkaadmin client.delete_topic_on_shutdown- flag to delete topic on broker shutdown.
Package: taskiq-elastic-apm
TaskIQ Elastic APM MiddlewareTaskIQ Elastic APM Middleware is a Python library providing easy integration of Elastic APM monitoring into TaskIQ task processing. This middleware enables efficient tracking and analysis of task performance and errors, leveraging the capabilities of Elastic APM.FeaturesEasy integration with TaskIQ workers.Automatic performance and error tracking for tasks.Customizable to fit various Elastic APM configurations.InstallationYou can install the TaskIQ Elastic APM Middleware directly from PyPI:pipinstalltaskiq-elastic-apmUsage
To use the middleware in your TaskIQ project, simply import and add it to your TaskIQ application:python
Copy code
from taskiq import Taskiq
from taskiq_elastic_apm_middleware import ElasticApmMiddlewareCreate your Taskiq instancebroker=AioPikaBroker(url=settings.CELERY_BROKER_URL,queue_name=settings.CELERY_DEFAULT_QUEUE,exchange_name=settings.CELERY_DEFAULT_QUEUE,).with_middlewares(ElasticApmMiddleware(client=apm,))Define your tasks and start your application as usualConfigurationYou can configure the middleware by passing parameters to ElasticApmMiddleware.The main parameters are:client: elasticapm.Client - The Elastic APM client instance to use. If not provided, a new client will be created using the following parameters.
Package: taskiq-faststream
Taskiq - FastStreamThe current package is just a wrapper forFastStreamobjects to make them compatible withTaskiqlibrary.The main goal of it - provideFastStreamwith a greatTaskiqtasksschedulingfeature.InstallationIf you already haveFastStreamproject to interact with your Message Broker, you can add scheduling to it by installing just ataskiq-faststreampipinstalltaskiq-faststreamIf you starting with a clear project, you can specifytaskiq-faststreambroker by the following distributions:pipinstalltaskiq-faststream[rabbit]# orpipinstalltaskiq-faststream[kafka]# orpipinstalltaskiq-faststream[nats]UsageThe package gives you two classes:AppWrapperandBrokerWrapperThese are just containers for the relatedFastStreamobjects to make themtaskiq-compatibleTo create scheduling tasks for your broker, just wrap it toBrokerWrapperand use it like a regulartaskiqBroker.# regular FastStream codefromfaststream.natsimportNatsBrokerbroker=NatsBroker()@broker.subscriber("test-subject")asyncdefhandler(msg:str):print(msg)# taskiq-faststream schedulingfromtaskiq.schedule_sourcesimportLabelScheduleSourcefromtaskiq_faststreamimportBrokerWrapper,StreamScheduler# wrap FastStream objecttaskiq_broker=BrokerWrapper(broker)# create periodic tasktaskiq_broker.task(message="Hi!",# If you are using RabbitBroker, then you need to replace subject with queue.# If you are using KafkaBroker, then you need to replace subject with topic.subject="test-subject",schedule=[{"cron":"* * * * *",}],)# create scheduler objectscheduler=StreamScheduler(broker=taskiq_broker,sources=[LabelScheduleSource(taskiq_broker)],)To run the scheduler, just use the following commandtaskiqschedulermodule:schedulerAlso, you can wrap yourFastStreamapplication the same way (allows to use lifespan events and AsyncAPI documentation):# regular FastStream codefromfaststreamimportFastStreamfromfaststream.natsimportNatsBrokerbroker=NatsBroker()app=FastStream(broker)@broker.subscriber("test-subject")asyncdefhandler(msg:str):print(msg)# wrap FastStream objectfromtaskiq_faststreamimportAppWrappertaskiq_broker=AppWrapper(app)# Code below omitted 👇A little feature: instead of using a finalmessageargument, you can set a message callback to collect information right before sending:asyncdefcollect_information_to_send():return"Message to send"taskiq_broker.task(message=collect_information_to_send,...,)Also, you can send a multiple message by one task call just using generator message callback withyieldasyncdefcollect_information_to_send():"""Sends 10 messages per task call."""foriinrange(10):yielditaskiq_broker.task(message=collect_information_to_send,...,)
Package: taskiq-litestar
Taskiq LitestarThis project adds integration withLitestartoTaskIQ.Mainly this project focuses on running starup and shutdown events declared in your litestar app
on worker nodes. This will allow you to access application's state and data from within your tasks.Also we add a few dependencies that you can depend on in your tasks.Statefromlitestar.datastructures;Litestarfromlitestar.Installationpipinstalltaskiq-litestarUsageHere we have a script calledtest_script.pyso the listestar app can be found attest_script:app. We use strings to resolve application to bypass circular imports.In the declared task I depend on a state.fromcontextlibimportasynccontextmanagerfromlitestarimportLitestar,getfromlitestar.datastructuresimportStatefromtaskiqimportTaskiqDependsfromtaskiq_redisimportListQueueBrokerimporttaskiq_litestarbroker=ListQueueBroker("redis://localhost:6379/0")taskiq_litestar.init(broker,"test_script:app",)@asynccontextmanagerasyncdefapp_lifespan(app:Litestar)->None:"""Lifespan generator."""ifnotbroker.is_worker_process:awaitbroker.startup()app.state.value="abc123"yieldifnotbroker.is_worker_process:awaitbroker.shutdown()@broker.task()asyncdefmy_task(state:State=TaskiqDepends())->None:"""My task."""print("a",state.dict())# noqa: T201@get("/")asyncdefindex()->str:"""Index get handler."""awaitmy_task.kiq()return"Task sent"app=Litestar([index],lifespan=[app_lifespan])Manually update dependency contextWhen using InMemoryBroker you can manually update the dependency context.
This might come handy when setting up tests.importtaskiq_litestarfromtaskiqimportInMemoryBrokerbroker=InMemoryBroker()app=FastAPI()taskiq_fastapi.init(broker,"test_script:app")taskiq_fastapi.populate_dependency_context(broker,app)
Package: taskiq-nats
Taskiq NATSTaskiq-nats is a plugin for taskiq that adds NATS broker.
This package has support for NATS JetStream.InstallationTo use this project you must have installed core taskiq library:pipinstalltaskiqtaskiq-natsUsageHere's a minimal setup example with a broker and one task.Default NATS broker.importasynciofromtaskiq_natsimportNatsBroker,JetStreamBrokerbroker=NatsBroker(["nats://nats1:4222","nats://nats2:4222",],queue="random_queue_name",)@broker.taskasyncdefmy_lovely_task():print("I love taskiq")asyncdefmain():awaitbroker.startup()awaitmy_lovely_task.kiq()awaitbroker.shutdown()if__name__=="__main__":asyncio.run(main())NATS broker based on JetStreamimportasynciofromtaskiq_natsimport(PushBasedJetStreamBroker,PullBasedJetStreamBroker)broker=PushBasedJetStreamBroker(servers=["nats://nats1:4222","nats://nats2:4222",],queue="awesome_queue_name",)# Or you can use pull based variantbroker=PullBasedJetStreamBroker(servers=["nats://nats1:4222","nats://nats2:4222",],durable="awesome_durable_consumer_name",)@broker.taskasyncdefmy_lovely_task():print("I love taskiq")asyncdefmain():awaitbroker.startup()awaitmy_lovely_task.kiq()awaitbroker.shutdown()if__name__=="__main__":asyncio.run(main())NatsBroker configurationHere's the constructor parameters:servers- a single string or a list of strings with nats nodes addresses.subject- name of the subect that will be used to exchange tasks betwee workers and clients.queue- optional name of the queue. By default NatsBroker broadcasts task to all workers,
but if you want to handle every task only once, you need to supply this argument.result_backend- custom result backend.task_id_generator- custom function to generate task ids.Every other keyword argument will be sent tonats.connectfunction.JetStreamBroker configurationCommonservers- a single string or a list of strings with nats nodes addresses.subject- name of the subect that will be used to exchange tasks betwee workers and clients.stream_name- name of the stream where subjects will be located.queue- a single string or a list of strings with nats nodes addresses.result_backend- custom result backend.task_id_generator- custom function to generate task ids.stream_config- a config for stream.consumer_config- a config for consumer.PushBasedJetStreamBrokerqueue- name of the queue. It's used to share messages between different consumers.PullBasedJetStreamBrokerdurable- durable name of the consumer. It's used to share messages between different consumers.pull_consume_batch- maximum number of message that can be fetched each time.pull_consume_timeout- timeout for messages fetch. If there is no messages, we start fetching messages again.
Package: taskiq-redis
TaskIQ-RedisTaskiq-redis is a plugin for taskiq that adds a new broker and result backend based on redis.InstallationTo use this project you must have installed core taskiq library:pipinstalltaskiqThis project can be installed using pip:pipinstalltaskiq-redisUsageLet's see the example with the redis broker and redis async result:# broker.pyimportasynciofromtaskiq_redisimportListQueueBroker,RedisAsyncResultBackendredis_async_result=RedisAsyncResultBackend(redis_url="redis://localhost:6379",)# Or you can use PubSubBroker if you need broadcastingbroker=ListQueueBroker(url="redis://localhost:6379",result_backend=redis_async_result,)@broker.taskasyncdefbest_task_ever()->None:"""Solve all problems in the world."""awaitasyncio.sleep(5.5)print("All problems are solved!")asyncdefmain():task=awaitbest_task_ever.kiq()print(awaittask.wait_result())if__name__=="__main__":asyncio.run(main())Launch the workers:taskiq worker broker:brokerThen run the main code:python3 broker.pyPubSubBroker and ListQueueBroker configurationWe have two brokers with similar interfaces, but with different logic.
The PubSubBroker uses redis' pubsub mechanism and is very powerful,
but it executes every task on all workers, because PUBSUB broadcasts message
to all subscribers.If you want your messages to be processed only once, please use ListQueueBroker.
It uses redis'LPUSHandBRPOPcommands to deal with messages.Brokers parameters:url- url to redis.task_id_generator- custom task_id genertaor.result_backend- custom result backend.queue_name- name of the pub/sub channel in redis.max_connection_pool_size- maximum number of connections in pool.RedisAsyncResultBackend configurationRedisAsyncResultBackend parameters:redis_url- url to redis.keep_results- flag to not remove results from Redis after reading.result_ex_time- expire time in seconds (by default - not specified)result_px_time- expire time in milliseconds (by default - not specified)IMPORTANT:It is highly recommended to use expire time ​​in RedisAsyncResultBackendIf you want to add expiration, eitherresult_ex_timeorresult_px_timemust be set.# First variantredis_async_result=RedisAsyncResultBackend(redis_url="redis://localhost:6379",result_ex_time=1000,)# Second variantredis_async_result=RedisAsyncResultBackend(redis_url="redis://localhost:6379",result_px_time=1000000,)