Saravanan666/pypi-org huggingface dataset:
Package: celery_mutex
Celey Mutex===========Celery Mutex is a mutex for Celery Tasks, optionally refined based on providedkeys. This mutex prevents execution of concurrent tasks, as opposed to delayingexecution.Installation------------Simply run `pip install celery_mutex`Requirements------------Celery Mutex relies on [ZooKeeper](http://zookeeper.apache.org/).Usage-----Celery Mutex takes advantage of Abstract Tasks for Celery. To utilize thisabstract task, add it as a base for the task:import celery_mutex@app.task(base=celery_mutex.MutexTask)def my_task(a, b):return a + bA new configuration variable is required in order to let Celery Mutex knowabout your ZooKeeper servers:ZOOKEEPER_HOSTS = 'localhost:2181'This is a comma-separated list of hosts to connect to.By default a mutex times out after one hour. This can be changed globally bysetting `MUTEX_TIMEOUT` or per-task by setting `mutex_timeout` on the task. Forboth the value is an integer for the number of seconds to set the time out.A second configuration option allows you to refine the mutex for a given task.By default, Celery Mutex only allows one instance of a task at a time. However,there may be a need to further refine what is controlled by the mutex. This canbe done by setting `mutex_keys` on the task. The value is a list of keys thatare to be used for determining exclusivity.Using our above example, adding the two optional parameters would yield:import celery_mutex@app.task(base=celery_mutex.MutexTask, mutex_timeout=30, mutex_keys=('a',))def my_task(a, b):return a + bThis would cause the mutex to only prevent execution for tasks that share thesame value for "a".
Package: celery-persistent-revokes
Celery task revokes are stored on memory or on file. This packages makes possible to easily customize how your revokes are stored (Ex.: Database).DocumentationThe full documentation is athttps://celery-persistent-revokes.readthedocs.io.QuickstartInstall Celery Persistent Revokes:pip install celery-persistent-revokesAdd it to yourINSTALLED_APPS:INSTALLED_APPS=(...'celery_persistent_revokes.apps.CeleryPersistentRevokesConfig',...)Add Celery Persistent Revokes’s URL patterns:fromcelery_persistent_revokesimporturlsascelery_persistent_revokes_urlsurlpatterns=[...url(r'^',include(celery_persistent_revokes_urls)),...]FeaturesBuilt-in tasks revokes using Django default databaseSupport for custom backendsRunning TestsDoes the code actually work?source <YOURVIRTUALENV>/bin/activate
(myenv) $ pip install tox
(myenv) $ toxCreditsTools used in rendering this package:Cookiecuttercookiecutter-djangopackageChange Log1.0.1 (2018-04-03)Adds warning in logs when a revoked tasks is run by a worker1.0.0 (2018-04-03)Updates Django revokes model to be timestamped0.1.0 (2018-04-25)First release on PyPI.
Package: celery-pipeline
Execution pipeline built on celery.pipelineis a library to facilitate the execution of shell scripts
and python functions, configured at runtime.QuickstartInstall Pipeline:pip install pipelineThen use it in a project:import pipelineTestsAll the tests fail due to a major refactor.
But if you like pretty red console text, run this:mkvirtualenv -pwhich python3pipeline
tox
Package: celery-prometheus
Celery PrometheusThis module expose the Prometheus HTTP server to expose metrics of your Celery backends.To installcelery-prometheuswith pip, use the command:pip install celery-prometheusWith Poetry:poetry add celery-prometheusUsageTo setupcelery-prometheusto your backend, simply call the methodadd_prometheus_optionafter the init of theCeleryobject.Example:fromceleryimportCeleryfromcelery_prometheusimportadd_prometheus_optionapp=Celery()add_prometheus_option(app)# Rest of your code ...Before starting your backend, you will need to expose thePROMETHEUS_MULTIPROC_DIRenvironment
variable to indicate which folder the Prometheus Client will use to store the metrics
(seeMultiprocess Mode (E.g. Gunicorn) of the Promehteus Client documentation).To start and expose the Prometheus HTTP Server, you need to use the--prometheus-collector-addrargument when starting your Celery backend:exportPROMETHEUS_MULTIPROC_DIR=/var/cache/my_celery_app
celeryworker-Amy_celery_backend.backend--prometheus-collector-addr0.0.0.0:6543Now that your backend is started, you can configure your Prometheus scrappers to scrappe your
Celery backend.ContributionsThis project is open to external contributions. Feel free to submit us aPull requestif you want to contribute and
improve with us this project.In order to maintain an overall good code quality, this project use the following tools:BlackIsortFlake8Linting and formatting tools are configured to match thecurrent default rules of Black.We also useMypyas a static type checker.Please ensure to run these tools before commiting and submiting a Pull request. In case one of
these mentionned tools report an error, the CI will automatically fail.If you're making your first contribution to this project, please add your name to thecontributors list.LicenseThis project is released byGandi.nettech team under theBSD-3 license.
Package: celery-redbeat
RedBeatis aCelery Beat Schedulerthat stores the scheduled tasks and runtime metadata inRedis.Why RedBeat?Dynamic live task creation and modification, without lengthy downtimeExternally manage tasks from any language with Redis bindingsShared data store; Beat isn’t tied to a single drive or machineFast startup even with a large task countPrevent accidentally running multiple Beat serversFor more background on the genesis of RedBeat see thisblog postGetting StartedInstall with pip:pip install celery-redbeatConfigure RedBeat settings in your Celery configuration file:redbeat_redis_url="redis://localhost:6379/1"Then specify the scheduler when running Celery Beat:celery beat -S redbeat.RedBeatSchedulerRedBeat uses a distributed lock to prevent multiple instances running.
To disable this feature, set:redbeat_lock_key=NoneMore details available onRead the DocsDevelopmentRedBeat is available onGitHubOnce you have the source you can run the tests with the following commands:pip install -r requirements-dev.txt
python -m unittest discover testsYou can also quickly fire up a sample Beat instance with:celery beat --config exampleconf
Package: celery-redbeat-meideng
RedBeatis aCelery Beat Schedulerthat stores the scheduled tasks and runtime metadata inRedis.Why RedBeat?Dynamic live task creation and modification, without lengthy downtimeExternally manage tasks from any language with Redis bindingsShared data store; Beat isn’t tied to a single drive or machineFast startup even with a large task countPrevent accidentally running multiple Beat serversFor more background on the genesis of RedBeat see thisblog postGetting StartedInstall with pip:pip install celery-redbeatConfigure RedBeat settings in your Celery configuration file:redbeat_redis_url="redis://localhost:6379/1"Then specify the scheduler when running Celery Beat:celery beat -S redbeat.RedBeatSchedulerRedBeat uses a distributed lock to prevent multiple instances running.
To disable this feature, set:redbeat_lock_key=NoneMore details available onRead the DocsDevelopmentRedBeat is available onGitHubOnce you have the source you can run the tests with the following commands:pip install -r requirements.dev.txt
py.test testsYou can also quickly fire up a sample Beat instance with:celery beat --config exampleconf
Package: celery-redisbeater
RedisBeateris aCelery Beat Schedulerthat stores the scheduled tasks and runtime metadata inRedis. It is a
fork ofRedBeatWhy RedisBeater?Dynamic live task creation and modification, without lengthy downtimeExternally manage tasks from any language with Redis bindingsShared data store; Beat isn’t tied to a single drive or machineFast startup even with a large task countPrevent accidentally running multiple Beat serversWork with any schedule class which provide required interfaceFor more background on the genesis of RedisBeater see thisblog postGetting StartedInstall with pip:pip install celery-redisbeaterConfigure RedisBeater settings in your Celery configuration file:redisbeater_redis_url="redis://localhost:6379/1"Then specify the scheduler when running Celery Beat:celery beat -S redisbeater.RedisBeaterSchedulerRedisBeater uses a distributed lock to prevent multiple instances running.
To disable this feature, set:redisbeater_lock_key=NoneMore details available onRead the DocsYou can initialize and use RedisBeater just as useforked project. You just need to replace
RedBeat with RedisBeater. For instance:RedisBeaterSchedulerEntry('task-name','tasks.some_task',interval,args=['arg1',2],).save()Custom ScheduleIf you want to use your custom schedule class, you must defineencode_beatermethod and return fields that your class needs when initialized byRedisBeaterSchedulerlater. For instance:classcustomecrontab(BaseSchedule):def__init__(self,minute='*',hour='*',day_of_week='*',day_of_month='*',month_of_year='*',**kwargs):self.hour=hourself.minute=minuteself.day_of_week=day_of_weekself.day_of_month=day_of_monthself.month_of_year=month_of_yearsuper(crontab,self).__init__(**kwargs)defencode_beater(self):return{'minute':self.minute,'hour':self.hour,'day_of_week':self.day_of_week,'day_of_month':self.day_of_month,'month_of_year':self.month_of_year,}DevelopmentRedisBeater is available onGitHubOnce you have the source you can run the tests with the following commands:pip install -r requirements.dev.txt
py.test testsYou can also quickly fire up a sample Beat instance with:celery beat --config exampleconf
Package: celery-redis-prometheus
Exports task execution metrics in Prometheus format: how many tasks were started
and have completed successfully or with failure, and how many tasks are still in
the queues (supported only for broker redis).Inspired byhttps://gitlab.com/kalibrr/celery-prometheusUsageStart HTTP serviceStart the HTTP server like this:$ bin/celery prometheus --host=127.0.0.1 --port=9691Configure Prometheusscrape_configs:
  - job_name: 'celery'
    static_configs:
      - targets: ['localhost:9691']We export the following metrics:celery_tasks_total{state="started|succeeded|failed|retried|retries-exceeded",queue="..."}, countercelery_task_queuetime_seconds{queue}, histogram (only iftask_send_sent_eventis enabled in Celery)celery_task_runtime_seconds{queue}, histogramIf you pass--queuelength-interval=xthen every x seconds the queue lengths will be checked (NOTE: this only works with redis as the broker), resulting in this additional metric:celery_queue_length{queue="..."}, gaugeRun testsUsingtoxandpy.test. Maybe installtox(e.g. viapip install tox)
and then simply runtox.celery_redis_prometheus changes1.5.1 (2023-10-27)Fix brown-bag release1.5.0 (2023-10-27)Be explicit about unregistering default collectors1.4.0 (2022-10-28)Addqueuelabel to task_queuetime and task_runtime metrics1.3.0 (2022-09-14)Addqueuelabel tocelery_tasks_totalmetricAdd separate stateretries-exceeded(previously usedfailed)1.2.0 (2022-06-23)Update to celery-5.x1.1.1 (2020-03-20)Fix py2 class syntax issues1.1.0 (2020-03-20)Make py2 compatible.1.0.0 (2020-03-20)Initial release
Package: celery-redis-sync
Synchronous Redis result store backend.This fixes the issue that the redis publish/subscribe channels are currently
not removed properly by the default (asynchronous) redis backend, seehttps://github.com/celery/celery/issues/3812. (Our “solution” is to never
create any channels in the first place.)UsageMake sure thecelery_redis_syncmodule is importable, and then simply
specify aredis+sync://orrediss+sync://URL in your celery configurationresult_backendsetting instead of the built-inredis://URL scheme.Run testsUsingtoxandpy.test. Maybe installtox(e.g. viapip install tox)
and then simply runtox.For the integration tests you need to have the redis binary installed (tests
starttheir own server).celery_redis_sync1.1.1 (2022-09-27)Properly pass through SSL parameters forrediss1.1.0 (2022-09-23)Also register forrediss+sync://URL1.0.1 (2019-11-28)Declare Python-3 compatibility1.0.0 (2017-10-05)Initial release
Package: celery-redis-unixsocket
This package provides a celery result backend and a kombu transport for Redis
using unix sockets instead of TCP.UsageAdd this to your settings:BROKER_TRANSPORT = 'celery_redis_unixsocket.broker.Transport'
BROKER_HOST = '/path/to/redis.sock'
BROKER_VHOST = 0

# if redis password is configured:
BROKER_PASSWORD = 'foobar'And if you want to store results:CELERY_RESULT_BACKEND = 'redisunixsocket'
CELERY_REDIS_HOST = '/path/to/redis.sock'

CELERY_IGNORE_RESULT = False

# if redis password is configured:
CELERY_REDIS_PASSWORD = 'foobar'

import celery_redis_unixsocketTheimport celery_redis_unixsocketis important because it registersredisunixsocketas a result backend.NoteThis code has only been tested with Django.
Package: celery-redundant-scheduler
# celery-redundant-scheduler[Celery](http://celeryproject.org) beat scheduler providing ability to run multiple `celerybeat` instances.# ProblemProduction level deployment requires redundancy and fault-tolerance environment. Unfortunately [Celery](http://celeryproject.org) doesn't provide periodic tasks scheduling redundancy out of the box. Running multiple `celerybeat` instances results multiple scheduled tasks queuing. This package provides synchronized scheduler class. By default `redis` backend used, but developers are free too use their own based on package primitives.# Installation```#bashpip install celery-redundant-schedulerpip install git+https://github.com/MnogoByte/celery-redundant-scheduler.git```# Usage1. Setup scheduler synchronization backend:```CELERYBEAT_REDUNDANT_BACKEND_OPTIONS = {'HOST': 'localhost','PORT': 6379,'DB': 0,'PASSWORD': 'secret'}```2. Provide `--scheduler=celery_redundant_scheduler:RedundantScheduler` option running your `worker` or `beat` instance.# Settings- `CELERYBEAT_REDUNDANT_BACKEND_OPTIONS` - redis connection options.- `CELERYBEAT_REDUNDANT_REDIS_KEYPREFIX` - prefix for redis keys. Default is `redundant-scheduler`.- `CELERYBEAT_REDUNDANT_REDIS_LOCK_EXPIRES` - redis lock timeout. Default is 60 seconds.# Using custom scheduler synchronization backend1. Define subclass of `celery_redundant_scheduler.backends.base:BaseBackend`.2. Override back-end interaction methods.3. Provide backend options with `CELERYBEAT_REDUNDANT_BACKEND_OPTIONS`4. Claim using your subclass providing `CELERYBEAT_REDUNDANT_BACKEND` setting or environment variable.# Author[Antonov Mikhail](https://github.com/atin65536)# LicenseBSD - 3
Package: celery-rmq
celery-rmqcelery-rmq is a python library to implement Celery and RabbitMQ message broker.
it's a standalone library required celery and kombu (python library for rabbitmq).
Celery and RabbitMQ producer and consumer made easy with celery-rmqQuick StartFirst of all, install rabbitmq and start.Create a python app and create an virtual environment too. Activate the virtual environment.1:: Install it:pipinstallcelery-rmq2:: Basic Usage:** app.py **fromcelery_rmq.appimportCeleryAppProviderfromcelery_rmq.exchangeimportregister_exchangefromcelery_rmq.queueimportregister_queuefrom.consumers.testconsumerimportBasicTestConsumerapps=['tests.testapp']app_provider=CeleryAppProvider(app_name='test_celery_broker',installed_apps=apps)app=app_provider.get_app()# adding exchangesdefregister_exchanges():register_exchange(app_provider,"test_exchange")defregister_queues():register_queue(app_provider,"test_queue","test_routing","test_exchange")defadd_consumers():app_provider.add_consumer(BasicTestConsumer)# execution of following two functions# "register_exchanges()" and "register_queues()"# needs to be synced# i.e one after another.# because queues are depended on exchangesregister_exchanges()register_queues()add_consumers()app.start()register as many as exchanges, queues and consumers3:: consumer.pyimportkombufromceleryimportbootstepsfromcelery_rmq.registryimportget_queueclassBasicTestConsumer(bootsteps.ConsumerStep):defhandle_message(self,body,message):print(body)message.ack()defget_consumers(self,channel):queue=get_queue("test_queue","test_routing")return[kombu.Consumer(channel,queues=[queue],callbacks=[self.handle_message],accept=['json'])]4:: If you want to add this with any framework,
create an app (example:Djangoapps) and create a tasks.py file under the app:from__future__importabsolute_import,unicode_literalsfromceleryimportshared_taskfromtests.appimportapp_provider@shared_taskdefsimple_json_message(message,exchange_name,route_key):producer=app_provider.get_producer()producer.publish(message,content_type='application/json',exchange=exchange_name,routing_key=route_key)and call thesimple_json_messagefunction from the view as per your need.5:: Run the celery worker:celeryworker-linfo-Aapphere app is app.py fileDone.for testing purpose, we would like to send messages directly from RabbitMQ web panel. Go the the queue section and publish message.
Package: celery-s3
# Celery-S3[![Build Status](https://travis-ci.org/robgolding/celery-s3.svg?branch=master)](https://travis-ci.org/robgolding/celery-s3)[![Coverage Status](https://coveralls.io/repos/github/robgolding/celery-s3/badge.svg?branch=master)](https://coveralls.io/github/robgolding/celery-s3?branch=master)Celery-S3 is a simple S3 result backend for Celery.If used in conjunction with the SQS broker, it allows for Celery deploymentsthat use only distributed AWS services -- with no dependency on individualmachines within your infrastructure.This backend probably isn't suitable for particularly high-traffic Celerydeployments, but it works just fine in general -- and imposes no limits on thenumber of workers in the pool.## InstallationInstall via pip:`pip install celery-s3`Then configure Celery to use the `S3Backend`:CELERY_RESULT_BACKEND = 'celery_s3.backends.S3Backend'CELERY_S3_BACKEND_SETTINGS = {'aws_access_key_id': '<your_aws_access_key_id>','aws_secret_access_key': '<your_aws_secret_access_key>','bucket': '<your_bucket_name>',}## ConfigurationTo use a folder within the specified bucket, set the `base_path` in your`CELERY_S3_BACKEND_SETTINGS`:CELERY_S3_BACKEND_SETTINGS = {...'base_path': '/celery/',...}To use a region other than the default (`us-east-1`), set the `aws_region`parameter:CELERY_S3_BACKEND_SETTINGS = {...'aws_region': 'us-east-1',...}To use [reduced redundancy storage](https://aws.amazon.com/s3/reduced-redundancy/),set the `reduced_redundancy` parameter:CELERY_S3_BACKEND_SETTINGS = {...'reduced_redundancy': True,...}To use [server-side encryption](http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html),set the `encrypt_key` parameter:CELERY_S3_BACKEND_SETTINGS = {...'encrypt_key': True,...}## NotesStoring Celery results with this backend will obviously result in API callsbeing made to Amazon S3.  For each result, at least one `PUT` request will bemade (priced at $0.01 per 1,000 requests at the time of writing).  Also, thedata contained within the result object will be stored indefinitely, unlessotherwise specified.To fetch a result for a task that has already finished, at least two requestswill be made (one `HEAD` and one `GET`).  If you use Celery's `result.get()` towait for a task to finish, S3 will be polled continuously until the task hasfinished.By default, the poll interval is set to 0.5 seconds, which could result ina lot of requests (two `HEAD` requests per second until the task has finished,then one `GET` request to fetch the result).  If you need to use`result.get()`, consider increasing the interval and using a timeout to preventpolling forever: `result.get(interval=5, timeout=600)`.Also, for tasks whose result you don't need, be sure to use `ignore_result`:@celery.task(ignore_result=True)def process_data(obj):obj.do_processing()Once task results have been used and are no longer needed, be sure to call`result.forget()` to delete the corresponding S3 key.  Otherwise, old resultswill remain forever and contribute to storage costs (storage is priced at$0.095 per GB per month at the time of writing).Also, the S3 lifecycle can be used to archive or delete old keys aftera certain period of time.
Package: celery-sentinel
celery-sentinelCelery broker forRedis SentinelInstallationAs simple as possible:pip install celery-sentinelUsageSetup celery broker:#  settings.pyBROKER_URL='redis-sentinel://redis-sentinel:26379/0'BROKER_TRANSPORT_OPTIONS={'sentinels':[('192.168.1.1',26379),('192.168.1.2',26379),('192.168.1.3',26379)],'service_name':'master','socket_timeout':0.1,}Configure celery app:# celery_app.pyimportosfromcelery_sentinelimportCelery# set the default Django settings module for the 'celery' program.os.environ.setdefault("DJANGO_SETTINGS_MODULE","config.settings.local")app=Celery("your-project")# Using a string here means the worker doesn't have to serialize# the configuration object to child processes.# - namespace='CELERY' means all celery-related configuration keys#   should have a `CELERY_` prefix.app.config_from_object("django.conf:settings",namespace="CELERY")# Load task modules from all registered Django app configs.app.autodiscover_tasks()Then use the celery as usual.
Package: celery-service-task
Celery service taskClasse permettant d'implémenter une tâche pour un worker Celery spécifiqueUsage# importfromcelery_service_task.taskimportTaskBase# implémentation d'une tâche simpleclassTask(TaskBase):deftask(self,payload:Dict[str,Any],meta:MessageMetadata)->bool:print(self.conf)# la configuration est donnée par le worker Celeryprint(payload)# le payload est déjà sous forme d'un dictionnaireprint(payload['transaction_id'])# identifiant de la transaction issue du payloadprint(meta)# Initialisation de la classe avec une configurationt=Task(conf={'token':'tk'},redis=redis_connection,redis_key=redis_task_key)# lance la tâche si l'id de transaction n'est pas déjà enregistré# Ici la tâche se lancet.run_task(payload=payload,meta=metadata,t_id=payload['transaction_id'])# Ici la tâche est considérée comme un replicat.run_task(payload=payload,meta=metadata,t_id=payload['transaction_id'])